installation.md

Installation and features

The current repository requires Rust 1.92 or newer (edition 2024). The package is silkworm-rs; Rust imports use silkworm. The commands below install the published crate. To use the current checkout, see development and verification. Start with the complete README quickstart.

Create a binary project with cargo new my-spider, then run these commands inside it. Native TLS dependencies require a C/C++ toolchain, CMake and libclang with its headers available to bindgen (see the development guide).

cargo add silkworm-rs

If you want to use the async API directly (instead of run_spider), add Tokio:

cargo add tokio --features rt-multi-thread,macros

The README quickstart and guide snippets also use serde_json for convenience:

cargo add serde_json

Enable XPath explicitly when needed:

cargo add silkworm-rs --features xpath

The default scraper-atomic feature caches parsed HTML. Without default features, CSS remains available but documents are parsed per selection. cli-examples enables clap-based repository examples; it is not required by application spiders.

Enable automatic detection of undeclared non-UTF-8 text with cargo add silkworm-rs --features charset-detection. This optional feature uses charset-norm and requires Rust 1.98 or newer; default builds retain Rust 1.92 support. BOM, HTTP charset and HTML/XML declarations remain authoritative, and valid UTF-8 bypasses statistical detection. See response decoding.

The optional tl-parser feature adds TlDocument, TlElement and HtmlResponse::tl_document() using rustedbytes-tl 0.3.0. It offers explicit lightweight parsing; existing scraper APIs keep their behavior. See TL integration and compatibility limits.

Tip: use silkworm::prelude::*; for the most common types.