Skip to content

http-and-logging.md

Latest commit

 

History

History
132 lines (102 loc) · 5.04 KB

http-and-logging.md

File metadata and controls

132 lines (102 loc) · 5.04 KB
Raw

HTTP Client, Utility API, and Logging

This document covers the HTTP layer, convenience fetch helpers, and structured logging output.

HttpClient

Silkworm's HttpClient wraps wreq and enforces concurrency with a semaphore. It also supports proxy routing and redirect handling.

Key behaviors:

  • Merges request headers with configured defaults.
  • Supports JSON and raw body payloads.
  • Merges query params into the URL.
  • Optional keep-alive header injection.
  • Redirect handling with a max redirect limit and loop detection.
  • Proxy-specific client caching keyed by proxy URL.

Code:

  • HttpClient: ../src/http.rs
use silkworm::Request;

let request = Request::<QuotesSpider>::get("https://example.com/search")
    .with_params([("q", "rust"), ("page", "1")])
    .with_header("Accept", "text/html");

Utility Fetch API

The api module exposes convenience functions for fetching and parsing HTML outside of the spider engine:

  • fetch_html: returns the raw HTML text and parsed document.
  • fetch_document: returns only the parsed document.
  • fetch_html_with / fetch_document_with: same helpers with configurable UtilityFetchOptions.
  • UtilityFetcher: reusable helper that keeps one configured HTTP client for multiple calls.

fetch_html and fetch_document use a shared, lazily initialized default HttpClient. fetch_*_with builds a client from the provided options for that call. Use UtilityFetcher when you want to reuse custom options across many requests. Default safety options are 15-second timeout, redirect following, and a 2 MB response-body cap.

Code:

  • Utility API: ../src/api.rs
let (html, doc) = silkworm::fetch_html("https://example.com").await?;
let document = silkworm::fetch_document("https://example.com").await?;
use std::time::Duration;
use silkworm::{UtilityFetchOptions, fetch_html_with};

let options = UtilityFetchOptions::new()
    .with_timeout(Duration::from_secs(8))
    .with_html_max_size_bytes(512_000)
    .with_header("User-Agent", "silkworm-rs/docs-example");
let (html, doc) = fetch_html_with("https://example.com", options).await?;
use std::time::Duration;
use silkworm::{UtilityFetchOptions, UtilityFetcher};

let options = UtilityFetchOptions::new()
    .with_timeout(Duration::from_secs(8))
    .with_header("User-Agent", "silkworm-rs/docs-example");
let fetcher = UtilityFetcher::new(options)?;
let (html, doc) = fetcher.fetch_html("https://example.com").await?;

Header Model Evaluation

The current public header type is still HashMap<String, String> for compatibility and ergonomics, but duplicate response header values are now normalized into merged comma-separated values.

Repeated values are joined with commas in the current implementation. This representation does not preserve separate Set-Cookie values; do not use it as a full-fidelity cookie jar. No multi-value header API or release migration is implemented here.

Performance Regression Checks

See benchmark commands and regression checks.

Logging

Silkworm uses a lightweight structured logger that prints key/value fields. The log level is controlled by SILKWORM_LOG_LEVEL and defaults to INFO.

Code:

  • Logger implementation: ../src/logging.rs
  • Engine and middleware logging usage: ../src/engine.rs, ../src/middlewares.rs
SILKWORM_LOG_LEVEL=DEBUG cargo run

URL parameter preparation and measurement

URL preparation borrows keys and values from Request.params through a private Cow<str> merge helper instead of cloning each payload. Existing URL query pairs remain owned so the parsed URL can be mutated safely. Params still replace all existing values of matching keys, appended params are sorted, canonical URLs sort all pairs and remove fragments. Request URL construction retains fragments in the returned string; they are not part of a canonical fingerprint. The returned URL is still an owned String; parsing, percent-encoding and output storage allocate as needed.

Run cargo bench --bench url_params for URL construction and canonicalization with 0, 1, 8 and 64 params, Unicode values, duplicate query keys, empty fields and invalid URL input. Each case measures 20,000 synchronous operations, two warmups and nine samples. Fixtures are excluded; parsing, merging, serialization, error construction where applicable and result destruction are included. This measures URL preparation rather than HTTP or whole-crawl throughput.

For allocation counts, run python scripts/measure_seen_allocations.py --benchmark url_params --toolchain 1.92.0. The temporary harness imports the production helper and error types, copies the project lockfile to preserve dependency resolution, and adds stats_alloc 0.1.10 only in that temporary package. Timing runs use no counting allocator. Counts cover Rust System allocator calls/reallocations, deallocations and requested bytes inside the workload, excluding fixture setup and reporting. Peak RSS, allocator usable size, networking and foreign allocations are not measured. The existing seen-cache allocation command remains the default.