Skip to content

Integrate rustedbytes-tl 0.3 through an explicit optional parser - #18

Merged
egorsmkv merged 2 commits into
masterfrom
investigate/tl-parser
Oct 8, 2026
Merged

egorsmkv merged 2 commits into
masterfrom
investigate/tl-parser

Conversation

@egorsmkv

@egorsmkv egorsmkv commented Oct 8, 2026 •

Copy link
Copy Markdown
Contributor

Closes #17 with a tested optional integration of the now-published rustedbytes-tl 0.3.0.

Add the non-default tl-parser feature, owned TlDocument and borrowing TlElement APIs, plus HtmlResponse::tl_document(). The response adapter snapshots the existing decoded, byte-limited source; callers keep the document for repeated and nested queries. It uses TL's safe owned constructor and preserves selector errors separately from empty results. Text decoding is enabled; raw_attr deliberately exposes raw attribute spelling. No unsafe code is introduced.

Existing scraper-based selection, compiled scraper::Selector APIs, utility return types, XPath and default dependency behavior remain unchanged. This is an explicit alternative, not transparent HTML5 equivalence: omitted tbody, malformed-tree recovery, raw attributes, serialization and unsupported CSS still differ. There is no automatic fallback. Test site fixtures before opting in.

Update the independent evaluation to exact published TL 0.3.0 + entities. The catalog's 14 checked selectors and decoded entity text now agree with scraper 0.27.0; retain tests demonstrating remaining differences. Document ownership, lifetimes, memory/parse boundaries and limitations. Add TL feature checks and a path-filtered independent evaluation workflow. Historical timings are clearly labeled as TL 0.2.0, not performance evidence for this adapter.

Validation:

  • Rust 1.99: 126 default unit tests, 131 with tl-parser, 131 no-default + tl-parser, 133 all-features; all passed. Enabled TL doctests and lifetime compile-fail test passed.
  • Adapter tests cover nested selection, decoded text, raw/boolean attributes, rejected versus empty queries, zero/byte limits, HTTP charset decoding, response mutation/drop, Send/Sync and moving the owner to a worker.
  • Rust 1.92: root check --all-targets --features tl-parser and strict Clippy passed; independent evaluation compilation passed.
  • Five differential tests, audit executable, independent strict Clippy and all 15 Criterion smoke cases passed.
  • Root all-target compilation, documentation build and 34 Markdown examples in three feature configurations passed. Rust 1.99 lint/doc commands allow only its pre-existing fetch_update deprecation; Rust 1.92 Clippy passes without allowances.
  • Formatting, diff checks and actionlint passed. Native TLS verification used cmake/libclang with local PATH and include-path setup.

No new timing, allocation or whole-crawler speedup claim is made. A broad real-site differential corpus and complete-adapter benchmarks remain prerequisites for replacing scraper as the default. GitHub Actions results are pending.

@egorsmkv egorsmkv closed this Oct 8, 2026
@egorsmkv egorsmkv reopened this Oct 8, 2026
Merge current master and add an optional TL adapter with compatibility tests.
@egorsmkv egorsmkv changed the title Investigate replacing scraper with scraper-rust's TL backend Integrate rustedbytes-tl 0.3 through an explicit optional parser Oct 8, 2026
@egorsmkv
egorsmkv merged commit 6cad287 into master Oct 8, 2026
4 checks passed
@egorsmkv
egorsmkv deleted the investigate/tl-parser branch October 8, 2026 16:06
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Investigate a possibility to replace scraper by what scraper-rust uses under the hood

1 participant