AdBlocker: request engine

[nikneym]

On-going effort; here's the recent benchmark made against EasyList + EasyPrivacy on this branch:

Adblock engine: EasyList + EasyPrivacy corpus report

Lists downloaded fresh from easylist.to the same day (135,908 lines total). Measured with
the per-request engine (tries + token-bucket indexes) after the adversarial-review fixes.

Coverage: what loads

	EasyList	EasyPrivacy	Combined
File lines	79,233	56,675	135,908
Comments / blank	13,920	771	14,691
Rules	65,313	55,904	121,217
Cosmetic rules (not network)	10,852	32	10,884
Network rules	54,461	55,872	110,333
Loaded	54,266 (99.6%)	55,858 (99.97%)	110,124 (99.8%)
→ in hostname tries	45,264	42,955	88,219
→ engine-indexed	9,002	12,903	21,905

The 209 network rules we don't apply, by reason

• 31 /regex/ filters (24 EL + 7 EP) — the only rules uBO enforces as
  blocks that we genuinely can't: there's still no regex engine. ~0.03% of the
  corpus.
• 168 cosmetic-realm exceptions ($generichide/$elemhide/
  $specifichide) — they modify cosmetic filtering, which we don't have yet,
  so there's nothing for them to except.
• 6 modifier-only rules ($removeparam-class) — uBO applies these as URL
  rewrites, never blocks; dropping them is the conservative, correct choice.
• 3 malformed rules (EasyPrivacy) — uBO rejects these too.
• 1 unsupported option (the @@…$redirect-rule google-analytics line —
  correctly treated as excepting nothing, so google-analytics.com stays
  blocked).

The ~10,884 "cosmetic" drops are element-hiding rules (##, #@# — including
944 with spaces in their CSS selectors that take the hosts-file parsing path);
no network engine applies those, so they're out of scope rather than failures.
Notably, zero hostnames land in the suppressed trie across the whole
corpus — every exception either evaluates per-request or provably unblocks
nothing.

Correctness on the loaded rules

• A 26-assertion doubleclick battery against uBO ground truth passes with the
  review fixes in: gpt.js blocked except on the listed carve-out sites, the
  /ssai/ exception honored, wrong-type/wrong-site/wrong-path variants all
  correct, and the apex doubleclick.net undecided rather than suppressed.
• End-to-end: lightpanda fetch on a page loading the real ad stack blocks
  gpt.js, td./googleads./fls.doubleclick.net and Google Analytics
  before they reach the network.

Bottom line

99.8% of the network rules in EasyList+EasyPrivacy load and enforce; the
only real gap is 31 regex filters.

[krichprollsch]

It seems ||coinimp.com ||coinimp.com^ and ||coinimp.com| are all the same in ublock: hostname is coinimp.com or a subdomain

[krichprollsch]

@@…$important seems to be invalid in ublock. we should return error.InvalidOption in this case.

[karlseguin]

Right now, AdBlocker.match does:

    if (self.trie.matches(self.suppressed, hostname) != null) return .none;

    if (self.trie.matches(self.blocked_important, hostname) != null) return .blocked;
    if (self.blocking_important.match(request) != null) return .blocked;

    if (self.exceptions.match(request) != null) return .allowed;
    if (self.trie.matches(self.allowed, hostname) != null) return .allowed;

    if (self.trie.matches(self.blocked, hostname) != null) return .blocked;
    if (self.blocking.match(request) != null) return .blocked;

There are cases where you won't even need the tokens that you've already split / hashed. AND, you can rewrite that to be even more efficient:

  if (self.trie.matches(self.suppressed, hostname) != null) return .none;

  if (self.trie.matches(self.blocked_important, hostname) != null) return .blocked;
  if (self.blocking_important.match(request) != null) return .blocked;
  
  if (self.trie.matches(self.blocked, hostname) != null or self.blocking.match(request) != null) {
      if (self.trie.matches(self.allowed, hostname) != null) return .allowed;
      if (self.exceptions.match(request) != null) return .allowed;
      return .blocked;
  }
  return .none;

There's no need to check exceptions unless something is blocked (.allowed vs .none is just a metrics difference) AND there are fewer cases where you need to tokenize/hash anything.

With that in mind, don't pre-tokenize. Tokenize on demand. Whether or not it's worth caching in the tokenizer, I don't know. But it isn't worth limiting it to 128 tokens. Since the tokenizer becomes streaming, you can cache the first 128 values, and then stream until the end of the URL.

[nikneym]

Since the tokenizer becomes streaming, you can cache the first 128 values, and then stream until the end of the URL.

I actually thought the same, Claude also recommended, but unsure how many cases would benefit from it. Currently overflow results to loss of tokens; comparing it with stream-the-rest approach.