blockLexer: frozen-prefix tail lexing diverges from a full lex on duplicate reference definitions

[sudongyuer]

Bug Description

createBlockLexer's frozen-prefix optimization re-lexes only the text after the frozen boundary. But marked.lexer keeps document-level state in tokens.links: a reference definition whose label was already defined earlier in the document is absorbed (its raw merges into the surrounding whitespace instead of producing its own token). A tail-window lex cannot see definitions that live in the frozen prefix, so when a stream contains the same label defined twice, the incremental block list diverges from what a full lex of the same text produces — different block count, different block boundaries.

Minimal reproduction

import { marked } from 'marked';
import { createBlockLexer } from '../src/blockLexer';

const full = [
  'See [foo] here.',
  '',
  '[foo]: https://example.com "first"',
  '',
  'More prose to freeze the prefix.',
  '',
  'Another paragraph so the defs are far apart.',
  '',
  '[foo]: https://example.com "duplicate"',
  '',
  'tail keeps streaming',
].join('\n');

const lex = createBlockLexer();
let out;
for (let pos = 20; pos <= full.length; pos += 20) out = lex(full.slice(0, pos));
out = lex(full);

console.log(out.blocks.length);                 // 11
console.log(marked.lexer(full).length);         // 10

Observed: the incremental result keeps the duplicate [foo]: … "duplicate" as its own block; a full lex absorbs it (marked drops the second definition of an existing label). The mis-split persists for the rest of the stream — every later append extends a block list whose boundaries no longer match a fresh parse, which also shifts downstream block indices/keys.

LLMs do emit repeated reference definitions (regenerated sections, repeated citation footers), so this is reachable with real streams.

Suggested fix

Treat reference-definition / footnote syntax in the re-lex window as a bail-out: if the tail (from the frozen boundary to the end) matches something like

/\[\^|^ {0,3}\[[^\S\n]*[^\n\]]+]:/m

skip the frozen-prefix path for that commit and re-lex the whole document. Definitions are rare enough that the occasional full lex costs little, and it restores exact equivalence with marked.lexer.

For context: we hit this while porting the frozen-prefix approach to LobeHub Mobile (lobehub-mobile#287), found it via a randomized streaming-replay fuzz (incremental vs full lex equivalence at every prefix), and the guard above eliminated all divergences. Happy to share the fuzz harness if useful.