Repository navigation
fix(docs): preserve source text through syntax highlighting - #971
rudycelekli wants to merge 3 commits into
Conversation
Signed-off-by: Rudy Celekli <47457359+rudycelekli@users.noreply.github.com>
Signed-off-by: Rudy Celekli <47457359+rudycelekli@users.noreply.github.com>
|
🦞👀 Pull request received. I will update this pull request when review starts. ClawSweeper review completeClawSweeper finished reviewing this revision. The review result is being finalized. |
PR SummaryLow Risk Overview The highlighter no longer substitutes matches with private-use Unicode placeholders (which could strip real characters in paths/YAML and break past ~6,400 tokens). It now tracks highlighted fragments separately from plain text in Adds Reviewed by Cursor Bugbot for commit f5e962d. Bugbot is set up for automated code reviews on this repo. Configure here. |
clawsweeper
Bot
added
P2
|
Codex review: needs maintainer review before merge. Reviewed October 5, 2026, 10:51 AM ET / 14:51 UTC. ClawSweeper reviewWhat this changesThe PR replaces hidden syntax-highlighting placeholders with separate text and HTML fragments, adds source-preservation regressions, and includes them in the docs test command. Merge readiness✅ Ready for maintainer review The fix remains necessary: current main and v4.8.0 retain the reported text-corruption mechanism. The introduced patch is focused, has relevant production-builder proof, and has no actionable correctness findings. Priority: P2 Review scores
Verification
How this fits togetherPeekaboo’s documentation builder converts Markdown into the HTML published by GitHub Pages. Its syntax highlighter formats fenced code blocks that readers view and copy. flowchart LR
A[Markdown documentation] --> B[Extract fenced code]
B --> C[Choose language patterns]
C --> D[Separate text and highlighted fragments]
D --> E[Escape source and render spans]
E --> F[Published HTML code blocks]
Before mergeNone. Agent review detailsSecurityNone. Review metrics
Technical reviewBest possible solution: Keep highlighted HTML separate from source text so published code remains copyable without character collisions or a fixed match limit. Do we have a high-confidence way to reproduce the issue? Yes: current-main source establishes collisions for private-use characters and failed restoration beyond 6,400 matches. The contributor supplies before/after builder observations; this review did not execute the builder. Is this the best way to solve the issue? Yes: separating generated spans from raw text directly removes both failure mechanisms while retaining HTML escaping and existing language dispatch. AGENTS.md: found and applied where relevant. Codex review notes: model internal, reasoning medium; reviewed against 91cd87e79bf0. LabelsLabel changes:
Label justifications:
EvidenceWhat I checked:
Likely related people:
Rating scale
Overall follows the weaker of proof and patch quality. Workflow
|
|
The owner consolidation is #992, combining the five related rendering repairs after independently reproducing them on current main. Its 24 renderer tests, metadata tests, lint, full site build, and P0–P2 review pass. It also includes actual Chrome before/after screenshots using identical synthetic input. This original will be closed as superseded once the combined candidate completes CI and lands; contributor credit is retained. |
|
ClawSweeper status: review started. I am starting a fresh review of this pull request: fix(docs): preserve source text through syntax highlighting This is item 1/1 in the current shard. Shard 0/1. This temporary status tracks the active review worker. The completed review will appear in the durable ClawSweeper review comment. Crustacean status: shell secured, claws on keyboard, evidence pebbles being sorted. |
|
Thanks for identifying source-text corruption during highlighting. Main reproduces the token/Unicode collisions. Owner PR #992 uses fragments instead of reserved placeholders, with exact decoded-text regressions and browser proof. Closing this duplicate while that successor completes its landing gates. |
![cursor[bot]](https://avatars.githubusercontent.com/in/1210556?s=60&v=4){kind=small}
There was a problem hiding this comment.
Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.
❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.
Reviewed by Cursor Bugbot for commit f5e962d. Configure here.
| }); | ||
| result.push(fragment.slice(previous)); | ||
| return result; | ||
| }); |
There was a problem hiding this comment.
Anchors bind to fragments not lines
Low Severity
highlightFragments runs each pattern on leftover source slices, so ^ and $ now mean fragment edges rather than the original line. Mid-line shell comments stop at an earlier quoted match, and later passes can restyle the remainder. Copyable text stays intact; only span assignment changes.
Additional Locations (1)
Reviewed by Cursor Bugbot for commit f5e962d. Configure here.
Consolidate the reproduced CRLF metadata, entity/link, heading identity, EOF fence, and highlighter token defects from #939, #931, #940, #969, and #971. Keep highlighted fragments separate from source text and retain the shared renderer gate. Reconcile the Unreleased note and menu-preparation documentation with current main. All 25 docs-site regressions pass; independent Codex review is clean through P2. Co-authored-by: Rudy Mizrahi Celekli <47457359+rudycelekli@users.noreply.github.com>
Documentation rendering lost literal code text, leaked CRLF front matter into articles, double-escaped TOC text and link queries, reused heading anchors, and discarded fenced code at EOF. This consolidates the verified fixes from #939, #931, #940, #969 and #971, with credit to @rudycelekli. The highlighter now keeps rendered fragments separate from source text, with no reserved source characters or 6,400-token limit. The page-wide heading allocator preserves natural anchors and assigns unique duplicate/fallback IDs. Front matter is normalized before extraction; renderer-owned entities are decoded once; link suffixes remain intact; EOF flushes the pending fence. Normal macOS CI now runs the complete shared renderer gate. Preserve the existing regression and platform proof from the PR. Reconcile with current main and retain the Unreleased changelog. Independent Codex review is clean through P2, and the final exact-head CI checks pass. Co-authored-by: Rudy Mizrahi Celekli <47457359+rudycelekli@users.noreply.github.com>
2 participants
Summary
The syntax highlighter used U+E000–U+F8FF as hidden replacement tokens. Actual private-use characters in code were silently deleted or replaced, and blocks containing more than 6,400 highlighted matches exceeded that token range. The builder still reported success while changing the text users copy.
Keep highlighted fragments separate from raw source text, applying later patterns only to unhighlighted fragments. This removes token collisions and the fixed token limit while retaining the existing span classes and escaping. Include the new producer regressions in test:docs-site/test:safe.
Observed production behavior
Actual docs builder entrypoint with owned Markdown fixtures:
Validation
node --test tests/docs-site-code-preservation.test.mjs tests/docs-site-toc.test.mjs— five tests passed.node scripts/build-docs-site.mjs— passed.git diff --check— passed.AI assistance
Prepared with Codex assistance; owned production fixtures and focused regressions verify the change.