fix(docs): preserve rendered source text and navigation

[steipete]

Documentation rendering lost literal code text, leaked CRLF front matter into articles, double-escaped TOC text and link queries, reused heading anchors, and discarded fenced code at EOF. This consolidates the verified fixes from #939, #931, #940, #969 and #971, with credit to @rudycelekli.

The highlighter now keeps rendered fragments separate from source text, with no reserved source characters or 6,400-token limit. The page-wide heading allocator preserves natural anchors and assigns unique duplicate/fallback IDs. Front matter is normalized before extraction; renderer-owned entities are decoded once; link suffixes remain intact; EOF flushes the pending fence. Normal macOS CI now runs the complete shared renderer gate.

Verification:

• Unchanged main: 12 failures across the five reported defects, with six unaffected controls passing. A TOC fixture was corrected to include the two headings needed to render the TOC, then independently rerun against unchanged main and the candidate.
• AWS candidate: 24 renderer tests, four metadata tests, docs lint, and the complete production site build pass.
• Actual Chrome before/after pages captured through the installed signed Peekaboo host. Both use the same synthetic input. The attached crops contain only the synthetic webpage; browser tabs, address bar and bookmarks are excluded.
• An independent HTML parser confirms decoded TOC text, unique heading IDs, exact code characters, the restored final fence, and absence of leaked metadata.
• Independent Codex P0–P2 review is clean.

Reconciled with current main after #977; the Unreleased entry is now included and the final diff has a clean post-reconciliation review. The original contributor PRs are closed in favor of this consolidated successor. It also corrects the newly landed menu documentation to distinguish serialized evidence across preparation/dispatch from the native leaf identity check within dispatch; the unchanged contract source was supplied to review. Exact-head macOS/CodeQL CI is required before merge. No website deployment, release, or version bump is performed by this work.

Co-authored-by: Rudy Mizrahi Celekli 47457359+rudycelekli@users.noreply.github.com

[clawsweeper]

🦞👀
ClawSweeper picked this up.

Pull request received. I will update this pull request when review starts.

ClawSweeper review complete

ClawSweeper finished reviewing this revision. The review result is being finalized.

View the workflow run.

[clawsweeper]

Codex review: needs changes before merge. Reviewed October 7, 2026, 6:10 AM ET / 10:10 UTC (Revision 4).

ClawSweeper review

What this changes

The PR repairs documentation text preservation, metadata parsing, link destinations and heading navigation, adds renderer regressions, and clarifies scoped menu-click documentation.

Merge readiness

⛔ Needs changes before merge - 2 items remain

The consolidated repairs remain useful and are not implemented on current main. The previously reported shell-comment styling defect remains on this head.

Likely related people: steipete, with high-confidence routing based on prior docs-site work.

Priority: P2
Reviewed head: 1adc7b7066d8f8630c5dde4990298f1518b8e4ce

Review scores

Measure	Result	What it means
Overall readiness	🐚 platinum hermit (4/6)	Strong visible before/after proof and focused regression coverage support the repairs, with one small previously reported styling defect remaining.
Proof confidence	🦞 diamond lobster (5/6) ✨ media proof bonus	Sufficient (screenshot): Inspected Chrome before/after screenshots directly show the production renderer’s corrected metadata, TOC text, example characters and EOF code on the same synthetic page; recorded production builds and HTML parsing supplement anchor and query checks. No stored-data contract changes.
Patch quality	🐚 platinum hermit (4/6)	1 actionable review finding remain.

Verification

Check	Result	Evidence
Real behavior	Verified	Sufficient (screenshot): Inspected Chrome before/after screenshots directly show the production renderer’s corrected metadata, TOC text, example characters and EOF code on the same synthetic page; recorded production builds and HTML parsing supplement anchor and query checks. No stored-data contract changes.
Evidence reviewed	8 items	Verified introduced changes: The pinned main-to-head delta contains the renderer repairs, documentation and regression tests. Current main still uses private-use highlighter placeholders and lacks the page-wide heading allocator and EOF fence flush; the central work is still necessary. Existing finding remains: Quoted spans are isolated before the comment regex runs on each remaining fragment. For echo x # comment "quoted" tail --flag 42, the comment ends at the first fragment boundary and the trailing tokens receive command, flag and number styling. The quoted-comment regression checks text fidelity without checking those styles. Re-review continuity: GitHub’s original source at the earlier reviewed head contains the same shell-highlighting implementation. This retains the prior finding and rank-up request rather than introducing a new concern. Local comparison could not load the older object; the exact GitHub contents read supplied the relevant historical source.
Findings	1 actionable finding	[P3] Preserve shell comment styling across quoted fragments
Security	None	None.

How this fits together

Peekaboo’s documentation builder converts Markdown into static website pages. Its renderer supplies article text, highlighted examples, heading anchors and table-of-contents links.

flowchart LR
  A[Markdown documents] --> B[Metadata extraction]
  B --> C[Article renderer]
  C --> D[Code highlighting]
  C --> E[Heading and link allocation]
  D --> F[Static documentation pages]
  E --> F

Loading

Before merge

• Preserve shell comment styling across quoted fragments (P3) - The prior finding remains: for echo x # comment "quoted" tail --flag 42, the quote pass isolates "quoted" before the comment pass runs. The comment regex consequently stops at that fragment boundary, and the trailing tail, --flag and 42 receive command, flag and number styling despite belonging to the comment. Preserve comment context across fragments while protecting hashes inside actual strings, and assert the trailing comment styling. This retains the late-discovered concern from the earlier review; the relevant implementation is unchanged.
• Complete next step (P2) - Resolve the existing shell-comment styling finding and add a regression that checks styling after quoted comment text.

Findings

• [P3] Preserve shell comment styling across quoted fragments — scripts/build-docs-site.mjs:806-807

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Renderer and test LOC	production +87/-46; tests +273/-22	The production growth implements five documented rendering repairs, with most added lines devoted to regression coverage.

Root-cause cluster

Relationship: canonical
Canonical: #992
Summary: This PR is the documented consolidated successor to five closed-unmerged renderer repairs; the merged source-byte repair overlaps only in supporting build tooling.

Members:

• superseded: fix(docs): preserve rendered entities and link query parameters #931 - Its entity and link repair is incorporated into the consolidated successor.
• superseded: fix(docs): parse CRLF front matter before rendering #939 - Its CRLF front-matter repair is incorporated into the consolidated successor.
• superseded: fix(docs): assign unique heading permalink targets #940 - Its unique heading allocation is incorporated into the consolidated successor.
• superseded: fix(docs): render fenced code at the end of a document #969 - Its EOF fence repair is incorporated into the consolidated successor.
• superseded: fix(docs): preserve source text through syntax highlighting #971 - Its highlighter source-preservation repair is incorporated into the consolidated successor.
• 1 more in the report.

Proposal only: this assessment does not dispatch repair, suppress jobs, mutate sibling items, close, or merge anything.

Technical review

Best possible solution:

Keep the fragment-based renderer while recognizing shell comments across the complete line and preserving literal hashes inside strings.

Do we have a high-confidence way to reproduce the issue?

Yes, source inspection establishes the remaining styling trigger: a quoted span after a shell comment marker splits the fragment that the comment regex consumes. No new runtime reproduction was executed.

Is this the best way to solve the issue?

Yes, separating source text from rendered fragments is a focused repair, but shell comment recognition must retain line-wide context to preserve existing styling.

Full review comments:

• [P3] Preserve shell comment styling across quoted fragments — scripts/build-docs-site.mjs:806-807
  The prior finding remains: for echo x # comment "quoted" tail --flag 42, the quote pass isolates "quoted" before the comment pass runs. The comment regex consequently stops at that fragment boundary, and the trailing tail, --flag and 42 receive command, flag and number styling despite belonging to the comment. Preserve comment context across fragments while protecting hashes inside actual strings, and assert the trailing comment styling. This retains the late-discovered concern from the earlier review; the relevant implementation is unchanged.
  Confidence: 0.99
  Late finding: first raised on code an earlier review cycle already covered.

Overall correctness: patch is correct
Overall confidence: 0.96

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning medium; reviewed against 3a9590594ea4.

Labels

Label changes:

No label changes.

Label justifications:

• P2: Documentation rendering currently corrupts copyable examples and navigation, with a bounded website-only repair.
• rating: 🐚 platinum hermit: Overall readiness is 🐚 platinum hermit; proof is 🦞 diamond lobster and patch quality is 🐚 platinum hermit.
• status: 👀 ready for maintainer look: ClawSweeper has no concrete contributor-facing blocker left for this PR. Sufficient (screenshot): Inspected Chrome before/after screenshots directly show the production renderer’s corrected metadata, TOC text, example characters and EOF code on the same synthetic page; recorded production builds and HTML parsing supplement anchor and query checks. No stored-data contract changes.
• proof: sufficient: Contributor real behavior proof is sufficient. Inspected Chrome before/after screenshots directly show the production renderer’s corrected metadata, TOC text, example characters and EOF code on the same synthetic page; recorded production builds and HTML parsing supplement anchor and query checks. No stored-data contract changes.
• proof: 📸 screenshot: Contributor real behavior proof includes screenshot evidence. Inspected Chrome before/after screenshots directly show the production renderer’s corrected metadata, TOC text, example characters and EOF code on the same synthetic page; recorded production builds and HTML parsing supplement anchor and query checks. No stored-data contract changes.

Evidence

Acceptance criteria:

• [P1] pnpm run test:docs-site.
• [P1] pnpm run docs:lint.
• [P1] pnpm run docs:site.
• [P1] git diff --check.

What I checked:

• Verified introduced changes: The pinned main-to-head delta contains the renderer repairs, documentation and regression tests. Current main still uses private-use highlighter placeholders and lacks the page-wide heading allocator and EOF fence flush; the central work is still necessary. (scripts/build-docs-site.mjs:439, 1adc7b7066d8)
• Existing finding remains: Quoted spans are isolated before the comment regex runs on each remaining fragment. For echo x # comment "quoted" tail --flag 42, the comment ends at the first fragment boundary and the trailing tokens receive command, flag and number styling. The quoted-comment regression checks text fidelity without checking those styles. (scripts/build-docs-site.mjs:806, 1adc7b7066d8)
• Re-review continuity: GitHub’s original source at the earlier reviewed head contains the same shell-highlighting implementation. This retains the prior finding and rank-up request rather than introducing a new concern. Local comparison could not load the older object; the exact GitHub contents read supplied the relevant historical source. (scripts/build-docs-site.mjs:806, 7a6092866899)
• Inspected browser proof: Both prepared screenshots were inspected locally. The before image shows leaked metadata, escaped TOC labels, corrupted example text and missing EOF code; the after image visibly corrects those results. The captured PR body additionally reports production builds and independent HTML parsing for exact characters, link parameters and unique heading IDs. Screenshot attachment URLs: https://github.com/user-attachments/assets/18bc392c-4d96-4911-8f38-91c743d3107e and https://github.com/user-attachments/assets/3f5c9e52-9c21-408d-90aa-992aaf51bc1b.
• Consolidation and existing gate: The provided discussion identifies this PR as the successor to the five closed-unmerged renderer PRs. build(docs): escape NUL stash placeholders in docs site builder #1001 separately merged the source-byte repair and shared test wiring; that wiring is already present on the pinned main and is not introduced here. (.github/workflows/macos-ci.yml:52, 3a9590594ea4)
• Menu documentation matches production contract: The unchanged scoped dispatch owner compares serialized selected-leaf evidence against preparation, then compares native snapshot identities between dispatch reads before submitting the AX action. This supports the documentation clarification without changing runtime authority. (Core/PeekabooAutomationKit/Sources/PeekabooAutomationKit/Services/UI/MenuService+ScopedMenuExtraPreparation.swift:116, 1adc7b7066d8)

Likely related people:

• steipete: Suggested for follow-up; no historical authorship or introduction is verified. (role: unverified routing candidate; confidence: low)

Rank-up moves

Optional improvements that raise the rating; they are not merge blockers.

• Repair quoted shell-comment styling and assert that all trailing comment text retains comment styling.

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (3 earlier review cycles)

• reviewed 2026-10-07T02:08:40.761Z sha 538b8d9 :: needs changes before merge. :: none
• reviewed 2026-10-07T04:16:04.374Z sha 7a60928 :: needs maintainer review before merge. :: none
• reviewed 2026-10-07T04:54:27.389Z sha 3cfafcb :: needs changes before merge. :: [P3] Preserve shell comment styling across quoted fragments

[cursor]

PR Summary

Low Risk
Changes are limited to the static docs build/renderer and tests; they do not affect the Peekaboo CLI runtime or security-sensitive paths.

Overview
Fixes several docs-site renderer bugs so published pages keep faithful source text and stable navigation.

The syntax highlighter no longer uses private-use placeholder characters (which could corrupt paths and large JSON blocks); it applies pattern passes on separate fragments so literal code, entities, and markup in fences stay intact. Front matter is normalized for CRLF before parsing so metadata does not leak into articles. Fenced code at end-of-file is flushed instead of dropped.

Links and TOC text decode renderer-owned entities once (no double-escaped & in query strings); relative .md rewrites keep ?/# suffixes. A page-wide heading ID pass assigns unique anchors—including nested blockquotes—without stealing natural slugs like usage-1. Link validation uses the same decoding rules.

Adds a shared buildDocsFixture harness and broad tests/docs-site-*.test.mjs coverage; macOS CI and test:safe run the full test:docs-site gate. CHANGELOG and building.md document the behavior; menubar.md clarifies scoped click evidence is revalidated between preparation and dispatch.

^{Reviewed by Cursor Bugbot for commit 1adc7b7. Bugbot is set up for automated code reviews on this repo. Configure here.}

[cursor]

Cursor Bugbot has reviewed your changes using default effort and found 1 potential issue.

^{❌ Bugbot Autofix is OFF. To automatically fix reported issues with cloud agents, enable autofix in the Cursor dashboard.}

^{Reviewed by Cursor Bugbot for commit 7a60928. Configure here.}