Prose-gate unpaired additions and deletions - #16
Merged
Conversation
An unpaired line with no target-script content — code, maths, markup or metadata arriving or leaving — now routes to code-or-markup instead of addition/deletion, mirroring what categorise() already does for non-prose paired lines, so the omission taxonomy counts prose only. The total pair count is unchanged (categories shift only), which keeps the regression's pinned corpus pair count green by construction. At programming.zh-cn @ b727528: additions 103 -> 30, deletions 80 -> 38, code-or-markup 25 -> 140, total 255 unchanged; composition and engine strata byte-identical. The fixture gains one purely additive and one purely deletive editor-shaped commit so the routing is asserted without depending on hunk pairing. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Pull request overview
This PR fixes a precision defect in the omission taxonomy by applying the same “prose gate” used for paired lines to unpaired additions/deletions during scanning, so that non-prose (code/maths/markup/metadata) lines no longer inflate addition/deletion.
Changes:
- Route unpaired added/deleted lines to
code-or-markupwhen they contain no target-script characters, otherwise keepaddition/deletion. - Add a synthetic baseline test that asserts the new routing for a purely-additive and purely-deletive edit shape.
- Update method documentation and the changelog to describe the rule and its intended effect (category-only shift; total pair count unchanged).
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated no comments.
| File | Description |
|---|---|
src/textstrata/scan.py |
Prose-gates unpaired additions/deletions and assigns taxonomy via CATEGORY_MAP to keep non-prose edits out of omission buckets. |
tests/test_baseline.py |
Extends the synthetic repo fixture and adds a focused test asserting unpaired-line routing and taxonomy for code lines vs prose. |
docs/method.md |
Documents the known-limit rule that unpaired non-prose lines are categorised as code-or-markup. |
CHANGELOG.md |
Records the behavioural change and clarifies that totals are preserved (only categories shift). |
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #14 — the dominant pair-precision cause measured by the three-edition adjudication (QuantEcon/project-translation#53).
An unpaired line with no target-script content — code, maths, markup, or
.translatefrontmatter arriving or leaving — now routes tocode-or-markupinstead ofaddition/deletion, mirroring whatcategorise()already does for non-prose paired lines. The omission taxonomy therefore counts prose only, which is what the adjudication showed it was failing to do (additionprecision 4/13 / 8/18 / 1/20 across intro / python / programming, the misses almost entirely code and metadata rows).As #14 recommended, non-prose lines are re-routed rather than dropped, so the total pair count is unchanged — the regression test pins the corpus pair count against the 2026-08-04 study, and a category-only shift keeps it green by construction (verified: full suite passes including the pinned regression).
Real-corpus effect at programming.zh-cn @
b727528, the pathological case:addition103 → 30,deletion80 → 38,code-or-markup25 → 140, total 255 unchanged; composition andengine_stratabyte-identical. The direction and scale match the adjudication sample, where 19 of 20 sampled additions were non-prose.The synthetic fixture gains one purely additive and one purely deletive editor-shaped commit, so the routing is asserted without depending on git's hunk alignment or the similarity pairing (which cross-pairs freely inside mixed hunks — the separate, smaller artefact #14 leaves out of scope).
docs/method.md's line-pairing known-limit now states the rule.🤖 Generated with Claude Code