Fix malformed math allocation and footnote ownership recovery

pdf_craft/extractor/chapter/chapter.py

@@ -43,6 +43,9 @@ class TextSource:
     text: str
     content: tuple
     first_line_inset: float | None = None
+    printed_line_count: int | None = None
+    printed_line_height: float | None = None
+    printed_line_lefts: tuple[int, ...] | None = None
 
 
 @dataclass

pdf_craft/extractor/chapter/data/page_repair.jinja

@@ -2,7 +2,9 @@ You repair ownership, reading order and the PRINTED paragraph structure of one O
 
 Begin with stream_assessment: briefly identify main-text and note streams and their cross-page continuations, quoting literal boundary words. Keep quotation SCOPE separate from paragraph grouping. Do NOT decide within-page paragraph joins in this global summary; decide them locally in each layout's paragraph_evidence instead. This is your reading overview, not a claim to be verified against a neighbour's eventual output.
 
-For EVERY layout entry, write paragraph_evidence BEFORE choosing continues_from_previous. Give one concise local boundary finding: quote the preceding retained ending and this part's opening, mention measured first-line inset when available, and state whether those TWO pieces form one paragraph. A complete statement followed by a newly indented explanation starts a separate paragraph even inside one quotation. A shared quotation/topic is not positive evidence of a paragraph join. For ignored entries or assets, state that they do not continue a text paragraph. For the first retained part, cite direct raw cross-page evidence if choosing incoming continuation. For sliced sources, the box's first-line observation applies only to the original first line, not invented sub-boxes. Do not copy a global quotation summary as the local finding. This field records your submitted decision; the program does not audit its semantic content against the original page or neighbours.
+Decide reading-stream OWNERSHIP before paragraph boundaries. A paragraph break answers "is this a new paragraph within its stream?", not "has the main text resumed?" A note may contain several separately indented paragraphs, equations and diagrams, without repeating its printed marker. First-line indentation, a complete sentence, a conjunction, or a change of paragraph can establish a new paragraph INSIDE that same note; none establishes a return to main text. Track the suspended main-text sentence across the raw page boundary separately from the intervening explanatory stream. Require positive reading evidence that a candidate really resumes that main narrative rather than elaborating the note. Do not use a paragraph-shape observation as an ownership argument.
+
+For EVERY layout entry, write ownership_evidence first, then ownership. In ownership_evidence identify its reading stream and the literal words that establish that stream. Next write paragraph_evidence, then continues_from_previous. Keep these findings distinct: paragraph_evidence concerns the PRINTED PARAGRAPH only, after ownership has been decided. A paragraph_evidence finding must not say "same quotation", "same argument" or "same note" as a reason to join paragraphs: these establish at most a shared stream, not one printed paragraph. A complete short statement followed by a FIRST-line-only indent relative to later lines in the next box is a new printed paragraph even when it directly explains that statement. Describe the physical/local printed seam before choosing its boolean; do not substitute argumentative cohesion for the page's typography. Give one concise local boundary finding: quote the preceding retained ending and this part's opening, mention measured first-line inset when available, and state whether those TWO pieces form one paragraph. A complete statement followed by a newly indented explanation starts a separate paragraph even inside one quotation. A shared quotation/topic is not positive evidence of a paragraph join. For ignored entries or assets, state that they do not continue a text paragraph. For the first retained part, cite direct raw cross-page evidence if choosing incoming continuation. For sliced sources, the box's first-line observation applies only to the original first line, not invented sub-boxes. Do not copy a global quotation summary as the local finding. This field records your submitted decision; the program does not audit its semantic content against the original page or neighbours.
 
 source_text_seams explicitly pairs the end/start of adjacent raw text observations. It is a reading aid, not an approved reading order or a list of paragraph joins; page numbers or reordered columns may intervene. Read these local pairs before writing your paragraph plan. In your assessment, cite BOTH literal sides of each incoming paragraph join you actually choose; do not paraphrase several boxes as one large paragraph first.
 
@@ -21,15 +23,21 @@ For every retained text part set paragraph_role=body and paragraph_level=-1, or
 {% if includes_footnotes %}
 Create one citation per logical note, not per OCR box. Multiple paragraphs, boxes and pages can belong to one note; several notes can share a box. Numbered definitions use page-local contiguous index=1,2,... and citation_id=new-PAGE-index. mark is the actual printed marker, or null if genuinely missing. Do not merge different printed notes or discard a definition to satisfy validation. All members of a note share its ID and citation_index. citations.stream_order defines note order. Every citation needs retained members.
 
-Each citation_id is a target-page local handle, not a claim about a neighbour's identity or origin. You may retain or replace initial IDs freely. Set citation.continues_previous=true only for the FIRST note that appends to the last open note output stream. This is an append operation, even when its first paragraph starts anew (continues_from_previous=false). Do not submit predecessor page/source ranges or copy a neighbour's membership as a requirement. Incoming notes have index=null and mark=null. If no output cursor exists (for example an excerpt starts here), their available content will be preserved as an explicitly unbound stream. A new unmarked independent note gets a new local index, with mark=null. Initial origins and grouping are guesses, not constraints.
+Each citation_id is a target-page local handle, not a claim about a neighbour's identity or origin. Decide the citation OPERATION before assigning index or mark: is this an independent new note, or an append to a preceding note? For an append, write continues_previous=true, index=null and mark=null. The earlier note's number is background evidence only; never repeat it as if freshly printed on this page. A page-turn continuation cue is not a new printed note marker. A separately indented paragraph of that continued note still uses the append operation even when its own continues_from_previous=false. Only an independent new note receives a new local index. If your reading assessment says "continues the previous note", the citation record must express append, not a fresh definition. On field-conflict feedback repair the index/mark to express that same reading decision; do not flip continues_previous just to satisfy the schema. You may retain or replace initial IDs freely. Set citation.continues_previous=true only for the FIRST note that appends to the last open note output stream. This is an append operation, even when its first paragraph starts anew (continues_from_previous=false). Do not submit predecessor page/source ranges or copy a neighbour's membership as a requirement. Incoming notes have index=null and mark=null. If no output cursor exists (for example an excerpt starts here), their available content will be preserved as an explicitly unbound stream. A new unmarked independent note gets a new local index, with mark=null. Initial origins and grouping are guesses, not constraints.
 
 References name the ORIGINAL layout_id, citation_index and anchor.before/mark/after taken consecutively from original layout.text. Make the anchor unique. They annotate content already owned by a paragraph; they do not consume another copy. A reference must fit within one retained body part. Associate only with the correct indexed target definition. Never swallow body text as a marker. A formula may be an OCR wrapper around a printed footnote superscript: for 2005^{9}, set extract_from_expression=true and select exactly ^{9} to retain the year 2005 and link note 9. Only an explicitly identified, complete terminal superscript with a real matching note can be extracted; do not reinterpret ordinary mathematical exponents as notes.
 
+{% endif %}
+
 Topic similarity does NOT distinguish a footnote from main prose: explanatory notes normally discuss the very same concepts as the body. When an unfinished main-text construction has a literal next-page completion, that is strong evidence for the main reading seam. Before calling an intervening unnumbered paragraph a return to body, mentally insert it between those two literal pieces and read the resulting main-text sentence. Do not interrupt a suspended list, quotation or parenthesis with an independent explanation merely because its topic resembles the body. Keep the explanatory note stream separate from that unfinished main construction.
 
 Audit the reading streams before returning. First follow the main narrative through the raw neighbouring text, including any unfinished parenthesis, quotation or sentence at the page break. Then account for everything between that main-text ending and its next-page continuation. Numbered notes can contain several independent paragraphs and diagrams; their later paragraphs normally have NO repeated printed marker. Determine whether such a paragraph elaborates the last numbered note, instead of promoting it to body just because it opens a complete sentence, starts with a conjunction, or has no number. A note's paragraph break does not create a new note. Carry explanations and any intervening diagram in their logical note, while keeping genuine internal paragraph breaks. Diagram labels belong to the same note when that note introduces the diagram; use preserve_as_image for genuinely non-linear artwork. Page-turn cues are ignored publication apparatus, not note prose or sources that must be retained to permit continuation. Use neighbours only as raw evidence for your own current-page decisions; no neighbour's eventual decision will approve or veto them.
-{% else %}
-Footnote output is disabled. citations and references must be empty; citation_id and citation_index must be null on every entry. Retain genuine body even if OCR classified it as footnote. Exclude clearly identified note content, page numbers and publication cues as ignored. For a mixed source, use ranges to retain body and ignore the note; do not delete the whole box. If semantic evidence is insufficient, preserve the initial ownership. Do not infer exclusion just from bottom-of-page location, size or numbering.
+{% if not includes_footnotes %}
+Footnote output is disabled, but identifying the full note stream is still necessary BEFORE excluding it. Internally follow each printed note through ALL of its unnumbered explanatory paragraphs and associated diagrams. Removing a numbered note does not make its next paragraph a body paragraph, and the ignored output itself is not evidence that the note has ended. Preserve genuine body that resumes after a note only when its words and the main narrative support that return. First-line indent is evidence about paragraphing, never evidence of body ownership. citations and references must be empty; citation_id and citation_index must be null on every entry. Retain genuine body even if OCR classified it as footnote. Exclude clearly identified note content, page numbers and publication cues as ignored. For a mixed source, use ranges to retain body and ignore the note; do not delete the whole box. If semantic evidence is insufficient, preserve the initial ownership. Do not infer exclusion just from bottom-of-page location, size or numbering.
 {% endif %}
 
 preserve_as_image=true is only for genuinely non-linear OCR diagrams whose labels cannot be read as prose. Preserve their ownership and original region; do not use it to avoid normal text or inline formulas. If asset_text_association is present, keep_asset_attachment may preserve or explicitly detach that provisional link. Never move or rewrite the asset itself. Follow all schema constraints and return a complete candidate, not a patch. On validation feedback reconsider the actual semantics; do not connect unrelated paragraphs or omit information just to pass.
+
+printed_lines reports direct raster observations: count is the number of horizontal ink bands and median_ink_height is their median ink height in this page's raster pixels. It is not a semantic label. Compare the candidate's printed ink height with BOTH the body text and neighboring notes on the same page, alongside their literal reading continuity. text_scale_estimate is only a coarse area-per-character estimate, not a measured type size; use the direct ink observation when available. A note's unnumbered continuation can have exactly the same type size as that note, despite a new paragraph indent, a generalizing sentence, or initial body classification. Do not infer ownership from size alone; explain how the typography corroborates or contradicts the proposed return to the body, especially when the main narrative is suspended until the next page. Missing observations are unknown, not zero.
+
+printed_lines.left_offsets gives each observed ink band's left edge in pixels RELATIVE TO THIS SAME OCR RECTANGLE, in printed top-to-bottom order. In particular, a first offset farther right than the later offsets is a first-line-only inset; shifting an entire quotation or column shifts the rectangle too and cannot explain that difference. Read these actual local line starts when deciding the incoming paragraph boundary, separately from argument/quotation continuity or an outgoing page wrap. For a single-line rectangle, its bbox left is its actual first-line start; for a multi-line rectangle, the bbox left alone is not. These are physical observations only, not program paragraph verdicts. Use literal incoming sentence continuity as well as the printed boundary; do not merge independent indented explanations just because they explain the preceding assertion.

pdf_craft/extractor/chapter/page_repair.py

@@ -77,6 +77,7 @@ class RepairedReference(_StrictModel):
 class RepairedLayout(_StrictModel):
     layout_id: str
     source_range: SourceRange | None = None
+    ownership_evidence: str = Field(min_length=1)
     ownership: Literal["paragraph", "citation", "ignored"]
     paragraph_evidence: str = Field(min_length=1)
     continues_from_previous: bool
@@ -91,10 +92,10 @@ class RepairedLayout(_StrictModel):
 
 class RepairedCitation(_StrictModel):
     citation_id: str
-    index: int | None = Field(default=None, ge=1)
-    mark: str | None
     stream_order: int = Field(ge=0)
     continues_previous: bool = False
+    index: int | None = Field(default=None, ge=1)
+    mark: str | None
 
 
 class RepairedPage(_StrictModel):
@@ -446,6 +447,7 @@ def _layout_packet(
         "source_type": getattr(layout.source, "ref", "text"),
         "bbox": list(layout.bbox),
         "first_line_extra_indent_vs_later_lines_in_same_box": getattr(getattr(layout.source, "origin", None), "first_line_inset", None),
+        "printed_lines": _printed_lines_packet(layout),
         "text": text,
         "text_scale_estimate": text_scale_estimate(text, layout.bbox) if layout.source_order is not None else None,
         "definition_mark": definition_mark,
@@ -468,6 +470,14 @@ def _layout_packet(
     return packet
 
 
+def _printed_lines_packet(layout):
+    origin = getattr(layout.source, 'origin', None)
+    count = getattr(origin, 'printed_line_count', None)
+    height = getattr(origin, 'printed_line_height', None)
+    return dict(count=count, median_ink_height=height,
+                left_offsets=getattr(origin, 'printed_line_lefts', None)) if count is not None and height is not None else None
+
+
 def _initial_stream_key(layout: AnalysedLayout) -> tuple[str, str | None] | None:
     if layout.annotations.keep_asset_attachment:
         return None
@@ -713,7 +723,11 @@ def _compile_citations(page_index, proposed, issues):
         if item.continues_previous and position != 0:
             issues.append(f'[CITATION_ORDER] {prefix}: only the first note may append to the output cursor')
         if item.continues_previous and item.index is not None:
-            issues.append(f'[CITATION_ORIGIN] {prefix}: an indexed origin cannot continue a previous note')
+            issues.append(f'[CITATION_ORIGIN] {prefix}: choose the actual note operation first. '
+                          'For a continuing note, retain continues_previous=true and set index=null, mark=null; '
+                          'do not copy the earlier printed marker into a new local definition. '
+                          'Use continues_previous=false with a local index only for an independent new note, '
+                          'not merely to resolve this field conflict.')
         if item.citation_id in seen_ids:
             issues.append(f'[CITATION_ID_DUPLICATE] {prefix}: duplicate local identity')
         seen_ids.add(item.citation_id)

pdf_craft/extractor/chapter/source.py

@@ -64,6 +64,9 @@ def capture_sources(pages: list[Page]) -> dict[tuple[int, int], tuple[int, TextS
                     stream, item.order, positions[id(item)], item.ref, item.text,
                     tuple(deepcopy(_parse_block_content(item.text))),
                     first_line_inset=item.first_line_inset,
+                    printed_line_count=item.printed_line_count,
+                    printed_line_height=item.printed_line_height,
+                    printed_line_lefts=item.printed_line_lefts,
                 ))
     return sources
 

pdf_craft/pdf/line_geometry.py

@@ -3,13 +3,41 @@
 from collections.abc import Iterable
 from statistics import median
 import re
+from typing import NamedTuple
 
 from PIL.Image import Image
 
 _MAX_CROP_PIXELS = 4_000_000
 _DARK_PIXELS = [255 if value < 110 else 0 for value in range(256)]
 
 
+class PrintedLineMetrics(NamedTuple):
+    line_count: int
+    median_ink_height: float
+    left_offsets: tuple[int, ...]
+
+
+def printed_line_metrics(image: Image, bbox: tuple[int, int, int, int]) -> PrintedLineMetrics | None:
+    """Observe band count, median height and local left edges, without classification."""
+    left, top, right, bottom = bbox
+    width, height = right - left, bottom - top
+    if (image.mode not in ('L', 'RGB') or not 0 <= left < right <= image.width
+            or not 0 <= top < bottom <= image.height
+            or width < height or width * height > _MAX_CROP_PIXELS):
+        return None
+    with image.crop(bbox) as region, region.convert('L') as crop:
+        with crop.point(_DARK_PIXELS, mode='1') as ink:
+            bands = _ink_bands(ink.getprojection()[1])
+            lefts = []
+            for start, end in bands:
+                with ink.crop((0, start, width, end)) as band:
+                    bounds = band.getbbox()
+                if bounds is None:
+                    return None
+                lefts.append(bounds[0])
+    return PrintedLineMetrics(len(bands), float(median(end-start for start, end in bands)), tuple(lefts)) if bands else None
+
+
 def first_line_inset(
     image: Image,
     bbox: tuple[int, int, int, int],