Pin signed sparse QRF outputs' net mass to the donor instrument (farm sign flip; #432)

[MaxGhenis]

Defect

The export-mass parity gate flagged farm_operations_income exporting net-positive on the Build M base pool (base-m: +$10.37B = +$34.84B positive leg, −$24.47B negative leg; 6,577 nonzero persons of 865,046, 2,851 negative). The source instrument is loss-heavy: SOI Schedule F sole-proprietor farm income is net-negative nationally, and the coverage campaign's own staging of the column was loss-heavy (the release's qbi_farm_operations_income_exclusion smoke probe scored −$4.16M there, vs +$14.55B on base-m — sign-flipped and ~4,000× the magnitude).

Root-cause mechanism

The staged base builder imputes the column with the regime-gated QRF (populace.fit), which models a sparse, sign-mixed, heavy-tailed target as an independent sign gate (a HistGradientBoostingClassifier) times per-sign magnitude forests. On such a column:

• the sign gate regresses the positive/negative/zero shares toward balance — it over-predicts the rarer leg — and
• the magnitude forests inflate each leg,

so nothing pins the aggregate signed total. The imputed net — a small difference of two large legs — regresses toward a fixed, slightly-positive balance point that is nearly independent of the donor's true net. A loss-heavy source is dragged toward or past zero and its export sign flips.

Ruled out during diagnosis (the task's checklist):

• Donor construction is correct — the donor carries the raw signed Schedule F E02100 (loss-heavy); no positive-only mapping.
• No abs/clipping destroys the negative leg — farm_operations_income is (correctly) absent from _PUF_TAX_DETAIL_NONNEGATIVE_OUTPUTS and the sparse/snap sets; finalize writes the signed total to the first person.
• The Bound QRF leaf storage for near-discrete targets #415 near-discrete leaf bound is not implicated — it never fires on these columns' continuous within-sign magnitudes, and the flip reproduces with the bound out of effect.
• The legs are imputed separately (gate + two forests) — this separation, with no aggregate constraint, is the structural cause.

Does it also explain #432?

Yes. partnership_self_employment_net_earnings is the same signed-sparse-column family. Its source is mostly positive (reference +$61.74B), so the identical "regression toward balance" manufactures a spurious cancelling negative leg, collapsing base-m to +$47.71B / −$48.06B (net −$0.35B) — exactly #432. The two columns share one code path, so one fix addresses both.

Fix

Calibrate the imputed positive and negative legs of the affected columns to the donor's per-unit-weight leg masses in finalize_us_puf_tax_detail_predictions — the signed generalization of the existing _sparsify_..._to_donor_positive_rate. The gate's sign assignment (which rows are +/−/0) and each leg's relative shape are untouched; only the two leg totals move, so the imputed net signed mass tracks the source instrument. The calibration runs on the PUF support channel only, never rescaling the measured ASEC leg (Schedule F operations income is measured on ASEC as FRSE_VAL).

Scoped to the mechanism: a _PUF_TAX_DETAIL_SIGNED_MASS_CALIBRATED_PERSON_OUTPUTS set carries the two evidenced columns; no new columns or targets are added to the chain. The fix lives in finalize_us_puf_tax_detail_predictions, so it applies to both the monolithic (impute_us_puf_tax_detail_support) and checkpointed (finalize_primary_puf_qrf_chain) paths.

Evidence: donor vs imputed leg masses

End-to-end through the real regime-gated QRF on a faithful loss-heavy Schedule-F synthetic donor (per-unit-weight leg masses, $/weight):

stage	positive leg	negative leg	net
donor (source instrument)	+302.6	−1761.9	−1459.2
raw regime-gated QRF draw	+448.8	−113.8	+335.0 ← sign flipped
finalized (with signed calibration)	+302.6	−1761.9	−1459.2 ← tracks donor

The raw draw collapses the loss leg by 94% (−1761.9 → −113.8) and inflates the positive leg, flipping the net; the calibration restores each leg to the donor exactly.

A controlled gate check (pure-noise features, so a faithful gate must reproduce the marginal odds) confirms the gate itself regresses the sign shares toward balance: donor weighted pos/neg odds 0.482 → gate 0.541, over-predicting the rarer positive leg 1.23× vs the negative 1.10×.

Teeth proof

Three new tests (packages/populace-build/tests/test_us_puf_support.py), parametrized over both columns:

• test_finalize_pins_signed_person_leg_masses_to_loss_heavy_donor[farm_operations_income | partnership_self_employment_net_earnings] — deterministic (monkeypatched QRF returns a sign-flipped draw); asserts each finalized PUF-channel leg equals the loss-heavy donor's per-unit-weight leg mass.
• test_regime_gated_qrf_farm_net_flips_and_calibration_restores_the_source_sign — fits the chain's real QRF on a loss-heavy donor, asserts the raw draw flips the national net sign, and that the finalized net tracks the source's net-negative sign.

With the calibration call disabled, all three fail with clean assertion errors (e.g. imputed leg 448.8 ≠ donor 302.6); with the fix they pass. The pre-existing test_weighted_qrf_preserves_asec_and_writes_signed_puf_support was updated to reflect the calibrated (still-signed, still-both-legs, gate-passing) PUF magnitudes.

Test results

• packages/populace-build + packages/populace-fit: 2312 passed, 76 skipped (the only 2 failures in the full run are pre-existing UK tests that require the uninstalled policyengine-uk package, unrelated to this change).
• uvx ruff@0.15.16 check and format --check: clean.

Rebuild of the base pool is intentionally not performed here — the conductor handles it after merge, at which point the reviewed parity exclusions for these columns lift (same pattern as the miscellaneous_income/#393 flow).

🤖 Generated with Claude Code