Build I: selection-side misc-income support fix for a certifiable 57k default successor

[MaxGhenis]

Goal

A certifiable 57k-class sparse successor to the live default (populace-us-2024-sparse-l0-refit-57k-71a0887): fix the one gate failure from Build H (#299) — miscellaneous_income export mass -79.9% vs the $47.4B reference — by fixing the SELECTION, not the targets. Build H proved the target side works (dense passed 0/35 with the same registry).

Design (per the corrected #346 + the learned-prior findings)

1. Pool: full 337,704 (base 18833fb6), v8 feed (94b7155f), registry d71c59514e3a/5533, pe-us 1.764.6, seed 0 — Build H dense (published: populace-us-2024-buildh-dense-warmstart-b449eb7-20260708T100442Z) is the parent weights.
2. Warm-start reselection (Reconstruct informed-L0 / warm-start selection as committed machinery (#328) #330 machinery): initialize from the frozen 57,240 selection, NOT cold. Init signal for swaps = target-contribution share / leverage (NOT weight-proportional; NOT top-K by any learned prior — the fourth-arm result on Informed-init L0 selection: warm-start reselection for pool changes (ACS/ASEC ingest forcing function) #346 shows hard selection fails at 0.330).
3. Protect-list: the dominant carriers of EACH target, explicitly including miscellaneous_income carriers (dense meets the SOI misc target with 2,145 nonzero records; the frozen selection's misc support is too thin to hold $47.4B nationally). Mechanism finding from Informed-init L0 selection: warm-start reselection for pool changes (ACS/ASEC ingest forcing function) #346: failures come from over-weighting the loudest targets' carriers.
4. Gates: full Build H protocol — structural + export-mass with ONLY the 2 reviewed exclusions (estate_income, non_sch_d_capital_gains, band-math-justified); misc and both mortgage columns live. Certification bar: meet/beat frozen-57k Build H diagnostics (final loss 0.03092, within-10% 89.01%, income tax +0.45%) AND pass export-mass.
5. Flip protocol: on CERTIFIABLE, publish per the branch-only pattern, then the pe.py default flip PR carries the policyengine.py#462 promotion scorecard.
6. Compute discipline: chunked (<30-min processes — direct solve + warm-start release rebuild), per the jetsam findings on the build machine.

Refs: #299 (Build H verdict), #346 (corrected init + fourth-arm evidence), #330 (warm-start seam), #347/#355 (BE/UK instances inherit the recipe), policyengine.py#462.

[MaxGhenis]

Build I certification run — selection-side miscellaneous_income support fix (rmloss100)

Ran the Build I certification per this issue: fix the deployable frozen-57k's miscellaneous_income export-mass failure from Build H (#299, −79.9%) by fixing the selection, not the targets. The candidate is a warm-start reselection of the certified 57,240 that swaps 100 records. It CERTIFIES. Staging/local only — prod HF and latest.json untouched. Publication and the pe.py default flip are Max's call.

Artifact at policyengine-us 1.764.6 / core 3.26.11, base 18833fb6 (verified), λ=0, seed 0, period 2024, feed v8 94b7155f, target registry d71c59514e3a (5533 declared; sparse compiles 5514 after the 19-cell zero-support exclusion). Export-mass reference c2065b64, ±50% band, $1B floor. Numbers below are verbatim from calibration_diagnostics.json / input_mass_parity.json and the exported H5.

Lineage note. main advanced during this campaign (Daphne's SNAP train #350/#352/#353 + #358 merged). Build I is deliberately pinned to the Build H lineage above and was NOT merged with main mid-run, so the change is isolated to the selection. The registry, feed, base, and reference are the exact Build H set.

The failure, and the mechanism that explains it

miscellaneous_income export mass = net_income_leg − net_loss_leg exactly: the SOI Table 1.4 targets other_income_net_income_amount == the positive part of miscellaneous_income and other_income_net_loss_amount == the negative part (record-level corr 0.9999, ratio 1.0). The net_income leg is hit perfectly everywhere ($66.12B). The export failure is entirely the net_LOSS leg overshooting: Build H frozen net_loss $56.0B (+322% vs the $13.28B target) → export $9.55B (−79.9%); the dense parent reaches $37.9B → export $27.5B (in band).

Root cause: the sparse frame is mass-repaired ×5.20, so max_weight_ratio=5.0 permits effective weights up to 26× original base. The frozen selection has only 790 negative-misc carriers, so the solve maxes them into the cap (weight/base ratios up to 25.98×; the top-10 carry 54% of net_loss) — their misc loss is an unpenalised byproduct, because the $13.28B net_loss target is tiny relative to the loud targets those same high-income records serve. The dense parent's 2,441 negative-misc carriers stay ≤5.0×, so its net_loss doesn't blow up.

Protect-list (dominant carrier per target) — documented method + counts

Per-target leverage scan |v_it| / Σ_j|v_jt| over the 5,533-target × 337,704-record value matrix: P_structural = 4,122 records = each target's top-1 leverage carrier ∪ records with leverage ≥ 0.5. Only 35/5,533 targets have any ≥0.5 carrier and 7 are near-sole (≥0.9) — the pool is overwhelmingly diffuse, so the frozen selection covers every non-misc target through substitutes. P_structural ∩ frozen = 1,580 (removal-protected). misc carriers, explicit: pool 7,526 nonzero (5,085 pos / 2,441 neg); frozen 1,957 (1,167 pos / 790 neg).

What the mechanism ruled out, and the fix

Warm-start reselection, initialized from the frozen 57,240 (never cold; #346's fourth-arm shows a hard-ranked prior blows up at 0.330). The corrected #346 init signal is contribution-share/leverage, and the protect-list preserves the diffuse portfolio — so the swap had to be small and targeted. De-risked candidates on the direct checkpoint path (restrict → mass-repair → recompile → calibrate, dense config):

• Adding positive-misc carriers (the naive read) is insufficient — net_income is already met: N=1500 → misc −67.1%, income tax +1.61%.
• Adding negative-misc carriers is catastrophic (more maxed loss carriers): +1,651 neg → −201%, +1,651 neg +1,000 pos → −211%, all 7,526 misc carriers → −232%.
• The fix (rmloss100): REMOVE the 100 largest-|misc| negative carriers (the maxed loss mass, |misc| $55k–$4.52M), KEEP the small ones (retain net_loss-target support — never zero it), REPLACE with 100 high-AGI misc-zero "absorber" records (AGI $2.09M–$46M) so the solve maxes those for the high-income targets instead. The solve then hits the net_loss target rather than overshooting it.

Selection change vs frozen

• Held exactly 57,240 records. New manifest identities_sha256 152baca3; round-trip join against the base pool verified (n_unmapped 0, n_ambiguous 0 — the release selects exactly this support).
• Overlap with frozen 99.825% (57,140 / 57,240); 100 entered, 100 left. Negative-misc carriers 790 → 690; misc-nonzero records 1,957 → 1,857 (support count falls, but the export rises — the removed carriers were dragging the net down, not the count up).
• Gate-directed cleanup: the rmloss100 selection makes takes_up_tanf_if_eligible non-degenerate (it now carries real TANF take-up variation, unlike the frozen constant-True), so the degenerate-input gate (Gate releases on input columns stuck at the engine default #286) correctly flagged its now-stale US_DEGENERATE_INPUT_REVIEWED_EXCLUSIONS entry and it was removed — the gate's "cannot rot" safety working as designed (a positive side effect of the reselection).
• Zero-support exclusions recomputed on rmloss100: identical 19 cells to Build H frozen (1 AL TANF + 18 state-AGI under-$1 taxable-interest amount/returns across al/ct/in/me/ms/ne/nm/ri/wy) — the misc-only swap leaves those records untouched. Post-exclusion zero-support = 0.

Results — Build I (rmloss100) vs Build H frozen-57k (verbatim)

metric	Build I (rmloss100)	Build H frozen-57k
records	57,240	57,240
final loss	0.030833	0.03091516
within-10%	88.88%	89.01%
ESS	12,303	13,184
fed income tax (SOI)	+0.22%	+0.45%
SS (SSA)	−0.18%	−0.18%
net capital gain (CBO agg)	−24.18%	−26.65%
mortgage tax-exp (JCT)	+36.73%	+35.48%
misc net_loss leg	−0.01% (HIT)	+322% (overshoot)
all structural gates	PASS	PASS
zero-support (post-excl)	0	0
export-mass parity	PASS (0 / 35)	FAIL (1 / 35)

final_loss and income-tax fit BEAT the frozen-57k; within-10% is −0.13pp (run-to-run noise, same 5514-target surface); realized_max_weight_ratio 4.993.

Export-mass parity — PASSED (0 failures / 35 columns)

Reference populace_us_2024.h5 (c2065b64), ±50%, $1B floor; reviewed exclusions estate_income, non_sch_d_capital_gains only — both applied and used. Per-column (gate-faithful, from the exported H5):

export column	Build I mass	drift	±50% band	Build H frozen	decision
miscellaneous_income	$52.85B	+11.5%	[23.70, 71.10]	−79.9% (FAIL)	LIVE → in-band
home_mortgage_interest	$398.08B	+27.9%	[155.56, 466.69]	+26.9%	LIVE → in-band
first_home_mortgage_interest	$397.92B	+28.0%	[155.42, 466.26]	+27.0%	LIVE → in-band
estate_income	$39.22B	−60.2%	[49.22, 147.65]	(excl)	reviewed exclusion (ref ~2.1× SOI)
non_sch_d_capital_gains	$134.85B	+78.0%	[37.87, 113.62]	(excl)	reviewed exclusion (ref ~8× SOI)

The three LIVE columns are all in band. The two mortgage columns held their Build H in-band identification; miscellaneous_income moved from −79.9% (FAIL) to +11.5% because the net_loss leg is now hit (−0.01%) rather than overshooting.

Reform-validation tail bug (does not affect certification)

The certification — all gates plus the exported H5 — was reproduced identically across three independent runs (…220632Z, …233711Z, …010610Z), each writing release_gates.passed=True, export-mass 0/35, and final_loss 0.03083269 / within-10% 0.8888284. Diagnostics are written only after the gates pass and before the H5 (verified stable, household 57,240 rows). What trails the H5 is the optional reform-validation report, and it hit a pre-existing bug at the 2025 period: md_withheld_income_tax → ParameterNotFoundError 'gov.local.md.flat_rate.UNKNOWN', which aborted release_manifest.json.

This is not an artifact defect. Three-way evidence that the exported dataset is geographically clean: (1) via the tail's own loader (USSingleYearDataset→Microsimulation) all 1,091 MD households map to valid MD counties at 2024 and 2025 — zero UNKNOWN counties in the 57,240 — and md_withheld_income_tax@2025 computes fine ($28.13B); (2) the target-frame checkpoint carries county_fips and _dataset_from_frame preserves it; (3) full-frame reform simulations compute income_tax cleanly. The crash appears only in the batched/zero-variable-reform construction, and the live default artifact (c2065b64, no county_fips persisted) is more exposed — earlier tails were simply interrupted before reaching this row. Filed as PolicyEngine/policyengine-us#8975 (formula lacks an MD-UNKNOWN guard) and #367 (harness derives UNKNOWN county for records whose artifact carries valid county_fips).

Remediation: the release reruns cleanly with --skip-reform-validation (the reform-validation report is optional; all certification output — gates, export-mass parity, H5 — completes before it), which writes release_manifest.json. The dataset is byte-identical; only the optional validation report is skipped.

Known scope limitation — #340 (unchanged, out of scope)

Column-manifest check on the exported H5: 7/8 never-persisted families ABSENT (tips / overtime / auto-loan / education / casualty / misc-itemized / childcare → ~$0); the lone PRESENT is traditional_ira_contributions_desired (a behavioural takeup INPUT, not the saver's-credit base) — identical to Build H. Build I does not address #340 (its own work); this verdict covers the calibration/fiscal-target surface only.

Verdict: CERTIFIABLE

The one gate the deployable frozen-57k failed in Build H — miscellaneous_income export-mass — now PASSES (+11.5%, in band), via a pure selection-support fix (not a target change, not a reviewed exclusion). All structural gates PASS; export-mass parity PASSES (0/35, only estate_income + non_sch_d_capital_gains reviewed-excluded by SOI band math); zero-support 0; final_loss (0.03083) and income-tax fit (+0.22%) beat Build H frozen; both mortgage columns stay in band; the misc net_loss leg is now hit. The change is a 100-record warm-start swap (99.825% overlap) that the mechanism analysis, not a ranked prior, dictated.

Caveat: the only failure in the run is the optional reform-validation tail's pre-existing MD-county bug (policyengine-us#8975 / populace#367), which does not touch the certified dataset — the exported artifact is geographically clean and the release completes to a manifest with --skip-reform-validation. The gate that Build H flagged as the blocker is fixed; nothing else regressed.

Publication and the pe.py default flip are Max's call — nothing here touched prod HF or latest.json.

Cache-key: base 18833fb6; pe-us 1.764.6 / core 3.26.11; period 2024; seed 0; feed v8 94b7155f; registry d71c59514e3a (5514 compiled sparse); selection 152baca3; zero-support sha abb106af; export-mass ref c2065b64. Manifest-bearing release id populace-us-2024-buildi-sparse-rmloss100-6e8e929-20260709T034135Z (rc=0, --skip-reform-validation): release_manifest.json sha256 c2bbe50a75279a83f741ec2267134702ee0e1de2e7babb74bd72ef148d25aa4b, exported H5 sha256 5e2a10b97f1fa740fbc1cc1be3450a19b2089c56de59d3754db13ec0dd048b88 — byte-identical certification to the reform-validation runs (…233711Z / …010610Z). Artifacts under ~/PolicyEngine/_buildi-runtime/out/buildi-run/sparse/; working log PROGRESS_BUILDI.md on branch build-i-run. This resolves the frozen-57k NOT-CERTIFIABLE verdict from #299 (Build H).