Follow-up to #372/#376, measured on the first build to calibrate the FNS caseload targets: Build K (populace-us-2024-buildk-snapval-1fe5fc4-20260713T171826Z, staging-only, all gates green, registry b9286d4dc8a3 + 16 CHIP zero-support exclusions).
The wedge
Three measurements of the same 52 state household-caseload targets:
| view |
result |
| calibrated column (release diagnostics) |
52/52 within 10% |
| benefit-dollar targets |
9 → 5 misses vs Build J, all shrunk |
snap > 0 simulated from the exported h5 (what any downstream PE-US user computes) |
23/52 outside 10% (Build J: 38/52) |
The improvement is real (national −1.0% vs −4.7%; CA −35%→−16%; IL −31%→−14%) but the calibrated view materially overstates what the artifact delivers: AK +563%, DE +102%, KS +65%, DC +55% on simulated households.
Mechanism (verified)
The engine's take-up gating is NOT the problem: zero benefit-receiving spm_units carry takes_up_snap_if_eligible = False, and the recalibrated flags ship in the export (87.5% True vs Build J's uniform 82.1%). Two things drive the wedge:
- Fill rates are derived on pre-mass-repair design weights.
us_snap_state_take_up.json shows AK with eligible_weight 28,689 vs target 31,319 → saturated: true, 100% flagged. But the release then mass-repairs household weights ×5.33 (Census person-population anchor) before calibration, so the real eligible pool is ~150k+ and the flag constrains nothing. A dozen states saturate this way at design weights.
- Build-frame
is_snap_eligible ≪ export-sim eligibility. With identical shipped weights, the materialized caseload column sums to 31,264 in AK while export-simulated snap > 0 sums to 207,693 — so most sim-eligible taker units carried a 0 in the build-time column. The export's input surface (b335241 coverage vintage) omits eligibility-narrowing inputs the build frame carried; downstream simulations therefore see a much broader eligible set than the one the calibration pinned.
Proposed fixes
- Run the take-up stage on post-repair weights (or apply the mass repair before the stage) so saturation is decided against real-world-scale pools.
- Close the eligibility-surface gap: either export every input
is_snap_eligible consumes on the frame, or persist the build-time eligibility/caseload indicator itself as a dataset column so the calibrated measure is reproducible from the artifact.
- Add an artifact-level regression check: simulated
snap > 0 per state vs the FNS table, tolerance-gated, so calibrated-vs-delivered drift fails closed instead of being discovered by audit. (Before/after scorecards from Build J/K available as the seed for this.)
Directly relevant to the SNAP work-requirements deliverable: reform runs simulate from the export, so they inherit view 3, not view 1.
Refs #370, #372, #376; Build K context in #410/#411/#412 and PR #414.
🤖 Generated with Claude Code
Follow-up to #372/#376, measured on the first build to calibrate the FNS caseload targets: Build K (
populace-us-2024-buildk-snapval-1fe5fc4-20260713T171826Z, staging-only, all gates green, registryb9286d4dc8a3+ 16 CHIP zero-support exclusions).The wedge
Three measurements of the same 52 state household-caseload targets:
snap > 0simulated from the exported h5 (what any downstream PE-US user computes)The improvement is real (national −1.0% vs −4.7%; CA −35%→−16%; IL −31%→−14%) but the calibrated view materially overstates what the artifact delivers: AK +563%, DE +102%, KS +65%, DC +55% on simulated households.
Mechanism (verified)
The engine's take-up gating is NOT the problem: zero benefit-receiving spm_units carry
takes_up_snap_if_eligible = False, and the recalibrated flags ship in the export (87.5% True vs Build J's uniform 82.1%). Two things drive the wedge:us_snap_state_take_up.jsonshows AK witheligible_weight28,689 vs target 31,319 →saturated: true, 100% flagged. But the release then mass-repairs household weights ×5.33 (Census person-population anchor) before calibration, so the real eligible pool is ~150k+ and the flag constrains nothing. A dozen states saturate this way at design weights.is_snap_eligible≪ export-sim eligibility. With identical shipped weights, the materialized caseload column sums to 31,264 in AK while export-simulatedsnap > 0sums to 207,693 — so most sim-eligible taker units carried a 0 in the build-time column. The export's input surface (b335241 coverage vintage) omits eligibility-narrowing inputs the build frame carried; downstream simulations therefore see a much broader eligible set than the one the calibration pinned.Proposed fixes
is_snap_eligibleconsumes on the frame, or persist the build-time eligibility/caseload indicator itself as a dataset column so the calibrated measure is reproducible from the artifact.snap > 0per state vs the FNS table, tolerance-gated, so calibrated-vs-delivered drift fails closed instead of being discovered by audit. (Before/after scorecards from Build J/K available as the seed for this.)Directly relevant to the SNAP work-requirements deliverable: reform runs simulate from the export, so they inherit view 3, not view 1.
Refs #370, #372, #376; Build K context in #410/#411/#412 and PR #414.
🤖 Generated with Claude Code