Repository navigation
fix(topology): retry a graph rebuild once when inventory is still collecting - #252
Conversation
…lecting The web graph timer and the inventory sync share a 15-minute cadence. After the dev web restart on 2026-11-10 the timer ticked 8 seconds after each sync dispatch, while the ec2 producer (60-80s) was still `running`, so every tick retained flow/infra and skipped trace; the phase never drifts on its own. - rebuildInventory reports `inventory_running` when an attempt is withheld while a producer is still running. - The timer wraps its cycle with withInventoryRetry: such a cycle schedules one retry after min(3 min, half the interval), at least 1 min; a retry never schedules another and the regular interval is unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
🤖 AI Code Review (two independent reviewers)Reviewer responses: codex/ALL claude/ALL Status: PASSED — No blocking issues found All load-bearing claims are now verified. I have what I need to synthesize the chair review. Chair Review — PR #252: fix(topology): retry a graph rebuild once when inventory is still collecting1. SummaryThe PR adds a bounded retry chain to the in-process graph timer: when a flow/infra rebuild is withheld because an inventory producer is still 2. Issues per lensL2 — Code correctnessMINOR (downgraded from one model's MAJOR; the other model explicitly disagreed) — retry amplification: each retry re-runs the full flow/infra/trace cycle, and the MINOR (both models flagged — agreement, chair-verified) — a retry colliding with an in-flight regular tick resets the retry budget. When a scheduled retry fires while a regular tick's cycle is still running, the retry clears MINOR (one model) — L3 — Security / AWS mutation safetyNo findings — both models agree, chair-confirmed. The diff adds no Terraform, IAM, AWS SDK call, secret, or credential handling; L4 — Observability / data-integration correctnessNo gating findings. The data-integration premises are verified: 420 s producer budget and shared 15-minute cadence match MINOR (one model) — test coverage gap: no test covers the flow-withheld/infra-published combination, which is exactly the case where a retry re-collects trace needlessly. L5 — Docs/ADR consistencyThe runbook paragraph matches the implementation on every checkable number (2-minute steps, 420 s + one step, four retries at 15 min, sub-5-minute intervals don't retry, log shape), and the CHANGELOG entries are correctly placed in both locales' Unreleased 3. Suggestions
4. VerdictAll panel CRITICAL/MAJOR candidates were reconciled against BASE plus the complete patch. The single MAJOR candidate (retry amplification) is factually grounded but is a bounded, read-only efficiency/hardening concern within the timer's already-supported operating envelope, with no incorrect published data, no ADR-005 implication, and an independent dissent from the second model — downgraded to MINOR. The image manifest is empty (no changed images, no unavailable/omitted entries), and both panel cells declared coverage not required. IMAGE_COVERAGE: NOT_REQUIRED Triggered by commit |
…ry chain Review: a single 3-minute retry cannot outlast the inventory sync Lambda's 420-second budget, so a long pass left the timer phase-locked as before. - withInventoryRetry now takes a plan: a waiting regular tick starts one chain of 2-minute retries that continues while cycles still wait, up to the producer budget plus one step (4 retries at 15 minutes), ending at least one step before the next regular tick; a second chain never starts while one is pending. - Intervals under 5 minutes plan no retries (their regular ticks already land outside a producer pass), fixing the floor that exceeded half the interval. - Runbook and CHANGELOG describe the bounded chain instead of a one-shot retry. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Why
The in-process graph timer (
web/instrumentation.ts) and the inventory sync both run every 15 minutes. After the dev web restart for #250, the new task's timer phase landed 8 seconds after each inventory dispatch::10:41 / :25:41 / :40:41 / :55:41:10:49 / :25:49 / :40:49 / :55:49The ec2 producer takes 60–80 s, so every tick saw
inventory:ec2withproducerStatus: running.inventoryAttemptcorrectly refused to publish, which means flow/infra were retained and trace was skipped (trace skipped: infra publication incomplete). Both cadences are exactly 15 minutes, so the phase never drifts and the graphs stayed frozen until a manual restart moved the timer.What changes
rebuildInventory(web/lib/graph-store.ts) adds the reasoninventory_runningwhen an account attempt is withheld while any of its sources reportsproducerStatus: running. The reason is added toGRAPH_REBUILD_REASONS, soexecuteGraphLayeraccepts it.web/lib/graph-rebuild-retry.ts:withInventoryRetrywraps the timer cycle.inventory_runningon flow or infra, it schedules one retry aftermin(3 min, half the interval), at least 1 minute.docs/runbooks/graph-read-contract.md) and a bilingual CHANGELOGFixedentry.Verification
npx tsc --noEmit -p .is clean. The full webnpm testpasses: 373 files, 5,730 tests.postgres:17container: 6 files, 526 tests passed. This includes the new test: arunningvpc producer yieldsinventory_running, and afailedproducer does not.graph-rebuild-runner.test.ts):degraded: 0).🤖 Generated with Claude Code