Skip to content

fix(topology): retry a graph rebuild once when inventory is still collecting - #252

Merged
Atom-oh merged 2 commits into
devfrom
fix/graph-rebuild-inventory-retry
Nov 10, 2026
Merged

Atom-oh merged 2 commits into
devfrom
fix/graph-rebuild-inventory-retry

Conversation

@Atom-oh

@Atom-oh Atom-oh commented Nov 10, 2026

Copy link
Copy Markdown
Contributor

Why

The in-process graph timer (web/instrumentation.ts) and the inventory sync both run every 15 minutes. After the dev web restart for #250, the new task's timer phase landed 8 seconds after each inventory dispatch:

timestamps (UTC)
inventory dispatch :10:41 / :25:41 / :40:41 / :55:41
graph rebuild (new task) :10:49 / :25:49 / :40:49 / :55:49

The ec2 producer takes 60–80 s, so every tick saw inventory:ec2 with producerStatus: running. inventoryAttempt correctly refused to publish, which means flow/infra were retained and trace was skipped (trace skipped: infra publication incomplete). Both cadences are exactly 15 minutes, so the phase never drifts and the graphs stayed frozen until a manual restart moved the timer.

What changes

  • rebuildInventory (web/lib/graph-store.ts) adds the reason inventory_running when an account attempt is withheld while any of its sources reports producerStatus: running. The reason is added to GRAPH_REBUILD_REASONS, so executeGraphLayer accepts it.
  • New web/lib/graph-rebuild-retry.ts: withInventoryRetry wraps the timer cycle.
    • When a regular tick reports inventory_running on flow or infra, it schedules one retry after min(3 min, half the interval), at least 1 minute.
    • At most one retry is pending, and a retry never schedules another. The regular interval, the overlap guard, the dependency gates and the publication rules are unchanged.
  • The timer callbacks now return the cycle promise (they previously discarded it). Test harnesses that drive the timer await it.
  • Runbook (docs/runbooks/graph-read-contract.md) and a bilingual CHANGELOG Fixed entry.

Verification

  • npx tsc --noEmit -p . is clean. The full web npm test passes: 373 files, 5,730 tests.
  • PostgreSQL suites in a disposable postgres:17 container: 6 files, 526 tests passed. This includes the new test: a running vpc producer yields inventory_running, and a failed producer does not.
  • Unit tests:
    • retry delay bounds
    • reason detection
    • exactly one retry per tick, and none from a retry
    • no retry when the cycle did not wait
  • Timer harness (graph-rebuild-runner.test.ts):
    • a running producer schedules exactly one extra 60 s timeout at a 1-minute interval
    • a failed producer schedules none
  • Live context: after the manual restart, the 12:19 UTC rebuild published flow/infra and the feat(topology): build the service map from whole-window ClickHouse trace aggregates #250 trace aggregate (3 ClickHouse queries, 67 nodes / 129 edges, degraded: 0).

🤖 Generated with Claude Code

…lecting

The web graph timer and the inventory sync share a 15-minute cadence. After the
dev web restart on 2026-11-10 the timer ticked 8 seconds after each sync
dispatch, while the ec2 producer (60-80s) was still `running`, so every tick
retained flow/infra and skipped trace; the phase never drifts on its own.

- rebuildInventory reports `inventory_running` when an attempt is withheld while
  a producer is still running.
- The timer wraps its cycle with withInventoryRetry: such a cycle schedules one
  retry after min(3 min, half the interval), at least 1 min; a retry never
  schedules another and the regular interval is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Nov 10, 2026 •

Copy link
Copy Markdown

🤖 AI Code Review (two independent reviewers)

Reviewer responses: codex/ALL claude/ALL

Status: PASSED — No blocking issues found

All load-bearing claims are now verified. I have what I need to synthesize the chair review.


Chair Review — PR #252: fix(topology): retry a graph rebuild once when inventory is still collecting

1. Summary

The PR adds a bounded retry chain to the in-process graph timer: when a flow/infra rebuild is withheld because an inventory producer is still running (new allow-listed reason inventory_running), the timer retries in 2-minute steps across the sync Lambda's 420-second budget instead of waiting a full interval — fixing a real phase-lock where a web restart could leave the graphs permanently stale. I verified the core wiring end-to-end against BASE plus the patch: executeGraphLayer returns totals.reasons (web/lib/graph-execution.ts:41-50), the new reason is added to GRAPH_REBUILD_REASONS so projectOutcome admits it, producerStatus: 'running' is a legal source value (web/lib/graph-inventory-read.ts:179), and the 420 s / rate(15 minutes) premises match terraform/foundation/steampipe.tf:372,403. No CRITICAL or MAJOR finding survives verification.

2. Issues per lens

L2 — Code correctness

MINOR (downgraded from one model's MAJOR; the other model explicitly disagreed) — retry amplification: each retry re-runs the full flow/infra/trace cycle, and the running evidence has no freshness bound. The claude panel flagged this as MAJOR; codex, reviewing the same lens independently, found the emission logic correct. Chair verification of the underlying facts: the facts are real — flow TYPES (web/lib/graph-store.ts:28-35) includes types absent from INFRA_TYPES (graph-inventory-read.ts:14-15), the sync lambda marks one type running at a time, so a flow-only withheld layer still triggers up to 4 retries that each re-collect trace from external datasources; and a ledger row stranded at running (residual failed-finalizer path, sync_lambda.py:1707-1712) is not age-bounded by the new check. But the severity is overstated: (a) the retry is needed for the withheld flow layer — re-running infra/trace is inefficiency, not incorrectness, and rebuilds are idempotent by the timer's own design contract; (b) the worst case (full cycles every 2 minutes for ≤8 minutes per interval) sits inside an operating envelope the timer already supports — GRAPH_REBUILD_INTERVAL_MINS=1 runs a full cycle every minute and is exercised by existing tests; (c) a stranded running row is repaired by the next 15-minute EventBridge sync run, whose ON CONFLICT DO UPDATE overwrites the ledger (sync_lambda.py:1388-1395) — "indefinite" amplification requires the sync schedule itself to be dead, and even then the cost is 4 bounded read-only cycles per interval. No incorrect data is ever published; attempt.publish semantics are unchanged. Worth fixing (bound evidence age with attemptedAtMs + PRODUCER_BUDGET_MS, and/or skip trace re-collection on a retry when the originating cycle already published it), but not gate-blocking.

MINOR (both models flagged — agreement, chair-verified) — a retry colliding with an in-flight regular tick resets the retry budget. When a scheduled retry fires while a regular tick's cycle is still running, the retry clears pending, hits the running overlap guard, gets false, and the chain dies; the slow in-flight tick then finishes with waited === true and pending === false and starts a fresh chain (web/lib/graph-rebuild-retry.ts:40-46). Total retries can exceed maxRetries per interval and the documented "ends at least one step before the next regular tick" bound is not actually enforced for cycles slower than 2 minutes. Cycles are deadline-bounded, so impact is small — distinguish "skipped by overlap" from "did not wait" in the cycle's return value.

MINOR (one model) — void run(retry + 1) in the scheduled callback has no rejection handler (graph-rebuild-retry.ts:44). The shipped cycle in instrumentation.ts is total (outer try/catch/finally), so no unhandled rejection is reachable today, but the helper is exported and generic; a .catch() costs nothing.

L3 — Security / AWS mutation safety

No findings — both models agree, chair-confirmed. The diff adds no Terraform, IAM, AWS SDK call, secret, or credential handling; graph-rebuild-retry.ts computes integers and calls setTimeout, and the graph-store.ts change appends a fixed string from a closed allow-list to in-memory reasons. ADR-005's read-only boundary and the frozen-flag surfaces are untouched. The retried work is the same read-only rebuild the timer already performs.

L4 — Observability / data-integration correctness

No gating findings. The data-integration premises are verified: 420 s producer budget and shared 15-minute cadence match steampipe.tf; the phase-lock mechanism is real; producerStatus !== 'succeeded' already withholds publication via the incomplete_collection blocker (graph-inventory-read.ts:186-190), so the new reason fires only while a producer is genuinely marked running, and the chain stops at the first cycle without one. Codex confirmed the emission logic; claude verified member types cannot produce a member-only false trigger. The missing freshness bound on running evidence (one model, MINOR) is folded into the L2 amplification item above. Steampipe queries, CIS checks and AgentCore diagnosis logic are untouched.

MINOR (one model) — test coverage gap: no test covers the flow-withheld/infra-published combination, which is exactly the case where a retry re-collects trace needlessly.

L5 — Docs/ADR consistency

The runbook paragraph matches the implementation on every checkable number (2-minute steps, 420 s + one step, four retries at 15 min, sub-5-minute intervals don't retry, log shape), and the CHANGELOG entries are correctly placed in both locales' Unreleased ### Fixed/### 수정 sections with no PR/round numbers and no duplicate of an existing bullet. Three MINORs (one model, chair-verified against the runbook): the ## Related files and decisions list (docs/runbooks/graph-read-contract.md:566-574) omits the two new graph-rebuild-retry files; the offline vitest command at line 338 omits lib/graph-rebuild-retry.test.ts; and the docs do not state that each retry re-executes the full cycle including external trace collection, which an operator sizing datasource budgets should know. Additionally (codex), if the overlap race above is retained, the runbook's "never started twice concurrently / ends at least one step before the next regular tick" claims should be softened, since they are not strictly guaranteed.

3. Suggestions

  1. Bound the running evidence age in graph-store.ts (e.g. require attemptedAtMs/started_at within PRODUCER_BUDGET_MS) so a stranded ledger row cannot drive retries, making the code enforce what the comment asserts.
  2. Return a three-state result from cycle (waited / did-not-wait / skipped-by-overlap) so a retry colliding with an in-flight tick neither kills nor restarts the chain, restoring the documented strict bound.
  3. On a retry, skip trace re-collection (or the already-published layers) when the originating cycle completed them — only the withheld layer needs the re-run.
  4. Add web/lib/graph-rebuild-retry.ts / .test.ts to the runbook's related-files list and the line-338 vitest command; add one sentence on per-retry cost.
  5. Add a .catch() inside the scheduled callback in withInventoryRetry, and a test with a flow-only running type.

4. Verdict

All panel CRITICAL/MAJOR candidates were reconciled against BASE plus the complete patch. The single MAJOR candidate (retry amplification) is factually grounded but is a bounded, read-only efficiency/hardening concern within the timer's already-supported operating envelope, with no incorrect published data, no ADR-005 implication, and an independent dissent from the second model — downgraded to MINOR. The image manifest is empty (no changed images, no unavailable/omitted entries), and both panel cells declared coverage not required.

IMAGE_COVERAGE: NOT_REQUIRED


Triggered by commit e92f8629ed45cc53f86b5b99469602e62c3998b4 · workflow: .github/workflows/pr-review.yml

…ry chain

Review: a single 3-minute retry cannot outlast the inventory sync Lambda's
420-second budget, so a long pass left the timer phase-locked as before.

- withInventoryRetry now takes a plan: a waiting regular tick starts one chain of
  2-minute retries that continues while cycles still wait, up to the producer
  budget plus one step (4 retries at 15 minutes), ending at least one step
  before the next regular tick; a second chain never starts while one is pending.
- Intervals under 5 minutes plan no retries (their regular ticks already land
  outside a producer pass), fixing the floor that exceeded half the interval.
- Runbook and CHANGELOG describe the bounded chain instead of a one-shot retry.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@Atom-oh
Atom-oh merged commit c0ecb27 into dev Nov 10, 2026
6 checks passed

This branch was previously deployed

1 inactive deployment
ci-review-auto — e92f8629 Deployed Nov 10, 2026 by Atom-oh via AI Code Review #1000
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant