Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0

### Fixed

- Graph refresh after a web restart: when a rebuild finds an inventory collection still in progress, the background graph timer retries every two minutes until the collection finishes (bounded by the collector's seven-minute limit) instead of keeping the previous request-flow, infrastructure and service-map graphs until the timing happens to shift, which could last indefinitely after a restart.
- Native observability connectors: connection tests verify bounded query access instead of public health endpoints. Explicit instance reads require ID-bound credentials; migrated defaults must be backfilled through the administrator save flow before rollout. Prometheus and Mimir default to a 10-second execution ceiling, configurable per datasource and tightened by Explore; provider response bytes and body reads are bounded. Valid short Jaeger trace IDs are retained, and local loss, search saturation and unknown results carry distinct completeness evidence.
- Security scanning hardening: deployment workflows select only explicit preview secrets; chat history deletion hides internal errors and session IDs always use cryptographic randomness. The configurator escapes literal HCL strings, screenshot login requires exact configured origins and valid TLS, presenter titles retain text semantics, datasource hostname classification checks the AWS DNS suffix including root-dot and China-partition forms, and rotation diagnostics omit unrelated secret identifiers.
- AgentCore Gateway inbound authentication: the provisioner creates every section gateway with IAM (SigV4) inbound authentication instead of none. Because a gateway's authorizer type cannot be changed after creation, `make agentcore` fails on any existing gateway that still accepts unauthenticated calls, and the AgentCore reference documents the operator replacement procedure.
Expand Down Expand Up @@ -718,6 +719,7 @@ First release of the **v2 line** (versioned independently from the v1 1.x line,

### 수정

- web 재시작 후 그래프 갱신: 그래프 재생성 시점에 인벤토리 수집이 진행 중이면 백그라운드 타이머가 수집이 끝날 때까지 2분 간격으로 다시 시도합니다(수집기의 7분 제한까지). 이전에는 재시작 시각에 따라 요청 흐름·인프라·서비스 맵 그래프가 계속 이전 상태로 남을 수 있었습니다.
- 외부 관측 데이터소스: 연결 테스트가 공개 상태 엔드포인트 대신 제한된 실제 조회 권한을 검증합니다. 인스턴스 ID 조회는 해당 ID의 자격 증명만 허용하며, 이전 기본 연결은 배포 전에 관리자 저장 흐름으로 자격 증명을 보완해야 합니다. Prometheus·Mimir의 기본 실행 제한은 10초이며 데이터소스별 설정과 Explore의 더 짧은 제한을 적용합니다. 응답 크기와 본문 읽기 시간을 제한하고, 짧은 Jaeger trace ID를 보존하며 실제 누락·조회 한도 도달·판별 불가 결과를 구분합니다.
- 보안 검사 개선: 배포 워크플로가 명시된 미리보기 시크릿만 선택하며, 채팅 기록 삭제 오류는 내부 정보를 숨기고 세션 ID는 암호학적 난수만 사용합니다. 설정 도구는 HCL 문자열을 안전하게 인코딩하고, 스크린샷 로그인은 지정한 정확한 출처와 유효한 TLS를 요구합니다. 발표자 제목은 텍스트로 보존하며, 데이터소스 호스트 분류는 DNS 루트 점과 중국 파티션을 포함한 AWS 도메인 접미사를 확인하고, 회전 진단에서는 무관한 시크릿 식별자를 생략합니다.
- AgentCore Gateway 유입 인증: 프로비저너가 모든 섹션 Gateway를 인증 없음 대신 IAM(SigV4) 유입 인증으로 생성합니다. Gateway의 인증 방식은 생성 후 바꿀 수 없으므로, 아직 인증 없는 호출을 받는 기존 Gateway가 있으면 `make agentcore`가 실패하며, 운영자 교체 절차는 AgentCore 참조 문서에 안내합니다.
Expand Down
11 changes: 11 additions & 0 deletions docs/runbooks/graph-read-contract.md
Original file line number Diff line number Diff line change
Expand Up @@ -376,6 +376,17 @@ itself or relax source freshness limits: `INVENTORY_STALE_AFTER_MINUTES` and the
existing source-quality/clock rules still apply. These steps describe operator
work to perform; timer activation and a live release require separate evidence.

The timer and the inventory sync share the 15-minute cadence, so a web restart can place
every tick while an inventory producer is still `running`; each tick then retains flow/infra
and skips trace. When a flow or infra attempt is withheld while a producer is `running`, the
rebuild reports `inventory_running` and the timer starts one retry chain in 2-minute steps. The
chain covers the inventory sync Lambda's 420-second budget plus one step (four retries at a
15-minute interval), stops at the first cycle that no longer waits, ends at least one step
before the next regular tick, and is never started twice concurrently. Intervals under five
minutes do not retry, because their regular ticks already land outside a producer pass. The
regular interval is unchanged. Logs show `[graph-rebuild] infra: {... "reasons":["inventory_running"]}`
for each waiting cycle.

## Verification commands

Use browser developer tools on an already-authorized page to distinguish HTTP503/busy,
Expand Down
21 changes: 15 additions & 6 deletions web/instrumentation.ts
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,7 @@ export async function register() {
const { loadGraphSources } = await import('./lib/graph-sources');
const { graphDiagnostic } = await import('./lib/graph-state');
const { executeGraphLayer } = await import('./lib/graph-execution');
const { inventoryRetryPlan, needsInventoryRetry, withInventoryRetry } = await import('./lib/graph-rebuild-retry');
const pool = getPool();
const execute = async (stage: string, action: () => ReturnType<typeof rebuildGraph>) => {
return executeGraphLayer(stage, action, (line, failed) => console[failed ? 'error' : 'log'](line));
Expand All @@ -34,18 +35,21 @@ export async function register() {
// would pile up duplicate concurrent read/source calls. Skipping a tick (not queuing it) is fine:
// the next interval fires regardless, and a rebuild is idempotent.
let running = false;
const run = async () => {
if (running) return;
// Returns whether a flow/infra layer waited on a running inventory producer.
const cycle = async (): Promise<boolean> => {
if (running) return false;
running = true;
let waited = false;
try {
await execute('flow', () => rebuildGraph(pool));
const flow = await execute('flow', () => rebuildGraph(pool));
const infra = await execute('infra', () => rebuildInfraGraph(pool));
waited = needsInventoryRetry([flow, infra]);
const qualification = infra.selfInfraStatus === 'degraded' || infra.selfInfraStatus === 'stale' ? infra.selfInfraStatus : undefined;
if (!infra.selfInfraComplete && !qualification) {
console.error(infra.failed ? '[graph-rebuild] trace skipped: infra execution failed'
: '[graph-rebuild] trace skipped: infra publication incomplete');
await execute('trace', () => recordTraceDependencySkip(pool));
return;
return waited;
}
if (qualification) console.error(`[graph-rebuild] trace qualified: self infra ${qualification}`);
// Registry-driven (2026-07-08): sources come from every registered datasource's pre-built
Expand All @@ -67,9 +71,14 @@ export async function register() {
} finally {
running = false;
}
return waited;
};
// A cycle withheld by a running inventory producer retries in bounded 2-minute steps across the
// producer's runtime budget (see lib/graph-rebuild-retry.ts); otherwise a restart can
// phase-lock every tick onto the sync.
const run = withInventoryRetry(cycle, inventoryRetryPlan(mins));

setTimeout(run, 60_000); // first attempt ~60s after boot; interval ticks keep the same overlap guard
setInterval(run, mins * 60_000);
setTimeout(() => run(), 60_000); // first attempt ~60s after boot; interval ticks keep the same overlap guard
setInterval(() => run(), mins * 60_000);
}
}
2 changes: 2 additions & 0 deletions web/lib/fixtures/graph-fatal-child.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@ const store = load(resolve('lib/graph-store.ts'));
const { graphTransaction } = load(resolve('lib/graph-inventory.ts'));
const state = load(resolve('lib/graph-state.ts'));
const execution = load(resolve('lib/graph-execution.ts'));
const retry = load(resolve('lib/graph-rebuild-retry.ts'));
let original;
let injected = false;
let failureAttempts = 0;
Expand Down Expand Up @@ -125,6 +126,7 @@ try {
const exports = specifier.includes('/db') ? { getPool: () => target }
: specifier.includes('graph-sources') ? { loadGraphSources: async () => ({ sources: [], metricsSources }) }
: specifier.includes('graph-execution') ? execution
: specifier.includes('graph-rebuild-retry') ? retry
: specifier.includes('graph-state') ? state : store;
const module = new vm.SyntheticModule(Object.keys(exports), function() {
for (const [key, value] of Object.entries(exports)) this.setExport(key, value);
Expand Down
57 changes: 57 additions & 0 deletions web/lib/graph-rebuild-retry.test.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,57 @@
import { describe, expect, it, vi } from 'vitest';
import { inventoryRetryPlan, needsInventoryRetry, withInventoryRetry } from './graph-rebuild-retry';

describe('inventory-running retry', () => {
it('covers the 420 s producer budget plus one step and stays clear of the next regular tick', () => {
expect(inventoryRetryPlan(15)).toEqual({ delayMs: 120_000, maxRetries: 4 }); // 2..8 min, past 7 min
expect(inventoryRetryPlan(60)).toEqual({ delayMs: 120_000, maxRetries: 4 });
expect(inventoryRetryPlan(6)).toEqual({ delayMs: 120_000, maxRetries: 2 }); // last retry at 4 of 6 min
for (const mins of [6, 15, 60]) {
const plan = inventoryRetryPlan(mins);
expect(plan.maxRetries * plan.delayMs).toBeLessThanOrEqual(mins * 60_000 - plan.delayMs);
}
});

it('plans no retries for short or invalid intervals, whose regular ticks already recover', () => {
for (const mins of [0.5, 1, 4, Number.NaN]) expect(inventoryRetryPlan(mins).maxRetries).toBe(0);
});

it('detects the reason on any layer result and ignores failed layers without totals', () => {
expect(needsInventoryRetry([{ totals: { reasons: [] } }, { totals: { reasons: ['inventory_running'] } }])).toBe(true);
expect(needsInventoryRetry([{ totals: { reasons: ['rebuild_busy'] } }, {}, undefined])).toBe(false);
});

it('runs one bounded chain while the producer keeps running and never a second concurrent chain', async () => {
const callbacks: (() => void)[] = [];
const schedule = vi.fn((callback: () => void, _ms: number) => { callbacks.push(callback); });
const cycle = vi.fn(async () => true);
const run = withInventoryRetry(cycle, { delayMs: 120_000, maxRetries: 3 }, schedule);
await run();
await run(); // a regular tick while the chain is pending does not start another chain
expect(schedule).toHaveBeenCalledTimes(1);
for (let step = 0; step < 3; step++) {
callbacks[step]();
await vi.waitFor(() => expect(cycle).toHaveBeenCalledTimes(3 + step));
}
// Three retries ran; the third was the last allowed step, so nothing more is scheduled.
expect(schedule).toHaveBeenCalledTimes(3);
expect(schedule.mock.calls.every(call => call[1] === 120_000)).toBe(true);
});

it('stops the chain at the first cycle that no longer waits', async () => {
const callbacks: (() => void)[] = [];
const schedule = vi.fn((callback: () => void) => { callbacks.push(callback); });
const cycle = vi.fn().mockResolvedValueOnce(true).mockResolvedValue(false);
await withInventoryRetry(cycle, { delayMs: 120_000, maxRetries: 4 }, schedule)();
callbacks[0]();
await vi.waitFor(() => expect(cycle).toHaveBeenCalledTimes(2));
expect(schedule).toHaveBeenCalledTimes(1);
});

it('schedules nothing when the cycle did not wait or the plan allows no retries', async () => {
const schedule = vi.fn();
await withInventoryRetry(async () => false, { delayMs: 120_000, maxRetries: 4 }, schedule)();
await withInventoryRetry(async () => true, { delayMs: 120_000, maxRetries: 0 }, schedule)();
expect(schedule).not.toHaveBeenCalled();
});
});
46 changes: 46 additions & 0 deletions web/lib/graph-rebuild-retry.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,46 @@
// Bounded follow-up for the in-process graph timer. The rebuild interval and the inventory sync
// share a cadence, so a restart can phase-lock rebuilds onto a running inventory producer: every
// tick then sees `running` and retains the previous graph. A regular tick that reports
// `inventory_running` starts a short retry chain that covers the producer's runtime budget; the
// chain stops at the first cycle that no longer waits, and never runs into the next regular tick.

export const INVENTORY_RETRY_REASON = 'inventory_running';

// steampipe.tf caps the inventory sync Lambda at 420 s, so a ledger row stays `running` for at most
// that long; the chain reaches one step past it.
const PRODUCER_BUDGET_MS = 420_000;
const RETRY_STEP_MS = 120_000;

export interface InventoryRetryPlan { delayMs: number; maxRetries: number }

/** Retry steps for an interval. Intervals under 5 minutes need none: regular ticks already land
* outside a producer pass within a few minutes. Otherwise 2-minute steps cover the producer budget
* plus one step, and the last retry still runs at least one step before the next regular tick. */
export function inventoryRetryPlan(intervalMins: number): InventoryRetryPlan {
const intervalMs = intervalMins * 60_000;
if (!Number.isFinite(intervalMs) || intervalMs < 5 * 60_000) return { delayMs: RETRY_STEP_MS, maxRetries: 0 };
const window = Math.min(PRODUCER_BUDGET_MS + RETRY_STEP_MS, intervalMs - RETRY_STEP_MS);
return { delayMs: RETRY_STEP_MS, maxRetries: Math.floor(window / RETRY_STEP_MS) };
}

/** True when any layer result says publication waited on a running inventory producer. */
export function needsInventoryRetry(results: ({ totals?: { reasons: string[] } } | undefined)[]): boolean {
return results.some(result => result?.totals?.reasons.includes(INVENTORY_RETRY_REASON) === true);
}

/** Wraps a rebuild cycle: a waiting regular tick starts one retry chain of at most `maxRetries`
* steps, a waiting retry continues it, and nothing starts a second chain while one is pending. */
export function withInventoryRetry(
cycle: () => Promise<boolean>, plan: InventoryRetryPlan,
schedule: (callback: () => void, ms: number) => unknown = (callback, ms) => setTimeout(callback, ms),
): (retry?: number) => Promise<void> {
let pending = false;
const run = async (retry = 0): Promise<void> => {
if (retry) pending = false;
const waited = await cycle();
if (!waited || pending || retry >= plan.maxRetries) return;
pending = true;
schedule(() => { void run(retry + 1); }, plan.delayMs);
};
return run;
}
20 changes: 19 additions & 1 deletion web/lib/graph-rebuild-runner.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -57,7 +57,8 @@ function run(options: Record<string, unknown> = {}) {
if (sql.includes('FROM inventory_sync_runs')) {
const now = fixtureNow - (stage === 'infra' && input.infraOutcome === 'stale' ? 3600000 : 0);
return { rows: args[0].map(resource_type => ({ resource_type, account_id: 'self',
status: stage === 'infra' && input.infraOutcome === 'retained' && resource_type === 'vpc' ? 'failed' : 'succeeded',
status: stage === 'infra' && input.infraOutcome === 'retained' && resource_type === 'vpc' ? 'failed'
: stage === 'infra' && input.infraOutcome === 'running' && resource_type === 'vpc' ? 'running' : 'succeeded',
row_count: stage === 'infra' && input.infraOutcome === 'degraded' && resource_type === 'vpc' ? 1 : 0,
unknown_attribute_count: stage === 'infra' && input.infraOutcome === 'degraded' && resource_type === 'vpc' ? 1 : 0, version: '1',
started_at: new Date(now - 2000).toISOString(),
Expand Down Expand Up @@ -160,6 +161,23 @@ function run(options: Record<string, unknown> = {}) {
}

const cycles = (timer: boolean) => timer ? 2 : 1;
describe('inventory-running retry on the timer', () => {
const env = { NEXT_RUNTIME: 'nodejs', GRAPH_REBUILD_INTERVAL_MINS: '15' };
it('starts one bounded retry chain when a running inventory producer withholds infra', () => {
const result = run({ timer: true, infraOutcome: 'running', env });
expect(result.errors).toContain('[graph-rebuild] trace skipped: infra publication incomplete');
const infra = result.logs.find((line: string) => line.startsWith('[graph-rebuild] infra:'));
expect(JSON.parse(infra.split(': ').slice(1).join(': ')).reasons).toContain('inventory_running');
// Boot timeout + interval, then exactly one pending 2-minute retry despite later ticks also waiting.
expect(result.scheduled).toEqual([['timeout', 60000], ['interval', 900000], ['timeout', 120000]]);
});

it('does not retry a failed (not running) producer', () => {
const result = run({ timer: true, infraOutcome: 'retained', env });
expect(result.scheduled).toEqual([['timeout', 60000], ['interval', 900000]]);
});
});

describe('graph execution and publication contract', () => {
it.each([false, true])('uses real publisher totals and writes trace (timer=%s)', timer => {
const result = run({ timer });
Expand Down
10 changes: 10 additions & 0 deletions web/lib/graph-store-postgres.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -347,6 +347,16 @@ describe.skipIf(!socket)('inventory graph publication on PostgreSQL', () => {
expect(await build('infra')).toMatchObject({ published: 0, retained: 1 });
expect(await state('infra')).toMatchObject({ status: 'error', retainedPrevious: true });
});
it('reports inventory_running only when a still-running producer withholds publication', async () => {
await seed('infra');
expect(await build('infra')).toMatchObject({ published: 1, reasons: [] });
await pool.query(`UPDATE inventory_sync_runs SET status='running' WHERE resource_type='vpc'`);
const waiting = await build('infra');
expect(waiting).toMatchObject({ published: 0, retained: 1 });
expect(waiting.reasons).toContain('inventory_running');
await pool.query(`UPDATE inventory_sync_runs SET status='failed' WHERE resource_type='vpc'`);
expect((await build('infra')).reasons).not.toContain('inventory_running');
});
it.each(['flow', 'infra'].flatMap(cls => ['failed', 'partial', 'running'].map(status => [cls, status])))(
'%s member first publication includes aggregate %s with no member rows/history for that type', async (cls, status) => {
await seed(cls, recent, '111122223333');
Expand Down
8 changes: 7 additions & 1 deletion web/lib/graph-store.ts
Original file line number Diff line number Diff line change
Expand Up @@ -69,7 +69,7 @@ export const SELF_INFRA_STATES = ['complete', 'degraded', 'stale', 'retained', '
export const GRAPH_REBUILD_REASONS: ReadonlySet<string> = new Set(['publication_busy', 'superseded', 'rebuild_busy', 'state_schema_missing',
'account_limit', 'time_limit', 'skip_record_busy', 'skip_record_failed', 'snapshot_limit',
'graph_limit', 'account_failed', 'rebuild_deadline', 'collection_ok', 'collection_empty', 'collection_partial',
'collection_unavailable', 'collection_error']);
'collection_unavailable', 'collection_error', 'inventory_running']);

const emptyResult = (): GraphRebuildResult =>
({ nodes: 0, edges: 0, published: 0, retained: 0, skipped: 0, degraded: 0, reasons: [] });
Expand Down Expand Up @@ -207,6 +207,12 @@ async function rebuildInventory(pool: Pool, cls: GraphClass, lock: number, runId
counts ??= await inventoryCounts(pool, types));
attempt = inventoryAttempt(snapshot, accountTypes, cls, account, attemptedAt);
if (snapshot.truncated) reason('snapshot_limit');
// A producer that is still writing withholds publication only until it finalizes; the
// timer uses this reason to retry once after the run instead of waiting a full interval.
const sources = Array.isArray(attempt.details.sources) ? attempt.details.sources as { producerStatus?: unknown }[] : [];
if (!attempt.publish && sources.some(source => source?.producerStatus === 'running')) {
reason('inventory_running');
}
const graph = attempt.publish ? build(snapshot.rows, account) : { nodes: [], edges: [] };
if (graph.nodes.length > 4000 || graph.edges.length > 8000
|| Buffer.byteLength(JSON.stringify(graph)) > 8 * 1024 * 1024) {
Expand Down
Loading