From ce1024faa9bbdccc778d2338f200a137b24ff82b Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=EC=98=A4=EC=A4=80=EC=84=9D=28Junseok=20Oh=29DevOps?= <52226147+Atom-oh@users.noreply.github.com> Date: Tue, 10 Nov 2026 09:48:32 +0000 Subject: [PATCH 1/4] feat(topology): build the service map from whole-window ClickHouse trace aggregates The ClickHouse trace source joined the newest 1,000 spans of a ~160k-span hour, so most parent/child pairs fell outside the sample: low-volume calls and databases dropped out and most spans showed as orphans. - ClickHouseOtelAggregateSource reads the catalog's exporter-schema table with three bounded read-only queries: service/kind/status attribute groups with span counts, parent->child service pairs and span-link pairs (1,000-row limit each; a saturated limit is cap_reached, any failed query fails the read). Generated/custom templates and Tempo keep span sampling. - buildTraceGraph weights aggregate rows and draws source-counted pairs as span evidence on the same service nodes. - Service-graph metric rows keep connection_type: database/messaging servers become db/queue nodes only without span evidence, and virtual_node rows draw the uninstrumented side as an external node. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 2 + docs/api-reference.md | 15 ++ scripts/v2/workers/graph_catalog.py | 4 +- scripts/v2/workers/test_graph_catalog.py | 2 +- web/lib/graph-sources.test.ts | 28 +++- web/lib/graph-sources.ts | 14 +- web/lib/graph-store-trace.test.ts | 22 +++ web/lib/graph-store.ts | 8 +- web/lib/trace-graph.test.ts | 71 ++++++++ web/lib/trace-graph.ts | 80 +++++++-- web/lib/trace-source.test.ts | 110 +++++++++++++ web/lib/trace-source.ts | 200 ++++++++++++++++++++++- 12 files changed, 525 insertions(+), 31 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e65963532..e127e870e 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -29,6 +29,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed +- Service map from complete trace counts: ClickHouse trace sources using the standard OTel exporter table now count service calls, database access, workloads and messaging across the whole one-hour window with three bounded read-only aggregate queries, instead of joining the newest 1,000 spans, so low-volume calls and databases no longer drop out and calls whose parent fell outside the sample no longer show as orphans. Custom or generated ClickHouse queries and Tempo keep span sampling. Prometheus/Mimir service-graph rows keep `connection_type`, so database and queue servers no longer appear as same-named services (trace spans name them when available) and uninstrumented callers or peers appear as external nodes. Rebuild the datasource graph catalog (`datasource_index`) to pick up the new metric query. - Web framework and runtime refresh: Next.js 16.4.5, React 19.3.1 and Node 26.13.0 retain authentication, ownership checks and standalone deployment; the request body guard uses the Next.js proxy convention. Refresh JavaScript/Python dependencies and AgentCore SDKs with compatibility pins, Terraform 1.17.1/provider locks, and Python 3.14 Lambda/container runtimes. The documentation guide builds and deploys on Node 26 with refreshed documentation security dependencies and Mermaid 12, preserving all four locales. Preserve gateway authentication, feature gates and release safeguards. - EKS member onboarding: the member cluster guide is now two owner commands — an Access Entry for the registered member read role and a cluster-scoped `AmazonEKSViewPolicy` association — with no Kubernetes group, kubeconfig, kubectl or RBAC step; View never reads Kubernetes Secrets, and the guide tells owners to remove any existing `AmazonEKSAdminViewPolicy`. Because View cannot list nodes, member node lists come from read-only EC2/node-group APIs through the same role and are labeled as an EC2/node-group view (no kubelet status, instance-type capacity as allocatable, no Fargate nodes); their Ready count shows as "EC2 running N · Ready unknown" and they are left out of reserved-% figures; a failed node read no longer marks the whole cluster unreachable. When an administrator opens EKS, member clusters whose Access Entry is verified are listed first in an "Access Entry verified" callout with a one-click [조회 등록] button; nothing is registered until the administrator clicks it (an app registration record only; AWSops creates no AWS or Kubernetes permissions), and an unregistered cluster stays unregistered. In diagnosis, the Kubernetes map and the node drilldown, EC2/node-group nodes read as "EC2 running · Ready unknown" with Pressure, Allocatable and Reserved shown as unknown instead of as unhealthy or fully allocatable. Entries created from the earlier guide keep working. - Bedrock models: chat answers with OpenAI GPT-6 Luna (without explicit Bedrock cache points, which GPT models reject); Explore "AI generate" writes queries with GPT-6.1 Terra and retries on Claude Sonnet 5.5 when Terra fails or its draft fails validation; primary, corrections and fallback share a 55-second request deadline with SDK cancellation; AI Diagnosis, routing, synthesis and the background workers move to Claude Sonnet 5.5, Opus 5.5 (AI Diagnosis Deep tier option) and Haiku 5.5. These models reject `temperature`, so requests no longer send it. Haiku calls and Sonnet calls with fixed output caps run without extended thinking so existing budgets still fit; Opus 5.5, which always thinks, gets extra output headroom for Deep report sections and invariant proposals. The web task now always holds a grant for its direct query-generation and assistant models (Terra, Sonnet 5.5, Haiku 5.5), so "AI generate" and the product-help assistant no longer fail with AccessDenied when hybrid routing is off; the chat runtime role additionally allows GPT-6 Luna. With `model_provider = mixed`, the Terraform variable `agentcore_chat_model_id = "global.anthropic.claude-sonnet-5-5"` rolls chat back to Sonnet 5.5 on every provisioning run, and the chat footer shows a model only when the answer carries its model provenance. @@ -709,6 +710,7 @@ First release of the **v2 line** (versioned independently from the v1 1.x line, ### Changed +- 전체 트레이스 집계 기반 서비스 맵: 표준 OTel exporter 테이블을 쓰는 ClickHouse 트레이스 원본은 최신 스팬 1,000개를 이어 붙이는 대신, 제한된 읽기 전용 집계 쿼리 3개로 1시간 구간 전체의 서비스 호출·DB 접근·워크로드·메시징을 셉니다. 호출량이 적은 연결이나 DB가 맵에서 빠지지 않고, 부모 스팬이 표본 밖에 있던 호출도 끊긴 스팬으로 표시되지 않습니다. 사용자 정의·생성된 ClickHouse 쿼리와 Tempo는 기존 스팬 표본 방식을 유지합니다. Prometheus/Mimir 서비스 그래프 지표는 `connection_type`을 보존해 DB·큐 서버가 같은 이름의 서비스로 그려지지 않으며(트레이스 스팬이 있으면 스팬 기준으로 표시), 계측되지 않은 호출자나 상대는 외부 노드로 표시됩니다. 새 지표 쿼리를 쓰려면 데이터소스 그래프 카탈로그(`datasource_index`)를 다시 만들어야 합니다. - 웹 프레임워크·런타임 최신화: Next.js 16.4.5, React 19.3.1, Node 26.13.0으로 인증·소유권 검사·standalone 배포를 유지하고 요청 본문 제한을 Next.js proxy 규약으로 옮깁니다. 호환성 버전 고정과 함께 JavaScript/Python 의존성·AgentCore SDK, Terraform 1.17.1·provider 잠금파일, Python 3.14 Lambda·컨테이너 런타임을 갱신합니다. 문서 가이드는 Node 26에서 빌드·배포하며 문서 사이트의 보안 의존성과 Mermaid 12를 갱신하고 4개 언어를 유지합니다. Gateway 인증·기능 게이트·릴리스 보호 장치는 유지합니다. - EKS 멤버 온보딩: 멤버 클러스터 가이드가 소유자 명령 두 개(등록된 멤버 읽기 역할의 Access Entry 생성, 클러스터 범위 `AmazonEKSViewPolicy` 연결)로 줄었으며 Kubernetes 그룹·kubeconfig·kubectl·RBAC 단계가 없습니다. View는 Kubernetes Secrets를 읽지 않으며, 가이드는 기존 `AmazonEKSAdminViewPolicy` 연결을 제거하도록 안내합니다. View로는 노드를 조회할 수 없으므로 멤버 노드 목록은 같은 역할로 읽기 전용 EC2/노드그룹 API에서 가져와 EC2/노드그룹 기준으로 표시합니다(kubelet 상태 없음, allocatable은 인스턴스 타입 용량, Fargate 노드 제외). 이 노드의 Ready 수는 "EC2 running N · Ready unknown"으로 표시되며 예약 비율 계산에서 제외됩니다. 노드 조회만 실패해도 클러스터 전체가 조회 불가로 표시되지 않습니다. 관리자가 EKS 화면을 열면 Access Entry가 확인된 멤버 클러스터가 "Access Entry 확인됨" 안내 영역에 먼저 표시되고 [조회 등록] 버튼 한 번으로 등록할 수 있으며, 관리자가 누르기 전에는 등록되지 않습니다(앱 등록 기록만 저장, AWSops는 AWS·Kubernetes 권한을 만들지 않음). 해제한 클러스터는 해제 상태로 유지됩니다. 진단 표·Kubernetes 맵·노드 상세에서도 EC2/노드그룹 노드는 "EC2 running · Ready unknown"으로 표시되며 Pressure·Allocatable·Reserved는 비정상이나 전량 할당 대신 알 수 없음으로 표시됩니다. 이전 가이드로 만든 항목도 그대로 동작합니다. - Bedrock 모델: 채팅은 OpenAI GPT-6 Luna로 답합니다(GPT 모델이 거부하는 명시적 Bedrock 캐시 포인트는 끔). Explore "AI로 생성"은 GPT-6.1 Terra로 쿼리를 만들고, Terra 호출이 실패하거나 초안이 검증에서 거부되면 Claude Sonnet 5.5로 다시 시도합니다. 기본 모델·수정·폴백은 하나의 55초 요청 제한 시간과 SDK 취소 신호를 공유합니다. AI 진단, 라우팅, 합성, 백그라운드 워커는 Claude Sonnet 5.5, Opus 5.5(AI 진단 Deep 단계 선택), Haiku 5.5로 바뀝니다. 이 모델들은 `temperature`를 거부하므로 요청에서 더 이상 보내지 않습니다. Haiku 호출과 출력 상한이 고정된 Sonnet 호출은 기존 상한이 유지되도록 확장 사고 없이 실행하고, 항상 사고하는 Opus 5.5는 Deep 리포트 섹션과 불변식 제안에 출력 여유분을 더 받습니다. 웹 태스크는 직접 호출하는 쿼리 생성·도움말 모델(Terra, Sonnet 5.5, Haiku 5.5) 권한을 항상 가지므로, 하이브리드 라우팅이 꺼져 있어도 "AI로 생성"과 제품 도움말이 AccessDenied로 실패하지 않습니다. 채팅 런타임 역할은 GPT-6 Luna도 허용합니다. `model_provider = mixed`일 때 Terraform 변수 `agentcore_chat_model_id = "global.anthropic.claude-sonnet-5-5"`로 채팅을 Sonnet 5.5로 되돌리며 매 프로비저닝마다 유지되고, 채팅 하단의 모델 표시는 답변에 모델 출처가 있을 때만 나타납니다. diff --git a/docs/api-reference.md b/docs/api-reference.md index 3695ca237..3171adf0d 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -544,6 +544,19 @@ runtime payloads for compatibility with older or malformed responses. | `readStatus`, `readReason`, `readTruncated` | API read availability/coverage, independent of collector status: `ok`, `partial` (`row_limit`), or `unavailable` (`busy`/`timeout`/`query_failed`). | | `sources[].scope/capturedAtMs/lastSuccessAtMs`, `publishedSources[]` | Current/saved source status, scope and reasons, plus optional capture/sweep clocks and saved provenance. Missing status is explicitly unknown; absent clocks are not fabricated. | +### Trace graph sources + +A ClickHouse datasource whose registry query is the catalog's exporter-schema `otel_v1` template +(or the unregistered default) is read as whole-window aggregates of the same table, not as the +newest spans. Three bounded read-only queries return service/kind/status attribute groups with +span counts, parent→child service pairs and span-link pairs, each limited to 1,000 rows; a +saturated limit adds `cap_reached`, and any failed query fails the whole source read. Node and +edge `spanCount` then count spans in the window, and source-counted pairs produce no +`orphanSpans`. Generated or custom templates, Tempo and other sources keep span sampling. +Service-graph metric rows keep Tempo's `connection_type`: `database` and `messaging_system` +servers become `db`/`queue` nodes only when no trace spans are present (spans carry the +host-qualified identity), and `virtual_node` rows draw the uninstrumented side as an `external` node. + ### Trace edge instance observations Trace `queries` edges (service → `db`) may carry three additive, optional `meta` fields built by @@ -556,6 +569,8 @@ no bound observation was recorded, not that no instance queried the database. | `instance_observations_omitted` | Count of further distinct pairs dropped by the 200-pair cap. | | `instance_observations_unbound` | Count of spans whose EC2 claim could not be bound: missing or invalid account/region/clock, or no unique inventory EC2 instance and RDS instance endpoint in the same account and region. | +For aggregate sources, `observation_count` and `instance_observations_unbound` count the spans behind each aggregate row, and `correlation_id` names one sample span of the row. + These are sampled telemetry claims bound to inventory, not authority: a pair shows that the sampled spans named that instance and endpoint within the trace window. It does not prove current connectivity, credentials or complete coverage of instances querying the database. diff --git a/scripts/v2/workers/graph_catalog.py b/scripts/v2/workers/graph_catalog.py index f4998b808..fdab80f4a 100644 --- a/scripts/v2/workers/graph_catalog.py +++ b/scripts/v2/workers/graph_catalog.py @@ -121,7 +121,9 @@ def _servicegraph_calls(kind, schema): if isinstance(schema, dict): metrics = {m for m in (schema.get("metrics") or []) if isinstance(m, str)} if _SERVICEGRAPH_METRIC in metrics: - labels = ",".join(("client", "server") + _IDENTITY_LABELS) + # connection_type separates database/messaging/virtual-node rows from service calls; dropping + # it would merge a db.name server label into a same-named service node. + labels = ",".join(("client", "server", "connection_type") + _IDENTITY_LABELS) expr = f"sum by ({labels}) (increase({_SERVICEGRAPH_METRIC}[{{window}}m]))" return {"query_key": "servicegraph_calls", "status": "ready", "query": {"tool": tool, "mapper": "servicegraph_v1", "args_template": {"query": expr}}, diff --git a/scripts/v2/workers/test_graph_catalog.py b/scripts/v2/workers/test_graph_catalog.py index a9335e9da..9d3da46b5 100644 --- a/scripts/v2/workers/test_graph_catalog.py +++ b/scripts/v2/workers/test_graph_catalog.py @@ -149,7 +149,7 @@ def test_grouping_preserves_every_identity_label_recognized_by_the_web_mapper(se assert aliases, "identity label aliases must be explicit for the catalog/mapper contract" suffixes = set(re.findall(r"'([a-z_]+)'", aliases.group(1))) for kind, metric, endpoints in [ - ("prometheus", "traces_service_graph_request_total", ("client", "server")), + ("prometheus", "traces_service_graph_request_total", ("client", "server", "connection_type")), ("mimir", "istio_requests_total", ("source_workload", "destination_workload")), ]: query = gc.build_graph_queries(kind, {"metrics": [metric]})[0]["query"]["args_template"]["query"] diff --git a/web/lib/graph-sources.test.ts b/web/lib/graph-sources.test.ts index 9cf251ba7..924816124 100644 --- a/web/lib/graph-sources.test.ts +++ b/web/lib/graph-sources.test.ts @@ -8,7 +8,7 @@ vi.mock('@/lib/datasources', () => ({ })); vi.mock('@/lib/mcp-lambda-invoke', () => ({ invokeMcpLambdaTool: vi.fn(async () => ({ rows: [] })) })); import { loadGraphSources } from './graph-sources'; -import { ClickHouseOtelTraceSource, TempoTraceSource, MetricsCallsSource } from './trace-source'; +import { ClickHouseOtelAggregateSource, ClickHouseOtelTraceSource, TempoTraceSource, MetricsCallsSource } from './trace-source'; // Adapter internals (connector calls, credential resolution) are covered by trace-source.test.ts — // this file only tests the mapper→adapter selection/wiring logic, so no need to mock datasources/ @@ -18,7 +18,7 @@ function mockPool(rows: unknown[]) { } describe('loadGraphSources', () => { - it('maps an otel_v1 ready row to a ClickHouseOtelTraceSource carrying the schema-driven SQL template', async () => { + it('keeps a non-catalog otel_v1 template on the span reader with its SQL template', async () => { const pool = mockPool([ { integration_id: 7, query: { tool: 'clickhouse_query', mapper: 'otel_v1', args_template: { sql: 'SELECT 1 FROM t LIMIT {cap}' } } }, ]); @@ -31,6 +31,18 @@ describe('loadGraphSources', () => { expect(s.sqlTemplate).toBe('SELECT 1 FROM t LIMIT {cap}'); }); + it('reads the catalog exporter-schema otel_v1 template as whole-window aggregates of the same table', async () => { + const sql = 'SELECT TraceId, SpanId, ParentSpanId, ServiceName, SpanKind, Timestamp, Duration, ' + + 'ResourceAttributes, SpanAttributes, StatusCode FROM `otel`.`otel_traces` WHERE Timestamp >= now() - ' + + 'INTERVAL {window} MINUTE AND Timestamp <= now() ORDER BY Timestamp DESC LIMIT {cap}'; + const pool = mockPool([{ integration_id: 7, query: { tool: 'clickhouse_query', mapper: 'otel_v1', args_template: { sql } } }]); + const { sources } = await loadGraphSources(pool); + expect(sources[0]).toBeInstanceOf(ClickHouseOtelAggregateSource); + expect(sources[0] as unknown).toMatchObject({ + instanceId: 7, shape: { table: '`otel`.`otel_traces`', spanKind: true, statusCode: true, links: false }, + }); + }); + it('maps a tempo_v1 ready row to a TempoTraceSource', async () => { const pool = mockPool([ { integration_id: 9, query: { tool: 'tempo_search', mapper: 'tempo_v1', args_template: { query: '{}', limit: 20 } } }, @@ -65,19 +77,19 @@ describe('loadGraphSources', () => { const { sources, metricsSources } = await loadGraphSources(pool); // none of the 3 rows produced a usable adapter → the empty-result fallback kicks in (1 bare default) expect(sources).toHaveLength(1); - expect(sources[0]).toBeInstanceOf(ClickHouseOtelTraceSource); + expect(sources[0]).toBeInstanceOf(ClickHouseOtelAggregateSource); expect((sources[0] as unknown as { instanceId?: number }).instanceId).toBeUndefined(); expect(metricsSources).toHaveLength(0); }); - it('falls back to a bare default ClickHouseOtelTraceSource when no ready rows exist yet', async () => { + it('falls back to a bare default aggregate ClickHouse source when no ready rows exist yet', async () => { const pool = mockPool([]); const { sources, metricsSources } = await loadGraphSources(pool); expect(sources).toHaveLength(1); - expect(sources[0]).toBeInstanceOf(ClickHouseOtelTraceSource); - const s = sources[0] as unknown as { instanceId?: number; sqlTemplate?: string }; + expect(sources[0]).toBeInstanceOf(ClickHouseOtelAggregateSource); + const s = sources[0] as unknown as { instanceId?: number; shape: { table: string } }; expect(s.instanceId).toBeUndefined(); - expect(s.sqlTemplate).toBeUndefined(); + expect(s.shape.table).toBe('otel_traces'); expect(metricsSources).toHaveLength(0); }); @@ -101,7 +113,7 @@ describe('loadGraphSources', () => { const pool = { query } as unknown as import('pg').Pool; const { sources } = await loadGraphSources(pool); expect(sources).toHaveLength(1); // no ready rows survived the join → bare default fallback - expect(sources[0]).toBeInstanceOf(ClickHouseOtelTraceSource); + expect(sources[0]).toBeInstanceOf(ClickHouseOtelAggregateSource); expect((sources[0] as unknown as { instanceId?: number }).instanceId).toBeUndefined(); }); }); diff --git a/web/lib/graph-sources.ts b/web/lib/graph-sources.ts index 4f47ec894..ed1b27396 100644 --- a/web/lib/graph-sources.ts +++ b/web/lib/graph-sources.ts @@ -6,7 +6,10 @@ // single hardcoded default ClickHouse instance. import type { Pool } from 'pg'; -import { ClickHouseOtelTraceSource, TempoTraceSource, MetricsCallsSource, type TraceSource } from '@/lib/trace-source'; +import { + ClickHouseOtelAggregateSource, ClickHouseOtelTraceSource, TempoTraceSource, MetricsCallsSource, + otelAggregateShape, type TraceSource, +} from '@/lib/trace-source'; interface GraphQueryRow { integration_id: number; @@ -57,7 +60,12 @@ export async function loadGraphSources(pool: Pool): Promise { const mapper = q?.mapper; if (mapper === 'otel_v1') { const sql = q?.args_template?.sql; - sources.push(new ClickHouseOtelTraceSource(row.integration_id, typeof sql === 'string' ? sql : undefined)); + const template = typeof sql === 'string' ? sql : undefined; + // The catalog's exporter-schema template is read as whole-window aggregates; any other + // (generated/custom) template keeps the bounded newest-span reader. + const shape = otelAggregateShape(template); + sources.push(shape ? new ClickHouseOtelAggregateSource(row.integration_id, shape) + : new ClickHouseOtelTraceSource(row.integration_id, template)); } else if (mapper === 'tempo_v1') { sources.push(new TempoTraceSource(row.integration_id)); } else if (mapper === 'servicegraph_v1' || mapper === 'istio_v1') { @@ -69,7 +77,7 @@ export async function loadGraphSources(pool: Pool): Promise { } if (sources.length === 0 && metricsSources.length === 0) { - sources.push(new ClickHouseOtelTraceSource()); + sources.push(new ClickHouseOtelAggregateSource()); } return { sources, metricsSources }; } diff --git a/web/lib/graph-store-trace.test.ts b/web/lib/graph-store-trace.test.ts index 0eb011bec..b272a142d 100644 --- a/web/lib/graph-store-trace.test.ts +++ b/web/lib/graph-store-trace.test.ts @@ -86,6 +86,28 @@ describe('trusted host account DB bridge', () => { }); }); +describe('source-aggregated trace edges', () => { + it('publishes span-pair edges a trace source aggregated beside its node rows', async () => { + const { pool, params } = mockPool([]); + const source = { + available: async () => true, + recentSpans: async (_w: number, _c: number, endMs = Date.now()) => ({ + sourceId: 'clickhouse:3', status: 'ok' as const, reasons: [], windowStartMs: endMs - 3_600_000, windowEndMs: endMs, + items: [span({ service: 'checkout', startMs: endMs - 1_000, weight: 9 })], + calls: [{ client: 'frontend', server: 'checkout', count: 17, evidence: 'trace' as const, rel: 'calls' as const }], + }), + }; + await rebuildTraceGraph(pool as never, [source]); + const services = params.filter(p => p[1] === 'service').map(p => JSON.parse(String(p[3]))); + expect(services.map(meta => [meta.service, meta.sourceId]).sort()).toEqual([ + ['checkout', 'clickhouse:3'], ['frontend', 'clickhouse:3'], + ]); + const edges = params.filter(p => p.includes('calls')); + expect(edges).toHaveLength(1); + expect(JSON.parse(String(edges[0][7]))).toEqual({ spanCount: 17, metricCount: 0 }); + }); +}); + describe('instance observation publication', () => { const account = '111122223333', region = 'ap-northeast-2', instanceId = 'i-0123456789abcdef0'; const infra = [ diff --git a/web/lib/graph-store.ts b/web/lib/graph-store.ts index d16406837..5f1537c4b 100644 --- a/web/lib/graph-store.ts +++ b/web/lib/graph-store.ts @@ -2,7 +2,7 @@ import { randomUUID } from 'node:crypto'; import type { Pool, PoolClient } from 'pg'; import { buildFlowGraph, type FlowInput, type FlowKind } from './flow-topology'; import { buildInfraGraph, type Row } from './infra-topology'; -import type { TraceSource, TraceSpan, ServiceGraphCall, SourceRead } from './trace-source'; +import type { TraceSource, TraceRead, ServiceGraphCall, SourceRead } from './trace-source'; import { buildTraceGraph, type InfraNodeLike } from './trace-graph'; import { graphDiagnostic, inventorySourcesStale, writeGraphState, type GraphAttempt, type GraphClass } from './graph-state'; import { currentAccountId } from './account'; @@ -303,7 +303,7 @@ export async function rebuildTraceGraph( // Adapter status, not a separate readiness probe, distinguishes absent config from a failed read. const spanReads = await Promise.all(sources.map(async (source, i) => { try { return await source.recentSpans(TRACE_WINDOW_MINS, TRACE_SPAN_CAP, endMs); } - catch { return failed(`trace:${i}`); } + catch { return failed(`trace:${i}`) as TraceRead; } })); const metricReads = await Promise.all(metricsSources.map(async (source, i) => { try { return await source.calls(TRACE_WINDOW_MINS, endMs); } @@ -318,7 +318,9 @@ export async function rebuildTraceGraph( const cannotSweep = reads.some((read) => read.canSweep === false); const partial = reads.some((read) => read.status === 'partial') || (cannotSweep && !hasFailure); const spans = spanReads.flatMap((read) => read.items.map((span) => ({ ...span, sourceId: span.sourceId ?? read.sourceId }))); - const calls = metricReads.flatMap((read) => read.items.map((call) => ({ + // Aggregate trace sources return span-pair edges beside their node rows; same sourceId stamping. + const calls = [...spanReads.map((read) => ({ ...read, items: read.calls ?? [] })), ...metricReads] + .flatMap((read) => read.items.map((call) => ({ ...call, clientIdentity: { ...call.clientIdentity, sourceId: call.clientIdentity?.sourceId ?? read.sourceId }, serverIdentity: { ...call.serverIdentity, sourceId: call.serverIdentity?.sourceId ?? read.sourceId }, diff --git a/web/lib/trace-graph.test.ts b/web/lib/trace-graph.test.ts index 1a9c463c4..6640fef61 100644 --- a/web/lib/trace-graph.test.ts +++ b/web/lib/trace-graph.test.ts @@ -226,3 +226,74 @@ describe('same-span EC2 and DB observations', () => { expect(queries[0].meta).not.toHaveProperty('instance_observations_unbound'); }); }); + +describe('source-aggregated rows and service-graph connection types', () => { + const svc = (graph: ReturnType, name: string) => + graph.nodes.find(node => node.kind === 'service' && node.label === name); + + it('counts an aggregate row as its weight on nodes, DB edges, pods and instance pairs', () => { + const graph = buildTraceGraph([dbSpan({ + weight: 40, firstStartMs: 1_793_999_000_000, status: 'error', durationMs: 120, + k8sNamespace: 'shop', k8sDeployment: 'checkout', k8sPods: ['checkout-a', 'checkout-b'], + })], [], inventory()); + expect(svc(graph, 'checkout')?.meta).toMatchObject({ spanCount: 40, errorSpanCount: 40, sampledDurationMs: 120 }); + const query = graph.edges.find(edge => edge.rel === 'queries')!; + expect(query.meta.spanCount).toBe(40); + expect(query.meta.instance_observations?.[0]).toMatchObject({ + observation_count: 40, first_observed_at_ms: 1_793_999_000_000, observed_at_ms: 1_794_000_000_000, + }); + expect(graph.edges.find(edge => edge.rel === 'runs_on')?.meta.spanCount).toBe(40); + expect(graph.nodes.find(node => node.kind === 'workload')?.meta.pods).toEqual(['checkout-a', 'checkout-b']); + }); + + it('draws source-counted span pairs as span evidence on the same service nodes, never as self-loops', () => { + const identity = { sourceId: 'clickhouse:3', environment: 'prod' }; + const graph = buildTraceGraph( + [dbSpan({ sourceId: 'clickhouse:3', environment: 'prod', accountId: undefined, region: undefined, weight: 5 })], + [ + { client: 'frontend', server: 'checkout', count: 17, evidence: 'trace', rel: 'calls', + clientIdentity: identity, serverIdentity: identity }, + { client: 'worker', server: 'checkout', count: 3, evidence: 'trace', rel: 'linked', + clientIdentity: identity, serverIdentity: identity }, + { client: 'checkout', server: 'checkout', count: 9, evidence: 'trace', rel: 'calls', + clientIdentity: identity, serverIdentity: identity }, + ], []); + const checkout = svc(graph, 'checkout')!; + expect(graph.nodes.filter(node => node.label === 'checkout')).toHaveLength(1); + const calls = graph.edges.find(edge => edge.rel === 'calls')!; + expect(calls).toMatchObject({ source: svc(graph, 'frontend')!.id, target: checkout.id, + meta: { spanCount: 17, metricCount: 0 } }); + expect(graph.edges.find(edge => edge.rel === 'linked')?.meta.spanCount).toBe(3); + expect(graph.edges.some(edge => edge.source === edge.target)).toBe(false); + expect(graph.orphanSpans).toBe(0); + }); + + it('keeps metric database/messaging servers off service nodes and defers to span evidence', () => { + const calls = [ + { client: 'review', server: 'reviews', count: 10, connectionType: 'database' }, + { client: 'order', server: 'orders-topic', count: 4, connectionType: 'messaging_system' }, + { client: 'user', server: 'payment', count: 6, connectionType: 'virtual_node' }, + { client: 'payment', server: 'stripe', count: 2, connectionType: 'virtual_node' }, + ]; + const metricsOnly = buildTraceGraph([], calls, []); + expect(metricsOnly.nodes.filter(node => node.kind === 'service').map(node => node.label).sort()) + .toEqual(['order', 'payment', 'review']); + expect(metricsOnly.nodes.find(node => node.label === 'reviews')).toMatchObject({ kind: 'db', + meta: { provenance: 'service_graph_metric', host: null } }); + expect(metricsOnly.nodes.find(node => node.label === 'orders-topic')?.kind).toBe('queue'); + expect(metricsOnly.nodes.filter(node => node.kind === 'external').map(node => node.label).sort()) + .toEqual(['stripe', 'user']); + const rel = (source: string, target: string) => metricsOnly.edges.find(edge => + metricsOnly.nodes.find(n => n.id === edge.source)?.label === source + && metricsOnly.nodes.find(n => n.id === edge.target)?.label === target); + expect(rel('review', 'reviews')).toMatchObject({ rel: 'queries', meta: { metricCount: 10, spanCount: 0 } }); + expect(rel('order', 'orders-topic')?.rel).toBe('publishes'); + expect(rel('user', 'payment')?.rel).toBe('calls'); + expect(rel('payment', 'stripe')?.rel).toBe('calls'); + + const withSpans = buildTraceGraph([dbSpan({ service: 'review', dbName: 'reviews' })], calls, inventory()); + expect(withSpans.nodes.filter(node => node.meta.provenance === 'service_graph_metric' && node.kind !== 'external')) + .toEqual([]); + expect(withSpans.nodes.filter(node => node.kind === 'db')).toHaveLength(1); + }); +}); diff --git a/web/lib/trace-graph.ts b/web/lib/trace-graph.ts index 43ebbbe30..7137bc643 100644 --- a/web/lib/trace-graph.ts +++ b/web/lib/trace-graph.ts @@ -126,7 +126,8 @@ function observedInstanceDatabasePair( }; } -/** A sampled dependency graph. Counts describe evidence, never a probability of correctness. */ +/** A dependency graph from sampled spans, source-aggregated span rows and metric calls. + * Counts describe evidence, never a probability of correctness. */ export function buildTraceGraph( spans: TraceSpan[], metricCalls: ServiceGraphCall[], @@ -179,11 +180,13 @@ export function buildTraceGraph( }; for (const span of byId.values()) { + // An aggregate row stands for `weight` spans; a raw span for one. + const w = Number.isSafeInteger(span.weight) && span.weight! > 0 ? span.weight! : 1; const sid = serviceNode(span.service, span); const meta = nodes.get(sid)!.meta; - meta.spanCount = Number(meta.spanCount) + 1; - if (span.status === 'error') meta.errorSpanCount = Number(meta.errorSpanCount) + 1; - if (!span.status || span.status === 'unset') meta.unknownStatusSpanCount = Number(meta.unknownStatusSpanCount) + 1; + meta.spanCount = Number(meta.spanCount) + w; + if (span.status === 'error') meta.errorSpanCount = Number(meta.errorSpanCount) + w; + if (!span.status || span.status === 'unset') meta.unknownStatusSpanCount = Number(meta.unknownStatusSpanCount) + w; meta.sampledDurationMs = Number(meta.sampledDurationMs) + Math.max(0, Number(span.durationMs) || 0); const traceIds = meta.traceIds as string[]; if (traceIds.length < 20 && !traceIds.includes(span.traceId)) traceIds.push(span.traceId); @@ -219,21 +222,25 @@ export function buildTraceGraph( accountId: span.accountId ?? null, region: span.region ?? null, ...(infraRef ? { infra_ref: infraRef } : {}) }, }); - const queryEdge = edge(sid, id, 'queries'); + const queryEdge = edge(sid, id, 'queries', w); // Only an outbound client call is evidence that this instance queried the database; // SERVER/PRODUCER/etc. spans carrying db attributes keep the service→DB edge only. const observation = span.kind === 'CLIENT' ? observedInstanceDatabasePair(span, infraNodes, trustedHostAccountId) : undefined; - if (observation === null) queryEdge.meta.instance_observations_unbound = (queryEdge.meta.instance_observations_unbound ?? 0) + 1; + if (observation === null) queryEdge.meta.instance_observations_unbound = (queryEdge.meta.instance_observations_unbound ?? 0) + w; else if (observation) { + observation.observation_count = w; + const first = Number.isSafeInteger(span.firstStartMs) && span.firstStartMs! >= 0 + ? Math.min(span.firstStartMs!, observation.observed_at_ms) : observation.observed_at_ms; + observation.first_observed_at_ms = first; const observations = queryEdge.meta.instance_observations ??= []; let index = pairIndex.get(queryEdge); if (!index) pairIndex.set(queryEdge, index = { kept: new Map(), omitted: new Set() }); const key = JSON.stringify([observation.instance_ref, observation.database_ref]); const existing = index.kept.get(key); if (existing) { - existing.observation_count++; - existing.first_observed_at_ms = Math.min(existing.first_observed_at_ms, observation.observed_at_ms); + existing.observation_count += w; + existing.first_observed_at_ms = Math.min(existing.first_observed_at_ms, first); if (observation.observed_at_ms > existing.observed_at_ms || (observation.observed_at_ms === existing.observed_at_ms && observation.correlation_id > existing.correlation_id)) { @@ -260,8 +267,8 @@ export function buildTraceGraph( sourceId: span.sourceId ?? null, pods: [] }, }); const pods = nodes.get(id)!.meta.pods as string[]; - if (span.k8sPod && !pods.includes(span.k8sPod)) pods.push(span.k8sPod); - edge(sid, id, 'runs_on'); + for (const pod of span.k8sPods ?? (span.k8sPod ? [span.k8sPod] : [])) if (!pods.includes(pod)) pods.push(pod); + edge(sid, id, 'runs_on', w); } if (span.messagingSystem && span.messagingDestination) { const destination = span.messagingDestination.trim(); @@ -290,14 +297,59 @@ export function buildTraceGraph( broker: qualified ? null : span.messagingBroker ?? null, cluster: qualified ? null : span.k8sCluster ?? null, sourceId: span.sourceId ?? null, environment: span.environment ?? null }) }); - if (span.kind === 'PRODUCER') edge(sid, id, 'publishes'); - if (span.kind === 'CONSUMER') edge(id, sid, 'consumes'); + if (span.kind === 'PRODUCER') edge(sid, id, 'publishes', w); + if (span.kind === 'CONSUMER') edge(id, sid, 'consumes', w); } } + // Trace spans name databases and queues with host-qualified identity. Metric rows only carry + // Tempo's server label (a db.name such as "admin"), so they draw those nodes only when no + // span evidence exists; otherwise the same database would appear twice under two names. + const spanEvidence = byId.size > 0; for (const call of metricCalls) { if (!Number.isFinite(call.count) || call.count <= 0) continue; - edge(serviceNode(call.client, call.clientIdentity ?? {}), - serviceNode(call.server, call.serverIdentity ?? {}), 'calls', 0, call.count); + if (call.evidence === 'trace') { + const client = serviceNode(call.client, call.clientIdentity ?? {}); + const server = serviceNode(call.server, call.serverIdentity ?? {}); + if (client !== server) edge(client, server, call.rel ?? 'calls', call.count, 0); + continue; + } + const clientIdentity = call.clientIdentity ?? {}, serverIdentity = call.serverIdentity ?? {}; + if (call.connectionType === 'database' || call.connectionType === 'messaging_system') { + if (spanEvidence) continue; + const database = call.connectionType === 'database'; + const scoped = resourceScope({ ...clientIdentity, ...serverIdentity }); + const id = nodeId(database ? 'db' : 'queue', scoped, `metric:${call.server}`); + if (!nodes.has(id)) nodes.set(id, database ? { + id, kind: 'db', label: call.server.slice(0, 240), + meta: { system: null, host: null, dbName: null, metricServer: call.server, + sourceId: scoped.sourceId ?? null, environment: scoped.environment ?? null, + accountId: scoped.accountId ?? null, region: scoped.region ?? null, provenance: 'service_graph_metric' }, + } : { + id, kind: 'queue', label: call.server.slice(0, 240), + meta: queueClaimMeta({ system: null, destination: call.server, broker: null, cluster: null, + sourceId: scoped.sourceId ?? null, environment: scoped.environment ?? null, + provenance: 'service_graph_metric' }), + }); + edge(serviceNode(call.client, clientIdentity), id, database ? 'queries' : 'publishes', 0, call.count); + continue; + } + if (call.connectionType === 'virtual_node') { + // Tempo names an uninstrumented caller "user"; otherwise the uninstrumented side is the server. + const clientVirtual = call.client === 'user'; + const external = (name: string, identity: TraceIdentity) => { + const id = nodeId('ext', resourceScope(identity), name); + if (!nodes.has(id)) nodes.set(id, { id, kind: 'external', label: name.slice(0, 240), + meta: { name, sourceId: identity.sourceId ?? null, environment: identity.environment ?? null, + provenance: 'service_graph_metric' } }); + return id; + }; + edge(clientVirtual ? external(call.client, clientIdentity) : serviceNode(call.client, clientIdentity), + clientVirtual ? serviceNode(call.server, serverIdentity) : external(call.server, serverIdentity), + 'calls', 0, call.count); + continue; + } + edge(serviceNode(call.client, clientIdentity), + serviceNode(call.server, serverIdentity), 'calls', 0, call.count); } return { nodes: [...nodes.values()], edges: [...edges.values()], orphanSpans, invalidSpans, unresolvedMessaging }; diff --git a/web/lib/trace-source.test.ts b/web/lib/trace-source.test.ts index 07d505577..e2c2e3a6e 100644 --- a/web/lib/trace-source.test.ts +++ b/web/lib/trace-source.test.ts @@ -19,7 +19,9 @@ vi.mock('@/lib/mcp-lambda-invoke', () => ({ import { FakeTraceSource, + ClickHouseOtelAggregateSource, ClickHouseOtelTraceSource, + otelAggregateShape, TempoTraceSource, MetricsCallsSource, mapOtelRow, @@ -81,6 +83,101 @@ describe('mapOtelRow (real nested-map otel_traces shape)', () => { }); }); +describe('ClickHouseOtelAggregateSource (whole-window aggregates)', () => { + const END = 1_794_300_000_000; + const resource = { 'service.name': 'checkout', 'deployment.environment': 'prod', 'k8s.cluster.name': 'c1', + 'k8s.namespace.name': 'shop', 'k8s.deployment.name': 'checkout' }; + const group = (over: Record = {}) => ({ + ServiceName: 'checkout', SpanKind: 'Client', StatusCode: 'Error', ResourceAttributes: resource, + SpanAttributes: { 'db.system': 'postgresql', 'server.address': 'db.example.internal', 'db.name': 'orders' }, + Weight: '42', DurationSum: '84000000', LastTimestamp: END - 1_000, FirstTimestamp: END - 3_000_000, + SampleTraceId: 't1', SampleSpanId: 's1', Pods: ['checkout-a', 'checkout-b', ''], ...over, + }); + const pair = (over: Record = {}) => ({ + ClientService: 'frontend', ClientResource: { 'service.name': 'frontend', 'deployment.environment': 'prod' }, + ServerService: 'checkout', ServerResource: resource, Weight: '17', ...over, + }); + beforeEach(() => { + getDefaultDatasource.mockReset(); getDatasource.mockReset(); resolveConnConfig.mockReset(); invokeMcpLambdaTool.mockReset(); + getDefaultDatasource.mockResolvedValue({ id: 3, kind: 'clickhouse', isDefault: true }); + resolveConnConfig.mockResolvedValue({ endpoint: 'http://ch' }); + }); + const answer = (groups: unknown[], calls: unknown[], links: unknown[] = []) => + invokeMcpLambdaTool.mockImplementation(async ({ args }: { args: { sql: string } }) => { + const rows = args.sql.includes('ARRAY JOIN') ? links : args.sql.includes('INNER JOIN') ? calls : groups; + return { rows, collectionStatus: rows.length ? 'ok' : 'empty' }; + }); + + it('accepts only the catalog exporter template and reads its optional columns', () => { + expect(otelAggregateShape(undefined)).toEqual({ table: 'otel_traces', spanKind: true, statusCode: true, links: true }); + const catalog = 'SELECT TraceId, SpanId, ParentSpanId, ServiceName, Timestamp, Duration, ResourceAttributes, ' + + 'SpanAttributes FROM `otel`.`otel_traces` WHERE Timestamp >= now() - INTERVAL {window} MINUTE AND ' + + 'Timestamp <= now() ORDER BY Timestamp DESC LIMIT {cap}'; + expect(otelAggregateShape(catalog)).toEqual({ table: '`otel`.`otel_traces`', spanKind: false, statusCode: false, links: false }); + expect(otelAggregateShape('SELECT trace_id AS TraceId FROM spans LIMIT {cap}')).toBeUndefined(); + expect(otelAggregateShape(catalog.replace('LIMIT {cap}', 'LIMIT {cap}; DROP TABLE x'))).toBeUndefined(); + }); + + it('queries node groups, parent/child pairs and links over the exact window and maps weighted rows', async () => { + answer([group()], [pair()], [pair({ ClientService: 'worker', ClientResource: { 'service.name': 'worker' } })]); + const out = await new ClickHouseOtelAggregateSource().recentSpans(60, 1000, END); + expect(invokeMcpLambdaTool).toHaveBeenCalledTimes(3); + for (const [{ args }] of invokeMcpLambdaTool.mock.calls) { + expect(args.sql).toContain(`fromUnixTimestamp64Milli(${END}) - INTERVAL 60 MINUTE`); + expect(args.sql).toContain('FROM otel_traces'); + expect(args.sql).toMatch(/LIMIT 1000$/); + expect(args.max_rows).toBe(1000); + } + expect(out).toMatchObject({ status: 'ok', sourceId: 'clickhouse:3', reasons: [] }); + expect(out.items).toEqual([expect.objectContaining({ + service: 'checkout', kind: 'CLIENT', status: 'error', weight: 42, durationMs: 84, + startMs: END - 1_000, firstStartMs: END - 3_000_000, k8sPods: ['checkout-a', 'checkout-b'], + dbSystem: 'postgresql', dbHost: 'db.example.internal', k8sCluster: 'c1', sourceId: 'clickhouse:3', + })]); + expect(out.calls).toEqual([ + { client: 'frontend', server: 'checkout', count: 17, evidence: 'trace', rel: 'calls', + clientIdentity: { sourceId: 'clickhouse:3', environment: 'prod' }, + serverIdentity: { sourceId: 'clickhouse:3', environment: 'prod', k8sCluster: 'c1', k8sNamespace: 'shop' } }, + expect.objectContaining({ client: 'worker', rel: 'linked', count: 17, evidence: 'trace' }), + ]); + }); + + it('skips the link query when the table has no Links columns', async () => { + answer([group()], []); + const shape = { table: 't', spanKind: true, statusCode: true, links: false }; + await new ClickHouseOtelAggregateSource(undefined, shape).recentSpans(60, 1000, END); + expect(invokeMcpLambdaTool).toHaveBeenCalledTimes(2); + expect(invokeMcpLambdaTool.mock.calls.some(([{ args }]) => args.sql.includes('ARRAY JOIN'))).toBe(false); + }); + + it('publishes nothing when any aggregate fails, so nodes never appear without their edges', async () => { + invokeMcpLambdaTool.mockImplementation(async ({ args }: { args: { sql: string } }) => + args.sql.includes('INNER JOIN') ? { error: 'boom', collectionStatus: 'error' } : { rows: [group()], collectionStatus: 'ok' }); + expect(await new ClickHouseOtelAggregateSource().recentSpans(60, 1000, END)).toMatchObject({ + status: 'error', items: [], reasons: ['query_failed'], + }); + invokeMcpLambdaTool.mockRejectedValue(new Error('timeout')); + expect(await new ClickHouseOtelAggregateSource().recentSpans(60, 1000, END)).toMatchObject({ + status: 'error', items: [], reasons: ['query_failed'], + }); + }); + + it('marks a saturated row limit and malformed rows as partial evidence', async () => { + answer([group(), group({ Weight: '0', SampleSpanId: 's2' }), group({ FirstTimestamp: END, SampleSpanId: 's3' })], + [pair(), pair({ ServerResource: 'oops' })]); + const out = await new ClickHouseOtelAggregateSource().recentSpans(60, 2, END); + expect(out.status).toBe('partial'); + expect(out.reasons).toEqual(expect.arrayContaining(['cap_reached', 'malformed_rows'])); + expect(out.items).toHaveLength(1); + expect(out.calls).toHaveLength(1); + }); + + it('requires proof before treating an empty window as empty', async () => { + invokeMcpLambdaTool.mockResolvedValue({ rows: [] }); + expect((await new ClickHouseOtelAggregateSource().recentSpans(60, 1000, END)).reasons).toContain('empty_not_confirmed'); + }); +}); + describe('ClickHouseOtelTraceSource', () => { beforeEach(() => { getDefaultDatasource.mockReset(); getDatasource.mockReset(); resolveConnConfig.mockReset(); invokeMcpLambdaTool.mockReset(); @@ -282,6 +379,19 @@ describe('extractServiceGraphCalls (Prometheus/Mimir instant-query vector result ]); }); + it('keeps Tempo connection_type so database and virtual-node rows are not drawn as services', () => { + const result = { resultType: 'vector', result: [ + { metric: { client: 'review', server: 'reviews', connection_type: 'database' }, value: [0, '4'] }, + { metric: { client: 'user', server: 'payment', connection_type: 'virtual_node' }, value: [0, '2'] }, + { metric: { client: 'a', server: 'b', connection_type: '' }, value: [0, '1'] }, + ] }; + expect(extractServiceGraphCalls(result)).toEqual([ + { client: 'review', server: 'reviews', count: 4, connectionType: 'database' }, + { client: 'user', server: 'payment', count: 2, connectionType: 'virtual_node' }, + { client: 'a', server: 'b', count: 1 }, + ]); + }); + it('extracts from an istio_v1-shaped vector (source_workload/destination_workload labels)', () => { const result = { result: [ { metric: { source_workload: 'checkout', destination_workload: 'orders' }, value: [0, '3'] }, diff --git a/web/lib/trace-source.ts b/web/lib/trace-source.ts index 92c4335f1..0aea52c2a 100644 --- a/web/lib/trace-source.ts +++ b/web/lib/trace-source.ts @@ -49,11 +49,17 @@ export interface TraceSpan extends TraceIdentity { k8sDeployment?: string; startMs: number; durationMs: number; + /** Aggregate rows only: the number of spans this row stands for (absent = one span). Its + * durationMs is then the summed duration, startMs the latest start and firstStartMs the earliest. */ + weight?: number; + firstStartMs?: number; + /** Aggregate rows only: bounded distinct pod names behind the row. */ + k8sPods?: string[]; } export interface TraceSource { available(): Promise; - recentSpans(windowMins: number, cap: number, endMs?: number): Promise>; + recentSpans(windowMins: number, cap: number, endMs?: number): Promise; } export interface ServiceGraphCall { @@ -62,6 +68,17 @@ export interface ServiceGraphCall { count: number; clientIdentity?: TraceIdentity; serverIdentity?: TraceIdentity; + /** 'trace': counted from parent/child (or link) span pairs, so count is a span count. + * Absent: a service-graph metric increase. */ + evidence?: 'trace'; + rel?: 'calls' | 'linked'; + /** Tempo service-graph connection_type label (database, messaging_system, virtual_node). */ + connectionType?: string; +} + +/** Trace sources may also return span-pair edges they already aggregated at the source. */ +export interface TraceRead extends SourceRead { + calls?: ServiceGraphCall[]; } // Never use backend exceptions, status messages, SQL, previews or credentials as reasons. @@ -364,6 +381,185 @@ export class ClickHouseOtelTraceSource implements TraceSource { } } +// --- ClickHouse aggregate (edges counted in ClickHouse over the whole window) ---------------- + +/** Which optional columns the catalog found; the aggregate reads the same table it introspected. */ +export interface OtelAggregateShape { table: string; spanKind: boolean; statusCode: boolean; links: boolean } + +const DEFAULT_OTEL_SHAPE: OtelAggregateShape = { table: 'otel_traces', spanKind: true, statusCode: true, links: true }; +const IDENT = '(?:`(?:[^`]|``)+`|[A-Za-z_][A-Za-z0-9_]*)'; +// Only graph_catalog.py's deterministic otel_v1 template qualifies; generated or custom SQL keeps +// the span reader, because its columns are not known to follow the exporter schema. +const CATALOG_OTEL_TEMPLATE = new RegExp( + `^SELECT (TraceId, SpanId, ParentSpanId, ServiceName[^;]*?) FROM (${IDENT}(?:\\.${IDENT})?) ` + + 'WHERE Timestamp >= now\\(\\) - INTERVAL \\{window\\} MINUTE AND Timestamp <= now\\(\\) ' + + 'ORDER BY Timestamp DESC LIMIT \\{cap\\}$'); + +/** The aggregate shape for a registry template, or undefined when it is not the catalog's. */ +export function otelAggregateShape(sqlTemplate?: string): OtelAggregateShape | undefined { + if (sqlTemplate === undefined) return DEFAULT_OTEL_SHAPE; + const match = CATALOG_OTEL_TEMPLATE.exec(sqlTemplate); + if (!match) return undefined; + const columns = match[1].split(',').map((c) => c.trim()); + return { + table: match[2], spanKind: columns.includes('SpanKind'), statusCode: columns.includes('StatusCode'), + links: columns.includes('`Links.TraceId`') && columns.includes('`Links.SpanId`'), + }; +} + +const list = (keys: readonly string[]) => `[${keys.map((k) => `'${k}'`).join(', ')}]`; +// Only the attributes spanMetadata reads are grouped. Pod names and per-process instance IDs would +// multiply rows, so pods are collected as a bounded array and service.instance.id is kept only +// when it is an EC2 instance claim. +const AGG_RESOURCE_KEYS = [ + 'service.name', 'cloud.account.id', 'cloud.region', 'deployment.environment.name', + 'deployment.environment', 'service.namespace', 'service.version', 'cloud.resource_id', 'host.id', + 'service.instance.id', 'k8s.namespace.name', 'k8s.cluster.name', 'k8s.deployment.name', +] as const; +const AGG_SPAN_KEYS = [ + 'db.system', 'db.name', 'server.address', 'net.peer.name', 'server.port', 'net.peer.port', + 'peer.service', 'messaging.system', 'messaging.destination.name', 'messaging.destination', +] as const; +// Node scope only (trace-graph.ts scope()); edge endpoints must land on the same service nodes. +const AGG_IDENTITY_KEYS = [ + 'service.name', 'cloud.account.id', 'cloud.region', 'deployment.environment.name', + 'deployment.environment', 'service.namespace', 'k8s.namespace.name', 'k8s.cluster.name', +] as const; +const AGG_POD_CAP = 20; + +function aggregateSql(shape: OtelAggregateShape, windowMins: number, endMs: number, limit: number) { + const window = `Timestamp >= fromUnixTimestamp64Milli(${endMs}) - INTERVAL ${windowMins} MINUTE ` + + `AND Timestamp <= fromUnixTimestamp64Milli(${endMs})`; + const kind = shape.spanKind ? 'SpanKind' : "'' AS SpanKind"; + const status = shape.statusCode ? 'StatusCode' : "'' AS StatusCode"; + // Non-db/messaging spans drop their peer attributes so HTTP peers do not multiply groups. + const groups = 'SELECT ServiceName, SpanKind, StatusCode, RA AS ResourceAttributes, SA AS SpanAttributes, ' + + 'count() AS Weight, sum(Duration) AS DurationSum, toUnixTimestamp64Milli(max(Timestamp)) AS LastTimestamp, ' + + 'toUnixTimestamp64Milli(min(Timestamp)) AS FirstTimestamp, any(TraceId) AS SampleTraceId, ' + + `any(SpanId) AS SampleSpanId, groupUniqArray(${AGG_POD_CAP})(Pod) AS Pods FROM (` + + `SELECT ServiceName, ${kind}, ${status}, Duration, Timestamp, TraceId, SpanId, ` + + "ResourceAttributes['k8s.pod.name'] AS Pod, " + + `mapFilter((k, v) -> has(${list(AGG_RESOURCE_KEYS)}, k) AND (k != 'service.instance.id' ` + + "OR match(v, '^i-[0-9a-f]{8}([0-9a-f]{9})?$')), ResourceAttributes) AS RA, " + + "if(SpanAttributes['db.system'] != '' OR SpanAttributes['messaging.system'] != '', " + + `mapFilter((k, v) -> has(${list(AGG_SPAN_KEYS)}, k), SpanAttributes), ` + + 'mapFilter((k, v) -> false, SpanAttributes)) AS SA ' + + `FROM ${shape.table} WHERE ${window}) ` + + `GROUP BY ServiceName, SpanKind, StatusCode, RA, SA ORDER BY Weight DESC LIMIT ${limit}`; + const pairs = (child: string, on: string) => + `SELECT p.ServiceName AS ClientService, mapFilter((k, v) -> has(${list(AGG_IDENTITY_KEYS)}, k), ` + + 'p.ResourceAttributes) AS ClientResource, c.ServiceName AS ServerService, ' + + `mapFilter((k, v) -> has(${list(AGG_IDENTITY_KEYS)}, k), c.ResourceAttributes) AS ServerResource, ` + + `count() AS Weight FROM (${child}) AS c INNER JOIN (SELECT TraceId, SpanId, ServiceName, ` + + `ResourceAttributes FROM ${shape.table} WHERE ${window}) AS p ON ${on} ` + + 'GROUP BY ClientService, ClientResource, ServerService, ServerResource ' + + 'HAVING NOT (ClientService = ServerService AND ClientResource = ServerResource) ' + + `ORDER BY Weight DESC LIMIT ${limit}`; + const calls = pairs( + `SELECT TraceId, ParentSpanId, ServiceName, ResourceAttributes FROM ${shape.table} ` + + `WHERE ${window} AND ParentSpanId != ''`, + 'c.TraceId = p.TraceId AND c.ParentSpanId = p.SpanId'); + const links = shape.links ? pairs( + 'SELECT LinkTraceId, LinkSpanId, ServiceName, ResourceAttributes FROM ' + + `${shape.table} ARRAY JOIN \`Links.TraceId\` AS LinkTraceId, \`Links.SpanId\` AS LinkSpanId ` + + `WHERE ${window}`, + 'c.LinkTraceId = p.TraceId AND c.LinkSpanId = p.SpanId') : undefined; + return { groups, calls, links }; +} + +function positiveCount(value: unknown): number | undefined { + const n = numeric(value); + return n !== undefined && Number.isSafeInteger(n) && n > 0 ? n : undefined; +} + +function resourceIdentity(value: unknown, sourceId: string): TraceIdentity | undefined { + const resource = object(value); + if (!resource) return undefined; + const span = mapOtelRow({ ResourceAttributes: resource }); + const out: TraceIdentity = { sourceId }; + for (const key of ['accountId', 'region', 'environment', 'serviceNamespace', 'k8sNamespace', 'k8sCluster'] as const) { + if (span[key] !== undefined) out[key] = span[key]; + } + return out; +} + +/** Reads service nodes and span-pair edges as ClickHouse aggregates over the whole window, instead + * of the newest raw spans. Counts are complete for the window unless a row limit is reached. */ +export class ClickHouseOtelAggregateSource implements TraceSource { + constructor(private readonly instanceId?: number, private readonly shape: OtelAggregateShape = DEFAULT_OTEL_SHAPE) {} + async available(): Promise { return !(await resolveSource('clickhouse', this.instanceId)).reason; } + + async recentSpans(windowMins: number, cap: number, endMs = Date.now()): Promise { + const window = readWindow(windowMins, endMs); + const { sourceId, connConfig, reason } = await resolveSource('clickhouse', this.instanceId); + if (reason) return readResult(sourceId, window, [], [reason], reason === 'missing_configuration' ? 'unavailable' : 'error'); + if (!validWindow(window) || !Number.isFinite(cap)) return readResult(sourceId, window, [], ['invalid_request']); + const limit = Math.min(CLICKHOUSE_ROW_CAP, Math.max(0, Math.floor(cap))); + if (!limit) return readResult(sourceId, window, [], ['cap_reached']); + const sql = aggregateSql(this.shape, Math.max(1, Math.floor(windowMins)), endMs, limit); + const query = (text: string) => invokeMcpLambdaTool({ + kind: 'clickhouse', tool: 'clickhouse_query', args: { sql: text, max_rows: limit }, connConfig, + }); + let payloads: unknown[]; + try { + payloads = await Promise.all([sql.groups, sql.calls, ...(sql.links ? [sql.links] : [])].map(query)); + } catch { + return readResult(sourceId, window, [], ['query_failed']); + } + const reasons: Reason[] = []; + const results = payloads.map(clickhouseRows); + // Any failed aggregate leaves the window unproven; never publish nodes without their edges. + for (const result of results) { + if (!result.rows) return readResult(sourceId, window, [], result.reasons); + reasons.push(...result.reasons); + if (result.rows.length >= limit) reasons.push('cap_reached'); + } + const [groupRows, callRows, linkRows = []] = results.map((r) => r.rows!); + const items: TraceSpan[] = []; + for (const value of groupRows) { + const row = object(value); + const weight = positiveCount(row?.Weight); + const last = numeric(row?.LastTimestamp), first = numeric(row?.FirstTimestamp); + const duration = numeric(row?.DurationSum); + if (!row || !text(row.ServiceName) || !text(row.SampleTraceId) || !text(row.SampleSpanId) + || weight === undefined || last === undefined || first === undefined || first > last + || duration === undefined || duration < 0 + || (row.ResourceAttributes !== undefined && !object(row.ResourceAttributes)) + || (row.SpanAttributes !== undefined && !object(row.SpanAttributes))) { + reasons.push('malformed_rows'); + continue; + } + const item = mapOtelRow({ + TraceId: row.SampleTraceId, SpanId: row.SampleSpanId, ServiceName: row.ServiceName, + SpanKind: row.SpanKind, StatusCode: row.StatusCode, Timestamp: last, Duration: duration, + ResourceAttributes: row.ResourceAttributes, SpanAttributes: row.SpanAttributes, + }); + if (this.shape.statusCode && text(row.StatusCode) && !item.status) reasons.push('malformed_rows'); + const pods = Array.isArray(row.Pods) + ? [...new Set(row.Pods.filter((p): p is string => !!text(p)))].slice(0, AGG_POD_CAP) : []; + if (!inWindow(item, window)) continue; + items.push({ ...item, sourceId, weight, firstStartMs: first, ...(pods.length ? { k8sPods: pods } : {}) }); + } + const calls: ServiceGraphCall[] = []; + for (const [rows, rel] of [[callRows, 'calls'], [linkRows, 'linked']] as const) { + for (const value of rows) { + const row = object(value); + const count = positiveCount(row?.Weight); + const client = text(row?.ClientService), server = text(row?.ServerService); + const clientIdentity = resourceIdentity(row?.ClientResource, sourceId); + const serverIdentity = resourceIdentity(row?.ServerResource, sourceId); + if (!client || !server || count === undefined || !clientIdentity || !serverIdentity) { + reasons.push('malformed_rows'); + continue; + } + calls.push({ client, server, count, clientIdentity, serverIdentity, evidence: 'trace', rel }); + } + } + requireEmptyProof(payloads[0], items, reasons); + return { ...readResult(sourceId, window, items, reasons), calls }; + } +} + // --- Tempo (search -> bounded per-trace fetch) ----------------------------------------------- const TEMPO_TRACE_CAP = 20; // graph_catalog.py tempo_v1; <=21 invokes per source @@ -570,6 +766,8 @@ function parseServiceGraphCalls(value: unknown): { items: ServiceGraphCall[]; re } if (count === 0) continue; // valid zero traffic; no edge, no data-quality failure const item: ServiceGraphCall = { client, server, count }; + const connectionType = text(metric.connection_type); + if (connectionType) item.connectionType = connectionType; const clientIdentity = metricIdentity(metric, 'client'); const serverIdentity = metricIdentity(metric, 'server'); if (clientIdentity) item.clientIdentity = clientIdentity; From 543c1dab1b28f2ac8536ba8021984bdb73d98437 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=EC=98=A4=EC=A4=80=EC=84=9D=28Junseok=20Oh=29DevOps?= <52226147+Atom-oh@users.noreply.github.com> Date: Tue, 10 Nov 2026 10:09:59 +0000 Subject: [PATCH 2/4] fix(topology): address review on trace aggregates - Bump graph CATALOG_VERSION to v4 so indexed Prometheus/Mimir instances rebuild and pick up the connection_type PromQL. - Defer a metric database/queue row only when trace spans show the same client service using a database/queue, instead of whenever any span exists; count deferrals as deferredMetricCalls. - Accept the aggregate path only for exactly the bare exporter columns, so generated or aliased otel_v1 templates keep the span reader. - Push the sorted identity map into both join sides (bounded hash build) and mapSort grouped maps so equal attribute sets group together. - Keep ARN service.instance.id claims so claimedInstance can still veto them. - Require empty proof from every aggregate; guard link-array lengths before ARRAY JOIN; sum durations in milliseconds within safe integers. - Metric-derived db nodes carry no account/region attribution; metric volume ranks nodes under the node cap with external peers ranked like services. - Document the two-plus-optional-links queries, caps and boundary pairs, and the runbook aggregate contract; add an external node color. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 4 +- docs/api-reference.md | 26 ++++++++---- docs/runbooks/graph-read-contract.md | 14 ++++++- scripts/v2/workers/graph_catalog.py | 3 +- web/app/topology/services/page.tsx | 1 + web/lib/graph-store.ts | 21 ++++++---- web/lib/trace-graph.test.ts | 10 ++++- web/lib/trace-graph.ts | 46 +++++++++++++++------ web/lib/trace-source.test.ts | 31 +++++++++++++- web/lib/trace-source.ts | 62 +++++++++++++++++++--------- 10 files changed, 159 insertions(+), 59 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index e127e870e..ca20bdb31 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -29,7 +29,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed -- Service map from complete trace counts: ClickHouse trace sources using the standard OTel exporter table now count service calls, database access, workloads and messaging across the whole one-hour window with three bounded read-only aggregate queries, instead of joining the newest 1,000 spans, so low-volume calls and databases no longer drop out and calls whose parent fell outside the sample no longer show as orphans. Custom or generated ClickHouse queries and Tempo keep span sampling. Prometheus/Mimir service-graph rows keep `connection_type`, so database and queue servers no longer appear as same-named services (trace spans name them when available) and uninstrumented callers or peers appear as external nodes. Rebuild the datasource graph catalog (`datasource_index`) to pick up the new metric query. +- Service map from complete trace counts: ClickHouse trace sources using the standard OTel exporter table now count service calls, database access, workloads and messaging across the whole one-hour window with bounded read-only aggregate queries (two, plus a third for span links when the table has them), instead of joining the newest 1,000 spans, so low-volume calls and databases no longer drop out and calls whose parent fell outside the sample no longer show as orphans. Custom or generated ClickHouse queries and Tempo keep span sampling. Prometheus/Mimir service-graph rows keep `connection_type`, so database and queue servers no longer appear as same-named services (trace spans name them when the same caller has them) and uninstrumented callers or peers appear as external nodes. Rebuild the datasource graph catalog (`datasource_index`) to pick up the new metric query. - Web framework and runtime refresh: Next.js 16.4.5, React 19.3.1 and Node 26.13.0 retain authentication, ownership checks and standalone deployment; the request body guard uses the Next.js proxy convention. Refresh JavaScript/Python dependencies and AgentCore SDKs with compatibility pins, Terraform 1.17.1/provider locks, and Python 3.14 Lambda/container runtimes. The documentation guide builds and deploys on Node 26 with refreshed documentation security dependencies and Mermaid 12, preserving all four locales. Preserve gateway authentication, feature gates and release safeguards. - EKS member onboarding: the member cluster guide is now two owner commands — an Access Entry for the registered member read role and a cluster-scoped `AmazonEKSViewPolicy` association — with no Kubernetes group, kubeconfig, kubectl or RBAC step; View never reads Kubernetes Secrets, and the guide tells owners to remove any existing `AmazonEKSAdminViewPolicy`. Because View cannot list nodes, member node lists come from read-only EC2/node-group APIs through the same role and are labeled as an EC2/node-group view (no kubelet status, instance-type capacity as allocatable, no Fargate nodes); their Ready count shows as "EC2 running N · Ready unknown" and they are left out of reserved-% figures; a failed node read no longer marks the whole cluster unreachable. When an administrator opens EKS, member clusters whose Access Entry is verified are listed first in an "Access Entry verified" callout with a one-click [조회 등록] button; nothing is registered until the administrator clicks it (an app registration record only; AWSops creates no AWS or Kubernetes permissions), and an unregistered cluster stays unregistered. In diagnosis, the Kubernetes map and the node drilldown, EC2/node-group nodes read as "EC2 running · Ready unknown" with Pressure, Allocatable and Reserved shown as unknown instead of as unhealthy or fully allocatable. Entries created from the earlier guide keep working. - Bedrock models: chat answers with OpenAI GPT-6 Luna (without explicit Bedrock cache points, which GPT models reject); Explore "AI generate" writes queries with GPT-6.1 Terra and retries on Claude Sonnet 5.5 when Terra fails or its draft fails validation; primary, corrections and fallback share a 55-second request deadline with SDK cancellation; AI Diagnosis, routing, synthesis and the background workers move to Claude Sonnet 5.5, Opus 5.5 (AI Diagnosis Deep tier option) and Haiku 5.5. These models reject `temperature`, so requests no longer send it. Haiku calls and Sonnet calls with fixed output caps run without extended thinking so existing budgets still fit; Opus 5.5, which always thinks, gets extra output headroom for Deep report sections and invariant proposals. The web task now always holds a grant for its direct query-generation and assistant models (Terra, Sonnet 5.5, Haiku 5.5), so "AI generate" and the product-help assistant no longer fail with AccessDenied when hybrid routing is off; the chat runtime role additionally allows GPT-6 Luna. With `model_provider = mixed`, the Terraform variable `agentcore_chat_model_id = "global.anthropic.claude-sonnet-5-5"` rolls chat back to Sonnet 5.5 on every provisioning run, and the chat footer shows a model only when the answer carries its model provenance. @@ -710,7 +710,7 @@ First release of the **v2 line** (versioned independently from the v1 1.x line, ### Changed -- 전체 트레이스 집계 기반 서비스 맵: 표준 OTel exporter 테이블을 쓰는 ClickHouse 트레이스 원본은 최신 스팬 1,000개를 이어 붙이는 대신, 제한된 읽기 전용 집계 쿼리 3개로 1시간 구간 전체의 서비스 호출·DB 접근·워크로드·메시징을 셉니다. 호출량이 적은 연결이나 DB가 맵에서 빠지지 않고, 부모 스팬이 표본 밖에 있던 호출도 끊긴 스팬으로 표시되지 않습니다. 사용자 정의·생성된 ClickHouse 쿼리와 Tempo는 기존 스팬 표본 방식을 유지합니다. Prometheus/Mimir 서비스 그래프 지표는 `connection_type`을 보존해 DB·큐 서버가 같은 이름의 서비스로 그려지지 않으며(트레이스 스팬이 있으면 스팬 기준으로 표시), 계측되지 않은 호출자나 상대는 외부 노드로 표시됩니다. 새 지표 쿼리를 쓰려면 데이터소스 그래프 카탈로그(`datasource_index`)를 다시 만들어야 합니다. +- 전체 트레이스 집계 기반 서비스 맵: 표준 OTel exporter 테이블을 쓰는 ClickHouse 트레이스 원본은 최신 스팬 1,000개를 이어 붙이는 대신, 제한된 읽기 전용 집계 쿼리(2개, 테이블에 span link가 있으면 1개 추가)로 1시간 구간 전체의 서비스 호출·DB 접근·워크로드·메시징을 셉니다. 호출량이 적은 연결이나 DB가 맵에서 빠지지 않고, 부모 스팬이 표본 밖에 있던 호출도 끊긴 스팬으로 표시되지 않습니다. 사용자 정의·생성된 ClickHouse 쿼리와 Tempo는 기존 스팬 표본 방식을 유지합니다. Prometheus/Mimir 서비스 그래프 지표는 `connection_type`을 보존해 DB·큐 서버가 같은 이름의 서비스로 그려지지 않으며(같은 호출 서비스의 트레이스 스팬이 있으면 스팬 기준으로 표시), 계측되지 않은 호출자나 상대는 외부 노드로 표시됩니다. 새 지표 쿼리를 쓰려면 데이터소스 그래프 카탈로그(`datasource_index`)를 다시 만들어야 합니다. - 웹 프레임워크·런타임 최신화: Next.js 16.4.5, React 19.3.1, Node 26.13.0으로 인증·소유권 검사·standalone 배포를 유지하고 요청 본문 제한을 Next.js proxy 규약으로 옮깁니다. 호환성 버전 고정과 함께 JavaScript/Python 의존성·AgentCore SDK, Terraform 1.17.1·provider 잠금파일, Python 3.14 Lambda·컨테이너 런타임을 갱신합니다. 문서 가이드는 Node 26에서 빌드·배포하며 문서 사이트의 보안 의존성과 Mermaid 12를 갱신하고 4개 언어를 유지합니다. Gateway 인증·기능 게이트·릴리스 보호 장치는 유지합니다. - EKS 멤버 온보딩: 멤버 클러스터 가이드가 소유자 명령 두 개(등록된 멤버 읽기 역할의 Access Entry 생성, 클러스터 범위 `AmazonEKSViewPolicy` 연결)로 줄었으며 Kubernetes 그룹·kubeconfig·kubectl·RBAC 단계가 없습니다. View는 Kubernetes Secrets를 읽지 않으며, 가이드는 기존 `AmazonEKSAdminViewPolicy` 연결을 제거하도록 안내합니다. View로는 노드를 조회할 수 없으므로 멤버 노드 목록은 같은 역할로 읽기 전용 EC2/노드그룹 API에서 가져와 EC2/노드그룹 기준으로 표시합니다(kubelet 상태 없음, allocatable은 인스턴스 타입 용량, Fargate 노드 제외). 이 노드의 Ready 수는 "EC2 running N · Ready unknown"으로 표시되며 예약 비율 계산에서 제외됩니다. 노드 조회만 실패해도 클러스터 전체가 조회 불가로 표시되지 않습니다. 관리자가 EKS 화면을 열면 Access Entry가 확인된 멤버 클러스터가 "Access Entry 확인됨" 안내 영역에 먼저 표시되고 [조회 등록] 버튼 한 번으로 등록할 수 있으며, 관리자가 누르기 전에는 등록되지 않습니다(앱 등록 기록만 저장, AWSops는 AWS·Kubernetes 권한을 만들지 않음). 해제한 클러스터는 해제 상태로 유지됩니다. 진단 표·Kubernetes 맵·노드 상세에서도 EC2/노드그룹 노드는 "EC2 running · Ready unknown"으로 표시되며 Pressure·Allocatable·Reserved는 비정상이나 전량 할당 대신 알 수 없음으로 표시됩니다. 이전 가이드로 만든 항목도 그대로 동작합니다. - Bedrock 모델: 채팅은 OpenAI GPT-6 Luna로 답합니다(GPT 모델이 거부하는 명시적 Bedrock 캐시 포인트는 끔). Explore "AI로 생성"은 GPT-6.1 Terra로 쿼리를 만들고, Terra 호출이 실패하거나 초안이 검증에서 거부되면 Claude Sonnet 5.5로 다시 시도합니다. 기본 모델·수정·폴백은 하나의 55초 요청 제한 시간과 SDK 취소 신호를 공유합니다. AI 진단, 라우팅, 합성, 백그라운드 워커는 Claude Sonnet 5.5, Opus 5.5(AI 진단 Deep 단계 선택), Haiku 5.5로 바뀝니다. 이 모델들은 `temperature`를 거부하므로 요청에서 더 이상 보내지 않습니다. Haiku 호출과 출력 상한이 고정된 Sonnet 호출은 기존 상한이 유지되도록 확장 사고 없이 실행하고, 항상 사고하는 Opus 5.5는 Deep 리포트 섹션과 불변식 제안에 출력 여유분을 더 받습니다. 웹 태스크는 직접 호출하는 쿼리 생성·도움말 모델(Terra, Sonnet 5.5, Haiku 5.5) 권한을 항상 가지므로, 하이브리드 라우팅이 꺼져 있어도 "AI로 생성"과 제품 도움말이 AccessDenied로 실패하지 않습니다. 채팅 런타임 역할은 GPT-6 Luna도 허용합니다. `model_provider = mixed`일 때 Terraform 변수 `agentcore_chat_model_id = "global.anthropic.claude-sonnet-5-5"`로 채팅을 Sonnet 5.5로 되돌리며 매 프로비저닝마다 유지되고, 채팅 하단의 모델 표시는 답변에 모델 출처가 있을 때만 나타납니다. diff --git a/docs/api-reference.md b/docs/api-reference.md index 3171adf0d..6c49fed8a 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -547,15 +547,23 @@ runtime payloads for compatibility with older or malformed responses. ### Trace graph sources A ClickHouse datasource whose registry query is the catalog's exporter-schema `otel_v1` template -(or the unregistered default) is read as whole-window aggregates of the same table, not as the -newest spans. Three bounded read-only queries return service/kind/status attribute groups with -span counts, parent→child service pairs and span-link pairs, each limited to 1,000 rows; a -saturated limit adds `cap_reached`, and any failed query fails the whole source read. Node and -edge `spanCount` then count spans in the window, and source-counted pairs produce no -`orphanSpans`. Generated or custom templates, Tempo and other sources keep span sampling. -Service-graph metric rows keep Tempo's `connection_type`: `database` and `messaging_system` -servers become `db`/`queue` nodes only when no trace spans are present (spans carry the -host-qualified identity), and `virtual_node` rows draw the uninstrumented side as an `external` node. +(exactly the bare exporter columns; generated or aliased templates are excluded), or the +unregistered default, is read as whole-window aggregates of the same table, not as the newest +spans. Two bounded read-only queries return service/kind/status attribute groups with span counts +and parent→child service pairs; a third returns span-link pairs when the table has `Links` +columns. Each is limited to 1,000 rows: a saturated limit adds `cap_reached`, an unproven empty +result adds `empty_not_confirmed`, and any failed query fails the whole source read. Node and edge +`spanCount` then count spans in the window, and source-counted pairs produce no `orphanSpans`. +A child whose parent started before the window is not counted as a pair and is not disclosed. +Publication still applies the 200-node/500-edge caps reported as `nodeDrops`/`edgeDrops`. +Generated or custom templates, Tempo and other sources keep span sampling. + +Service-graph metric rows keep Tempo's `connection_type`. A `database` or `messaging_system` row +becomes a `db`/`queue` node with no account/region attribution, unless trace spans already show the +same client service using a database (or queue); those deferred rows are counted in +`deferredMetricCalls`. `virtual_node` rows draw the uninstrumented side as an `external` node +(Tempo names an uninstrumented caller `user`). Metric volume is kept as node `metricCount` and +ranks with `spanCount` under the node cap. ### Trace edge instance observations diff --git a/docs/runbooks/graph-read-contract.md b/docs/runbooks/graph-read-contract.md index 15761f65f..2adb98ae0 100644 --- a/docs/runbooks/graph-read-contract.md +++ b/docs/runbooks/graph-read-contract.md @@ -220,6 +220,17 @@ Oversized valid Tempo children keep a bounded structured OTLP projection and can The shared query normalizer carries collection status into Explore. Marked partial, unknown or failed empty responses show an uncertainty/failure note instead of an ordinary empty-result claim; useful rows remain visible with the same disclosure. Scalar format failures remain distinct from empty responses. Non-boolean truncation metadata is unverified, never silently interpreted as complete output. +### Whole-window ClickHouse aggregates + +The ClickHouse adapter for the catalog's exporter-schema template reads node groups, +parent→child pairs and (when present) span links as whole-window aggregates; see +[trace graph sources](../api-reference.md#trace-graph-sources). The aggregates form one source +read: a failed or malformed query result fails it, and every aggregate needs its own empty proof, +so nodes never publish without their edges and an unproven pair query keeps the read partial. +Row limits disclose `cap_reached` like other bounded sources. The join sides project only the +sorted identity attributes. Metric-derived `external` nodes and deferred metric rows +(`deferredMetricCalls`) are additive trace metadata. + ## Rebuild capacity Flow and infra input allow 8192 rows within the same bounded read envelope. The 64KiB-per-row and 8MiB projected-data/identifier limits still apply, as do the @@ -409,7 +420,8 @@ docker exec "$graph_test_container" psql -U postgres -d awsops \ -c "COMMENT ON DATABASE awsops IS 'awsops-disposable-graph-test'" cd web npx vitest run lib/trace-source.test.ts lib/graph-read-postgres.test.ts lib/graph-inventory-read-postgres.test.ts \ - lib/graph-store-postgres.test.ts app/api/graph/route.test.ts lib/graph-state.test.ts + lib/graph-store-postgres.test.ts app/api/graph/route.test.ts lib/graph-state.test.ts \ + lib/trace-graph.test.ts lib/graph-store-trace.test.ts lib/graph-sources.test.ts docker rm -f "$graph_test_container" ``` diff --git a/scripts/v2/workers/graph_catalog.py b/scripts/v2/workers/graph_catalog.py index fdab80f4a..521e34006 100644 --- a/scripts/v2/workers/graph_catalog.py +++ b/scripts/v2/workers/graph_catalog.py @@ -26,7 +26,8 @@ # v3: preserve span operation/status/links and metric identity scopes. Rebuild cached queries so # source adapters receive metadata before aggregation, and both time bounds share one anchor. -CATALOG_VERSION = "v3" +# v4: service-graph PromQL keeps connection_type; rebuild so indexed instances pick it up. +CATALOG_VERSION = "v4" # The OTel ClickHouse exporter's default `otel_traces` column shape. A table matching this column # SET (regardless of its actual name) is treated as the span source. diff --git a/web/app/topology/services/page.tsx b/web/app/topology/services/page.tsx index 884f302ca..53507ea15 100644 --- a/web/app/topology/services/page.tsx +++ b/web/app/topology/services/page.tsx @@ -26,6 +26,7 @@ const COLORS: Record = { db: ['#FBEFE0', '#C8902F'], // amber workload: ['#F1E9FF', '#8A5BD0'], // purple queue: ['#E9F5EE', '#4D906C'], + external: ['#F4F4F1', '#7A7A6E'], // uninstrumented service-graph peer }; const RESOURCE = ['#EEF0F2', '#9AA6B2'] as const; const QUEUE_HEIGHT = 144; diff --git a/web/lib/graph-store.ts b/web/lib/graph-store.ts index 5f1537c4b..004ecea87 100644 --- a/web/lib/graph-store.ts +++ b/web/lib/graph-store.ts @@ -2,7 +2,7 @@ import { randomUUID } from 'node:crypto'; import type { Pool, PoolClient } from 'pg'; import { buildFlowGraph, type FlowInput, type FlowKind } from './flow-topology'; import { buildInfraGraph, type Row } from './infra-topology'; -import type { TraceSource, TraceRead, ServiceGraphCall, SourceRead } from './trace-source'; +import type { TraceSource, TraceRead, TraceSpan, ServiceGraphCall, SourceRead } from './trace-source'; import { buildTraceGraph, type InfraNodeLike } from './trace-graph'; import { graphDiagnostic, inventorySourcesStale, writeGraphState, type GraphAttempt, type GraphClass } from './graph-state'; import { currentAccountId } from './account'; @@ -301,9 +301,9 @@ export async function rebuildTraceGraph( windowStartMs: startMs, windowEndMs: endMs, }); // Adapter status, not a separate readiness probe, distinguishes absent config from a failed read. - const spanReads = await Promise.all(sources.map(async (source, i) => { + const spanReads = await Promise.all(sources.map(async (source, i): Promise => { try { return await source.recentSpans(TRACE_WINDOW_MINS, TRACE_SPAN_CAP, endMs); } - catch { return failed(`trace:${i}`) as TraceRead; } + catch { return failed(`trace:${i}`); } })); const metricReads = await Promise.all(metricsSources.map(async (source, i) => { try { return await source.calls(TRACE_WINDOW_MINS, endMs); } @@ -321,10 +321,10 @@ export async function rebuildTraceGraph( // Aggregate trace sources return span-pair edges beside their node rows; same sourceId stamping. const calls = [...spanReads.map((read) => ({ ...read, items: read.calls ?? [] })), ...metricReads] .flatMap((read) => read.items.map((call) => ({ - ...call, - clientIdentity: { ...call.clientIdentity, sourceId: call.clientIdentity?.sourceId ?? read.sourceId }, - serverIdentity: { ...call.serverIdentity, sourceId: call.serverIdentity?.sourceId ?? read.sourceId }, - }))); + ...call, + clientIdentity: { ...call.clientIdentity, sourceId: call.clientIdentity?.sourceId ?? read.sourceId }, + serverIdentity: { ...call.serverIdentity, sourceId: call.serverIdentity?.sourceId ?? read.sourceId }, + }))); // Unproven empty/lost-data reads retain the prior generation. Valid nonempty // bounded reads continue through the existing atomic partial-snapshot publisher. if (!reads.length || hasFailure || cannotSweep || (partial && !spans.length && !calls.length)) { @@ -355,9 +355,11 @@ export async function rebuildTraceGraph( } const graph = buildTraceGraph(spans, calls, infraNodes, currentAccountId()); // Preserve structurally important DB/queue/workload nodes before ranking service volume. - const rank = (kind: string) => kind === 'service' ? 0 : 1; + // Metric-only external peers rank with services by volume, so many virtual nodes cannot evict them. + const rank = (kind: string) => kind === 'service' || kind === 'external' ? 0 : 1; + const volume = (meta: Record) => Number(meta.spanCount ?? 0) + Number(meta.metricCount ?? 0); const nodes = graph.nodes.sort((a, b) => rank(b.kind) - rank(a.kind) - || Number(b.meta.spanCount ?? 0) - Number(a.meta.spanCount ?? 0)).slice(0, TRACE_NODE_CAP); + || volume(b.meta) - volume(a.meta)).slice(0, TRACE_NODE_CAP); const kept = new Set(nodes.map((node) => node.id)); const edges = graph.edges.filter((edge) => kept.has(edge.source) && kept.has(edge.target)) .sort((a, b) => (b.meta.spanCount + b.meta.metricCount) - (a.meta.spanCount + a.meta.metricCount)) @@ -373,6 +375,7 @@ export async function rebuildTraceGraph( sources: sourceDetails, retainedPrevious: false, windowStartMs: startMs, windowEndMs: endMs, nodeDrops, edgeDrops, orphanSpans: graph.orphanSpans, invalidSpans: graph.invalidSpans, unresolvedMessaging: graph.unresolvedMessaging, + ...(graph.deferredMetricCalls ? { deferredMetricCalls: graph.deferredMetricCalls } : {}), infraUnavailable, }, }); diff --git a/web/lib/trace-graph.test.ts b/web/lib/trace-graph.test.ts index 6640fef61..7e506b29f 100644 --- a/web/lib/trace-graph.test.ts +++ b/web/lib/trace-graph.test.ts @@ -291,9 +291,15 @@ describe('source-aggregated rows and service-graph connection types', () => { expect(rel('user', 'payment')?.rel).toBe('calls'); expect(rel('payment', 'stripe')?.rel).toBe('calls'); + expect(metricsOnly.nodes.find(node => node.label === 'reviews')?.meta).toMatchObject({ + accountId: null, region: null, identityProvenance: 'telemetry_claim' }); + expect(svc(metricsOnly, 'payment')?.meta.metricCount).toBe(8); + + // Span evidence for review's database defers only review's metric DB row; order's queue row + // (no span evidence for that client) is still drawn, and the deferral is counted. const withSpans = buildTraceGraph([dbSpan({ service: 'review', dbName: 'reviews' })], calls, inventory()); - expect(withSpans.nodes.filter(node => node.meta.provenance === 'service_graph_metric' && node.kind !== 'external')) - .toEqual([]); expect(withSpans.nodes.filter(node => node.kind === 'db')).toHaveLength(1); + expect(withSpans.nodes.find(node => node.label === 'orders-topic')?.kind).toBe('queue'); + expect(withSpans.deferredMetricCalls).toBe(1); }); }); diff --git a/web/lib/trace-graph.ts b/web/lib/trace-graph.ts index 7137bc643..96a32d106 100644 --- a/web/lib/trace-graph.ts +++ b/web/lib/trace-graph.ts @@ -302,9 +302,20 @@ export function buildTraceGraph( } } // Trace spans name databases and queues with host-qualified identity. Metric rows only carry - // Tempo's server label (a db.name such as "admin"), so they draw those nodes only when no - // span evidence exists; otherwise the same database would appear twice under two names. - const spanEvidence = byId.size > 0; + // Tempo's server label (a db.name such as "admin"), so a metric database/queue row is deferred + // when trace spans already show that same client service querying a database (or using a + // queue); otherwise the same dependency would appear twice under two names. Other services keep + // their metric rows, and every deferral is counted. + const spanDbClients = new Set(), spanQueueClients = new Set(); + for (const span of byId.values()) { + if (span.dbSystem) spanDbClients.add(span.service); + if (span.messagingSystem && span.messagingDestination) spanQueueClients.add(span.service); + } + let deferredMetricCalls = 0; + const metricWeight = (id: string, count: number) => { + const meta = nodes.get(id)!.meta; + meta.metricCount = Number(meta.metricCount ?? 0) + count; + }; for (const call of metricCalls) { if (!Number.isFinite(call.count) || call.count <= 0) continue; if (call.evidence === 'trace') { @@ -315,26 +326,33 @@ export function buildTraceGraph( } const clientIdentity = call.clientIdentity ?? {}, serverIdentity = call.serverIdentity ?? {}; if (call.connectionType === 'database' || call.connectionType === 'messaging_system') { - if (spanEvidence) continue; const database = call.connectionType === 'database'; + if ((database ? spanDbClients : spanQueueClients).has(call.client)) { deferredMetricCalls++; continue; } + // Caller labels scope the node ID (no cross-scope merge) but are not attribution: like + // queue claims, the database carries no account/region and never bridges to inventory. const scoped = resourceScope({ ...clientIdentity, ...serverIdentity }); const id = nodeId(database ? 'db' : 'queue', scoped, `metric:${call.server}`); if (!nodes.has(id)) nodes.set(id, database ? { id, kind: 'db', label: call.server.slice(0, 240), meta: { system: null, host: null, dbName: null, metricServer: call.server, sourceId: scoped.sourceId ?? null, environment: scoped.environment ?? null, - accountId: scoped.accountId ?? null, region: scoped.region ?? null, provenance: 'service_graph_metric' }, + accountId: null, region: null, identityProvenance: 'telemetry_claim', + provenance: 'service_graph_metric' }, } : { id, kind: 'queue', label: call.server.slice(0, 240), meta: queueClaimMeta({ system: null, destination: call.server, broker: null, cluster: null, sourceId: scoped.sourceId ?? null, environment: scoped.environment ?? null, provenance: 'service_graph_metric' }), }); - edge(serviceNode(call.client, clientIdentity), id, database ? 'queries' : 'publishes', 0, call.count); + const client = serviceNode(call.client, clientIdentity); + edge(client, id, database ? 'queries' : 'publishes', 0, call.count); + metricWeight(client, call.count); metricWeight(id, call.count); continue; } if (call.connectionType === 'virtual_node') { - // Tempo names an uninstrumented caller "user"; otherwise the uninstrumented side is the server. + // Tempo's service-graph processor names an uninstrumented caller "user" (its fixed default); + // otherwise the uninstrumented side is the server. A real service named "user" that calls an + // uninstrumented peer is indistinguishable in these labels and is drawn as the caller node. const clientVirtual = call.client === 'user'; const external = (name: string, identity: TraceIdentity) => { const id = nodeId('ext', resourceScope(identity), name); @@ -343,14 +361,16 @@ export function buildTraceGraph( provenance: 'service_graph_metric' } }); return id; }; - edge(clientVirtual ? external(call.client, clientIdentity) : serviceNode(call.client, clientIdentity), - clientVirtual ? serviceNode(call.server, serverIdentity) : external(call.server, serverIdentity), - 'calls', 0, call.count); + const source = clientVirtual ? external(call.client, clientIdentity) : serviceNode(call.client, clientIdentity); + const target = clientVirtual ? serviceNode(call.server, serverIdentity) : external(call.server, serverIdentity); + edge(source, target, 'calls', 0, call.count); + metricWeight(source, call.count); metricWeight(target, call.count); continue; } - edge(serviceNode(call.client, clientIdentity), - serviceNode(call.server, serverIdentity), 'calls', 0, call.count); + const client = serviceNode(call.client, clientIdentity), server = serviceNode(call.server, serverIdentity); + edge(client, server, 'calls', 0, call.count); + metricWeight(client, call.count); metricWeight(server, call.count); } return { nodes: [...nodes.values()], edges: [...edges.values()], orphanSpans, invalidSpans, - unresolvedMessaging }; + unresolvedMessaging, deferredMetricCalls }; } diff --git a/web/lib/trace-source.test.ts b/web/lib/trace-source.test.ts index e2c2e3a6e..859bf9230 100644 --- a/web/lib/trace-source.test.ts +++ b/web/lib/trace-source.test.ts @@ -90,7 +90,7 @@ describe('ClickHouseOtelAggregateSource (whole-window aggregates)', () => { const group = (over: Record = {}) => ({ ServiceName: 'checkout', SpanKind: 'Client', StatusCode: 'Error', ResourceAttributes: resource, SpanAttributes: { 'db.system': 'postgresql', 'server.address': 'db.example.internal', 'db.name': 'orders' }, - Weight: '42', DurationSum: '84000000', LastTimestamp: END - 1_000, FirstTimestamp: END - 3_000_000, + Weight: '42', DurationMsSum: '84', LastTimestamp: END - 1_000, FirstTimestamp: END - 3_000_000, SampleTraceId: 't1', SampleSpanId: 's1', Pods: ['checkout-a', 'checkout-b', ''], ...over, }); const pair = (over: Record = {}) => ({ @@ -116,6 +116,10 @@ describe('ClickHouseOtelAggregateSource (whole-window aggregates)', () => { expect(otelAggregateShape(catalog)).toEqual({ table: '`otel`.`otel_traces`', spanKind: false, statusCode: false, links: false }); expect(otelAggregateShape('SELECT trace_id AS TraceId FROM spans LIMIT {cap}')).toBeUndefined(); expect(otelAggregateShape(catalog.replace('LIMIT {cap}', 'LIMIT {cap}; DROP TABLE x'))).toBeUndefined(); + // Generated templates share the otel_v1 mapper but alias columns; aliases are not exporter Maps. + expect(otelAggregateShape(catalog.replace('ResourceAttributes,', 'attrs AS ResourceAttributes,'))).toBeUndefined(); + expect(otelAggregateShape(catalog.replace(', SpanAttributes FROM', ', SpanAttributes, extra FROM'))).toBeUndefined(); + expect(otelAggregateShape(catalog.replace('Duration, ', ''))).toBeUndefined(); }); it('queries node groups, parent/child pairs and links over the exact window and maps weighted rows', async () => { @@ -142,6 +146,28 @@ describe('ClickHouseOtelAggregateSource (whole-window aggregates)', () => { ]); }); + it('keeps the join build to sorted identity maps and guards link arrays before ARRAY JOIN', async () => { + answer([group()], []); + await new ClickHouseOtelAggregateSource().recentSpans(60, 1000, END); + const sqls = invokeMcpLambdaTool.mock.calls.map(([{ args }]) => args.sql as string); + for (const sql of sqls.filter(q => q.includes('INNER JOIN'))) { + expect(sql).not.toMatch(/SELECT TraceId, SpanId, ServiceName, ResourceAttributes FROM/); + expect(sql.match(/mapSort\(mapFilter/g)).toHaveLength(2); + } + const groups = sqls.find(q => !q.includes('JOIN'))!; + expect(groups).toContain("startsWith(v, 'arn:')"); + expect(groups).toContain('intDiv(sum(Duration), 1000000) AS DurationMsSum'); + expect(sqls.find(q => q.includes('ARRAY JOIN'))).toContain('length(`Links.TraceId`) = length(`Links.SpanId`)'); + }); + + it('requires empty proof from each aggregate, not only the node query', async () => { + invokeMcpLambdaTool.mockImplementation(async ({ args }: { args: { sql: string } }) => + args.sql.includes('INNER JOIN') ? { rows: [] } : { rows: [group()], collectionStatus: 'ok' }); + const out = await new ClickHouseOtelAggregateSource().recentSpans(60, 1000, END); + expect(out.reasons).toContain('empty_not_confirmed'); + expect(out.status).toBe('partial'); + }); + it('skips the link query when the table has no Links columns', async () => { answer([group()], []); const shape = { table: 't', spanKind: true, statusCode: true, links: false }; @@ -163,7 +189,8 @@ describe('ClickHouseOtelAggregateSource (whole-window aggregates)', () => { }); it('marks a saturated row limit and malformed rows as partial evidence', async () => { - answer([group(), group({ Weight: '0', SampleSpanId: 's2' }), group({ FirstTimestamp: END, SampleSpanId: 's3' })], + answer([group(), group({ Weight: '0', SampleSpanId: 's2' }), group({ FirstTimestamp: END, SampleSpanId: 's3' }), + group({ DurationMsSum: '9007199254740993', SampleSpanId: 's4' })], [pair(), pair({ ServerResource: 'oops' })]); const out = await new ClickHouseOtelAggregateSource().recentSpans(60, 2, END); expect(out.status).toBe('partial'); diff --git a/web/lib/trace-source.ts b/web/lib/trace-source.ts index 0aea52c2a..36a9d4d94 100644 --- a/web/lib/trace-source.ts +++ b/web/lib/trace-source.ts @@ -395,12 +395,24 @@ const CATALOG_OTEL_TEMPLATE = new RegExp( 'WHERE Timestamp >= now\\(\\) - INTERVAL \\{window\\} MINUTE AND Timestamp <= now\\(\\) ' + 'ORDER BY Timestamp DESC LIMIT \\{cap\\}$'); +// Exactly the bare exporter columns graph_catalog.py can select. Generated templates share the +// otel_v1 mapper but alias other columns (`attrs AS ResourceAttributes`); those are not proof of +// exporter Map columns, so any alias, expression or unknown column keeps the span reader. +const EXPORTER_COLUMNS = new Set([ + 'TraceId', 'SpanId', 'ParentSpanId', 'ServiceName', 'SpanKind', 'SpanName', 'StatusCode', + '`Links.TraceId`', '`Links.SpanId`', 'Timestamp', 'Duration', 'ResourceAttributes', 'SpanAttributes', +]); +const REQUIRED_EXPORTER_COLUMNS = ['TraceId', 'SpanId', 'ParentSpanId', 'ServiceName', 'Timestamp', + 'Duration', 'ResourceAttributes', 'SpanAttributes']; + /** The aggregate shape for a registry template, or undefined when it is not the catalog's. */ export function otelAggregateShape(sqlTemplate?: string): OtelAggregateShape | undefined { if (sqlTemplate === undefined) return DEFAULT_OTEL_SHAPE; const match = CATALOG_OTEL_TEMPLATE.exec(sqlTemplate); if (!match) return undefined; const columns = match[1].split(',').map((c) => c.trim()); + if (new Set(columns).size !== columns.length || columns.some((c) => !EXPORTER_COLUMNS.has(c)) + || REQUIRED_EXPORTER_COLUMNS.some((c) => !columns.includes(c))) return undefined; return { table: match[2], spanKind: columns.includes('SpanKind'), statusCode: columns.includes('StatusCode'), links: columns.includes('`Links.TraceId`') && columns.includes('`Links.SpanId`'), @@ -409,8 +421,9 @@ export function otelAggregateShape(sqlTemplate?: string): OtelAggregateShape | u const list = (keys: readonly string[]) => `[${keys.map((k) => `'${k}'`).join(', ')}]`; // Only the attributes spanMetadata reads are grouped. Pod names and per-process instance IDs would -// multiply rows, so pods are collected as a bounded array and service.instance.id is kept only -// when it is an EC2 instance claim. +// multiply rows, so pods are collected as a bounded array. service.instance.id keeps every value +// claimedInstance can act on: bare EC2 IDs bind, and ARNs bind or veto. Other values (process +// UUIDs) are ignored by that check, so dropping them changes no binding. const AGG_RESOURCE_KEYS = [ 'service.name', 'cloud.account.id', 'cloud.region', 'deployment.environment.name', 'deployment.environment', 'service.namespace', 'service.version', 'cloud.resource_id', 'host.id', @@ -434,35 +447,37 @@ function aggregateSql(shape: OtelAggregateShape, windowMins: number, endMs: numb const status = shape.statusCode ? 'StatusCode' : "'' AS StatusCode"; // Non-db/messaging spans drop their peer attributes so HTTP peers do not multiply groups. const groups = 'SELECT ServiceName, SpanKind, StatusCode, RA AS ResourceAttributes, SA AS SpanAttributes, ' + - 'count() AS Weight, sum(Duration) AS DurationSum, toUnixTimestamp64Milli(max(Timestamp)) AS LastTimestamp, ' + + 'count() AS Weight, intDiv(sum(Duration), 1000000) AS DurationMsSum, toUnixTimestamp64Milli(max(Timestamp)) AS LastTimestamp, ' + 'toUnixTimestamp64Milli(min(Timestamp)) AS FirstTimestamp, any(TraceId) AS SampleTraceId, ' + `any(SpanId) AS SampleSpanId, groupUniqArray(${AGG_POD_CAP})(Pod) AS Pods FROM (` + `SELECT ServiceName, ${kind}, ${status}, Duration, Timestamp, TraceId, SpanId, ` + "ResourceAttributes['k8s.pod.name'] AS Pod, " + - `mapFilter((k, v) -> has(${list(AGG_RESOURCE_KEYS)}, k) AND (k != 'service.instance.id' ` + - "OR match(v, '^i-[0-9a-f]{8}([0-9a-f]{9})?$')), ResourceAttributes) AS RA, " + + `mapSort(mapFilter((k, v) -> has(${list(AGG_RESOURCE_KEYS)}, k) AND (k != 'service.instance.id' ` + + "OR match(v, '^i-[0-9a-f]{8}([0-9a-f]{9})?$') OR startsWith(v, 'arn:')), ResourceAttributes)) AS RA, " + "if(SpanAttributes['db.system'] != '' OR SpanAttributes['messaging.system'] != '', " + - `mapFilter((k, v) -> has(${list(AGG_SPAN_KEYS)}, k), SpanAttributes), ` + + `mapSort(mapFilter((k, v) -> has(${list(AGG_SPAN_KEYS)}, k), SpanAttributes)), ` + 'mapFilter((k, v) -> false, SpanAttributes)) AS SA ' + `FROM ${shape.table} WHERE ${window}) ` + `GROUP BY ServiceName, SpanKind, StatusCode, RA, SA ORDER BY Weight DESC LIMIT ${limit}`; + // Both join sides carry only the sorted identity map, so the hash build holds one small row per + // span rather than every span's raw attribute map, and equal identities compare equal. + const identity = `mapSort(mapFilter((k, v) -> has(${list(AGG_IDENTITY_KEYS)}, k), ResourceAttributes)) AS R`; const pairs = (child: string, on: string) => - `SELECT p.ServiceName AS ClientService, mapFilter((k, v) -> has(${list(AGG_IDENTITY_KEYS)}, k), ` + - 'p.ResourceAttributes) AS ClientResource, c.ServiceName AS ServerService, ' + - `mapFilter((k, v) -> has(${list(AGG_IDENTITY_KEYS)}, k), c.ResourceAttributes) AS ServerResource, ` + - `count() AS Weight FROM (${child}) AS c INNER JOIN (SELECT TraceId, SpanId, ServiceName, ` + - `ResourceAttributes FROM ${shape.table} WHERE ${window}) AS p ON ${on} ` + + 'SELECT p.ServiceName AS ClientService, p.R AS ClientResource, c.ServiceName AS ServerService, ' + + `c.R AS ServerResource, count() AS Weight FROM (${child}) AS c INNER JOIN (SELECT TraceId, SpanId, ` + + `ServiceName, ${identity} FROM ${shape.table} WHERE ${window}) AS p ON ${on} ` + 'GROUP BY ClientService, ClientResource, ServerService, ServerResource ' + 'HAVING NOT (ClientService = ServerService AND ClientResource = ServerResource) ' + `ORDER BY Weight DESC LIMIT ${limit}`; const calls = pairs( - `SELECT TraceId, ParentSpanId, ServiceName, ResourceAttributes FROM ${shape.table} ` + + `SELECT TraceId, ParentSpanId, ServiceName, ${identity} FROM ${shape.table} ` + `WHERE ${window} AND ParentSpanId != ''`, 'c.TraceId = p.TraceId AND c.ParentSpanId = p.SpanId'); + // ARRAY JOIN raises on unequal nested-array lengths; such rows are filtered before it. const links = shape.links ? pairs( - 'SELECT LinkTraceId, LinkSpanId, ServiceName, ResourceAttributes FROM ' + - `${shape.table} ARRAY JOIN \`Links.TraceId\` AS LinkTraceId, \`Links.SpanId\` AS LinkSpanId ` + - `WHERE ${window}`, + `SELECT LinkTraceId, LinkSpanId, ServiceName, R FROM (SELECT \`Links.TraceId\` AS LT, ` + + `\`Links.SpanId\` AS LS, ServiceName, ${identity} FROM ${shape.table} WHERE ${window} ` + + 'AND length(`Links.TraceId`) = length(`Links.SpanId`)) ARRAY JOIN LT AS LinkTraceId, LS AS LinkSpanId', 'c.LinkTraceId = p.TraceId AND c.LinkSpanId = p.SpanId') : undefined; return { groups, calls, links }; } @@ -520,10 +535,10 @@ export class ClickHouseOtelAggregateSource implements TraceSource { const row = object(value); const weight = positiveCount(row?.Weight); const last = numeric(row?.LastTimestamp), first = numeric(row?.FirstTimestamp); - const duration = numeric(row?.DurationSum); + const duration = numeric(row?.DurationMsSum); if (!row || !text(row.ServiceName) || !text(row.SampleTraceId) || !text(row.SampleSpanId) || weight === undefined || last === undefined || first === undefined || first > last - || duration === undefined || duration < 0 + || duration === undefined || !Number.isSafeInteger(duration) || duration < 0 || (row.ResourceAttributes !== undefined && !object(row.ResourceAttributes)) || (row.SpanAttributes !== undefined && !object(row.SpanAttributes))) { reasons.push('malformed_rows'); @@ -531,14 +546,15 @@ export class ClickHouseOtelAggregateSource implements TraceSource { } const item = mapOtelRow({ TraceId: row.SampleTraceId, SpanId: row.SampleSpanId, ServiceName: row.ServiceName, - SpanKind: row.SpanKind, StatusCode: row.StatusCode, Timestamp: last, Duration: duration, + SpanKind: row.SpanKind, StatusCode: row.StatusCode, Timestamp: last, Duration: 0, ResourceAttributes: row.ResourceAttributes, SpanAttributes: row.SpanAttributes, }); if (this.shape.statusCode && text(row.StatusCode) && !item.status) reasons.push('malformed_rows'); const pods = Array.isArray(row.Pods) ? [...new Set(row.Pods.filter((p): p is string => !!text(p)))].slice(0, AGG_POD_CAP) : []; if (!inWindow(item, window)) continue; - items.push({ ...item, sourceId, weight, firstStartMs: first, ...(pods.length ? { k8sPods: pods } : {}) }); + items.push({ ...item, sourceId, weight, durationMs: duration, firstStartMs: first, + ...(pods.length ? { k8sPods: pods } : {}) }); } const calls: ServiceGraphCall[] = []; for (const [rows, rel] of [[callRows, 'calls'], [linkRows, 'linked']] as const) { @@ -555,7 +571,13 @@ export class ClickHouseOtelAggregateSource implements TraceSource { calls.push({ client, server, count, clientIdentity, serverIdentity, evidence: 'trace', rel }); } } - requireEmptyProof(payloads[0], items, reasons); + // Each aggregate needs its own empty proof: confirmed nodes with an unproven-empty pair query + // would otherwise publish as a complete graph without its edges. + for (const [i, payload] of payloads.entries()) { + if (results[i].rows!.length) continue; + const outer = object(payload), status = outer?.collectionStatus ?? object(outer?.result)?.collectionStatus; + if (status !== 'ok' && status !== 'empty' && !reasons.includes('empty_not_confirmed')) reasons.push('empty_not_confirmed'); + } return { ...readResult(sourceId, window, items, reasons), calls }; } } From bc76f85596c917dafcf6260c14c0f72f7cd28ff1 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=EC=98=A4=EC=A4=80=EC=84=9D=28Junseok=20Oh=29DevOps?= <52226147+Atom-oh@users.noreply.github.com> Date: Tue, 10 Nov 2026 10:31:02 +0000 Subject: [PATCH 3/4] fix(topology): keep trace aggregate groups bounded and metric dedup scoped - Reduce host.id/service.instance.id to EC2 IDs/ARNs and any other cloud.resource_id to one 'unbindable' marker, so host UUIDs stop splitting groups while claimedInstance keeps its cloud.resource_id veto. - Treat a metric database/queue row as a duplicate only when a span from a compatibly scoped client of the same name names that dependency; other dependencies and scopes keep their metric rows. The internal counter is no longer documented as published metadata. - Rank label-only metric db/queue nodes by volume with services so a high-cardinality metric cannot evict service nodes. - Clamp aggregate rows to the limit, disclose out-of-window rows, cap merged workload pods and bound metric node names. - Docs: distinguish failed envelopes from skipped rows, qualify counts by cap_reached, and move the mocked suites out of the PostgreSQL block. Co-Authored-By: Claude Opus 5.5 --- docs/api-reference.md | 28 +++++++++++++--------- docs/runbooks/graph-read-contract.md | 17 +++++++------ web/lib/graph-store-trace.test.ts | 14 +++++++++++ web/lib/graph-store.ts | 9 +++---- web/lib/trace-graph.test.ts | 25 +++++++++++++++++++ web/lib/trace-graph.ts | 36 +++++++++++++++++++--------- web/lib/trace-source.test.ts | 10 ++++++++ web/lib/trace-source.ts | 21 +++++++++------- 8 files changed, 119 insertions(+), 41 deletions(-) diff --git a/docs/api-reference.md b/docs/api-reference.md index 6c49fed8a..0845886fb 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -551,19 +551,25 @@ A ClickHouse datasource whose registry query is the catalog's exporter-schema `o unregistered default, is read as whole-window aggregates of the same table, not as the newest spans. Two bounded read-only queries return service/kind/status attribute groups with span counts and parent→child service pairs; a third returns span-link pairs when the table has `Links` -columns. Each is limited to 1,000 rows: a saturated limit adds `cap_reached`, an unproven empty -result adds `empty_not_confirmed`, and any failed query fails the whole source read. Node and edge -`spanCount` then count spans in the window, and source-counted pairs produce no `orphanSpans`. -A child whose parent started before the window is not counted as a pair and is not disclosed. -Publication still applies the 200-node/500-edge caps reported as `nodeDrops`/`edgeDrops`. -Generated or custom templates, Tempo and other sources keep span sampling. +columns. Each is limited to 1,000 rows. A failed query or malformed response envelope fails the +whole source read; malformed individual rows are skipped as `malformed_rows`, a saturated limit +adds `cap_reached`, rows outside the window add `incomplete_collection`, and an unproven empty +result adds `empty_not_confirmed` (a partial read that may publish nodes without edges). Without +`cap_reached`, node and edge `spanCount` count the window's spans, and source-counted pairs produce +no `orphanSpans`. A child whose parent started before the window is not counted as a pair and is +not disclosed. Grouping keeps only EC2 instance IDs and ARNs from the instance claim attributes, +plus a fixed marker for any other `cloud.resource_id` (which still vetoes binding). Publication +still applies the 200-node/500-edge caps reported as `nodeDrops`/`edgeDrops`. Generated or custom +templates, Tempo and other sources keep span sampling, and a graph mixing them sums both kinds of +count. Service-graph metric rows keep Tempo's `connection_type`. A `database` or `messaging_system` row -becomes a `db`/`queue` node with no account/region attribution, unless trace spans already show the -same client service using a database (or queue); those deferred rows are counted in -`deferredMetricCalls`. `virtual_node` rows draw the uninstrumented side as an `external` node -(Tempo names an uninstrumented caller `user`). Metric volume is kept as node `metricCount` and -ranks with `spanCount` under the node cap. +becomes a label-only `db`/`queue` node (`provenance: "service_graph_metric"`, no account/region +attribution), unless a trace span from a compatibly scoped client of the same name already names +that dependency (db name, system or host; queue destination), in which case the duplicate row is +not drawn. `virtual_node` rows draw the uninstrumented side as an `external` node (Tempo names an +uninstrumented caller `user`). Metric volume is kept as node `metricCount`; metric-derived nodes +rank by volume with service nodes under the node cap. ### Trace edge instance observations diff --git a/docs/runbooks/graph-read-contract.md b/docs/runbooks/graph-read-contract.md index 2adb98ae0..716d500ea 100644 --- a/docs/runbooks/graph-read-contract.md +++ b/docs/runbooks/graph-read-contract.md @@ -225,11 +225,15 @@ The shared query normalizer carries collection status into Explore. Marked parti The ClickHouse adapter for the catalog's exporter-schema template reads node groups, parent→child pairs and (when present) span links as whole-window aggregates; see [trace graph sources](../api-reference.md#trace-graph-sources). The aggregates form one source -read: a failed or malformed query result fails it, and every aggregate needs its own empty proof, -so nodes never publish without their edges and an unproven pair query keeps the read partial. -Row limits disclose `cap_reached` like other bounded sources. The join sides project only the -sorted identity attributes. Metric-derived `external` nodes and deferred metric rows -(`deferredMetricCalls`) are additive trace metadata. +read. A failed query or malformed response envelope fails the whole read, so a failed pair query +never publishes nodes without their edges. Malformed individual rows are skipped as partial +evidence, and each aggregate needs its own empty proof: confirmed nodes with an unproven-empty pair +query publish as `partial` with `empty_not_confirmed`. Row limits disclose `cap_reached` like other +bounded sources. The join sides project only the sorted identity attributes, and the connector's +`max_execution_time` bounds each query; a query that keeps exceeding it retains the previous graph. +Metric-derived `external` and label-only `db`/`queue` nodes are additive trace node kinds. +The adapter and builder contracts run without PostgreSQL: +`cd web && npx vitest run lib/trace-source.test.ts lib/trace-graph.test.ts lib/graph-store-trace.test.ts lib/graph-sources.test.ts`. ## Rebuild capacity @@ -420,8 +424,7 @@ docker exec "$graph_test_container" psql -U postgres -d awsops \ -c "COMMENT ON DATABASE awsops IS 'awsops-disposable-graph-test'" cd web npx vitest run lib/trace-source.test.ts lib/graph-read-postgres.test.ts lib/graph-inventory-read-postgres.test.ts \ - lib/graph-store-postgres.test.ts app/api/graph/route.test.ts lib/graph-state.test.ts \ - lib/trace-graph.test.ts lib/graph-store-trace.test.ts lib/graph-sources.test.ts + lib/graph-store-postgres.test.ts app/api/graph/route.test.ts lib/graph-state.test.ts docker rm -f "$graph_test_container" ``` diff --git a/web/lib/graph-store-trace.test.ts b/web/lib/graph-store-trace.test.ts index b272a142d..92645dd2b 100644 --- a/web/lib/graph-store-trace.test.ts +++ b/web/lib/graph-store-trace.test.ts @@ -108,6 +108,20 @@ describe('source-aggregated trace edges', () => { }); }); +describe('trace node cap ranking', () => { + it('ranks label-only metric db nodes by volume with services, so they cannot evict services', async () => { + const { pool, params } = mockPool([]); + const calls = Array.from({ length: 260 }, (_, i) => ({ + client: 'tenant-api', server: `tenant_${i}`, count: 1, connectionType: 'database' })); + const metrics = { available: async () => true, calls: async (_w: number, endMs = Date.now()) => ({ + sourceId: 'prometheus:1', status: 'ok' as const, reasons: [], items: calls, + windowStartMs: endMs - 3_600_000, windowEndMs: endMs }) }; + await rebuildTraceGraph(pool as never, [new FakeTraceSource([])], undefined, [metrics]); + const services = params.filter(p => p[1] === 'service').map(p => JSON.parse(String(p[3])).service); + expect(services).toContain('tenant-api'); + }); +}); + describe('instance observation publication', () => { const account = '111122223333', region = 'ap-northeast-2', instanceId = 'i-0123456789abcdef0'; const infra = [ diff --git a/web/lib/graph-store.ts b/web/lib/graph-store.ts index 004ecea87..bdbcfd673 100644 --- a/web/lib/graph-store.ts +++ b/web/lib/graph-store.ts @@ -355,10 +355,12 @@ export async function rebuildTraceGraph( } const graph = buildTraceGraph(spans, calls, infraNodes, currentAccountId()); // Preserve structurally important DB/queue/workload nodes before ranking service volume. - // Metric-only external peers rank with services by volume, so many virtual nodes cannot evict them. - const rank = (kind: string) => kind === 'service' || kind === 'external' ? 0 : 1; + // Metric-derived nodes (external peers, label-only db/queue) rank with services by volume, so a + // high-cardinality service-graph metric cannot fill the cap ahead of service nodes. + const rank = (node: { kind: string; meta: Record }) => node.kind === 'service' + || node.kind === 'external' || node.meta.provenance === 'service_graph_metric' ? 0 : 1; const volume = (meta: Record) => Number(meta.spanCount ?? 0) + Number(meta.metricCount ?? 0); - const nodes = graph.nodes.sort((a, b) => rank(b.kind) - rank(a.kind) + const nodes = graph.nodes.sort((a, b) => rank(b) - rank(a) || volume(b.meta) - volume(a.meta)).slice(0, TRACE_NODE_CAP); const kept = new Set(nodes.map((node) => node.id)); const edges = graph.edges.filter((edge) => kept.has(edge.source) && kept.has(edge.target)) @@ -375,7 +377,6 @@ export async function rebuildTraceGraph( sources: sourceDetails, retainedPrevious: false, windowStartMs: startMs, windowEndMs: endMs, nodeDrops, edgeDrops, orphanSpans: graph.orphanSpans, invalidSpans: graph.invalidSpans, unresolvedMessaging: graph.unresolvedMessaging, - ...(graph.deferredMetricCalls ? { deferredMetricCalls: graph.deferredMetricCalls } : {}), infraUnavailable, }, }); diff --git a/web/lib/trace-graph.test.ts b/web/lib/trace-graph.test.ts index 7e506b29f..0009187b4 100644 --- a/web/lib/trace-graph.test.ts +++ b/web/lib/trace-graph.test.ts @@ -301,5 +301,30 @@ describe('source-aggregated rows and service-graph connection types', () => { expect(withSpans.nodes.filter(node => node.kind === 'db')).toHaveLength(1); expect(withSpans.nodes.find(node => node.label === 'orders-topic')?.kind).toBe('queue'); expect(withSpans.deferredMetricCalls).toBe(1); + + // Deferral is per dependency and per compatible scope: review's metrics-only cache and a + // staging review's reviews DB are still drawn beside the production span evidence. + const mixed = buildTraceGraph([dbSpan({ service: 'review', dbName: 'reviews', environment: 'prod' })], [ + { client: 'review', server: 'reviews', count: 3, connectionType: 'database' }, + { client: 'review', server: 'cache', count: 2, connectionType: 'database' }, + { client: 'review', server: 'reviews', count: 1, connectionType: 'database', + clientIdentity: { environment: 'staging' } }, + ], inventory()); + expect(mixed.deferredMetricCalls).toBe(1); + expect(mixed.nodes.filter(node => node.meta.provenance === 'service_graph_metric').map(node => node.label).sort()) + .toEqual(['cache', 'reviews']); + }); + + it('keeps the cloud.resource_id veto for the aggregate unbindable marker and caps merged pods', () => { + const vetoed = buildTraceGraph([dbSpan({ cloudResourceId: 'unbindable', weight: 3 })], [], inventory()); + const query = vetoed.edges.find(edge => edge.rel === 'queries')!; + expect(query.meta.instance_observations).toBeUndefined(); + expect(query.meta.instance_observations_unbound).toBe(3); + const pods = Array.from({ length: 30 }, (_, i) => `pod-${i}`); + const graph = buildTraceGraph([ + dbSpan({ spanId: 'a', k8sNamespace: 'shop', k8sDeployment: 'checkout', k8sPods: pods }), + dbSpan({ spanId: 'b', k8sNamespace: 'shop', k8sDeployment: 'checkout', k8sPods: pods.map(p => `${p}-x`) }), + ], [], inventory()); + expect(graph.nodes.find(node => node.kind === 'workload')?.meta.pods).toHaveLength(50); }); }); diff --git a/web/lib/trace-graph.ts b/web/lib/trace-graph.ts index 96a32d106..ed25f48ae 100644 --- a/web/lib/trace-graph.ts +++ b/web/lib/trace-graph.ts @@ -32,6 +32,7 @@ export interface TraceEdge { } export interface InfraNodeLike { id: string; kind?: string; meta?: Record | null } const INSTANCE_OBSERVATION_CAP = 200; +const WORKLOAD_POD_CAP = 50; const INSTANCE_ID = /^i-(?:[0-9a-f]{8}|[0-9a-f]{17})$/; const EC2_ARN = /^arn:(aws(?:-[a-z0-9-]+)?):ec2:([a-z0-9-]+):(\d{12}):instance\/(i-(?:[0-9a-f]{8}|[0-9a-f]{17}))$/; @@ -267,7 +268,10 @@ export function buildTraceGraph( sourceId: span.sourceId ?? null, pods: [] }, }); const pods = nodes.get(id)!.meta.pods as string[]; - for (const pod of span.k8sPods ?? (span.k8sPod ? [span.k8sPod] : [])) if (!pods.includes(pod)) pods.push(pod); + for (const pod of span.k8sPods ?? (span.k8sPod ? [span.k8sPod] : [])) { + if (pods.length >= WORKLOAD_POD_CAP) break; + if (!pods.includes(pod)) pods.push(pod); + } edge(sid, id, 'runs_on', w); } if (span.messagingSystem && span.messagingDestination) { @@ -302,15 +306,25 @@ export function buildTraceGraph( } } // Trace spans name databases and queues with host-qualified identity. Metric rows only carry - // Tempo's server label (a db.name such as "admin"), so a metric database/queue row is deferred - // when trace spans already show that same client service querying a database (or using a - // queue); otherwise the same dependency would appear twice under two names. Other services keep - // their metric rows, and every deferral is counted. - const spanDbClients = new Set(), spanQueueClients = new Set(); + // Tempo's server label (the span's db.name, db.system or peer host for a database, or the + // destination for a queue), so a metric row is a duplicate only when a span from a compatibly + // scoped client of the same name names that same dependency. Only then is the row not drawn; + // any other metric row, including another dependency of the same client, is kept. + const scopeKeys = ['accountId', 'region', 'environment', 'serviceNamespace', 'k8sNamespace', 'k8sCluster'] as const; + const compatible = (span: TraceSpan, identity: TraceIdentity) => + scopeKeys.every(key => span[key] === undefined || identity[key] === undefined || span[key] === identity[key]); + const names = (span: TraceSpan, database: boolean): string[] => database + ? [span.dbName, span.dbSystem, span.dbHost, span.dbHost?.split('.')[0]].filter((v): v is string => !!v) + : span.messagingDestination ? [span.messagingDestination.trim()] : []; + const spansByService = new Map(); for (const span of byId.values()) { - if (span.dbSystem) spanDbClients.add(span.service); - if (span.messagingSystem && span.messagingDestination) spanQueueClients.add(span.service); + if (!span.dbSystem && !(span.messagingSystem && span.messagingDestination)) continue; + const list = spansByService.get(span.service); + if (list) list.push(span); else spansByService.set(span.service, [span]); } + const shownBySpans = (call: ServiceGraphCall, database: boolean) => + (spansByService.get(call.client) ?? []).some(span => (database ? !!span.dbSystem : !!span.messagingSystem) + && compatible(span, call.clientIdentity ?? {}) && names(span, database).includes(call.server)); let deferredMetricCalls = 0; const metricWeight = (id: string, count: number) => { const meta = nodes.get(id)!.meta; @@ -327,14 +341,14 @@ export function buildTraceGraph( const clientIdentity = call.clientIdentity ?? {}, serverIdentity = call.serverIdentity ?? {}; if (call.connectionType === 'database' || call.connectionType === 'messaging_system') { const database = call.connectionType === 'database'; - if ((database ? spanDbClients : spanQueueClients).has(call.client)) { deferredMetricCalls++; continue; } + if (shownBySpans(call, database)) { deferredMetricCalls++; continue; } // Caller labels scope the node ID (no cross-scope merge) but are not attribution: like // queue claims, the database carries no account/region and never bridges to inventory. const scoped = resourceScope({ ...clientIdentity, ...serverIdentity }); const id = nodeId(database ? 'db' : 'queue', scoped, `metric:${call.server}`); if (!nodes.has(id)) nodes.set(id, database ? { id, kind: 'db', label: call.server.slice(0, 240), - meta: { system: null, host: null, dbName: null, metricServer: call.server, + meta: { system: null, host: null, dbName: null, metricServer: call.server.slice(0, 240), sourceId: scoped.sourceId ?? null, environment: scoped.environment ?? null, accountId: null, region: null, identityProvenance: 'telemetry_claim', provenance: 'service_graph_metric' }, @@ -357,7 +371,7 @@ export function buildTraceGraph( const external = (name: string, identity: TraceIdentity) => { const id = nodeId('ext', resourceScope(identity), name); if (!nodes.has(id)) nodes.set(id, { id, kind: 'external', label: name.slice(0, 240), - meta: { name, sourceId: identity.sourceId ?? null, environment: identity.environment ?? null, + meta: { name: name.slice(0, 240), sourceId: identity.sourceId ?? null, environment: identity.environment ?? null, provenance: 'service_graph_metric' } }); return id; }; diff --git a/web/lib/trace-source.test.ts b/web/lib/trace-source.test.ts index 859bf9230..eb7760bb7 100644 --- a/web/lib/trace-source.test.ts +++ b/web/lib/trace-source.test.ts @@ -156,6 +156,8 @@ describe('ClickHouseOtelAggregateSource (whole-window aggregates)', () => { } const groups = sqls.find(q => !q.includes('JOIN'))!; expect(groups).toContain("startsWith(v, 'arn:')"); + expect(groups).toContain("k NOT IN ('host.id', 'service.instance.id')"); + expect(groups).toContain("'unbindable'"); expect(groups).toContain('intDiv(sum(Duration), 1000000) AS DurationMsSum'); expect(sqls.find(q => q.includes('ARRAY JOIN'))).toContain('length(`Links.TraceId`) = length(`Links.SpanId`)'); }); @@ -199,6 +201,14 @@ describe('ClickHouseOtelAggregateSource (whole-window aggregates)', () => { expect(out.calls).toHaveLength(1); }); + it('clamps rows to the limit and discloses out-of-window aggregate rows', async () => { + answer([group({ LastTimestamp: END + 60_000, FirstTimestamp: END + 1 }), group({ SampleSpanId: 's2' }), + group({ SampleSpanId: 's3' })], [pair()]); + const out = await new ClickHouseOtelAggregateSource().recentSpans(60, 2, END); + expect(out.items).toHaveLength(1); + expect(out.reasons).toEqual(expect.arrayContaining(['incomplete_collection', 'cap_reached'])); + }); + it('requires proof before treating an empty window as empty', async () => { invokeMcpLambdaTool.mockResolvedValue({ rows: [] }); expect((await new ClickHouseOtelAggregateSource().recentSpans(60, 1000, END)).reasons).toContain('empty_not_confirmed'); diff --git a/web/lib/trace-source.ts b/web/lib/trace-source.ts index 36a9d4d94..b489b034a 100644 --- a/web/lib/trace-source.ts +++ b/web/lib/trace-source.ts @@ -420,10 +420,12 @@ export function otelAggregateShape(sqlTemplate?: string): OtelAggregateShape | u } const list = (keys: readonly string[]) => `[${keys.map((k) => `'${k}'`).join(', ')}]`; -// Only the attributes spanMetadata reads are grouped. Pod names and per-process instance IDs would -// multiply rows, so pods are collected as a bounded array. service.instance.id keeps every value -// claimedInstance can act on: bare EC2 IDs bind, and ARNs bind or veto. Other values (process -// UUIDs) are ignored by that check, so dropping them changes no binding. +// Only the attributes spanMetadata reads are grouped. Pod names and per-host/per-process claims +// would multiply rows, so pods are collected as a bounded array and the three EC2 claim keys are +// reduced to what claimedInstance (trace-graph.ts) can act on: bare EC2 IDs bind and ARNs bind or +// veto, both kept verbatim. Other host.id/service.instance.id values are ignored by that check and +// dropped. Any other cloud.resource_id value vetoes binding, so it becomes one fixed 'unbindable' +// value: the veto survives while host UUIDs stop splitting groups. const AGG_RESOURCE_KEYS = [ 'service.name', 'cloud.account.id', 'cloud.region', 'deployment.environment.name', 'deployment.environment', 'service.namespace', 'service.version', 'cloud.resource_id', 'host.id', @@ -439,6 +441,7 @@ const AGG_IDENTITY_KEYS = [ 'deployment.environment', 'service.namespace', 'k8s.namespace.name', 'k8s.cluster.name', ] as const; const AGG_POD_CAP = 20; +const EC2_CLAIM = "match(v, '^i-[0-9a-f]{8}([0-9a-f]{9})?$') OR startsWith(v, 'arn:')"; function aggregateSql(shape: OtelAggregateShape, windowMins: number, endMs: number, limit: number) { const window = `Timestamp >= fromUnixTimestamp64Milli(${endMs}) - INTERVAL ${windowMins} MINUTE ` + @@ -452,8 +455,9 @@ function aggregateSql(shape: OtelAggregateShape, windowMins: number, endMs: numb `any(SpanId) AS SampleSpanId, groupUniqArray(${AGG_POD_CAP})(Pod) AS Pods FROM (` + `SELECT ServiceName, ${kind}, ${status}, Duration, Timestamp, TraceId, SpanId, ` + "ResourceAttributes['k8s.pod.name'] AS Pod, " + - `mapSort(mapFilter((k, v) -> has(${list(AGG_RESOURCE_KEYS)}, k) AND (k != 'service.instance.id' ` + - "OR match(v, '^i-[0-9a-f]{8}([0-9a-f]{9})?$') OR startsWith(v, 'arn:')), ResourceAttributes)) AS RA, " + + `mapSort(mapApply((k, v) -> (k, if(k = 'cloud.resource_id' AND NOT (${EC2_CLAIM}), 'unbindable', v)), ` + + `mapFilter((k, v) -> has(${list(AGG_RESOURCE_KEYS)}, k) AND (k NOT IN ('host.id', 'service.instance.id') ` + + `OR ${EC2_CLAIM}), ResourceAttributes))) AS RA, ` + "if(SpanAttributes['db.system'] != '' OR SpanAttributes['messaging.system'] != '', " + `mapSort(mapFilter((k, v) -> has(${list(AGG_SPAN_KEYS)}, k), SpanAttributes)), ` + 'mapFilter((k, v) -> false, SpanAttributes)) AS SA ' + @@ -529,7 +533,8 @@ export class ClickHouseOtelAggregateSource implements TraceSource { reasons.push(...result.reasons); if (result.rows.length >= limit) reasons.push('cap_reached'); } - const [groupRows, callRows, linkRows = []] = results.map((r) => r.rows!); + // Defensive clamp, as in the span reader: a connector ignoring max_rows stays bounded. + const [groupRows, callRows, linkRows = []] = results.map((r) => r.rows!.slice(0, limit)); const items: TraceSpan[] = []; for (const value of groupRows) { const row = object(value); @@ -552,7 +557,7 @@ export class ClickHouseOtelAggregateSource implements TraceSource { if (this.shape.statusCode && text(row.StatusCode) && !item.status) reasons.push('malformed_rows'); const pods = Array.isArray(row.Pods) ? [...new Set(row.Pods.filter((p): p is string => !!text(p)))].slice(0, AGG_POD_CAP) : []; - if (!inWindow(item, window)) continue; + if (!inWindow(item, window)) { reasons.push('incomplete_collection'); continue; } items.push({ ...item, sourceId, weight, durationMs: duration, firstStartMs: first, ...(pods.length ? { k8sPods: pods } : {}) }); } From 891df138ff7c2aef8c03ff612ed0d7a1d2fa1760 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?=EC=98=A4=EC=A4=80=EC=84=9D=28Junseok=20Oh=29DevOps?= <52226147+Atom-oh@users.noreply.github.com> Date: Tue, 10 Nov 2026 10:52:22 +0000 Subject: [PATCH 4/4] fix(topology): keep metric sources to service calls, select capped nodes by edge, fall back to sampling - Service-graph metrics contribute service->service calls only, as graph-store's metrics-source contract already states. Rows Tempo marks database, messaging_system or virtual_node are not drawn (they are not services, and their label-only nodes would carry no provenance through the agent SQL reader view); databases and queues come from trace spans with host-qualified identity. This removes the metric db/queue/external node kinds and the span-based deferral. - Trace node-cap selection takes the heaviest edges' endpoints first, so a kept db/queue/workload always keeps its caller and many low-volume dependencies cannot evict their services. - A failed aggregate query falls back to the bounded newest-span reader for that rebuild (partial, incomplete_collection) instead of retaining a frozen graph; a malformed envelope still fails the read. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 4 +- docs/api-reference.md | 40 ++++++++------- docs/runbooks/graph-read-contract.md | 15 +++--- web/app/topology/services/page.tsx | 1 - web/lib/graph-sources.ts | 2 +- web/lib/graph-store-trace.test.ts | 26 ++++++---- web/lib/graph-store.ts | 27 ++++++---- web/lib/trace-graph.test.ts | 52 ++++--------------- web/lib/trace-graph.ts | 75 ++++------------------------ web/lib/trace-source.test.ts | 27 ++++++++-- web/lib/trace-source.ts | 21 ++++++-- 11 files changed, 125 insertions(+), 165 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index ca20bdb31..122fb9e21 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -29,7 +29,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ### Changed -- Service map from complete trace counts: ClickHouse trace sources using the standard OTel exporter table now count service calls, database access, workloads and messaging across the whole one-hour window with bounded read-only aggregate queries (two, plus a third for span links when the table has them), instead of joining the newest 1,000 spans, so low-volume calls and databases no longer drop out and calls whose parent fell outside the sample no longer show as orphans. Custom or generated ClickHouse queries and Tempo keep span sampling. Prometheus/Mimir service-graph rows keep `connection_type`, so database and queue servers no longer appear as same-named services (trace spans name them when the same caller has them) and uninstrumented callers or peers appear as external nodes. Rebuild the datasource graph catalog (`datasource_index`) to pick up the new metric query. +- Service map from complete trace counts: ClickHouse trace sources using the standard OTel exporter table now count service calls, database access, workloads and messaging across the whole one-hour window with bounded read-only aggregate queries (two, plus a third for span links when the table has them), instead of joining the newest 1,000 spans, so low-volume calls and databases no longer drop out and calls whose parent fell outside the sample no longer show as orphans. If an aggregate query fails, that rebuild falls back to the newest-span sample and is marked partial. Custom or generated ClickHouse queries and Tempo keep span sampling. The node cap keeps each kept database, queue or workload together with its caller. Prometheus/Mimir service-graph metrics add service-to-service calls only: rows Tempo marks as database, queue or uninstrumented-peer connections no longer appear as same-named services. Rebuild the datasource graph catalog (`datasource_index`) to pick up the new metric query. - Web framework and runtime refresh: Next.js 16.4.5, React 19.3.1 and Node 26.13.0 retain authentication, ownership checks and standalone deployment; the request body guard uses the Next.js proxy convention. Refresh JavaScript/Python dependencies and AgentCore SDKs with compatibility pins, Terraform 1.17.1/provider locks, and Python 3.14 Lambda/container runtimes. The documentation guide builds and deploys on Node 26 with refreshed documentation security dependencies and Mermaid 12, preserving all four locales. Preserve gateway authentication, feature gates and release safeguards. - EKS member onboarding: the member cluster guide is now two owner commands — an Access Entry for the registered member read role and a cluster-scoped `AmazonEKSViewPolicy` association — with no Kubernetes group, kubeconfig, kubectl or RBAC step; View never reads Kubernetes Secrets, and the guide tells owners to remove any existing `AmazonEKSAdminViewPolicy`. Because View cannot list nodes, member node lists come from read-only EC2/node-group APIs through the same role and are labeled as an EC2/node-group view (no kubelet status, instance-type capacity as allocatable, no Fargate nodes); their Ready count shows as "EC2 running N · Ready unknown" and they are left out of reserved-% figures; a failed node read no longer marks the whole cluster unreachable. When an administrator opens EKS, member clusters whose Access Entry is verified are listed first in an "Access Entry verified" callout with a one-click [조회 등록] button; nothing is registered until the administrator clicks it (an app registration record only; AWSops creates no AWS or Kubernetes permissions), and an unregistered cluster stays unregistered. In diagnosis, the Kubernetes map and the node drilldown, EC2/node-group nodes read as "EC2 running · Ready unknown" with Pressure, Allocatable and Reserved shown as unknown instead of as unhealthy or fully allocatable. Entries created from the earlier guide keep working. - Bedrock models: chat answers with OpenAI GPT-6 Luna (without explicit Bedrock cache points, which GPT models reject); Explore "AI generate" writes queries with GPT-6.1 Terra and retries on Claude Sonnet 5.5 when Terra fails or its draft fails validation; primary, corrections and fallback share a 55-second request deadline with SDK cancellation; AI Diagnosis, routing, synthesis and the background workers move to Claude Sonnet 5.5, Opus 5.5 (AI Diagnosis Deep tier option) and Haiku 5.5. These models reject `temperature`, so requests no longer send it. Haiku calls and Sonnet calls with fixed output caps run without extended thinking so existing budgets still fit; Opus 5.5, which always thinks, gets extra output headroom for Deep report sections and invariant proposals. The web task now always holds a grant for its direct query-generation and assistant models (Terra, Sonnet 5.5, Haiku 5.5), so "AI generate" and the product-help assistant no longer fail with AccessDenied when hybrid routing is off; the chat runtime role additionally allows GPT-6 Luna. With `model_provider = mixed`, the Terraform variable `agentcore_chat_model_id = "global.anthropic.claude-sonnet-5-5"` rolls chat back to Sonnet 5.5 on every provisioning run, and the chat footer shows a model only when the answer carries its model provenance. @@ -710,7 +710,7 @@ First release of the **v2 line** (versioned independently from the v1 1.x line, ### Changed -- 전체 트레이스 집계 기반 서비스 맵: 표준 OTel exporter 테이블을 쓰는 ClickHouse 트레이스 원본은 최신 스팬 1,000개를 이어 붙이는 대신, 제한된 읽기 전용 집계 쿼리(2개, 테이블에 span link가 있으면 1개 추가)로 1시간 구간 전체의 서비스 호출·DB 접근·워크로드·메시징을 셉니다. 호출량이 적은 연결이나 DB가 맵에서 빠지지 않고, 부모 스팬이 표본 밖에 있던 호출도 끊긴 스팬으로 표시되지 않습니다. 사용자 정의·생성된 ClickHouse 쿼리와 Tempo는 기존 스팬 표본 방식을 유지합니다. Prometheus/Mimir 서비스 그래프 지표는 `connection_type`을 보존해 DB·큐 서버가 같은 이름의 서비스로 그려지지 않으며(같은 호출 서비스의 트레이스 스팬이 있으면 스팬 기준으로 표시), 계측되지 않은 호출자나 상대는 외부 노드로 표시됩니다. 새 지표 쿼리를 쓰려면 데이터소스 그래프 카탈로그(`datasource_index`)를 다시 만들어야 합니다. +- 전체 트레이스 집계 기반 서비스 맵: 표준 OTel exporter 테이블을 쓰는 ClickHouse 트레이스 원본은 최신 스팬 1,000개를 이어 붙이는 대신, 제한된 읽기 전용 집계 쿼리(2개, 테이블에 span link가 있으면 1개 추가)로 1시간 구간 전체의 서비스 호출·DB 접근·워크로드·메시징을 셉니다. 호출량이 적은 연결이나 DB가 맵에서 빠지지 않고, 부모 스팬이 표본 밖에 있던 호출도 끊긴 스팬으로 표시되지 않습니다. 집계 쿼리가 실패하면 그 수집은 최신 스팬 표본으로 대신 읽고 부분 수집으로 표시합니다. 사용자 정의·생성된 ClickHouse 쿼리와 Tempo는 기존 스팬 표본 방식을 유지합니다. 노드 상한을 적용할 때도 남기는 DB·큐·워크로드는 호출하는 서비스와 함께 남깁니다. Prometheus/Mimir 서비스 그래프 지표는 서비스 간 호출만 더하며, Tempo가 DB·큐·계측되지 않은 상대와의 연결로 표시한 행은 더 이상 같은 이름의 서비스로 그려지지 않습니다. 새 지표 쿼리를 쓰려면 데이터소스 그래프 카탈로그(`datasource_index`)를 다시 만들어야 합니다. - 웹 프레임워크·런타임 최신화: Next.js 16.4.5, React 19.3.1, Node 26.13.0으로 인증·소유권 검사·standalone 배포를 유지하고 요청 본문 제한을 Next.js proxy 규약으로 옮깁니다. 호환성 버전 고정과 함께 JavaScript/Python 의존성·AgentCore SDK, Terraform 1.17.1·provider 잠금파일, Python 3.14 Lambda·컨테이너 런타임을 갱신합니다. 문서 가이드는 Node 26에서 빌드·배포하며 문서 사이트의 보안 의존성과 Mermaid 12를 갱신하고 4개 언어를 유지합니다. Gateway 인증·기능 게이트·릴리스 보호 장치는 유지합니다. - EKS 멤버 온보딩: 멤버 클러스터 가이드가 소유자 명령 두 개(등록된 멤버 읽기 역할의 Access Entry 생성, 클러스터 범위 `AmazonEKSViewPolicy` 연결)로 줄었으며 Kubernetes 그룹·kubeconfig·kubectl·RBAC 단계가 없습니다. View는 Kubernetes Secrets를 읽지 않으며, 가이드는 기존 `AmazonEKSAdminViewPolicy` 연결을 제거하도록 안내합니다. View로는 노드를 조회할 수 없으므로 멤버 노드 목록은 같은 역할로 읽기 전용 EC2/노드그룹 API에서 가져와 EC2/노드그룹 기준으로 표시합니다(kubelet 상태 없음, allocatable은 인스턴스 타입 용량, Fargate 노드 제외). 이 노드의 Ready 수는 "EC2 running N · Ready unknown"으로 표시되며 예약 비율 계산에서 제외됩니다. 노드 조회만 실패해도 클러스터 전체가 조회 불가로 표시되지 않습니다. 관리자가 EKS 화면을 열면 Access Entry가 확인된 멤버 클러스터가 "Access Entry 확인됨" 안내 영역에 먼저 표시되고 [조회 등록] 버튼 한 번으로 등록할 수 있으며, 관리자가 누르기 전에는 등록되지 않습니다(앱 등록 기록만 저장, AWSops는 AWS·Kubernetes 권한을 만들지 않음). 해제한 클러스터는 해제 상태로 유지됩니다. 진단 표·Kubernetes 맵·노드 상세에서도 EC2/노드그룹 노드는 "EC2 running · Ready unknown"으로 표시되며 Pressure·Allocatable·Reserved는 비정상이나 전량 할당 대신 알 수 없음으로 표시됩니다. 이전 가이드로 만든 항목도 그대로 동작합니다. - Bedrock 모델: 채팅은 OpenAI GPT-6 Luna로 답합니다(GPT 모델이 거부하는 명시적 Bedrock 캐시 포인트는 끔). Explore "AI로 생성"은 GPT-6.1 Terra로 쿼리를 만들고, Terra 호출이 실패하거나 초안이 검증에서 거부되면 Claude Sonnet 5.5로 다시 시도합니다. 기본 모델·수정·폴백은 하나의 55초 요청 제한 시간과 SDK 취소 신호를 공유합니다. AI 진단, 라우팅, 합성, 백그라운드 워커는 Claude Sonnet 5.5, Opus 5.5(AI 진단 Deep 단계 선택), Haiku 5.5로 바뀝니다. 이 모델들은 `temperature`를 거부하므로 요청에서 더 이상 보내지 않습니다. Haiku 호출과 출력 상한이 고정된 Sonnet 호출은 기존 상한이 유지되도록 확장 사고 없이 실행하고, 항상 사고하는 Opus 5.5는 Deep 리포트 섹션과 불변식 제안에 출력 여유분을 더 받습니다. 웹 태스크는 직접 호출하는 쿼리 생성·도움말 모델(Terra, Sonnet 5.5, Haiku 5.5) 권한을 항상 가지므로, 하이브리드 라우팅이 꺼져 있어도 "AI로 생성"과 제품 도움말이 AccessDenied로 실패하지 않습니다. 채팅 런타임 역할은 GPT-6 Luna도 허용합니다. `model_provider = mixed`일 때 Terraform 변수 `agentcore_chat_model_id = "global.anthropic.claude-sonnet-5-5"`로 채팅을 Sonnet 5.5로 되돌리며 매 프로비저닝마다 유지되고, 채팅 하단의 모델 표시는 답변에 모델 출처가 있을 때만 나타납니다. diff --git a/docs/api-reference.md b/docs/api-reference.md index 0845886fb..dc2b606af 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -551,25 +551,27 @@ A ClickHouse datasource whose registry query is the catalog's exporter-schema `o unregistered default, is read as whole-window aggregates of the same table, not as the newest spans. Two bounded read-only queries return service/kind/status attribute groups with span counts and parent→child service pairs; a third returns span-link pairs when the table has `Links` -columns. Each is limited to 1,000 rows. A failed query or malformed response envelope fails the -whole source read; malformed individual rows are skipped as `malformed_rows`, a saturated limit -adds `cap_reached`, rows outside the window add `incomplete_collection`, and an unproven empty -result adds `empty_not_confirmed` (a partial read that may publish nodes without edges). Without -`cap_reached`, node and edge `spanCount` count the window's spans, and source-counted pairs produce -no `orphanSpans`. A child whose parent started before the window is not counted as a pair and is -not disclosed. Grouping keeps only EC2 instance IDs and ARNs from the instance claim attributes, -plus a fixed marker for any other `cloud.resource_id` (which still vetoes binding). Publication -still applies the 200-node/500-edge caps reported as `nodeDrops`/`edgeDrops`. Generated or custom -templates, Tempo and other sources keep span sampling, and a graph mixing them sums both kinds of -count. - -Service-graph metric rows keep Tempo's `connection_type`. A `database` or `messaging_system` row -becomes a label-only `db`/`queue` node (`provenance: "service_graph_metric"`, no account/region -attribution), unless a trace span from a compatibly scoped client of the same name already names -that dependency (db name, system or host; queue destination), in which case the duplicate row is -not drawn. `virtual_node` rows draw the uninstrumented side as an `external` node (Tempo names an -uninstrumented caller `user`). Metric volume is kept as node `metricCount`; metric-derived nodes -rank by volume with service nodes under the node cap. +columns. Each is limited to 1,000 rows and to the connector's execution budget. If an aggregate +query fails (for example the join exceeds that budget), the same table is read with the bounded +newest-span reader instead and the source is `partial` with `incomplete_collection`. A malformed +response envelope fails the read; malformed individual rows are skipped as `malformed_rows`, a +saturated limit adds `cap_reached`, rows outside the window add `incomplete_collection`, and an +unproven empty result adds `empty_not_confirmed` (a partial read that may publish nodes without +edges). Without those reasons, node and edge `spanCount` count the window's spans, and +source-counted pairs produce no `orphanSpans`. A child whose parent started before the window is +not counted as a pair and is not disclosed. Grouping keeps only EC2 instance IDs and ARNs from the +instance claim attributes, plus a fixed marker for any other `cloud.resource_id` (which still +vetoes binding). Generated or custom templates, Tempo and other sources keep span sampling, and a +graph mixing them sums both kinds of count. + +Publication keeps at most 200 nodes and 500 edges (`nodeDrops`/`edgeDrops`). Nodes are selected +from the heaviest edges first, both endpoints together, so a kept database, queue or workload +always keeps its caller; remaining capacity goes to other nodes by span and metric volume. + +Service-graph metrics contribute service→service `calls` only. Rows whose Tempo +`connection_type` is `database`, `messaging_system` or `virtual_node` name a database, queue or +uninstrumented peer rather than a service and are not drawn; databases and queues come from trace +spans with their host-qualified identity. ### Trace edge instance observations diff --git a/docs/runbooks/graph-read-contract.md b/docs/runbooks/graph-read-contract.md index 716d500ea..8fce9d43a 100644 --- a/docs/runbooks/graph-read-contract.md +++ b/docs/runbooks/graph-read-contract.md @@ -225,13 +225,14 @@ The shared query normalizer carries collection status into Explore. Marked parti The ClickHouse adapter for the catalog's exporter-schema template reads node groups, parent→child pairs and (when present) span links as whole-window aggregates; see [trace graph sources](../api-reference.md#trace-graph-sources). The aggregates form one source -read. A failed query or malformed response envelope fails the whole read, so a failed pair query -never publishes nodes without their edges. Malformed individual rows are skipped as partial -evidence, and each aggregate needs its own empty proof: confirmed nodes with an unproven-empty pair -query publish as `partial` with `empty_not_confirmed`. Row limits disclose `cap_reached` like other -bounded sources. The join sides project only the sorted identity attributes, and the connector's -`max_execution_time` bounds each query; a query that keeps exceeding it retains the previous graph. -Metric-derived `external` and label-only `db`/`queue` nodes are additive trace node kinds. +read. A failed aggregate query (including one that exceeds the connector's `max_execution_time`) +falls back to the bounded newest-span reader for that rebuild and is disclosed as +`incomplete_collection`, so a busy table degrades to sampling instead of retaining a frozen graph; +a malformed response envelope fails the read. Nodes therefore never publish from a partial set of +aggregates. Malformed individual rows are skipped as partial evidence, and each aggregate needs its +own empty proof: confirmed nodes with an unproven-empty pair query publish as `partial` with +`empty_not_confirmed`. Row limits disclose `cap_reached` like other bounded sources. The join sides +project only the sorted identity attributes. Node-cap selection keeps edge endpoints together. The adapter and builder contracts run without PostgreSQL: `cd web && npx vitest run lib/trace-source.test.ts lib/trace-graph.test.ts lib/graph-store-trace.test.ts lib/graph-sources.test.ts`. diff --git a/web/app/topology/services/page.tsx b/web/app/topology/services/page.tsx index 53507ea15..884f302ca 100644 --- a/web/app/topology/services/page.tsx +++ b/web/app/topology/services/page.tsx @@ -26,7 +26,6 @@ const COLORS: Record = { db: ['#FBEFE0', '#C8902F'], // amber workload: ['#F1E9FF', '#8A5BD0'], // purple queue: ['#E9F5EE', '#4D906C'], - external: ['#F4F4F1', '#7A7A6E'], // uninstrumented service-graph peer }; const RESOURCE = ['#EEF0F2', '#9AA6B2'] as const; const QUEUE_HEIGHT = 144; diff --git a/web/lib/graph-sources.ts b/web/lib/graph-sources.ts index ed1b27396..de0294155 100644 --- a/web/lib/graph-sources.ts +++ b/web/lib/graph-sources.ts @@ -64,7 +64,7 @@ export async function loadGraphSources(pool: Pool): Promise { // The catalog's exporter-schema template is read as whole-window aggregates; any other // (generated/custom) template keeps the bounded newest-span reader. const shape = otelAggregateShape(template); - sources.push(shape ? new ClickHouseOtelAggregateSource(row.integration_id, shape) + sources.push(shape ? new ClickHouseOtelAggregateSource(row.integration_id, shape, template) : new ClickHouseOtelTraceSource(row.integration_id, template)); } else if (mapper === 'tempo_v1') { sources.push(new TempoTraceSource(row.integration_id)); diff --git a/web/lib/graph-store-trace.test.ts b/web/lib/graph-store-trace.test.ts index 92645dd2b..7cbb98c01 100644 --- a/web/lib/graph-store-trace.test.ts +++ b/web/lib/graph-store-trace.test.ts @@ -108,17 +108,25 @@ describe('source-aggregated trace edges', () => { }); }); -describe('trace node cap ranking', () => { - it('ranks label-only metric db nodes by volume with services, so they cannot evict services', async () => { +describe('trace node cap selection', () => { + it('keeps the calling service when many span-derived databases exceed the node cap', async () => { const { pool, params } = mockPool([]); - const calls = Array.from({ length: 260 }, (_, i) => ({ - client: 'tenant-api', server: `tenant_${i}`, count: 1, connectionType: 'database' })); - const metrics = { available: async () => true, calls: async (_w: number, endMs = Date.now()) => ({ - sourceId: 'prometheus:1', status: 'ok' as const, reasons: [], items: calls, + const now = Date.now(); + const spans = Array.from({ length: 260 }, (_, i) => span({ + service: 'tenant-api', spanId: `s${i}`, kind: 'CLIENT', dbSystem: 'postgresql', + dbHost: `tenant-${i}.db.example.internal`, startMs: now - 1_000, weight: i + 1, + })); + const source = { available: async () => true, recentSpans: async (_w: number, _c: number, endMs = Date.now()) => ({ + sourceId: 'clickhouse:3', status: 'ok' as const, reasons: [], items: spans, windowStartMs: endMs - 3_600_000, windowEndMs: endMs }) }; - await rebuildTraceGraph(pool as never, [new FakeTraceSource([])], undefined, [metrics]); - const services = params.filter(p => p[1] === 'service').map(p => JSON.parse(String(p[3])).service); - expect(services).toContain('tenant-api'); + await rebuildTraceGraph(pool as never, [source]); + const kinds = params.filter(p => p[1] === 'service' || p[1] === 'db').map(p => p[1]); + expect(kinds.filter(kind => kind === 'service')).toHaveLength(1); + expect(kinds).toHaveLength(200); + // Every published query edge keeps its caller, and the heaviest databases are the ones kept. + const hosts = params.filter(p => p[1] === 'db').map(p => JSON.parse(String(p[3])).host as string); + expect(hosts).toContain('tenant-259.db.example.internal'); + expect(hosts).not.toContain('tenant-0.db.example.internal'); }); }); diff --git a/web/lib/graph-store.ts b/web/lib/graph-store.ts index bdbcfd673..897017062 100644 --- a/web/lib/graph-store.ts +++ b/web/lib/graph-store.ts @@ -354,18 +354,23 @@ export async function rebuildTraceGraph( infraUnavailable = true; } const graph = buildTraceGraph(spans, calls, infraNodes, currentAccountId()); - // Preserve structurally important DB/queue/workload nodes before ranking service volume. - // Metric-derived nodes (external peers, label-only db/queue) rank with services by volume, so a - // high-cardinality service-graph metric cannot fill the cap ahead of service nodes. - const rank = (node: { kind: string; meta: Record }) => node.kind === 'service' - || node.kind === 'external' || node.meta.provenance === 'service_graph_metric' ? 0 : 1; + // Node selection follows edge volume: the heaviest edges claim both endpoints first, so a kept + // dependency (db/queue/workload) always keeps its caller and many low-volume dependencies cannot + // evict the services that use them. Remaining capacity goes to the remaining nodes by volume. + const weight = (edge: { meta: { spanCount: number; metricCount: number } }) => edge.meta.spanCount + edge.meta.metricCount; const volume = (meta: Record) => Number(meta.spanCount ?? 0) + Number(meta.metricCount ?? 0); - const nodes = graph.nodes.sort((a, b) => rank(b) - rank(a) - || volume(b.meta) - volume(a.meta)).slice(0, TRACE_NODE_CAP); - const kept = new Set(nodes.map((node) => node.id)); - const edges = graph.edges.filter((edge) => kept.has(edge.source) && kept.has(edge.target)) - .sort((a, b) => (b.meta.spanCount + b.meta.metricCount) - (a.meta.spanCount + a.meta.metricCount)) - .slice(0, TRACE_EDGE_CAP); + const byWeight = [...graph.edges].sort((a, b) => weight(b) - weight(a)); + const kept = new Set(); + for (const edge of byWeight) { + const need = Number(!kept.has(edge.source)) + Number(!kept.has(edge.target)); + if (need && kept.size + need <= TRACE_NODE_CAP) { kept.add(edge.source); kept.add(edge.target); } + } + for (const node of [...graph.nodes].sort((a, b) => volume(b.meta) - volume(a.meta))) { + if (kept.size >= TRACE_NODE_CAP) break; + kept.add(node.id); + } + const nodes = graph.nodes.filter((node) => kept.has(node.id)); + const edges = byWeight.filter((edge) => kept.has(edge.source) && kept.has(edge.target)).slice(0, TRACE_EDGE_CAP); const nodeDrops = graph.nodes.length - nodes.length; const edgeDrops = graph.edges.length - edges.length; const incomplete = partial || infraUnavailable || nodeDrops > 0 || edgeDrops > 0 diff --git a/web/lib/trace-graph.test.ts b/web/lib/trace-graph.test.ts index 0009187b4..221edbd0f 100644 --- a/web/lib/trace-graph.test.ts +++ b/web/lib/trace-graph.test.ts @@ -268,51 +268,19 @@ describe('source-aggregated rows and service-graph connection types', () => { expect(graph.orphanSpans).toBe(0); }); - it('keeps metric database/messaging servers off service nodes and defers to span evidence', () => { - const calls = [ + it('draws only service calls from metrics; database, queue and virtual-node rows are not services', () => { + const graph = buildTraceGraph([], [ { client: 'review', server: 'reviews', count: 10, connectionType: 'database' }, { client: 'order', server: 'orders-topic', count: 4, connectionType: 'messaging_system' }, { client: 'user', server: 'payment', count: 6, connectionType: 'virtual_node' }, - { client: 'payment', server: 'stripe', count: 2, connectionType: 'virtual_node' }, - ]; - const metricsOnly = buildTraceGraph([], calls, []); - expect(metricsOnly.nodes.filter(node => node.kind === 'service').map(node => node.label).sort()) - .toEqual(['order', 'payment', 'review']); - expect(metricsOnly.nodes.find(node => node.label === 'reviews')).toMatchObject({ kind: 'db', - meta: { provenance: 'service_graph_metric', host: null } }); - expect(metricsOnly.nodes.find(node => node.label === 'orders-topic')?.kind).toBe('queue'); - expect(metricsOnly.nodes.filter(node => node.kind === 'external').map(node => node.label).sort()) - .toEqual(['stripe', 'user']); - const rel = (source: string, target: string) => metricsOnly.edges.find(edge => - metricsOnly.nodes.find(n => n.id === edge.source)?.label === source - && metricsOnly.nodes.find(n => n.id === edge.target)?.label === target); - expect(rel('review', 'reviews')).toMatchObject({ rel: 'queries', meta: { metricCount: 10, spanCount: 0 } }); - expect(rel('order', 'orders-topic')?.rel).toBe('publishes'); - expect(rel('user', 'payment')?.rel).toBe('calls'); - expect(rel('payment', 'stripe')?.rel).toBe('calls'); - - expect(metricsOnly.nodes.find(node => node.label === 'reviews')?.meta).toMatchObject({ - accountId: null, region: null, identityProvenance: 'telemetry_claim' }); - expect(svc(metricsOnly, 'payment')?.meta.metricCount).toBe(8); - - // Span evidence for review's database defers only review's metric DB row; order's queue row - // (no span evidence for that client) is still drawn, and the deferral is counted. - const withSpans = buildTraceGraph([dbSpan({ service: 'review', dbName: 'reviews' })], calls, inventory()); - expect(withSpans.nodes.filter(node => node.kind === 'db')).toHaveLength(1); - expect(withSpans.nodes.find(node => node.label === 'orders-topic')?.kind).toBe('queue'); - expect(withSpans.deferredMetricCalls).toBe(1); - - // Deferral is per dependency and per compatible scope: review's metrics-only cache and a - // staging review's reviews DB are still drawn beside the production span evidence. - const mixed = buildTraceGraph([dbSpan({ service: 'review', dbName: 'reviews', environment: 'prod' })], [ - { client: 'review', server: 'reviews', count: 3, connectionType: 'database' }, - { client: 'review', server: 'cache', count: 2, connectionType: 'database' }, - { client: 'review', server: 'reviews', count: 1, connectionType: 'database', - clientIdentity: { environment: 'staging' } }, - ], inventory()); - expect(mixed.deferredMetricCalls).toBe(1); - expect(mixed.nodes.filter(node => node.meta.provenance === 'service_graph_metric').map(node => node.label).sort()) - .toEqual(['cache', 'reviews']); + { client: 'payment', server: 'ledger', count: 2 }, + { client: 'pricing', server: 'catalog', count: 3, connectionType: 'unknown' }, + ], []); + expect(graph.nodes.map(node => node.label).sort()).toEqual(['catalog', 'ledger', 'payment', 'pricing']); + expect(graph.nodes.every(node => node.kind === 'service')).toBe(true); + expect(graph.edges.map(edge => edge.meta.metricCount).sort()).toEqual([2, 3]); + expect(graph.nonServiceMetricCalls).toBe(3); + expect(svc(graph, 'payment')?.meta.metricCount).toBe(2); }); it('keeps the cloud.resource_id veto for the aggregate unbindable marker and caps merged pods', () => { diff --git a/web/lib/trace-graph.ts b/web/lib/trace-graph.ts index ed25f48ae..188ddd08d 100644 --- a/web/lib/trace-graph.ts +++ b/web/lib/trace-graph.ts @@ -305,27 +305,11 @@ export function buildTraceGraph( if (span.kind === 'CONSUMER') edge(id, sid, 'consumes', w); } } - // Trace spans name databases and queues with host-qualified identity. Metric rows only carry - // Tempo's server label (the span's db.name, db.system or peer host for a database, or the - // destination for a queue), so a metric row is a duplicate only when a span from a compatibly - // scoped client of the same name names that same dependency. Only then is the row not drawn; - // any other metric row, including another dependency of the same client, is kept. - const scopeKeys = ['accountId', 'region', 'environment', 'serviceNamespace', 'k8sNamespace', 'k8sCluster'] as const; - const compatible = (span: TraceSpan, identity: TraceIdentity) => - scopeKeys.every(key => span[key] === undefined || identity[key] === undefined || span[key] === identity[key]); - const names = (span: TraceSpan, database: boolean): string[] => database - ? [span.dbName, span.dbSystem, span.dbHost, span.dbHost?.split('.')[0]].filter((v): v is string => !!v) - : span.messagingDestination ? [span.messagingDestination.trim()] : []; - const spansByService = new Map(); - for (const span of byId.values()) { - if (!span.dbSystem && !(span.messagingSystem && span.messagingDestination)) continue; - const list = spansByService.get(span.service); - if (list) list.push(span); else spansByService.set(span.service, [span]); - } - const shownBySpans = (call: ServiceGraphCall, database: boolean) => - (spansByService.get(call.client) ?? []).some(span => (database ? !!span.dbSystem : !!span.messagingSystem) - && compatible(span, call.clientIdentity ?? {}) && names(span, database).includes(call.server)); - let deferredMetricCalls = 0; + // Metric sources contribute service→service calls only. Tempo marks rows whose server is a + // database or queue (its label is a db.name or destination, not a service) or whose other side + // is uninstrumented (virtual_node); those are not service calls and are not drawn. Databases and + // queues come from trace spans, which carry their host-qualified identity. + let nonServiceMetricCalls = 0; const metricWeight = (id: string, count: number) => { const meta = nodes.get(id)!.meta; meta.metricCount = Number(meta.metricCount ?? 0) + count; @@ -338,53 +322,12 @@ export function buildTraceGraph( if (client !== server) edge(client, server, call.rel ?? 'calls', call.count, 0); continue; } - const clientIdentity = call.clientIdentity ?? {}, serverIdentity = call.serverIdentity ?? {}; - if (call.connectionType === 'database' || call.connectionType === 'messaging_system') { - const database = call.connectionType === 'database'; - if (shownBySpans(call, database)) { deferredMetricCalls++; continue; } - // Caller labels scope the node ID (no cross-scope merge) but are not attribution: like - // queue claims, the database carries no account/region and never bridges to inventory. - const scoped = resourceScope({ ...clientIdentity, ...serverIdentity }); - const id = nodeId(database ? 'db' : 'queue', scoped, `metric:${call.server}`); - if (!nodes.has(id)) nodes.set(id, database ? { - id, kind: 'db', label: call.server.slice(0, 240), - meta: { system: null, host: null, dbName: null, metricServer: call.server.slice(0, 240), - sourceId: scoped.sourceId ?? null, environment: scoped.environment ?? null, - accountId: null, region: null, identityProvenance: 'telemetry_claim', - provenance: 'service_graph_metric' }, - } : { - id, kind: 'queue', label: call.server.slice(0, 240), - meta: queueClaimMeta({ system: null, destination: call.server, broker: null, cluster: null, - sourceId: scoped.sourceId ?? null, environment: scoped.environment ?? null, - provenance: 'service_graph_metric' }), - }); - const client = serviceNode(call.client, clientIdentity); - edge(client, id, database ? 'queries' : 'publishes', 0, call.count); - metricWeight(client, call.count); metricWeight(id, call.count); - continue; - } - if (call.connectionType === 'virtual_node') { - // Tempo's service-graph processor names an uninstrumented caller "user" (its fixed default); - // otherwise the uninstrumented side is the server. A real service named "user" that calls an - // uninstrumented peer is indistinguishable in these labels and is drawn as the caller node. - const clientVirtual = call.client === 'user'; - const external = (name: string, identity: TraceIdentity) => { - const id = nodeId('ext', resourceScope(identity), name); - if (!nodes.has(id)) nodes.set(id, { id, kind: 'external', label: name.slice(0, 240), - meta: { name: name.slice(0, 240), sourceId: identity.sourceId ?? null, environment: identity.environment ?? null, - provenance: 'service_graph_metric' } }); - return id; - }; - const source = clientVirtual ? external(call.client, clientIdentity) : serviceNode(call.client, clientIdentity); - const target = clientVirtual ? serviceNode(call.server, serverIdentity) : external(call.server, serverIdentity); - edge(source, target, 'calls', 0, call.count); - metricWeight(source, call.count); metricWeight(target, call.count); - continue; - } - const client = serviceNode(call.client, clientIdentity), server = serviceNode(call.server, serverIdentity); + if (call.connectionType && call.connectionType !== 'unknown') { nonServiceMetricCalls++; continue; } + const client = serviceNode(call.client, call.clientIdentity ?? {}); + const server = serviceNode(call.server, call.serverIdentity ?? {}); edge(client, server, 'calls', 0, call.count); metricWeight(client, call.count); metricWeight(server, call.count); } return { nodes: [...nodes.values()], edges: [...edges.values()], orphanSpans, invalidSpans, - unresolvedMessaging, deferredMetricCalls }; + unresolvedMessaging, nonServiceMetricCalls }; } diff --git a/web/lib/trace-source.test.ts b/web/lib/trace-source.test.ts index eb7760bb7..9b390dc0f 100644 --- a/web/lib/trace-source.test.ts +++ b/web/lib/trace-source.test.ts @@ -178,16 +178,35 @@ describe('ClickHouseOtelAggregateSource (whole-window aggregates)', () => { expect(invokeMcpLambdaTool.mock.calls.some(([{ args }]) => args.sql.includes('ARRAY JOIN'))).toBe(false); }); - it('publishes nothing when any aggregate fails, so nodes never appear without their edges', async () => { + it('falls back to the bounded span reader when an aggregate query fails, never nodes without edges', async () => { + const spanRow = { TraceId: 't', SpanId: 's', Timestamp: END - 1_000, Duration: 1, ServiceName: 'a', + SpanAttributes: {}, ResourceAttributes: {} }; invokeMcpLambdaTool.mockImplementation(async ({ args }: { args: { sql: string } }) => - args.sql.includes('INNER JOIN') ? { error: 'boom', collectionStatus: 'error' } : { rows: [group()], collectionStatus: 'ok' }); + args.sql.includes('INNER JOIN') ? { error: 'boom', collectionStatus: 'error' } + : args.sql.includes('GROUP BY') ? { rows: [group()], collectionStatus: 'ok' } + : { rows: [spanRow], collectionStatus: 'ok' }); + const out = await new ClickHouseOtelAggregateSource().recentSpans(60, 1000, END); + expect(out).toMatchObject({ status: 'partial', reasons: ['incomplete_collection'] }); + expect(out.items).toEqual([expect.objectContaining({ service: 'a', spanId: 's' })]); + expect(out.items[0].weight).toBeUndefined(); + expect(out.calls).toBeUndefined(); + const last = invokeMcpLambdaTool.mock.calls.at(-1)![0].args.sql as string; + expect(last).toMatch(/ORDER BY Timestamp DESC LIMIT 1000$/); + + invokeMcpLambdaTool.mockReset(); + invokeMcpLambdaTool.mockRejectedValue(new Error('timeout')); expect(await new ClickHouseOtelAggregateSource().recentSpans(60, 1000, END)).toMatchObject({ status: 'error', items: [], reasons: ['query_failed'], }); - invokeMcpLambdaTool.mockRejectedValue(new Error('timeout')); + }); + + it('fails the read on a malformed aggregate envelope without falling back', async () => { + invokeMcpLambdaTool.mockImplementation(async ({ args }: { args: { sql: string } }) => + args.sql.includes('INNER JOIN') ? { unexpected: true } : { rows: [group()], collectionStatus: 'ok' }); expect(await new ClickHouseOtelAggregateSource().recentSpans(60, 1000, END)).toMatchObject({ - status: 'error', items: [], reasons: ['query_failed'], + status: 'error', items: [], reasons: ['malformed_payload'], }); + expect(invokeMcpLambdaTool).toHaveBeenCalledTimes(3); }); it('marks a saturated row limit and malformed rows as partial evidence', async () => { diff --git a/web/lib/trace-source.ts b/web/lib/trace-source.ts index b489b034a..c32102efc 100644 --- a/web/lib/trace-source.ts +++ b/web/lib/trace-source.ts @@ -503,9 +503,22 @@ function resourceIdentity(value: unknown, sourceId: string): TraceIdentity | und } /** Reads service nodes and span-pair edges as ClickHouse aggregates over the whole window, instead - * of the newest raw spans. Counts are complete for the window unless a row limit is reached. */ + * of the newest raw spans. Counts are complete for the window unless a row limit is reached. When + * an aggregate query fails (for example the join exceeds the connector's execution budget on a + * busy table), the same table is read with the bounded newest-span reader instead, disclosed as + * incomplete_collection, so a busy datasource degrades to sampling rather than a frozen graph. */ export class ClickHouseOtelAggregateSource implements TraceSource { - constructor(private readonly instanceId?: number, private readonly shape: OtelAggregateShape = DEFAULT_OTEL_SHAPE) {} + constructor( + private readonly instanceId?: number, private readonly shape: OtelAggregateShape = DEFAULT_OTEL_SHAPE, + private readonly spanTemplate?: string, + ) {} + + private async sampled(windowMins: number, cap: number, endMs: number): Promise { + const read = await new ClickHouseOtelTraceSource(this.instanceId, this.spanTemplate).recentSpans(windowMins, cap, endMs); + if (read.status === 'error' || read.status === 'unavailable') return read; + const reasons = read.reasons.includes('incomplete_collection') ? read.reasons : [...read.reasons, 'incomplete_collection']; + return { ...read, reasons, status: 'partial' }; + } async available(): Promise { return !(await resolveSource('clickhouse', this.instanceId)).reason; } async recentSpans(windowMins: number, cap: number, endMs = Date.now()): Promise { @@ -523,11 +536,13 @@ export class ClickHouseOtelAggregateSource implements TraceSource { try { payloads = await Promise.all([sql.groups, sql.calls, ...(sql.links ? [sql.links] : [])].map(query)); } catch { - return readResult(sourceId, window, [], ['query_failed']); + return this.sampled(windowMins, cap, endMs); } const reasons: Reason[] = []; const results = payloads.map(clickhouseRows); // Any failed aggregate leaves the window unproven; never publish nodes without their edges. + // A failed query falls back to sampling; a malformed response envelope fails the read. + if (results.some((result) => result.reasons.includes('query_failed'))) return this.sampled(windowMins, cap, endMs); for (const result of results) { if (!result.rows) return readResult(sourceId, window, [], result.reasons); reasons.push(...result.reasons);