diff --git a/CHANGELOG.md b/CHANGELOG.md index 70ed5efc..48735376 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -24,7 +24,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 - Chat connector auto-registration: with the optional `connector_target_sync_enabled` flag, adding a ClickHouse, Prometheus, Loki, Tempo or Mimir datasource registers that connector on its AgentCore gateway in the background, so chat can use it once the gateway finishes registering the target, without rerunning the AgentCore provisioner. This is ADR-023, an ADR-005 owner-override exception scoped to creating or updating this stack's own gateway targets. The worker first confirms the gateway belongs to this stack (its AgentCore role and IAM authentication), and a missing gateway fails the job visibly. It only creates or updates awsops's own gateway targets that point at this stack's connector functions, never deletes, and leaves a target owned by another stack untouched. - Cross-account observability over PrivateLink: an optional `privatelink_endpoints` Terraform input creates interface endpoints, with a restricted security group, to ClickHouse, Prometheus, Loki, Tempo or Mimir endpoint services exposed by another AWS account, so the connector Lambdas reach them privately without peering or a public path. An optional `privatelink_zone_name` adds a private DNS zone with short, stable names such as `clickhouse.obs.awsops.internal`. The zone's CNAME records are a second step (`privatelink_zone_records_enabled`) once the provider accepts each connection. All of it defaults to off; the provider-side setup, acceptance and rollback are in the new PrivateLink runbook. - AI Diagnosis evidence-backed six-pillar review (default off): `diagnosis_evidence_loop_enabled` routes reports through bounded, read-only collection of inventory, graph, 14-day metric, measured-memory and Compute Optimizer evidence, reviews it per Well-Architected pillar and renders only evidence-checked facts locally. Analysis runs only on a supported Bedrock Claude diagnosis model under `mixed` or `bedrock_claude`; other providers get a `partial` report without model calls, and nothing is remediated. `diagnosis_network_evidence_enabled` adds Resolver query-log, Network Flow Monitor and RDS Proxy observations. Optional `diagnosis_managed_discovery_enabled` reads existing Application Signals dependencies; the private Omni adapter stays inactive until a read-only, per-space credential IAM boundary is verified, and Terraform rejects nonempty space maps. Service membership and DNS aliases remain configuration, and shared telemetry does not independently confirm instance-to-DB calls. Member accounts are read only through allowlisted `AWSopsReadOnlyRole` ARNs, sending the registered ExternalId. Direct member X-Ray/network-source gaps use `member_xray_network_sources_unsupported`, separately from managed Signals coverage. Capped, unconfirmed or failed evidence is reported as incomplete, missing or degraded rather than healthy. -- Topology discovery from ENIs and database endpoints: inventory now collects network interfaces, RDS/Aurora clusters and RDS Proxies (Network and Storage & DB groups). A load-balancer IP target that is neither an EKS pod nor an ECS task resolves to the synced resource owning that IP's network interface: a load balancer (matched by its ARN, and the request-flow chain continues into that load balancer), Lambda function (marked inferred, because one interface can serve several functions), VPC endpoint, NAT gateway, the instance for its primary IP, or an RDS instance or cluster inferred from VPC and Availability Zone placement and marked inferred. Secondary instance IPs such as VPC CNI pod IPs, RDS interfaces shared with an RDS Proxy subnet and owners missing from inventory stay unresolved with a stated reason. Trace database hosts now bind to Aurora writer, reader and custom endpoints and to RDS Proxy endpoints in addition to instance endpoints. In the saved request-flow and infrastructure graphs these sources only add detail and never change collection status or freshness: one that is missing, failed, stale or cut by the snapshot row limit is left out, along with every inference it guards (RDS placement needs complete RDS instance and RDS Proxy reads). The live request-flow page lists them with its other inventory reads, uses only reads from a succeeded sync and leaves RDS placement inference to the saved graph. Network interface data keeps only private addresses. Placement-inferred owners are not used as identity evidence by the Service + Network view, and network interfaces do not count toward dashboard fleet totals. The new collection uses the existing read permissions; the inventory sync Lambda that runs it is updated by the reviewed Terraform plan. +- Topology discovery from ENIs and database endpoints: inventory now collects network interfaces, RDS/Aurora clusters and RDS Proxies (Network and Storage & DB groups). A load-balancer IP target that is neither an EKS pod nor an ECS task resolves to the synced resource owning that IP's network interface: a load balancer (matched by its ARN, and the request-flow chain continues into that load balancer), Lambda function (marked inferred, because one interface can serve several functions), VPC endpoint, NAT gateway, the instance for its primary IP, or an RDS instance or cluster inferred from VPC and Availability Zone placement and marked inferred. Secondary instance IPs such as VPC CNI pod IPs, RDS interfaces shared with an RDS Proxy subnet, an IP claimed by two different interfaces and owners missing from inventory stay unresolved with a stated reason; the same interface seen under several accounts of a shared VPC counts once when its copies agree. Trace database hosts now bind to Aurora writer, reader and custom endpoints and to RDS Proxy endpoints in addition to instance endpoints. In the saved request-flow and infrastructure graphs these sources only add detail and never change collection status or freshness: one that is missing, failed, stale or cut by the snapshot row limit is left out, along with every inference it guards (RDS placement needs complete RDS instance and RDS Proxy reads). The live request-flow page reads network interfaces in full pages, uses only reads from a succeeded sync and does not read RDS sources, so an RDS-owned IP target stays unresolved there and RDS placement inference is left to the saved graph. Network interface data keeps only private addresses. Placement-inferred owners are not used as identity evidence by the Service + Network view, and network interfaces do not count toward dashboard fleet totals, are never a Core Resources chip in the resource trend; they are charted in the default-hidden Other Resources group. The new collection uses the existing read permissions; the inventory sync Lambda that runs it is updated by the reviewed Terraform plan. - Topology navigation: the sidebar's Topology entry expands to Request flow, Infra layout, Network topology and Service map, and opens automatically on any topology page, so the infrastructure layout graph and service map no longer require the small in-page buttons. Large request-flow and infrastructure maps start with entry-point or VPC selection. The default infrastructure placement graph filters by VPC, subnet and resource kind and displays readable pages with optional security-group edges; service maps focus on a two-hop service path with optional workload context and explicit full-map access. Search, responsive fitting and source-count disclosure retain scoped identities and collection warnings; shared gateways or databases do not expand unrelated sibling paths. - User guide: `docs-site/` is published as this repository's GitHub Pages site (deployed automatically on pushes to `dev`, or manually via workflow dispatch), retargeted from the previous personal-domain deployment — site URL, navigation/landing GitHub links, robots.txt sitemap and alert runbook cross-references now point at this repository, and the previous dashboard link and personal analytics tracker are removed. @@ -709,7 +709,7 @@ First release of the **v2 line** (versioned independently from the v1 1.x line, - 채팅 커넥터 자동 등록: 선택 플래그 `connector_target_sync_enabled`를 켜면 ClickHouse·Prometheus·Loki·Tempo·Mimir 데이터소스를 추가할 때 해당 커넥터를 백그라운드에서 AgentCore 게이트웨이에 등록하므로, AgentCore 프로비저너를 다시 실행하지 않아도 게이트웨이가 target 등록을 마치면 채팅에서 사용할 수 있습니다. 이는 이 스택 자신의 게이트웨이 target 생성·갱신으로 범위를 한정한 ADR-005 owner-override 예외(ADR-023)입니다. worker는 먼저 게이트웨이가 이 스택 소유(이 스택의 AgentCore role과 IAM 인증)인지 확인하고, 게이트웨이가 없으면 job을 눈에 보이게 실패 처리합니다. 또한 이 스택의 커넥터 함수를 가리키는 awsops 소유 게이트웨이 target만 생성·갱신하고, 삭제하지 않으며, 다른 스택이 소유한 target은 건드리지 않습니다. - PrivateLink 기반 교차 계정 관측 데이터소스 연결: 선택 입력 `privatelink_endpoints`가 다른 AWS 계정이 노출한 ClickHouse·Prometheus·Loki·Tempo·Mimir 엔드포인트 서비스로 가는 인터페이스 엔드포인트와 제한된 보안 그룹을 만들어, 커넥터 Lambda가 피어링이나 공용 경로 없이 비공개로 접속합니다. 선택 입력 `privatelink_zone_name`은 `clickhouse.obs.awsops.internal`처럼 짧고 안정적인 이름의 프라이빗 DNS 영역을 추가하며, CNAME 레코드는 제공자가 연결을 수락한 뒤의 두 번째 단계(`privatelink_zone_records_enabled`)입니다. 모두 기본값은 꺼짐이며, 제공자 측 설정·승인·롤백 절차는 새 PrivateLink 런북에 있습니다. - AI 진단 근거 기반 6개 필러 리뷰(기본 꺼짐): `diagnosis_evidence_loop_enabled`를 켜면 리포트가 인벤토리·그래프·14일 지표·메모리 실측·Compute Optimizer 증거를 읽기 전용으로 제한된 범위만 수집하고, Well-Architected 필러별로 검토한 뒤 근거가 확인된 사실만 로컬에서 렌더링합니다. 분석은 `mixed` 또는 `bedrock_claude`의 지원되는 Bedrock Claude 진단 모델에서만 실행하며, 다른 제공자는 모델 호출 없이 `partial` 리포트를 받고 어떤 복구도 실행하지 않습니다. `diagnosis_network_evidence_enabled`는 Resolver 쿼리 로그·Network Flow Monitor·RDS Proxy 관측을 추가합니다. 선택 기능인 `diagnosis_managed_discovery_enabled`는 기존 Application Signals 의존성을 읽습니다. Omni 내부 어댑터는 space별 읽기 전용 자격 증명 IAM 범위가 검증될 때까지 비활성 상태이며 Terraform은 비어 있지 않은 space 맵을 거부합니다. 서비스 소속과 DNS 별칭은 구성 근거로 유지하며, 같은 텔레메트리로 인스턴스↔DB 호출을 중복 확정하지 않습니다. 멤버 계정은 허용 목록에 있는 `AWSopsReadOnlyRole`로만 읽고, 등록된 ExternalId를 함께 보냅니다. 멤버의 직접 X-Ray/네트워크 소스 누락은 managed Signals 수집 범위와 별도로 `member_xray_network_sources_unsupported`로 표시합니다. 한도에 걸리거나 확인되지 않았거나 실패한 증거는 정상이 아니라 불완전·누락·저하로 표시합니다. -- ENI와 DB 엔드포인트 기반 토폴로지 디스커버리: 인벤토리가 네트워크 인터페이스, RDS/Aurora 클러스터, RDS Proxy를 수집합니다(네트워크, 스토리지·DB 그룹). EKS 파드도 ECS 태스크도 아닌 로드밸런서 IP 타깃은 그 IP를 가진 네트워크 인터페이스의 소유 리소스로 판별됩니다. 대상은 로드밸런서(ARN으로 매칭하며, 요청 흐름이 그 로드밸런서로 이어집니다), Lambda 함수(인터페이스 하나가 여러 함수에 쓰일 수 있어 inferred로 표시), VPC 엔드포인트, NAT 게이트웨이, 기본 IP를 가진 인스턴스, 그리고 VPC·가용 영역 배치로 추론한 RDS 인스턴스 또는 클러스터입니다. RDS 추론 결과도 inferred로 표시됩니다. VPC CNI 파드 IP 같은 인스턴스의 보조 IP, RDS Proxy와 서브넷을 공유하는 RDS 인터페이스, 인벤토리에 없는 소유 리소스는 판별하지 않고 사유를 남깁니다. 트레이스의 DB 호스트는 인스턴스 엔드포인트뿐 아니라 Aurora writer·reader·사용자 지정 엔드포인트와 RDS Proxy 엔드포인트에도 연결됩니다. 저장된 요청 흐름·인프라 그래프에서 이 원천들은 세부 정보만 보강하며 수집 상태나 최신성을 바꾸지 않습니다. 원천이 없거나 실패했거나 오래됐거나 스냅샷 행 상한에 잘리면, 그 행과 그 원천에 기대는 추론을 모두 제외합니다(RDS 배치 추론에는 RDS 인스턴스와 RDS Proxy 읽기가 모두 완전해야 합니다). 라이브 요청 흐름 화면은 이 원천들을 다른 인벤토리 읽기와 함께 표시하고, 성공한 동기화의 읽기만 사용하며, RDS 배치 추론은 저장된 그래프에 맡깁니다. 네트워크 인터페이스 데이터는 private 주소만 보관합니다. 배치로 추론한 소유자는 서비스 + 네트워크 보기에서 식별 근거로 쓰지 않으며, 네트워크 인터페이스는 대시보드의 전체 리소스 합계에 포함되지 않습니다. 새 수집은 기존 읽기 권한을 사용하며, 이를 실행하는 인벤토리 동기화 Lambda는 검토된 Terraform plan으로 업데이트됩니다. +- ENI와 DB 엔드포인트 기반 토폴로지 디스커버리: 인벤토리가 네트워크 인터페이스, RDS/Aurora 클러스터, RDS Proxy를 수집합니다(네트워크, 스토리지·DB 그룹). EKS 파드도 ECS 태스크도 아닌 로드밸런서 IP 타깃은 그 IP를 가진 네트워크 인터페이스의 소유 리소스로 판별됩니다. 대상은 로드밸런서(ARN으로 매칭하며, 요청 흐름이 그 로드밸런서로 이어집니다), Lambda 함수(인터페이스 하나가 여러 함수에 쓰일 수 있어 inferred로 표시), VPC 엔드포인트, NAT 게이트웨이, 기본 IP를 가진 인스턴스, 그리고 VPC·가용 영역 배치로 추론한 RDS 인스턴스 또는 클러스터입니다. RDS 추론 결과도 inferred로 표시됩니다. VPC CNI 파드 IP 같은 인스턴스의 보조 IP, RDS Proxy와 서브넷을 공유하는 RDS 인터페이스, 서로 다른 두 인터페이스가 같은 IP를 가진 경우, 인벤토리에 없는 소유 리소스는 판별하지 않고 사유를 남깁니다. 공유 VPC에서 여러 계정에 같은 인터페이스가 잡히면 내용이 일치할 때 하나로 셉니다. 트레이스의 DB 호스트는 인스턴스 엔드포인트뿐 아니라 Aurora writer·reader·사용자 지정 엔드포인트와 RDS Proxy 엔드포인트에도 연결됩니다. 저장된 요청 흐름·인프라 그래프에서 이 원천들은 세부 정보만 보강하며 수집 상태나 최신성을 바꾸지 않습니다. 원천이 없거나 실패했거나 오래됐거나 스냅샷 행 상한에 잘리면, 그 행과 그 원천에 기대는 추론을 모두 제외합니다(RDS 배치 추론에는 RDS 인스턴스와 RDS Proxy 읽기가 모두 완전해야 합니다). 라이브 요청 흐름 화면은 네트워크 인터페이스를 모든 페이지에 걸쳐 읽고, 성공한 동기화의 읽기만 사용하며, RDS 원천은 읽지 않으므로 RDS가 소유한 IP 타깃은 그 화면에서 판별되지 않고 RDS 배치 추론은 저장된 그래프에 맡깁니다. 네트워크 인터페이스 데이터는 private 주소만 보관합니다. 배치로 추론한 소유자는 서비스 + 네트워크 보기에서 식별 근거로 쓰지 않으며, 네트워크 인터페이스는 대시보드의 전체 리소스 합계에 포함되지 않고 리소스 추세에서 Core Resources 칩이 되지 않고 기본 숨김인 Other Resources 그룹에 표시됩니다. 새 수집은 기존 읽기 권한을 사용하며, 이를 실행하는 인벤토리 동기화 Lambda는 검토된 Terraform plan으로 업데이트됩니다. - 토폴로지 내비게이션: 사이드바의 토폴로지 항목을 펼치면 요청 흐름·인프라 배치·네트워크 토폴로지·서비스 맵으로 바로 이동할 수 있고, 토폴로지 화면에 있으면 자동으로 펼쳐집니다. 인프라 배치 그래프와 서비스 맵을 찾기 위해 화면 안의 작은 버튼을 쓰지 않아도 됩니다. 큰 요청 흐름·인프라 맵은 진입점 또는 VPC 선택부터 시작합니다. 기본 인프라 배치 그래프는 VPC·서브넷·리소스 종류로 범위를 좁히고 큰 카드 페이지로 탐색하며 보안 그룹 연결선은 선택하여 표시합니다. 서비스 맵은 서비스 상·하위 두 단계 경로와 선택형 워크로드 문맥을 표시하며 전체 그래프도 열 수 있습니다. 검색·반응형 맞춤·표시 건수는 범위별 식별자와 수집 경고를 보존하고, 공유 게이트웨이·DB를 거쳐 무관한 형제 경로까지 펼치지 않습니다. - 사용자 가이드: `docs-site/`를 이 저장소의 GitHub Pages 사이트로 게시합니다(`dev` 브랜치 push 시 자동 배포, 또는 워크플로 수동 실행). 이전 개인 도메인 배포 대상을 이 저장소 기준으로 재조정하여 사이트 URL, 내비게이션·랜딩 페이지의 GitHub 링크, robots.txt sitemap, 알림 런북 상호 참조가 이 저장소를 가리키며, 기존 대시보드 링크와 개인 애널리틱스 트래커는 제거합니다. diff --git a/docs-site/docs/monitoring/inventory.md b/docs-site/docs/monitoring/inventory.md index d7fc9d58..a64228c9 100644 --- a/docs-site/docs/monitoring/inventory.md +++ b/docs-site/docs/monitoring/inventory.md @@ -30,6 +30,7 @@ AWS 리소스의 수량 변화를 일별로 추적하고 비용 영향을 추정 차트 시리즈는 고정 목록이 아니라 최신 스냅샷 수량 기준으로 동적으로 순위가 매겨집니다: - **Core Resources**: 수량 상위 5개 실제 리소스 타입 — 기본 표시 - **Other Resources**: 다음 순위 최대 3개 타입 — 기본 숨김(칩 클릭으로 표시) +- 네트워크 인터페이스(ENI)는 전체 합계에 포함되지 않으므로 Core에 들지 않고, 순위와 관계없이 Other에 추가됩니다 - 나머지 타입은 차트에는 표시되지 않지만 아래 수량 변화 테이블에는 전부 나열됩니다 ### 보안 시리즈 (기본 숨김, 별도 토글 그룹) - Public S3 Buckets, Open Security Groups, Unencrypted EBS — 보안 페이지와 동일 판정 기준의 파생 카운트, 전체 합계(total) 미포함 diff --git a/docs-site/i18n/en/docusaurus-plugin-content-docs/current/monitoring/inventory.md b/docs-site/i18n/en/docusaurus-plugin-content-docs/current/monitoring/inventory.md index 77004337..6e76f90a 100644 --- a/docs-site/i18n/en/docusaurus-plugin-content-docs/current/monitoring/inventory.md +++ b/docs-site/i18n/en/docusaurus-plugin-content-docs/current/monitoring/inventory.md @@ -30,6 +30,7 @@ A page for tracking daily changes in AWS resource counts and estimating cost imp Chart series are ranked dynamically by the latest snapshot counts, not a fixed list: - **Core Resources**: the top 5 real resource types by count — shown by default - **Other Resources**: up to the next 3 types — hidden by default (click a chip to show) +- Network interfaces (ENIs) are left out of the fleet total, so they never join Core and are always added to Other - Remaining types don't chart, but all of them appear in the delta table below ### Security Series (hidden by default, own toggle group) - Public S3 Buckets, Open Security Groups, Unencrypted EBS — derived counts using the Security page's criteria, excluded from the overall total diff --git a/docs-site/i18n/ja/docusaurus-plugin-content-docs/current/monitoring/inventory.md b/docs-site/i18n/ja/docusaurus-plugin-content-docs/current/monitoring/inventory.md index c897feed..70aeaf5f 100644 --- a/docs-site/i18n/ja/docusaurus-plugin-content-docs/current/monitoring/inventory.md +++ b/docs-site/i18n/ja/docusaurus-plugin-content-docs/current/monitoring/inventory.md @@ -30,6 +30,7 @@ AWS リソースの数量変化を日次で追跡し、コスト影響を推定 チャート系列は固定リストではなく、最新スナップショット数量で動的にランク付けされます: - **Core Resources**: 数量上位 5 つの実リソースタイプ — デフォルト表示 - **Other Resources**: 続く最大 3 タイプ — デフォルト非表示(チップをクリックで表示) +- ネットワークインターフェイス(ENI)は合計に含まれないため Core には入らず、順位に関係なく Other に追加されます - 残りのタイプはチャートには表示されませんが、下の数量変化テーブルにはすべて表示されます ### セキュリティ系列(デフォルト非表示、独立トグルグループ) - Public S3 Buckets, Open Security Groups, Unencrypted EBS — セキュリティページと同じ判定基準の派生カウント、合計(total)には含まれません diff --git a/docs-site/i18n/zh/docusaurus-plugin-content-docs/current/monitoring/inventory.md b/docs-site/i18n/zh/docusaurus-plugin-content-docs/current/monitoring/inventory.md index 55aecdda..7ee8663e 100644 --- a/docs-site/i18n/zh/docusaurus-plugin-content-docs/current/monitoring/inventory.md +++ b/docs-site/i18n/zh/docusaurus-plugin-content-docs/current/monitoring/inventory.md @@ -30,6 +30,7 @@ import Screenshot from '@site/src/components/Screenshot'; 图表序列按最新快照数量动态排序,而不是固定列表: - **Core Resources**: 数量前 5 的实际资源类型 — 默认显示 - **Other Resources**: 其后的最多 3 个类型 — 默认隐藏(点击标签显示) +- 网络接口(ENI)不计入总数,因此不会进入 Core,并且无论排名如何都会加入 Other - 其余类型不出现在图表中,但全部列在下方的数量变化表中 ### 安全序列(默认隐藏,独立开关组) - Public S3 Buckets、Open Security Groups、Unencrypted EBS — 采用安全页面判定标准的派生计数,不计入总数 diff --git a/docs/api-reference.md b/docs/api-reference.md index dc2b606a..198f35f5 100644 --- a/docs/api-reference.md +++ b/docs/api-reference.md @@ -60,7 +60,7 @@ without a manually held transaction. PostgreSQL supplies one MVCC snapshot for t statement; `statement-snapshot` describes this guarantee, not a requested transaction isolation level. Parameters and scope filters are unchanged. `view=agg` remains separate. -Topology reads target groups/ECS tasks/subnets in at most 20 pages of 500 under one +Topology reads target groups/ECS tasks/subnets/network interfaces in at most 20 pages of 500 under one 30-second browser load budget shared with EKS. All inventory and VPC/security-group enrichment requests share two lanes per load; critical pages run sequentially within a lane. This bounds this loader’s fan-out against the shared pool. Each critical page must carry the marker; diff --git a/docs/runbooks/graph-read-contract.md b/docs/runbooks/graph-read-contract.md index baf5844b..1ebf794d 100644 --- a/docs/runbooks/graph-read-contract.md +++ b/docs/runbooks/graph-read-contract.md @@ -245,7 +245,7 @@ Flow and infra input allow 8192 rows within the same bounded read envelope. The All listed sources feed the builders: a failed/missing source cannot be dropped to authorize replacement, and elapsed retentions never authorize an unproven sweep. Larger generations need a separately reviewed capacity path. -**Enrichment sources.** The one exception is an explicit enrichment tier that only adds detail to nodes built from core sources. Flow enrichment is `eni`, `rds`, `rds_cluster`, `rds_proxy`, `vpc_endpoint` and `nat_gateway`; it resolves load-balancer IP-target ownership. Infra enrichment is `rds_cluster` and `rds_proxy`; it adds DB endpoint nodes for trace binding. These types are evaluated outside `details.sources` and kept in the unprojected `details.enrichmentSources`. A missing, failed, running, unconfirmed, stale or capped enrichment source is dropped whole. The resolver receives it as incomplete, so lookups report `owner_inventory_incomplete` and RDS placement inference (which needs `rds` and `rds_proxy`) is not made. Dropping enrichment never changes the publication status, freshness or reason vocabulary. A core source keeps the rule above. +**Enrichment sources.** The one exception is an explicit enrichment tier that only adds detail to nodes built from core sources. Flow enrichment is `eni`, `rds`, `rds_cluster`, `rds_proxy`, `vpc_endpoint` and `nat_gateway`; it resolves load-balancer IP-target ownership. Infra enrichment is `rds_cluster` and `rds_proxy`; it adds DB endpoint nodes for trace binding. These types are evaluated outside `details.sources` and kept in the unprojected `details.enrichmentSources`. A missing, failed, running, unconfirmed, stale or capped enrichment source is dropped whole. The resolver receives it as incomplete, so lookups report `owner_inventory_incomplete` and RDS placement inference (which needs `rds` and `rds_proxy`) is not made. Dropping enrichment never changes the publication status, freshness or reason vocabulary. A core source keeps the rule above. These rules describe the materialized graph only. The live `/topology` page reads `eni`, `vpc_endpoint` and `nat_gateway` itself, accepts them only from a succeeded sync, and never reads the RDS sources: it passes them to the resolver as incomplete, so RDS placement inference happens only in the saved graph. ## Browser recovery and source evidence diff --git a/docs/runbooks/source-sync-observability.md b/docs/runbooks/source-sync-observability.md index f25d9a8d..23f53ae7 100644 --- a/docs/runbooks/source-sync-observability.md +++ b/docs/runbooks/source-sync-observability.md @@ -219,7 +219,7 @@ Refresh retains the previous graph, or source details do not explain partial cov **Interpretation:** inventory rows and the global per-type sweep ledger are read by one SQL statement through `pool.query`, so each page uses one PostgreSQL statement snapshot without holding a connection across application-managed transaction commands. Critical -target-group/ECS-task/subnet pages require `consistency: "statement-snapshot"` and a stable +target-group/ECS-task/subnet/network-interface pages require `consistency: "statement-snapshot"` and a stable succeeded ledger version across pages. The ledger is keyed under `self` for the whole account sweep; its count is neither the selected account's count nor this page's count. The collector marks that ledger running before mutating rows. This supports cross-page diff --git a/web/app/api/inventory/trend/route.test.ts b/web/app/api/inventory/trend/route.test.ts index 1ba75996..d5a4789c 100644 --- a/web/app/api/inventory/trend/route.test.ts +++ b/web/app/api/inventory/trend/route.test.ts @@ -67,6 +67,20 @@ describe('GET /api/inventory/trend', () => { expect(body.types).toEqual(['lambda', 'ec2']); }); + it('ranks ENIs (left out of the fleet total) below the resources they outnumber', async () => { + verifyUser.mockResolvedValue({ sub: 'u' }); + query.mockResolvedValueOnce({ rows: [ + { d: '2026-07-01', resource_type: 'eni', n: 970 }, + { d: '2026-07-01', resource_type: 'ec2', n: 40 }, + { d: '2026-07-01', resource_type: 'lambda', n: 12 }, + ] }); + query.mockResolvedValue({ rows: [] }); // coverage query + const { GET } = await import('./route'); + const body = await (await GET(req())).json(); + expect(body.trend[0].total).toBe(52); + expect(body.types).toEqual(['ec2', 'lambda', 'eni']); + }); + // every request issues 2 data queries (trend GROUP BY + per-day account coverage); // '__all__' prepends the accounts-table resolution query it('clamps days into [1, 90] and defaults to 14 (accounts default: self)', async () => { diff --git a/web/app/api/inventory/trend/route.ts b/web/app/api/inventory/trend/route.ts index a1bd20f3..a8ac6d1e 100644 --- a/web/app/api/inventory/trend/route.ts +++ b/web/app/api/inventory/trend/route.ts @@ -1,6 +1,6 @@ import { verifyUser } from '@/lib/auth'; import { getPool } from '@/lib/db'; -import { countsTowardFleetTotal, isDerivedTrendType } from '@/lib/trend-utils'; +import { countsTowardFleetTotal } from '@/lib/trend-utils'; export const dynamic = 'force-dynamic'; @@ -135,10 +135,12 @@ export async function GET(request: Request) { const recentPts = trend.filter((pt) => lastMs - new Date(pt.date).getTime() < 2 * 86_400_000); const recent = (t: string) => recentPts.some((pt) => typeof (pt as Record)[t] === 'number'); const types = [...latestByType.keys()].sort((a, b) => { - // Derived security series rank BELOW every real resource type: they must never claim a - // Core top-5 chip slot from an actual resource (their counts overlap the base series). - const da = isDerivedTrendType(a) ? 1 : 0; - const db = isDerivedTrendType(b) ? 1 : 0; + // Types outside the fleet total rank BELOW every fleet-counted type: derived security + // series (their counts overlap the base series) and attachment-only ENIs must never + // outrank the resources whose total they are left out of. Within that lower group they + // order by the same recency/count rules. + const da = countsTowardFleetTotal(a) ? 0 : 1; + const db = countsTowardFleetTotal(b) ? 0 : 1; if (da !== db) return da - db; const ra = recent(a) ? 1 : 0; const rb = recent(b) ? 1 : 0; diff --git a/web/app/page.tsx b/web/app/page.tsx index efefd2c9..65dffc58 100644 --- a/web/app/page.tsx +++ b/web/app/page.tsx @@ -21,7 +21,7 @@ import MultiLineTrend from '@/components/charts/MultiLineTrend'; import SegmentedControl from '@/components/ui/SegmentedControl'; import AiOps from '@/components/overview/AiOps'; import { useActiveScope, scopeParams, type ScopeSelection } from '@/lib/account-context'; -import { nearestSnapshot, netChange, covCompleteForScope, countsTowardFleetTotal, isDerivedTrendType, sameAccountSet, DERIVED_TREND_TYPES, type TrendCoverage } from '@/lib/trend-utils'; +import { nearestSnapshot, netChange, covCompleteForScope, countsTowardFleetTotal, sameAccountSet, trendLegendGroups, DERIVED_TREND_TYPES, type TrendCoverage } from '@/lib/trend-utils'; import { estimateCostImpact, COST_IMPACT_WEIGHTS } from '@/lib/cost-impact'; import { useI18n } from '@/components/shell/LanguageProvider'; import { localeOf } from '@/lib/i18n'; @@ -350,11 +350,9 @@ function ScopedHome({ scope }: { scope: ScopeSelection }) { // the chart entirely. The palette has 8 hues; indices past 8 wrap (colorFor is modulo), a // conscious trade — derived series are default-hidden, so duplicated hues only co-occur // when a user opts a security series in. Colors stay pinned to each series' index. - const realTrendTypes = (resTrend?.types ?? []).filter((t) => !isDerivedTrendType(t)); - const derivedTrendTypes = (resTrend?.types ?? []).filter((t) => isDerivedTrendType(t)); - const trendTypes = [...realTrendTypes.slice(0, 8), ...derivedTrendTypes]; + // Attachment-only types (ENIs) are appended like the derived series, into Other. + const { trendTypes, coreTypes, otherTypes, derivedTrendTypes } = trendLegendGroups(resTrend?.types ?? []); const trendSeries = trendTypes.map((t) => ({ key: t, label: INV_LABEL(t) })); - const coreTypes = realTrendTypes.slice(0, 5); // never a derived series, even on a tiny fleet // Chart honesty (gap L124): a (day, type) whose account coverage is less than the RESOLVED // scope is DROPPED from that day's point — the summed line would otherwise dip on an // account's silent day, presenting a sync artifact as a fleet change (the same rule the @@ -391,7 +389,6 @@ function ScopedHome({ scope }: { scope: ScopeSelection }) { const trendScopeNarrowed = Boolean(resTrend?.degraded) || Boolean( resTrend?.accounts && selectedValidAccts && resTrend.accounts.length < selectedValidAccts.length, ); - const otherTypes = trendTypes.slice(5).filter((t) => !isDerivedTrendType(t)); // L127: 7d net change (lib/trend-utils netChange — coverage-parity diff with honest-degrade // branches; see its doc). The trend data IS account-scoped now (gap L124 — the fetch above // passes the accounts scope), but snapshots still have no REGION dimension — so a narrowed diff --git a/web/app/topology/page-ownership.test.tsx b/web/app/topology/page-ownership.test.tsx index 924bfb81..6d24106d 100644 --- a/web/app/topology/page-ownership.test.tsx +++ b/web/app/topology/page-ownership.test.tsx @@ -214,7 +214,7 @@ describe('bounded ownership inventory paging', () => { const pending: (() => void)[] = []; let active = 0, peak = 0, completed = 0; const requests = serve({ - inventory: Object.fromEntries(['target_group', 'ecs_task', 'subnet'].map(type => [type, largeInventory(type)])), + inventory: Object.fromEntries(['target_group', 'ecs_task', 'subnet', 'eni'].map(type => [type, largeInventory(type)])), inventoryReply: async (_url, body, signal) => { active++; peak = Math.max(peak, active); await new Promise(resolve => { @@ -229,17 +229,19 @@ describe('bounded ownership inventory paging', () => { try { await waitFor(() => expect(pending.length).toBeGreaterThan(0)); expect(peak).toBeLessThanOrEqual(2); - while (completed < 27) { + while (completed < 25) { await waitFor(() => expect(pending.length).toBeGreaterThan(0)); await act(async () => { pending.splice(0).forEach(resolve => resolve()); }); expect(peak).toBeLessThanOrEqual(2); } await screen.findByRole('option', { name: 'ECS · ecs-app' }); const inventoryReads = requests.filter(url => url.pathname.startsWith('/api/inventory/')); - expect(new Set(inventoryReads.map(url => url.pathname)).size).toBe(24); - expect(inventoryReads).toHaveLength(27); + expect(new Set(inventoryReads.map(url => url.pathname)).size).toBe(21); + expect(inventoryReads).toHaveLength(25); + // RDS placement inference is saved-graph-only: the live page never reads RDS sources. + expect(inventoryReads.some(url => /\/rds/.test(url.pathname))).toBe(false); expect(inventoryReads.every(url => url.searchParams.get('accounts') === 'self')).toBe(true); - for (const type of ['target_group', 'ecs_task', 'subnet']) { + for (const type of ['target_group', 'ecs_task', 'subnet', 'eni']) { expect(inventoryReads.filter(url => url.pathname.endsWith(`/${type}`)) .map(url => url.searchParams.get('offset'))).toEqual(['0', '500']); } diff --git a/web/app/topology/page.test.tsx b/web/app/topology/page.test.tsx index be962406..fcbec756 100644 --- a/web/app/topology/page.test.tsx +++ b/web/app/topology/page.test.tsx @@ -155,9 +155,9 @@ describe('sample topology evidence', () => { render(); await screen.findByLabelText('Inventory collection evidence'); const text = screen.getByLabelText('Inventory collection evidence').textContent; - expect(text).toContain('Aggregate sync runs: unknown (21)'); + expect(text).toContain('Aggregate sync runs: unknown (17)'); expect(text).toContain('Run health unknown'); - expect(text).toContain('Inventory read failures: target_group: failed, ecs_task: failed, subnet: failed'); + expect(text).toContain('Inventory read failures: target_group: failed, ecs_task: failed, subnet: failed, eni: failed'); expect(screen.queryByTestId('target')).toBeNull(); }); it.each(['entry-only', 'no-entry'])('separates %s onboarding coverage from EKS read failure', async clusterAccess => { @@ -211,7 +211,7 @@ describe('sample topology evidence', () => { serve({ runStatus: 'running' }); await ready(); const text = screen.getByLabelText('Inventory collection evidence').textContent; - expect(text).toContain('Aggregate sync runs: running (24)'); + expect(text).toContain('Aggregate sync runs: running (21)'); expect(text).not.toContain('Aggregate sync issues:'); expect(text).not.toContain('Run health unknown'); }); @@ -228,7 +228,7 @@ describe('sample topology evidence', () => { serve({ runStatus, rowCapture: null }); await ready(); const evidence = screen.getByLabelText('Inventory collection evidence'); - expect(evidence.textContent).toContain(`Aggregate sync runs: ${runStatus} (24)`); + expect(evidence.textContent).toContain(`Aggregate sync runs: ${runStatus} (21)`); expect(evidence.textContent).toContain(`Aggregate sync issues: route53: ${runStatus}`); expect(evidence.textContent).toContain('Run health unknown for this account scope'); expect(evidence.textContent).not.toContain(new Date(captured).toLocaleString()); @@ -238,7 +238,7 @@ describe('sample topology evidence', () => { serve({ runStatus: 'succeeded' }); await ready(); const text = screen.getByLabelText('Inventory collection evidence').textContent; - expect(text).toContain('Aggregate sync runs: succeeded (24)'); + expect(text).toContain('Aggregate sync runs: succeeded (21)'); expect(text).toContain('Run health unknown for this account scope'); expect(text).not.toContain('Aggregate sync issues:'); }); @@ -249,7 +249,7 @@ describe('sample topology evidence', () => { await act(async () => setActiveScope({ ...DEFAULT_SCOPE, regions: ['ap-northeast-2'] })); const requests = vi.mocked(fetch).mock.calls.slice(count).map(([url]) => new URL(String(url), 'http://localhost')) .filter(url => url.pathname.startsWith('/api/inventory/')); - expect(requests).toHaveLength(24); + expect(requests).toHaveLength(21); expect(requests.every(url => url.searchParams.get('regions') === 'ap-northeast-2' && url.searchParams.get('accounts') === 'self' && url.searchParams.get('includeGlobal') === '1')).toBe(true); }); diff --git a/web/app/topology/page.tsx b/web/app/topology/page.tsx index 9b681935..9b15e358 100644 --- a/web/app/topology/page.tsx +++ b/web/app/topology/page.tsx @@ -26,11 +26,14 @@ const ReactFlow = dynamic(() => import('@xyflow/react').then((m) => m.ReactFlow) const TYPES = ['route53', 'cloudfront', 'alb', 'nlb', 'target_group', 'waf', 'ec2', 'lambda', 'ecs_task', 's3', 'subnet', 'apigatewayv2_api', 'apigatewayv2_integration', 'cloudfront_vpc_origin', 'apigatewayv2_route', 'alb_listener_rule', - 'eni', 'rds', 'rds_cluster', 'rds_proxy', 'vpc_endpoint', 'nat_gateway'] as const; + 'eni', 'vpc_endpoint', 'nat_gateway'] as const; type InvType = (typeof TYPES)[number]; -// Every inventory read the ENI owner resolver consults (eni-owner.ts). -const OWNER_TYPES: InvType[] = ['eni', 'rds', 'rds_cluster', 'rds_proxy', 'vpc_endpoint', 'nat_gateway', - 'alb', 'nlb', 'lambda', 'ec2']; +// Every inventory read the ENI owner resolver consults (eni-owner.ts) that this page fetches. +const OWNER_TYPES: InvType[] = ['eni', 'vpc_endpoint', 'nat_gateway', 'alb', 'nlb', 'lambda', 'ec2']; +// The page cannot prove freshness without comparing clocks, so RDS placement inference (which +// needs complete, current DB and proxy lists) is left to the materialized graph: these reads are +// not fetched and always count as incomplete. +const SAVED_GRAPH_ONLY_OWNER_TYPES = ['rds', 'rds_cluster', 'rds_proxy']; type Row = Record; // InvType → FlowInput key (target_group→tg, ecs_task→ecsTask). ec2/lambda/ecs enrich target labels. @@ -39,15 +42,14 @@ type Row = Record; // resolve IP targets that are neither EKS pods nor ECS tasks (eni-owner.ts). type RowKey = 'route53' | 'cloudfront' | 'alb' | 'nlb' | 'tg' | 'waf' | 'ec2' | 'lambda' | 'ecsTask' | 's3' | 'subnet' | 'apigatewayv2_api' | 'apigatewayv2_integration' | 'cloudfront_vpc_origin' | 'apigatewayv2_route' | 'alb_listener_rule' - | 'eni' | 'rds' | 'rds_cluster' | 'rds_proxy' | 'vpc_endpoint' | 'nat_gateway'; + | 'eni' | 'vpc_endpoint' | 'nat_gateway'; const FLOW_KEY: Record = { route53: 'route53', cloudfront: 'cloudfront', alb: 'alb', nlb: 'nlb', target_group: 'tg', waf: 'waf', ec2: 'ec2', lambda: 'lambda', ecs_task: 'ecsTask', s3: 's3', subnet: 'subnet', apigatewayv2_api: 'apigatewayv2_api', apigatewayv2_integration: 'apigatewayv2_integration', cloudfront_vpc_origin: 'cloudfront_vpc_origin', apigatewayv2_route: 'apigatewayv2_route', alb_listener_rule: 'alb_listener_rule', - eni: 'eni', rds: 'rds', rds_cluster: 'rds_cluster', rds_proxy: 'rds_proxy', - vpc_endpoint: 'vpc_endpoint', nat_gateway: 'nat_gateway', + eni: 'eni', vpc_endpoint: 'vpc_endpoint', nat_gateway: 'nat_gateway', }; // Node fill/border per FlowKind. Light + dark variants (ReactFlow dark colorMode flips default @@ -145,7 +147,9 @@ function nodeLabel(n: FlowNode): ReactNode { const ROW_CAP = 500; // /api/inventory caps limit at 500 const INVENTORY_LANES = 2; // Shared max:3 PG pool also serves authentication and other reads. const CRITICAL_PAGES = 20; -const CRITICAL_TYPES = new Set(['target_group', 'ecs_task', 'subnet']); +// Paged to completion: ownership reads that a cut list would turn into "unknown". ENIs run +// about 500 per account, so a single 500-row page would leave IP-target owners unresolved. +const CRITICAL_TYPES = new Set(['target_group', 'ecs_task', 'subnet', 'eni']); const rowLimit = (type: string) => ROW_CAP * (CRITICAL_TYPES.has(type) ? CRITICAL_PAGES : 1); const ISSUE_STATUSES = ['failed', 'partial'] as const; type InventoryIssue = { type: string; status: typeof ISSUE_STATUSES[number] }; @@ -417,17 +421,15 @@ function ScopedTopologyPage({ activeAccount, inventoryScope }: { activeAccount: const result = res[TYPES.indexOf(type)]; return result.readFailed || result.incomplete ? 'failed' : result.capped ? 'capped' : undefined; }; - // Owner reads are usable only from a succeeded sweep. The page cannot prove freshness - // without comparing clocks, so RDS placement inference (which needs complete, current DB - // and proxy lists) is left to the materialized graph: rds_proxy always counts as incomplete. - const ownerIssue = (type: InvType) => type === 'rds_proxy' || readIssue(type) !== undefined + // Owner reads are usable only from a succeeded sweep. + const ownerIssue = (type: InvType) => readIssue(type) !== undefined || !res[TYPES.indexOf(type)].runSucceeded; const out: FlowInput = { ipResolved: eks?.map, ownershipRead: { targetGroup: readIssue('target_group'), ecsTask: readIssue('ecs_task'), subnet: readIssue('subnet'), eksScopes: eks?.blockedScopes, eksUnknown: eks?.globalUnknown, ...(account === 'self' ? { eksRegions: eks?.coveredRegions } : { configurationOnly: true }), // Capped or failed owner reads: the ENI resolver refuses misses and inferences they guard. - ownerIncomplete: OWNER_TYPES.filter(ownerIssue), + ownerIncomplete: [...OWNER_TYPES.filter(ownerIssue), ...SAVED_GRAPH_ONLY_OWNER_TYPES], } }; TYPES.forEach((t, i) => { out[FLOW_KEY[t]] = res[i].rows; }); const nextHasNodes = buildFlowGraph(out).nodes.length > 0; diff --git a/web/lib/eni-owner.test.ts b/web/lib/eni-owner.test.ts index 02137907..3a8e9257 100644 --- a/web/lib/eni-owner.test.ts +++ b/web/lib/eni-owner.test.ts @@ -14,8 +14,29 @@ describe('eniOwnerResolver', () => { expect(resolve({ eni: [eni('eni-1', '10.0.0.1')] }, '10.0.0.1', 'vpc-other')).toBeNull(); }); - it('treats an IP claimed by two ENIs as contradictory, never the first match', () => { - expect(resolve({ eni: [eni('eni-1', '10.0.0.1'), eni('eni-2', '10.0.0.1')] }, '10.0.0.1')).toBeNull(); + it('treats an IP claimed by two ENIs as ambiguous, never the first match', () => { + expect(resolve({ eni: [eni('eni-1', '10.0.0.1'), eni('eni-2', '10.0.0.1')] }, '10.0.0.1')) + .toEqual({ reason: 'owner_ambiguous', eniId: null }); + }); + + it('merges one ENI seen under several accounts (a RAM-shared VPC) instead of vetoing it', () => { + const shared = (account: string) => eni('eni-s', '10.0.0.9', { account_id: account, + attached_instance_id: 'i-0abc', requester_managed: false }); + const input = { eni: [shared('111122223333'), shared('444455556666')], + ec2: [{ resource_id: 'i-0abc', region: R, name: 'node-a' }] }; + expect(resolve(input, '10.0.0.9')).toEqual({ owner: expect.objectContaining({ kind: 'ec2', eniId: 'eni-s' }) }); + // Copies that disagree on an owner-identifying field (sweep skew) are ambiguous, in either order. + const moved = { ...shared('444455556666'), attached_instance_id: 'i-0def' }; + for (const order of [[shared('111122223333'), moved], [moved, shared('111122223333')]]) + expect(resolve({ ...input, eni: order }, '10.0.0.9')).toEqual({ reason: 'owner_ambiguous', eniId: null }); + }); + + it('recognizes a NAT ENI by its InterfaceType even without the description prefix', () => { + const input = { nat_gateway: [{ resource_id: 'nat-0123456789abcdef0', region: R }], + eni: [eni('eni-n', '10.0.2.9', { interface_type: 'natGateway', description: 'nat-0123456789abcdef0' })] }; + expect(resolve(input, '10.0.2.9')).toEqual({ owner: expect.objectContaining({ kind: 'nat_gateway' }) }); + const snake = { ...input, eni: [{ ...input.eni[0], interface_type: 'nat_gateway' }] }; + expect(resolve(snake, '10.0.2.9')).toEqual({ owner: expect.objectContaining({ kind: 'nat_gateway' }) }); }); it('resolves an ELB ENI by the exact ARN suffix, not by name alone', () => { @@ -43,7 +64,7 @@ describe('eniOwnerResolver', () => { nat_gateway: [{ resource_id: 'nat-0123456789abcdef0', region: R, name: 'egress' }], eni: [ eni('eni-e', '10.0.2.1', { interface_type: 'vpc_endpoint', description: 'VPC Endpoint Interface vpce-0123456789abcdef0' }), - eni('eni-n', '10.0.2.2', { interface_type: 'nat_gateway', description: 'Interface for NAT Gateway nat-0123456789abcdef0' }), + eni('eni-n', '10.0.2.2', { interface_type: 'natGateway', description: 'Interface for NAT Gateway nat-0123456789abcdef0' }), eni('eni-x', '10.0.2.3', { interface_type: 'vpc_endpoint', description: 'VPC Endpoint Interface vpce-0fedcba987654321f' }), ], }; @@ -69,6 +90,8 @@ describe('eniOwnerResolver', () => { expect(resolve(input, '10.0.3.4')).toEqual({ owner: expect.objectContaining({ kind: 'rds_cluster', resourceId: 'aurora' }) }); expect(resolve({ ...input, rds: [db('aurora-1', 'aurora'), db('b', 'other')] }, '10.0.3.4')) .toEqual({ reason: 'owner_ambiguous', eniId: 'eni-r' }); + // Every DB names one cluster that the complete cluster read lacks: not synced, not ambiguous. + expect(resolve({ ...input, rds_cluster: [] }, '10.0.3.4')).toEqual({ reason: 'owner_not_synced', eniId: 'eni-r' }); }); it('refuses to guess when an RDS Proxy shares the ENI subnet', () => { @@ -102,7 +125,7 @@ describe('eniOwnerResolver', () => { }); it('reports an incomplete owner read instead of claiming the owner is not synced', () => { - const input = { eni: [eni('eni-n', '10.0.7.1', { interface_type: 'nat_gateway', description: 'Interface for NAT Gateway nat-0123456789abcdef0' })] }; + const input = { eni: [eni('eni-n', '10.0.7.1', { interface_type: 'natGateway', description: 'Interface for NAT Gateway nat-0123456789abcdef0' })] }; expect(eniOwnerResolver(input, new Set(['nat_gateway'])).resolve(R, VPC, '10.0.7.1')) .toEqual({ reason: 'owner_inventory_incomplete', eniId: 'eni-n' }); expect(eniOwnerResolver(input).resolve(R, VPC, '10.0.7.1')).toEqual({ reason: 'owner_not_synced', eniId: 'eni-n' }); diff --git a/web/lib/eni-owner.ts b/web/lib/eni-owner.ts index d5d4924f..123fd718 100644 --- a/web/lib/eni-owner.ts +++ b/web/lib/eni-owner.ts @@ -19,7 +19,7 @@ export type EniOwnerKind = 'ec2' | 'alb' | 'nlb' | 'lambda' | 'rds' | 'rds_clust export type EniUnresolved = | 'ecs_managed' // ECS task ENI — the ECS task map owns this attribution | 'secondary_ip' // secondary IP on an instance ENI (e.g. VPC CNI pod IP) — not the instance - | 'owner_not_synced' // the ENI names an owner that is absent from inventory + | 'owner_not_synced' // the owner the ENI names (or the one cluster its DBs name) is absent from inventory | 'owner_ambiguous' // several inventory resources could own the ENI | 'owner_inventory_incomplete' // the owner lookup (or an inference guard) relies on an incomplete read | 'owner_unknown'; // no rule identifies the owner @@ -51,6 +51,13 @@ const scoped = (region: string, vpc: string, value: string) => JSON.stringify([r const LAMBDA_PREFIX = 'AWS Lambda VPC ENI-'; const UUID_SUFFIX = /-[0-9a-f]{8}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{4}-[0-9a-f]{12}$/; const ELB_DESC = /^ELB (app|net)\/([^/\s]+)\/([0-9a-f]+)$/; +/** EC2 reports `natGateway`; accept the snake_case spelling a normalizing source could emit. */ +const NAT_INTERFACE_TYPES = new Set(['natGateway', 'nat_gateway']); +/** Every field the owner rules read: two copies of one ENI merge only when all of them agree. */ +const OWNER_FIELDS = ['resource_id', 'interface_type', 'description', 'requester_id', 'requester_managed', + 'attached_instance_id', 'availability_zone', 'subnet_id'] as const; +const sameEni = (a: Row, b: Row) => str(a.resource_id) !== '' + && OWNER_FIELDS.every(field => str(a[field]) === str(b[field])); function privateIps(eni: Row): { ip: string; primary: boolean }[] { const primary = str(eni.private_ip_address); @@ -69,15 +76,20 @@ function privateIps(eni: Row): { ip: string; primary: boolean }[] { /** Build a resolver once per graph; `resolve` is called per IP target. */ export function eniOwnerResolver(input: EniOwnerInput, incomplete: IncompleteTypes = new Set()) { const gap = (...types: string[]) => types.some(type => incomplete.has(type)); - // IP → ENI, scoped by region and VPC (a private IP is unique inside one VPC). A duplicate key - // means overlapping snapshots; it resolves to null (ambiguous) instead of the first match. - const byIp = new Map(); + // IP → ENI, scoped by region and VPC (a private IP is unique inside one VPC). The same ENI can + // appear once per account in a multi-account scope (a RAM-shared VPC), so a repeat of the same + // ENI id merges when every owner-identifying field agrees; copies that disagree (sweep skew) and + // two different ENIs claiming one IP are contradictory and resolve as ambiguous. + const byIp = new Map(); for (const eni of input.eni ?? []) { const region = str(eni.region), vpc = str(eni.vpc_id); if (!region || !vpc) continue; for (const { ip, primary } of privateIps(eni)) { const k = scoped(region, vpc, ip); - byIp.set(k, byIp.has(k) ? null : { eni, primary }); + const seen = byIp.get(k); + if (seen === undefined) byIp.set(k, { eni, primary }); + else if (seen !== 'conflict' && sameEni(seen.eni, eni) && seen.primary === primary) continue; + else byIp.set(k, 'conflict'); } } const byRegionId = (rows: Row[] | undefined) => { @@ -114,7 +126,8 @@ export function eniOwnerResolver(input: EniOwnerInput, incomplete: IncompleteTyp const row = id ? vpce.get(key(id)) : undefined; return row ? owner('vpc_endpoint', row, eniId) : id ? absent(eniId, 'vpc_endpoint') : miss('owner_unknown', eniId); } - if (type === 'nat_gateway' || desc.startsWith('Interface for NAT Gateway')) { + // InterfaceType first; the description is the fallback. + if (NAT_INTERFACE_TYPES.has(type) || desc.startsWith('Interface for NAT Gateway')) { const id = /\b(nat-[0-9a-f]{8,})\b/.exec(desc)?.[1]; const row = id ? nat.get(key(id)) : undefined; return row ? owner('nat_gateway', row, eniId) : id ? absent(eniId, 'nat_gateway') : miss('owner_unknown', eniId); @@ -150,6 +163,8 @@ export function eniOwnerResolver(input: EniOwnerInput, incomplete: IncompleteTyp const cluster = only ? clusters.get(key(only)) : undefined; if (dbs.length > 1 && only && gap('rds_cluster')) return miss('owner_inventory_incomplete', eniId); if (dbs.length > 1 && cluster) return owner('rds_cluster', cluster, eniId, 'inferred'); + // Every DB here names one cluster and the complete cluster read lacks it: not synced. + if (dbs.length > 1 && only) return miss('owner_not_synced', eniId); return miss(dbs.length ? 'owner_ambiguous' : 'owner_not_synced', eniId); } // A customer-attached instance ENI owns its primary IP. Secondary IPs (VPC CNI pod IPs, @@ -164,12 +179,13 @@ export function eniOwnerResolver(input: EniOwnerInput, incomplete: IncompleteTyp } return { - /** null = no synced ENI holds this IP in that region/VPC (or the snapshot is contradictory). */ + /** null = no synced ENI holds this IP in that region/VPC. */ resolve(region: string, vpc: string, ip: string): EniResolution | null { if (!region || !vpc || !ip) return null; // A cut or withheld ENI list cannot prove a single holder (duplicate claims may be missing). if (gap('eni')) return { reason: 'owner_inventory_incomplete', eniId: null }; const hit = byIp.get(scoped(region, vpc, ip)); + if (hit === 'conflict') return { reason: 'owner_ambiguous', eniId: null }; return hit ? resolveEni(hit.eni, hit.primary) : null; }, }; diff --git a/web/lib/trend-utils.test.ts b/web/lib/trend-utils.test.ts index 7f020744..27f67fc2 100644 --- a/web/lib/trend-utils.test.ts +++ b/web/lib/trend-utils.test.ts @@ -1,5 +1,5 @@ import { describe, it, expect, vi, beforeEach, afterEach } from 'vitest'; -import { nearestSnapshot, netChange, typeCovEqual, covComplete, covCompleteForScope, expectedAccounts, isDerivedTrendType, sameAccountSet } from './trend-utils'; +import { nearestSnapshot, netChange, typeCovEqual, covComplete, covCompleteForScope, expectedAccounts, isDerivedTrendType, sameAccountSet, trendLegendGroups } from './trend-utils'; const day = (offset: number) => new Date(Date.now() - offset * 86_400_000).toISOString().slice(0, 10); @@ -196,3 +196,22 @@ describe('fleet totals exclude attachment-only types', () => { expect(countsTowardFleetTotal('public_s3_buckets')).toBe(false); }); }); + +describe('trendLegendGroups', () => { + const derived = 'open_security_groups'; + it('puts every charted series in exactly one group, with ENIs in Other on a small fleet', () => { + expect(isDerivedTrendType(derived)).toBe(true); + const g = trendLegendGroups(['ec2', 'lambda', 's3', 'subnet', 'eni', derived]); + expect(g.coreTypes).toEqual(['ec2', 'lambda', 's3', 'subnet']); + expect(g.otherTypes).toEqual(['eni']); + expect(g.derivedTrendTypes).toEqual([derived]); + expect([...g.coreTypes, ...g.otherTypes, ...g.derivedTrendTypes].sort()).toEqual([...g.trendTypes].sort()); + }); + it('keeps ENIs reachable after the top 8 fleet-counted types', () => { + const counted = ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i']; + const g = trendLegendGroups([...counted, 'eni']); + expect(g.coreTypes).toEqual(['a', 'b', 'c', 'd', 'e']); + expect(g.otherTypes).toEqual(['f', 'g', 'h', 'eni']); + expect(g.trendTypes).not.toContain('i'); + }); +}); diff --git a/web/lib/trend-utils.ts b/web/lib/trend-utils.ts index b14c872d..44823be6 100644 --- a/web/lib/trend-utils.ts +++ b/web/lib/trend-utils.ts @@ -86,6 +86,20 @@ export function countsTowardFleetTotal(k: string): boolean { return !isDerivedTrendType(k) && !FLEET_TOTAL_EXCLUDED_TYPES.has(k); } +/** Resource-trend chart series and legend groups from the route's ranked type list. + * The top 8 fleet-counted types are charted: Core = top 5 (visible by default), Other = the + * next 3. Attachment-only types (ENIs) and derived security series are appended after them so + * they stay reachable however many types the fleet has: ENIs join Other, derived series keep + * their own group. Every charted series is in exactly one group; Other and derived default hidden. */ +export function trendLegendGroups(ranked: readonly string[]) { + const counted = ranked.filter(countsTowardFleetTotal); + const attachmentOnly = ranked.filter(t => FLEET_TOTAL_EXCLUDED_TYPES.has(t) && !isDerivedTrendType(t)); + const derivedTrendTypes = ranked.filter(isDerivedTrendType); + const coreTypes = counted.slice(0, 5); + const otherTypes = [...counted.slice(5, 8), ...attachmentOnly]; + return { trendTypes: [...coreTypes, ...otherTypes, ...derivedTrendTypes], coreTypes, otherTypes, derivedTrendTypes }; +} + /** Derived security-count trend series (gap L129), written by the sync lambda's * DERIVED_SNAPSHOTS (scripts/v2/steampipe/sync_lambda.py — LOCKSTEP: keys must match) from * the /security predicates in security-findings.ts. Excluded from the trend `total` (the