DEP: Unified CI reliability, diagnostics, and capacity governance

[xianlubird]

Summary

Dynamo already has change-based CI routing, test markers, domain-specific wait helpers, retries, and result artifacts. These mechanisms solve local problems, but do not share one reliability contract across E2E and integration suites.

This DEP proposes four common contracts:

1. wait for observable invariants instead of assuming scheduling order;
2. classify failures and allow only explicit, bounded retries;
3. capture failure-time evidence before destructive teardown;
4. publish comparable quality and capacity records.

Subsystems keep their domain-specific helpers. Adoption is incremental, starting with one deployment and one non-deployment pilot.

Motivation

Recent CI fixes repeatedly addressed the same underlying gaps: tests observed availability before the required state had converged; transport failures and contract failures were reported similarly; cleanup removed short-lived evidence; and existing metrics did not provide one view of first-attempt reliability, failure class, diagnostic completeness, queue time, and GPU cost.

The goal is not another universal test framework. It is a consistent lifecycle for how test harnesses wait, fail, diagnose, clean up, and report results.

Overview

Proposal

1. Invariant-based convergence

Define common sync and async waiting behavior: monotonic deadlines, bounded polling, optional stability windows, explicit transient observation errors, last observation retention, and timeout evidence hooks. Domain modules continue to provide helpers such as wait_for_dgd_ready or wait_for_router_topology.

2. Failure taxonomy and retry envelope

Record failure phase, failure class, and retry scope separately. Any retry must declare its operation, safety assumption, maximum attempts, first failure, and final result. Generic infrastructure recovery must not retry contract violations.

3. Evidence before teardown

A harness that owns destructive cleanup must first capture a bounded, allowlisted failure snapshot. Evidence collection is best effort and never replaces the primary exception. Workflow-level cluster dumps remain a fallback.

4. Quality and capacity telemetry

Applicable suites emit a backend-neutral result record containing timing, requested resources, first and final outcomes, retry information, failure class, evidence status, and cleanup result. Existing JUnit, Datadog, Allure, or OpenSearch-based systems may consume it.

Expected outcomes

• fewer timing failures caused by checking state before convergence;
• faster triage from consistent failure-time evidence;
• recovered retries remain visible instead of looking like clean passes;
• comparable reliability, queue-time, and requested GPU-time data by suite;
• shared behavior without erasing subsystem-specific test semantics.

Non-goals

• Redesign changed-file filters, pytest markers, or CI test tiers.
• Decide a particular compatibility, backend, or topology matrix.
• Replace domain-specific readiness and diagnostic helpers.
• Automatically retry all failures or ban every fixed delay.
• Require a particular metrics vendor.

Detailed design

The issue body remains the stable overview. Ordered comments cover:

1. invariant-based convergence;
2. failure taxonomy and retry semantics;
3. pre-teardown evidence;
4. quality and capacity metrics;
5. incremental implementation.

Decisions requested

1. Are these four contracts the right common boundary?
2. Which deployment and non-deployment paths should be the pilots?
3. What evidence allowlist, retention, and sanitization rules are required?
4. Which existing telemetry sink and team should own the durable view?
5. Is incremental adoption sufficient, or does any existing test class need a migration deadline?

References

• Isolate slow CPU E2E tests
• Wait for discovery convergence
• Retry transient vCluster API failures
• Isolate protocol output from tracing output
• Wait for the router worker watcher
• Recover deploy tests from transient vCluster tunnel drops
• Capture deployment discovery state before cleanup
• Synchronize cancellation tests with worker acknowledgement

[xianlubird]

Detailed design 1: Invariant-based convergence

Current problem

Dynamo has useful wait helpers, but their polling, deadlines, exception handling, stability requirements, and timeout diagnostics are implemented repeatedly. Recent fixes show that service availability is often weaker than the state a test needs: expected workers must be discovered, the router's watcher must converge, or cancellation must be acknowledged.

The problem is not multiple domain helpers. It is the lack of a minimum shared behavior underneath them.

Proposed contract

The shared sync/async primitive owns:

• monotonic deadlines and bounded polling;
• explicit transient observation errors;
• optional consecutive-success or stability requirements;
• bounded observation history and the last observation/error;
• structured timeout results and bounded evidence callbacks;
• preservation of the primary exception.

Domain code owns the observation and predicate, for example: wait_for_dgd_condition, wait_for_router_topology, or wait_for_worker_acknowledgement.

A fixed sleep remains valid when time is part of the scenario, such as lease expiry. A sleep used only to hope that state has converged should use an invariant wait instead.

Benefits

• Tests wait for required state rather than guessed scheduler timing.
• Timeouts identify the unmet invariant and last observed state.
• Deadline and stability behavior can be tested once with fake time.
• Domain terminology and semantics remain local.

Pilot and acceptance

Implement the primitive with deterministic unit tests, then migrate one deployment wait and one router or fault-tolerance wait. Pilots must not increase their external timeout budgets merely to pass.

Acceptance requires monotonic deadlines, structured timeout evidence, bounded callbacks, and at least two domain adapters.

Reviewer decisions

1. Should this live in tests/utils, or behind separate deploy/runtime adapters?
2. Is a stability window required in the initial API?
3. Which two existing waits are the best pilots?
4. Should unexplained E2E sleeps be review guidance or a lightweight check?

[xianlubird]

Detailed design 2: Failure taxonomy and retry semantics

Current problem

Dynamo already documents narrow retry and quarantine policies. However, “retry” still describes very different actions: another convergence observation, an idempotent API retry, another model request, a pytest rerun, or a workflow rerun. A final pass does not consistently show what first failed or how much recovery occurred.

Proposed model

Initial failure classes:

Class	Meaning	Default recovery
CONTRACT_VIOLATION	Observed behavior violated the tested contract	No generic retry
CONVERGENCE_TIMEOUT	Required state missed its deadline	Report last state
INFRA_TRANSPORT	Connection, tunnel, registry, or equivalent transport error	Bounded idempotent operation retry
INFRA_CAPACITY	Runner, GPU, scheduling, or quota unavailable	Classify separately
RUNNER_FAILURE	Runner/container/action failed independently of the test	Workflow handling
TEST_HARNESS	Fixture, helper, test data, or collector failed	Route to suite owner
NONDETERMINISTIC_OUTPUT	Supported output varied within a documented envelope	Narrow request retry
UNKNOWN	Not classified	Visible failure, no broad retry

Each retry record includes operation, phase, class, scope, attempt budget, delay, idempotency/safety assumption, first error, and final outcome.

Core rules:

• generic infrastructure helpers never retry contract violations;
• mutating retries document why repetition is safe;
• streaming requests are not retried after visible output begins;
• transient observations do not reset the convergence deadline;
• cleanup failures never replace the primary failure;
• test, job, and workflow reruns remain separate visible attempts.

Benefits

• Recovered failures no longer disappear behind a green result.
• Contract, harness, capacity, and transport failures can be routed separately.
• Retry safety, cost, and effectiveness become reviewable and measurable.
• UNKNOWN becomes an explicit improvement backlog.

Pilot and acceptance

Instrument one existing vCluster or DGDR recovery path and one model-output retry. Both must preserve native exceptions, first failure, class, scope, and final outcome. Product-contract failures must be rejected by infrastructure retry paths.

Reviewer decisions

1. Are phase, class, and scope sufficient dimensions?
2. Which classes may trigger an automatic job rerun?
3. Should recovered infrastructure failures produce green-with-annotation or a neutral/unstable signal?
4. Where should retry ownership be recorded?

[xianlubird]

Detailed design 3: Evidence before teardown

Current problem

Deployment tests already collect logs and workflow-level cluster dumps. However, ManagedDeployment cleanup deletes the DGD before pytest exits, while the workflow dump runs afterward. Failure-time DGD, DWM, EndpointSlice, ownership, and readiness relationships may therefore be gone or changed.

The resource-owning harness knows the exact deployment and lifecycle point, so it must own the first diagnostic snapshot.

Proposed lifecycle

For deployment failures, the minimum snapshot should include, when available:

• DGD/DGDR and DynamoWorkerMetadata;
• Pods, container state, Services, EndpointSlices, and relevant events;
• current/previous logs and last convergence observation;
• object identity and ownership fields;
• retry history, capture errors, and cleanup outcome.

An evidence.json manifest records each expected item as captured, absent, or collection-failed. Collection is allowlisted, redacted, size-bounded, and cannot replace the primary exception. Secrets, tokens, kubeconfigs, unrestricted environment values, and cluster-wide dumps are excluded from this first tier.

Workflow-level cluster-info dump remains a fallback. Non-Kubernetes harnesses use the same lifecycle with different evidence, such as process tree, stdout/stderr, exit status, and last protocol observation.

Benefits

• Investigations start with failure-time state instead of reconstruction.
• Missing evidence is distinguishable from a resource that was truly absent.
• Cleanup remains bounded and independent of diagnostic collection.
• A common manifest supports both human triage and tooling.

Pilot and acceptance

Integrate an allowlisted collector with ManagedDeployment and inject one deliberate failure. The test must prove evidence precedes deletion, capture errors do not mask the failure, cleanup outcome is recorded, and success paths avoid the full snapshot cost.

Reviewer decisions

1. What is the minimum evidence required for every deployment failure?
2. Should request/response bodies be opt-in?
3. What size and retention limits apply?
4. Which component owns Kubernetes and log redaction?

[xianlubird]

Detailed design 4: CI quality and capacity metrics

Current problem

Dynamo already emits JUnit and Allure results, and shared runtime tests use Datadog Test Optimization. Allure artifacts are produced, but report publication is disabled because of gh-pages growth. Deploy tests do not use the same Datadog path, and the workflow-metrics uploader is not wired into a current workflow.

No common cross-suite record currently combines first-attempt outcome, retry scope, failure class, evidence status, queue/run time, and requested GPU time.

Proposed data flow

The result record complements rather than replaces JUnit. It contains:

• suite, workflow tier, platform, and requested resources;
• queue, reservation, and execution timing when available;
• first outcome, final outcome, and attempt count;
• failure phase/class and retry scope;
• evidence status and cleanup result.

Initial metrics:

• first-attempt and final pass rate;
• retry recovery by scope and failure class;
• product/contract, harness, infrastructure, and UNKNOWN distribution;
• p50/p95 queue and run duration;
• cleanup failure and evidence completeness rate;
• requested GPU-hours by suite and workflow tier;
• GPU time spent on runs ending in infrastructure failure.

Basic capacity accounting is:

requested_gpu_seconds = requested_gpu_count * reservation_seconds

When only job execution time is available, it may be used as an explicitly labelled estimate. Requested capacity is not GPU utilization.

Benefits

• Reliability and capacity priorities can be based on trends, not anecdotes.
• Queue pressure, long tests, retries, and infrastructure waste are separable.
• Retry policies can be evaluated by both recovery rate and added cost.
• Numerical SLOs can be set after a baseline period rather than guessed upfront.

Pilot and acceptance

Emit the same schema from one runtime and one deployment job, connect one durable sink, and collect a baseline before introducing gates. The view must distinguish first from final outcome and expose failure class, evidence status, timing, and requested GPU time.

Reviewer decisions

1. Which existing sink should own the record?
2. Which dimensions are useful without excessive cardinality?
3. Should GPU budgets be organized by suite, workflow tier, or owner?
4. How long should the baseline run before SLOs are proposed?

[xianlubird]

Detailed design 5: Incremental implementation plan

Rollout

Phase 0: Agree on contracts

Finalize the convergence behavior, failure taxonomy, evidence manifest, result schema, storage owner, and privacy/cardinality rules.

Exit: reviewers agree on semantics and ownership. This does not approve numerical gates or repository-wide migration.

Phase 1: Shared primitives and schemas

Implement sync/async invariant waits, retry records, evidence/result schemas, and deterministic tests using fake clocks and observation sequences.

Exit: helpers require neither GPU nor Kubernetes and preserve native exceptions.

Phase 2: Deployment pilot

Migrate one ManagedDeployment wait, capture allowlisted evidence before deletion, and emit retry/evidence/cleanup/result records.

Exit: an injected failure preserves DGD/DWM/Pod/Service/EndpointSlice state; cleanup still runs; successful-path overhead is measured.

Phase 3: Non-deployment pilot

Adopt the same contracts in one router, cancellation, subprocess, or fault-tolerance path.

Exit: the design works without Kubernetes concepts leaking into the API.

Phase 4: Metrics baseline

Ingest both pilots into one durable view covering reliability, classification, diagnostic completeness, timing, and requested GPU time.

Exit: first-attempt success and retry recovery are separable over an agreed baseline window.

Phase 5: Incremental adoption

Publish a migration checklist, prioritize suites using failure rate, diagnostic gaps, and resource cost, then propose SLOs from observed data. New and materially changed harnesses adopt the contracts first; existing tests migrate when touched or justified by data.

Dependencies and risks

• durable metrics sink and owner;
• stable run/test identifiers across records and JUnit;
• artifact sanitization, retention, and size limits;
• deployment, runtime/router, and CI maintainer participation.

The design is additive and versioned. Pilots can revert to previous control flow without requiring a flag-day rollback. Missing evidence or telemetry must never turn a failing test green.

Completion criteria

The DEP is complete when both pilots run in CI, pre-cleanup capture is proven, first-attempt/retry/failure/timing/resource trends are durable, and migration ownership is documented. Numerical SLOs remain a follow-up based on the baseline.

Reviewer decisions

1. Who owns common utilities, schemas, and the metrics sink?
2. Which two pilots provide the most value with the smallest review surface?
3. Should the evidence pilot build on PR fix(discovery): log metadata publication and capture deploy failure state #14837 or land the common schema first?
4. What must be complete before moving from proposed to implementing?