Skip to content

runtime-shard: check on every build that no test dropped out of its shards - #884

Draft
Thangnguyenvn98 wants to merge 52 commits into
mainfrom
agent/shard-coverage-check
Draft

Thangnguyenvn98 wants to merge 52 commits into
mainfrom
agent/shard-coverage-check

Conversation

@Thangnguyenvn98

@Thangnguyenvn98 Thangnguyenvn98 commented Oct 30, 2026 •

Copy link
Copy Markdown
Collaborator

Trunk change: needs line-by-line review. Changes how every runtime-sharded command collects and verifies tests.

Summary

Automatic sharding needs a coverage check that does not depend on yesterday's unsharded baseline. Each command's expected test IDs are collected in a separate pytest process on a real worker; each shard records its outcomes; a build-level check compares the union and fails when a test or the evidence needed to prove coverage is missing. Deferred steps have no shard plan and continue to run whole and unsharded.

Decisions to check

  • Expected IDs use an isolated subprocess on the same worker. It inherits the command's environment and selectors, but not the shard plan. Its import path is captured when the plugin loads, before conftest imports change it, and it uses the original working directory. Other shards' imports cannot affect the live shard. A failed, timed-out, empty or incomplete inventory is unverified, never a trusted partial list. A reported test absent from that inventory is unverified too.
  • The inventory must not change test accounting. Function and kernel recorder variables, coverage.py subprocess variables, the test-analytics token and trace IDs are stripped so inventory collection cannot alter the job's recorded counts, timings or uploaded analytics. The least-loaded owner records the inventory and receives tests absent from the plan.
  • Keep the isolated inventory instead of only unioning per-shard collected lists. Process-global imports can change which tests exist. A union cannot recover a test that only exists when another file is imported first, and tethering cannot detect that missing node ID.
  • Coverage fails closed, preserving allowed soft-fail outcomes. Definite misses always fail, including a miss owned by a finished shard while another is unfinished. Planned IDs attribute misses when known; collection errors leave them null, not a false empty assignment. Unfinished shards and missing evidence warn only for a terminal state and allowed failure outcome on a soft-fail step; a running state, passed, neutral or unavailable outcome still fails. Duplicates warn. Only records matching the current build ID count; missing IDs are unverified, and legacy AMD dind jobs remain unsharded. The expected list comes from the latest valid inventory across owner attempts, not blindly from the latest attempt.
  • The check is a hard gate with bounded infrastructure retries. soft_fail is false for the coverage job. Removing only the old soft-fail would not have been enough, because unverified coverage used to return success. The absolute deadline prevents another wait window; agent loss and other infrastructure failures retain normal bounded retries.
  • Polling preserves coverage after cancellations and dependency failures. The job waits on shard state every 30 seconds, including terminal and canceled soft-fail steps. Direct dependencies cannot replace it: Buildkite does not run dependents of canceled jobs, even with allow_dependency_failure. Its deadline is 15 minutes before its 360-minute job timeout and includes pipeline and shard queue time. A long queue can therefore fail coverage at the deadline even if the shard later passes; retry coverage after the shard finishes.
  • The checker is frozen at generator time. Its source is packed with the existing helper and decoded locally with json.loads, then executed by an isolated Python process rather than a shared temporary script or a moving branch download.
  • Retry advice is limited to blocking, potentially retryable evidence gaps. It appears for unfinished shards with a collected plan or a missing shard alongside reported outcomes, after that shard finishes. A per-job marker distinguishes a deadline with shards still pending from terminal steps with no records: the former advises retrying after they finish, the latter does not. Deterministic collection or inventory errors and definite misses get no retry advice.
  • The existing test-tethering check stays. Tethering catches files no CI command selects; this catches tests lost between a selected command and its shards. “Ran” means pytest reported an outcome, including a skip or setup failure.

What changed

  • The plugin writes per-command records atomically, with expected, planned and reported IDs, retry metadata and build ID, in agent-cleanable directories. The inventory has a separate process group and a temporary cancellation handler; every exit stops the group, waits briefly and closes its pipes. Timeout diagnostics use captured partial output rather than reading again.
  • The comparison downloads every attempt, rejects empty, inconsistent and malformed evidence, and reports missed, unexpected, unfinished, unverified and duplicate cases. Download failures get three bounded attempts with 0/10/20-second backoff and then a distinct hard error with retry advice, rather than blame on the shards; an empty artifact set still uses the normal comparison and soft-fail outcome rules. Annotations use stdin so large reports stay within the argument-size limit.
  • The generator adds record uploads and the hard-failing, retry-aware coverage job for runtime-sharded steps. Recorder collection can stop on canceled or waiting_failed steps and polls once even after its deadline; upstream image or pre-commit failures produce an unverified-coverage report instead of silently skipping the check.

Not done

  • One-job plans and deferred unsharded jobs need no shard-union check. This PR adds the safety gate; ci-infra#885 enabled default-on sharding separately.
  • Every sharded build holds one small-CPU agent while it waits and adds a success annotation after verification. Extra inventory collection runs serially with the owner's own collection and has a fixed 300-second timeout; a command that cannot collect within it is unverified and fails coverage. Report-writing options and collection-time plugins run twice, including JUnit/JSON and pytest-cov outputs; a shard that dies after inventory can leave an inventory-generated report behind.
  • A descendant that starts its own session can outlive the inventory's process group. Closing inherited pipes prevents it from hanging the shard, but cleanup of its external resources is not guaranteed.
  • A shard stopped by -x or --maxfail stays unfinished, so coverage may fail alongside the test job even when the stop happened at the last planned test. This is fail-closed behavior, not proof that a particular test was skipped.
  • Test IDs must be deterministic across collection processes. Random or process-specific IDs can read as misses even when the same tests ran.
  • The live experiments below cover two engine steps and their mirrors, not every command shape. Live dependency-failure, agent-loss, soft-fail, retried-shard, download-failure and long-queue deadline cases remain unexercised. Timeout and process-cleanup cases are tested locally, not in a live worker failure.
  • This remains a draft for the human submitter's line-by-line review; nothing has been merged or rolled out by the agent.

Testing

  • Current rebased head: 735 generator/recorder tests passed; 1,430 selector tests passed and four skipped. Ruff and diff checks pass.
  • Scoped mutation runs: 466 mutants killed, 22 documented equivalents, and one phantom registry row. The earlier review delta killed 30/30 mutants. The bounded-wait delta killed 41/43, with two log/timing equivalents. Evidence validation killed 44/48 in the full run; a targeted regression killed the remaining behavioral mutant, leaving three falsy/control-flow/log-text equivalents. The cleanup delta killed all six behavioral mutants and left one log-text equivalent. The comparison and retry-guidance deltas killed all 13 and 11 mutants, respectively, with no survivors. The analytics and structured-retry delta killed all eight behavioral mutants. The per-command heading delta killed all five mutants. The failure-handling delta killed all seven mutants. The coverage-data isolation delta killed all six mutants. The retry-inventory delta killed all 19 mutants. The no-match download correction killed all four mutants with no survivors. The packaging-only delta has no Python logic to mutate. The timeout/deadline delta killed all 47 mutants with no survivors or skipped entries. The download/terminal-state/collection-error delta killed all 68 mutants with no survivors or skipped entries. The final review delta killed all 21 mutants with no survivors, timeouts or skipped entries. The unified cleanup and comparison delta killed all 25 mutants with no survivors, timeouts or skipped entries.
  • Peer review found soft-fail, annotation-size, dependency-wait, script-version-skew, missed-attribution, cancellation, recorder-contamination and import-path issues. Follow-up findings for retries, pending outcomes, empty or inconsistent inventories, unfinished-shard outcomes and malformed records are fixed; current main already rejects xdist sharding. The packaging review found the installed-wheel omission, byte-valued timeout output and missing deadline retry guidance; their fixes have regressions. The latest download, terminal-state, collection-error and unexpected-ID miss findings also have regressions, including a real child-process cleanup check. Recorder waits match main again. Union-only inventories and direct dependencies were rejected because they cannot prove the required coverage after import-dependent collection or cancellation.
  • Generator comparison against current main adds only the coverage job and its group, removes no jobs, and keeps both selected steps at two shards. It inherits every sharded step's dependencies plus pre-commit on PR builds and embeds the exact packed checker source. The absolute deadline prevents another wait window; agent loss and other infrastructure failures retain bounded retries.
  • Final head c22b299e: control 102749 and same-code main-generator baseline 102756 used the exact vLLM commit, two shards per selected step and identical test assignments. All four shards passed with no retries; the embedded plugin and checker matched this head byte for byte. The two comparisons found 423 and 378 node IDs (801 total), with zero missing, added, changed or duplicated tests. The slowest shard was 16.1 minutes, and machine time was 1% below baseline. The hard coverage job succeeded with 801 outcomes.
  • 102735 failed in bootstrap before any test shard: the installed package omitted the checker. A wheel-install reproduction was red before the manifest fix and green afterward, including an isolated installed-package import and full scoped generation with the exact embedded checker. The source-tree test path hid this omission; a new CI check exercises the installed package instead.
  • Earlier isolated-inventory control 102375: all four selected shards and the hard coverage job passed with no retries; it verified 797 outcomes on the older test set. Independent log comparison against main found 421 and 376 node IDs, with no missing, added, changed or duplicated tests.
  • Planted gap 102532: all four test shards passed and all 797 tests ran, but the coverage job failed the build and named the four deliberately omitted IDs. These earlier builds exercised the initial isolated-inventory path; later review fixes are covered by the final head's tests and control above.
  • Test jobs used ephemeral agents and disconnected. Every small-CPU instance used by the final control and baseline terminated. Temporary mutation worktrees are removed without deleting the evidence logs or the runtime-shard: check on every build that no test dropped out of its shards #884 branch.

AI assistance

Written by Claude Code and Kimi Code. The human submitter has not yet reviewed it.

🤖 Generated with Claude Code

Thangnguyenvn98 and others added 24 commits November 13, 2026 22:28
…hards

Each shard records what it ran, and the first shard that runs a command
collects all of it and records the tests it should run. One soft-failing job
compares them and posts a red annotation listing any test no shard ran.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
The coverage check flagged tests no shard ran but not tests that more than
one shard ran. Count, per command, the finished shards whose latest attempt
reported each test ID, and list any test with two or more under its own
heading with the shards that ran it. A duplicate alone makes the annotation
yellow: it costs a shard's time but loses no test, so a miss stays the only
red. A retried shard and a test pytest reruns inside one shard don't count.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
…ries

Three gaps from review. The artifact download returned only each job's
latest attempt, so an earlier attempt that finished never reached
latest_records(); set BUILDKITE_AGENT_INCLUDE_RETRIED_JOBS, which an agent
too old for it ignores rather than failing the download. A shard killed
while writing its record left cut-off JSON that stopped the check for every
step; the plugin now writes aside and swaps the file in, and the check skips
an unreadable record, so only that shard reads as unfinished. The log now
lists every duplicate with its shards, as the capped annotation says.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
… ran

A shard that left no record, a missing expected list and an unfinished
shard all exited 0 as warnings, so a gap in what the shards recorded
passed silently. Compare now marks a shard with no record unverified
(apart from one that recorded it didn't finish, whose tests stay
not_run, listed apart from definite misses), and every state but a
duplicate is an error. Duplicates stay warnings.

Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
The recording shard collected the command's every file in its own live
pytest run, so an import side effect of another shard's file could change
what that shard's own tests see (seen in review: the fallback shard fails
only when coverage recording is on). The shard's own collection now stays
restricted to its own files like every shard's, and the expected list
comes from a second pytest the shard starts on the same worker: the same
python, arguments and environment, but with the shard plan, coverage
directory and marker stripped from its environment, so it applies no plan
and writes no record of its own. The list it writes is the one recorded,
in the same format as before.

Collection there runs with --continue-on-collection-errors, and any
failure or error leaves the expected list unavailable: the coverage check
then reports the command unverified instead of comparing against a
partial list. The recording shard's own run no longer needs
continue-on-collection-errors, since it imports no other shard's file.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Spawning the inventory's pytest can itself fail (a worker out of
processes): catch the OSError so the shard's own tests run on, with the
expected list left unavailable for the coverage check to report
unverified. Tests: a real run whose spawn raises (own tests pass, record
written with no expected list) and a unit test of the return. Three more
assertions from the mutation run: the inventory applies the command's -k
deselection (kills the trylast mutants), its pytest's output never leaks
into the shard's log (kills capture_output=False), and both failure
warnings are matched in full. Remaining mutmut survivors are equivalent
(the inventory env-var and registration names, the tempfile name,
text=True on unread output, the None stand-in) plus one that does not
compile when applied (verified: the runner exits 2 with it; mutmut
misclassifies it because its run phase never writes it).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
The pipeline's coverage job skips deferred steps, so a sharded step
uploaded by "Run more jobs" had nothing checking its shards ran every
test. After each dynamic selection, upload a hard-failing coverage job
(runtime-shard-coverage-<round>) behind the picked sharded steps, with
RUNTIME_SHARD_COVERAGE_STEPS limited to them, running the same
runtime_shard_coverage.py over the same artifacts.

Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Every coverage job annotated under the fixed context
runtime-shard-coverage, so a deferred round's report replaced the
pipeline's. The script takes the context from
RUNTIME_SHARD_COVERAGE_CONTEXT (default unchanged), and each "Run more
jobs" round's coverage job sets it to its own key.

Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
The agent already sets BUILDKITE_STEP_KEY to the job's key, which is the
context each job should annotate under: "runtime-shard-coverage" for the
pipeline's job, "runtime-shard-coverage-<round>" for a deferred round's.
Drops the custom RUNTIME_SHARD_COVERAGE_CONTEXT variable, so only the
script carries the behavior.

Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
…ords count

Two review fixes on the coverage check. A command spec may carry
soft_fail: a definite miss stays red even for a soft-fail step (the
sharding itself is then broken), but an unfinished shard or a command
that can't be checked warns, matching the build's soft-fail semantics;
absent the key the behavior is unchanged. And compare()/main() take the
current build's ID (BUILDKITE_BUILD_ID) and ignore records that name
another build or none, so a stale artifact can't pass this build's
check; with no ID given every record counts, as before.

Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Peer review of the coverage inventory: the recording shard's collection
subprocess ran with no timeout, so a hung collection could hang the job;
it ran without the command's own directory; and a failed collection
reported only its exit code, leaving the broken file undiagnosable.

The inventory subprocess now runs in the command's invocation directory,
bounded by INVENTORY_TIMEOUT_SECONDS (300): a timeout, like a spawn
failure, leaves the expected list unknown and the shard's own run goes
on. A failed inventory prints a bounded tail of its captured stdout and
stderr, where pytest names the file that broke.

Each coverage record also carries the build's id (BUILDKITE_BUILD_ID,
null off Buildkite), so the coverage check can filter out records a
stale job wrote.

Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
plan_step picked owners()[0] as a command's fallback shard, so shard 0
also collected every command's full test inventory for coverage on top
of its own tests. Pick the owner with the smallest estimate instead;
ties still go to the first owner, and unplanned files keep running in
the fallback shard as before.

Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
The pipeline's coverage job waited on its shards up to a deadline fixed
at generation time (COLLECT_AFTER_MINUTES after the pipeline was
generated): a build whose shards start late gave up waiting early and
compared against records that were still coming. The wait now polls the
shards' states with no deadline of its own, every 30 s, bounded only by
the job's own timeout; the recorders' COLLECT_WAIT keeps its deadline.

The deferred 'Run more jobs' coverage job waited on the shards
themselves, so a cancelled or timed-out shard ended it waiting_failed
and an automatic retry started behind it. It now waits like the
pipeline's job: on the shards' prerequisites, polling the shards' states
through their retries before comparing, with an explicit timeout. Both
paths share the one wait string from unblock_jobs.py so they cannot
drift. Each round's job keeps its own key and annotation context.

Both coverage specs now carry each step's soft_fail, so the comparison
can keep a soft-failing step's failures a warning.

Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
@Thangnguyenvn98
Thangnguyenvn98 force-pushed the agent/shard-coverage-check branch from 02f8f78 to 024617e Compare November 14, 2026 06:31
Package the source read by the coverage job and test it from isolated Python so source-tree imports cannot hide an incomplete wheel.

Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Decode captured timeout bytes without stopping the live shard, and distinguish pending-at-deadline evidence gaps when giving retry advice.

Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Retry artifact transport errors without blaming shards, require a terminal state before relaxing soft-fail gaps, and avoid retry advice for deterministic collection failures.

Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Count definite misses beside unexpected outcomes, stop the inventory process group on timeout, preserve recorder waits, and back off artifact retries.

Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Clean up inventory processes on every exit and preserve definite misses beside unexpected IDs and unfinished shards.

Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant