Repository navigation
runtime-shard: check on every build that no test dropped out of its shards - #884
Draft
Thangnguyenvn98 wants to merge 52 commits into
Draft
Thangnguyenvn98 wants to merge 52 commits into
Thangnguyenvn98 wants to merge 52 commits into
Conversation
Thangnguyenvn98
force-pushed
the
agent/shard-coverage-check
branch
2 times, most recently
from
November 5, 2026 21:03
7a5685a to
19d664c
Compare
Thangnguyenvn98
force-pushed
the
agent/shard-coverage-check
branch
from
November 13, 2026 04:12
19d664c to
50e9696
Compare
This was referenced Nov 13, 2026
…hards Each shard records what it ran, and the first shard that runs a command collects all of it and records the tests it should run. One soft-failing job compares them and posts a red annotation listing any test no shard ran. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
The coverage check flagged tests no shard ran but not tests that more than one shard ran. Count, per command, the finished shards whose latest attempt reported each test ID, and list any test with two or more under its own heading with the shards that ran it. A duplicate alone makes the annotation yellow: it costs a shard's time but loses no test, so a miss stays the only red. A retried shard and a test pytest reruns inside one shard don't count. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
…ries Three gaps from review. The artifact download returned only each job's latest attempt, so an earlier attempt that finished never reached latest_records(); set BUILDKITE_AGENT_INCLUDE_RETRIED_JOBS, which an agent too old for it ignores rather than failing the download. A shard killed while writing its record left cut-off JSON that stopped the check for every step; the plugin now writes aside and swaps the file in, and the check skips an unreadable record, so only that shard reads as unfinished. The log now lists every duplicate with its shards, as the capped annotation says. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
… ran A shard that left no record, a missing expected list and an unfinished shard all exited 0 as warnings, so a gap in what the shards recorded passed silently. Compare now marks a shard with no record unverified (apart from one that recorded it didn't finish, whose tests stay not_run, listed apart from definite misses), and every state but a duplicate is an error. Duplicates stay warnings. Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
The recording shard collected the command's every file in its own live pytest run, so an import side effect of another shard's file could change what that shard's own tests see (seen in review: the fallback shard fails only when coverage recording is on). The shard's own collection now stays restricted to its own files like every shard's, and the expected list comes from a second pytest the shard starts on the same worker: the same python, arguments and environment, but with the shard plan, coverage directory and marker stripped from its environment, so it applies no plan and writes no record of its own. The list it writes is the one recorded, in the same format as before. Collection there runs with --continue-on-collection-errors, and any failure or error leaves the expected list unavailable: the coverage check then reports the command unverified instead of comparing against a partial list. The recording shard's own run no longer needs continue-on-collection-errors, since it imports no other shard's file. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Spawning the inventory's pytest can itself fail (a worker out of processes): catch the OSError so the shard's own tests run on, with the expected list left unavailable for the coverage check to report unverified. Tests: a real run whose spawn raises (own tests pass, record written with no expected list) and a unit test of the return. Three more assertions from the mutation run: the inventory applies the command's -k deselection (kills the trylast mutants), its pytest's output never leaks into the shard's log (kills capture_output=False), and both failure warnings are matched in full. Remaining mutmut survivors are equivalent (the inventory env-var and registration names, the tempfile name, text=True on unread output, the None stand-in) plus one that does not compile when applied (verified: the runner exits 2 with it; mutmut misclassifies it because its run phase never writes it). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
The pipeline's coverage job skips deferred steps, so a sharded step uploaded by "Run more jobs" had nothing checking its shards ran every test. After each dynamic selection, upload a hard-failing coverage job (runtime-shard-coverage-<round>) behind the picked sharded steps, with RUNTIME_SHARD_COVERAGE_STEPS limited to them, running the same runtime_shard_coverage.py over the same artifacts. Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Every coverage job annotated under the fixed context runtime-shard-coverage, so a deferred round's report replaced the pipeline's. The script takes the context from RUNTIME_SHARD_COVERAGE_CONTEXT (default unchanged), and each "Run more jobs" round's coverage job sets it to its own key. Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
The agent already sets BUILDKITE_STEP_KEY to the job's key, which is the context each job should annotate under: "runtime-shard-coverage" for the pipeline's job, "runtime-shard-coverage-<round>" for a deferred round's. Drops the custom RUNTIME_SHARD_COVERAGE_CONTEXT variable, so only the script carries the behavior. Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
…ords count Two review fixes on the coverage check. A command spec may carry soft_fail: a definite miss stays red even for a soft-fail step (the sharding itself is then broken), but an unfinished shard or a command that can't be checked warns, matching the build's soft-fail semantics; absent the key the behavior is unchanged. And compare()/main() take the current build's ID (BUILDKITE_BUILD_ID) and ignore records that name another build or none, so a stale artifact can't pass this build's check; with no ID given every record counts, as before. Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Peer review of the coverage inventory: the recording shard's collection subprocess ran with no timeout, so a hung collection could hang the job; it ran without the command's own directory; and a failed collection reported only its exit code, leaving the broken file undiagnosable. The inventory subprocess now runs in the command's invocation directory, bounded by INVENTORY_TIMEOUT_SECONDS (300): a timeout, like a spawn failure, leaves the expected list unknown and the shard's own run goes on. A failed inventory prints a bounded tail of its captured stdout and stderr, where pytest names the file that broke. Each coverage record also carries the build's id (BUILDKITE_BUILD_ID, null off Buildkite), so the coverage check can filter out records a stale job wrote. Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
plan_step picked owners()[0] as a command's fallback shard, so shard 0 also collected every command's full test inventory for coverage on top of its own tests. Pick the owner with the smallest estimate instead; ties still go to the first owner, and unplanned files keep running in the fallback shard as before. Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
The pipeline's coverage job waited on its shards up to a deadline fixed at generation time (COLLECT_AFTER_MINUTES after the pipeline was generated): a build whose shards start late gave up waiting early and compared against records that were still coming. The wait now polls the shards' states with no deadline of its own, every 30 s, bounded only by the job's own timeout; the recorders' COLLECT_WAIT keeps its deadline. The deferred 'Run more jobs' coverage job waited on the shards themselves, so a cancelled or timed-out shard ended it waiting_failed and an automatic retry started behind it. It now waits like the pipeline's job: on the shards' prerequisites, polling the shards' states through their retries before comparing, with an explicit timeout. Both paths share the one wait string from unblock_jobs.py so they cannot drift. Each round's job keeps its own key and annotation context. Both coverage specs now carry each step's soft_fail, so the comparison can keep a soft-failing step's failures a warning. Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Thangnguyenvn98
force-pushed
the
agent/shard-coverage-check
branch
from
November 14, 2026 06:31
02f8f78 to
024617e
Compare
Package the source read by the coverage job and test it from isolated Python so source-tree imports cannot hide an incomplete wheel. Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Decode captured timeout bytes without stopping the live shard, and distinguish pending-at-deadline evidence gaps when giving retry advice. Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Retry artifact transport errors without blaming shards, require a terminal state before relaxing soft-fail gaps, and avoid retry advice for deterministic collection failures. Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Count definite misses beside unexpected outcomes, stop the inventory process group on timeout, preserve recorder waits, and back off artifact retries. Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Clean up inventory processes on every exit and preserve definite misses beside unexpected IDs and unfinished shards. Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Trunk change: needs line-by-line review. Changes how every runtime-sharded command collects and verifies tests.
Summary
Automatic sharding needs a coverage check that does not depend on yesterday's unsharded baseline. Each command's expected test IDs are collected in a separate pytest process on a real worker; each shard records its outcomes; a build-level check compares the union and fails when a test or the evidence needed to prove coverage is missing. Deferred steps have no shard plan and continue to run whole and unsharded.
Decisions to check
null, not a false empty assignment. Unfinished shards and missing evidence warn only for a terminal state and allowed failure outcome on a soft-fail step; a running state, passed, neutral or unavailable outcome still fails. Duplicates warn. Only records matching the current build ID count; missing IDs are unverified, and legacy AMD dind jobs remain unsharded. The expected list comes from the latest valid inventory across owner attempts, not blindly from the latest attempt.soft_failis false for the coverage job. Removing only the old soft-fail would not have been enough, because unverified coverage used to return success. The absolute deadline prevents another wait window; agent loss and other infrastructure failures retain normal bounded retries.allow_dependency_failure. Its deadline is 15 minutes before its 360-minute job timeout and includes pipeline and shard queue time. A long queue can therefore fail coverage at the deadline even if the shard later passes; retry coverage after the shard finishes.json.loads, then executed by an isolated Python process rather than a shared temporary script or a moving branch download.What changed
waiting_failedsteps and polls once even after its deadline; upstream image or pre-commit failures produce an unverified-coverage report instead of silently skipping the check.Not done
-xor--maxfailstays unfinished, so coverage may fail alongside the test job even when the stop happened at the last planned test. This is fail-closed behavior, not proof that a particular test was skipped.Testing
c22b299e: control 102749 and same-code main-generator baseline 102756 used the exact vLLM commit, two shards per selected step and identical test assignments. All four shards passed with no retries; the embedded plugin and checker matched this head byte for byte. The two comparisons found 423 and 378 node IDs (801 total), with zero missing, added, changed or duplicated tests. The slowest shard was 16.1 minutes, and machine time was 1% below baseline. The hard coverage job succeeded with 801 outcomes.AI assistance
Written by Claude Code and Kimi Code. The human submitter has not yet reviewed it.
🤖 Generated with Claude Code