Repository navigation
pipeline-generator: plan shards from the S3 timings snapshot - #911
Open
Thangnguyenvn98 wants to merge 1 commit into
Open
Thangnguyenvn98 wants to merge 1 commit into
Thangnguyenvn98 wants to merge 1 commit into
Conversation
This was referenced Nov 13, 2026
Builds called /api/timings/latest once per enrolled step while they bootstrap; a slow or failing dashboard database left those steps unsharded. The generator now downloads the hourly timings snapshot from the public vllm-ci-timings bucket once per run and plans every step it covers from it, whatever its age. The live endpoint is used only for a step the snapshot lacks (or whose entry is malformed) and for every step when the snapshot cannot be read, so a build is never worse off than before. A stored 404 is honoured without a live read. Plans from timings over 24 hours old are annotated as a warning. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
Thangnguyenvn98
force-pushed
the
agent/timings-snapshot
branch
from
November 14, 2026 03:40
d35d017 to
f9ba4cf
Compare
Thangnguyenvn98
marked this pull request as ready for review
November 14, 2026 03:44
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Trunk change: needs line-by-line review (pipeline generator, every build).
Summary
The generator calls
/api/timings/latestonce per enrolled step during bootstrap. When the dashboard database is slow or returns 500, the step runs as one job and often times out, and after 3 failures every later step goes unsharded (3.4% of CI builds over the last two weeks).After this merges, the generator downloads one timings snapshot from S3 per build and plans every step it covers from it, whatever its age. The alerting worker refreshes that snapshot hourly (vllm-dashboard#297). The live endpoint is still called for a step the snapshot lacks, and for every step if the snapshot can't be read, so this is never worse than today.
Decisions to check
fileslist) falls through to the endpoint, like a missing one.testsOverMsis ignored: its per-test lists would not match this planner.What changed
runtime_shard.fetch_timings: snapshot lookup by<step key>|<queue>; unchanged live path after it.Not done
Testing
buildkite/testsall pass; mutation (mutmut, diff lines of runtime_shard.py): 0 survivors.basic-correctness-cpu-offloadandamd-v1-sample: the bootstrap read the snapshot once (timings snapshot of 2026-11-13T20:05:18Z, 563 steps) with no live reads, both steps planned 2 shards from it, all 4 shards passed with no automatic retries (6+7 and 136+189 tests), and the slots it used are clean.AI assistance
Written by Claude Code at Thang's direction; Thang has not yet reviewed every line.
🤖 Generated with Claude Code