Skip to content

pipeline-generator: plan shards from the S3 timings snapshot - #911

Open
Thangnguyenvn98 wants to merge 1 commit into
mainfrom
agent/timings-snapshot
Open

Thangnguyenvn98 wants to merge 1 commit into
mainfrom
agent/timings-snapshot

Conversation

@Thangnguyenvn98

@Thangnguyenvn98 Thangnguyenvn98 commented Nov 13, 2026 •

Copy link
Copy Markdown
Collaborator

Trunk change: needs line-by-line review (pipeline generator, every build).

Summary

The generator calls /api/timings/latest once per enrolled step during bootstrap. When the dashboard database is slow or returns 500, the step runs as one job and often times out, and after 3 failures every later step goes unsharded (3.4% of CI builds over the last two weeks).

After this merges, the generator downloads one timings snapshot from S3 per build and plans every step it covers from it, whatever its age. The alerting worker refreshes that snapshot hourly (vllm-dashboard#297). The live endpoint is still called for a step the snapshot lacks, and for every step if the snapshot can't be read, so this is never worse than today.

Decisions to check

  • Snapshot first, not live-first with a snapshot fallback: a slow endpoint is the failure, so trying it first keeps the timeout.
  • Old entries are used as is: the annotation names the snapshot time and warns past 24 h; age never triggers a live read.
  • A stored 404 is honoured (no live read for a step main never passed), so steps without timings stop costing a request.
  • Public bucket over HTTPS, no credentials: agents in every cloud read it like the ci-selector records.
  • An entry that isn't a stored answer (no files list) falls through to the endpoint, like a missing one.
  • Snapshot made with another testsOverMs is ignored: its per-test lists would not match this planner.

What changed

  • runtime_shard.fetch_timings: snapshot lookup by <step key>|<queue>; unchanged live path after it.
  • Runtime-shard summaries add "Timings from the snapshot fetched ". A plan from timings over 24 h old is posted as a warning annotation, even when the plan itself is fine.

Not done

  • Steps with no timings at all still run as one job (separate follow-up: split without timings).

Testing

  • buildkite/tests all pass; mutation (mutmut, diff lines of runtime_shard.py): 0 survivors.
  • Generator diff, nightly run-all on vLLM main, base (live endpoint) vs head (snapshot): same shard count on all 181 sharded steps; 24 steps pack files differently, each because the live read saw a newer main build than the snapshot (fetched about an hour earlier). Head planned 423 steps from the snapshot and read the endpoint for the 27 the seed couldn't fetch.
  • Peer review (Codex), two rounds: round 1 found two issues (a malformed entry skipped the live fallback; the staleness warning never became an annotation), both fixed with a test that fails without the fix; round 2 found nothing.
  • Staging build 102629 on vLLM main with this branch, scoped to basic-correctness-cpu-offload and amd-v1-sample: the bootstrap read the snapshot once (timings snapshot of 2026-11-13T20:05:18Z, 563 steps) with no live reads, both steps planned 2 shards from it, all 4 shards passed with no automatic retries (6+7 and 136+189 tests), and the slots it used are clean.

AI assistance

Written by Claude Code at Thang's direction; Thang has not yet reviewed every line.

🤖 Generated with Claude Code

Builds called /api/timings/latest once per enrolled step while they
bootstrap; a slow or failing dashboard database left those steps
unsharded. The generator now downloads the hourly timings snapshot from
the public vllm-ci-timings bucket once per run and plans every step it
covers from it, whatever its age. The live endpoint is used only for a
step the snapshot lacks (or whose entry is malformed) and for every step
when the snapshot cannot be read, so a build is never worse off than
before. A stored 404 is honoured without a live read. Plans from timings
over 24 hours old are annotated as a warning.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Signed-off-by: Thang Nguyen <thangnguyenvn647@gmail.com>
@Thangnguyenvn98
Thangnguyenvn98 marked this pull request as ready for review November 14, 2026 03:44
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant