Skip to content

feat(benchmarks): add UUID-aware vision EC sweep - #14834

Draft
furionw wants to merge 6 commits into
mainfrom
qiwa/vllm-ec-uuid-benchmark
Draft

furionw wants to merge 6 commits into
mainfrom
qiwa/vllm-ec-uuid-benchmark

Conversation

@furionw

@furionw furionw commented Sep 15, 2026

Copy link
Copy Markdown
Contributor

Why

The multimodal sweep could not exercise AIPerf image-UUID reuse, so repeated conversational images were always resent and EC comparisons mixed request transport with encoder-cache behavior. A deterministic UUID workload and an explicit harness switch let standalone vLLM, native CPU EC, and Dynamo EC receive the same requests while repeated image content is stripped after first use. Transport validation makes that benchmark invariant observable instead of inferred.

What Change

  • Propagate UUID stripping from sweep configuration into AIPerf.
  • Generate and validate deterministic image UUID workloads.
  • Add a three-arm Qwen3.5 embedding-cache benchmark and repetition wrapper.
classDiagram
    SweepConfig --> SweepOrchestrator : selects UUID transport
    SweepOrchestrator --> AIPerfRunner : configures each arm
    AIPerfRunner ..> AIPerf : passes UUID stripping

    note for SweepConfig "Owns the UUID transport switch"
    note for AIPerfRunner "Builds the UUID-aware profile command"
Loading

Test Plan

  • 11 focused tests pass for UUID propagation, command construction, enrichment, and validation.
  • Relevant pre-commit hooks pass.
  • Official nightly H100x2 sweep completed for vllm-serve, vllm-serve-native-ec, and vllm-serve-dynamo-ec: three repetitions per arm, 240 requests per run, zero errors.
  • Rep 1 transport validation passes for every arm: 360 content-bearing image occurrences, 840 UUID-only stripped occurrences, zero missing UUIDs.
  • Median versus baseline:
    • native EC: +3.72% output throughput, -6.83% p50 TTFT, -2.81% p50 ITL, -2.88% p50 request latency.
    • Dynamo EC: +3.80% output throughput, -4.96% p50 TTFT, -3.24% p50 ITL, -3.54% p50 request latency.
  • Runtime: Dynamo 1.5.0.dev20260914, vLLM 0.28.0, AIPerf 0.12.0, NVTX off.
  • Image digest: sha256:9149bd7780ccd59291f1978d273188c086cfc796785f087f2353ac9c43a7895a.

@github-actions github-actions Bot added feat documentation Improvements or additions to documentation labels Sep 15, 2026
…C arm

Signed-off-by: furionw <qiwa@nvidia.com>
Signed-off-by: furionw <qiwa@nvidia.com>
Signed-off-by: furionw <qiwa@nvidia.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

documentation Improvements or additions to documentation feat size/XL

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant