ARM/CPU inference tier: vLLM on Graviton (PRD-67) - #92
Merged
Conversation
New vllm-cpu Runtime, the third accelerator tier alongside gpu + neuron,
mirroring VLLMneuron. Single-node, quantization-forward, no discrete
accelerator device.
- DefaultImage → vllm/vllm-openai-cpu:v<ver>-arm64 (the -arm64 suffix is
load-bearing), routed through the EXISTING Docker Hub ECR pull-through
cache like vllm-gpu — no new Terraform. VLLM_CPU_IMAGE override (mirrors
VLLM_IMAGE / PRD-49).
- ResolveVersion reads a DISTINCT tool_versions field (VLLMCPUVersion, added
to the runtime ToolVersions projection), NOT FrameworkVersion — the CPU
image is a different repo with its own arm64 cadence. DefaultVLLMCPUVersion
fallback.
- BuildArgs forces --dtype bfloat16 (float16 unstable on torch CPU), TP =
NUMA-node count (caller-supplied), and NEVER emits --gpu-memory-utilization
or --kv-cache-dtype (GPU-only). Reuses the shared S3 + Run:ai streamer path.
- MapQuantization → compressed-tensors for int8/int4 (W8A8/W4A8, Arm KleidiAI),
not the GPU's bitsandbytes; suppressed when streaming.
- streamerExtraConfig now gates distributed:true off for cpu (new
ContainerParams.Accelerator field): distributed streaming is a CUDA/ROCm
torch-distributed feature, meaningless for single-node NUMA "ranks" (§5b).
GPU/D-P paths unchanged (Accelerator empty ≠ "cpu").
- Registered in registry; ForAccelerator("cpu") → vllm-cpu.
Grounding caveat (verified this session): ARM W8A8/W4A8 is documented in the
vLLM Arm blog but not yet in the stable CPU support matrix — validate on the
real arm64 image before the §11c guard hard-rejects unquantized.
Tests: registration, ForAccelerator, DefaultImage (+ pull-through), version
resolution, forced bfloat16, GPU-only knobs absent, and the key §5b assertion
that CPU streaming at TP>1 omits distributed:true.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Jeremy Cowan <jicowan@hotmail.com>
…checks (PRD-67 §2) Migration 043: add the 'cpu' accelerator tier and seed Graviton3/4/5 rows. - Expand instance_types.accelerator_type CHECK to include 'cpu' and benchmark_runs.framework CHECK to include 'vllm-cpu' (drop-and-recreate, mirroring migration 034's llm-d addition). Both constraints rejected the new values. - Seed 17 rows: r8g/m8g/c8g (Graviton4, primary), r7g/m7g/c7g (Graviton3, fallback), m9g/c9g (Graviton5, GA in us-east-2). Graviton2 EXCLUDED (no SVE/I8MM/BF16). accelerator_count=0 + accelerator_memory_gib=0 (no discrete accelerator; host DRAM in memory_gib is the serving memory). accelerator_name = Graviton generation. vCPU/memory verified against EC2 spec pages. accelerator_count=0 audit (recorded in PRD-67 Risks): - report/compare.go perGPU() already guards count<=0 → nil (no ÷0); raw CSV column prints 0. Safe. - recommend.go:688 (AcceleratorMemoryGiB/AcceleratorCount) IS ÷0-reachable for CPU because the dispatch (handlers.go:1338, handlers_estimate.go:215, seeder.go:221) is `if neuron else GPU` — a CPU instance falls into the GPU recommender. MUST be fixed by §4 (add a cpu→RecommendCPU branch) before CPU rows are usable; §4 is next. - catalog.go accelerator_type filter is user-param only, doesn't drop cpu rows. Applied + verified live: 17 rows, idempotent (ON CONFLICT), r8g.16xlarge = 64 vCPU/512 GiB/count 0. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Jeremy Cowan <jicowan@hotmail.com>
…RD-67 §4)
New RecommendCPU (internal/recommend/cpu.go), a SEPARATE path from the GPU
recommender because the memory model differs: CPU has no VRAM and no
gpu_memory_utilization; host DRAM IS the serving memory and the KV cache is a
fixed absolute allocation (VLLM_CPU_KVCACHE_SPACE). Fit is recomposed on the
host-memory map:
weights(quantized) + kv_cache_space + activation <= node_RAM * hostAllocatableFrac
- Reuses modelMemoryBytes / kvCachePerTokenBytes / hostAllocatableFrac /
inferenceOverheadBytes / isUnsupportedModelKind / roundDownContext — recomposition,
not new physics.
- TP = NUMA-node count via cpuNumaNodes(), which returns 1: Graviton is
single-socket, so TP=1 and intra-node parallelism comes from OMP thread
binding, NOT tensor-parallel ranks. NOT derived from AcceleratorCount (0).
- dtype always bfloat16 (float16 unstable on torch CPU); no fp8 KV.
- §11c: refuses DENSE UNQUANTIZED models above ~13B, steering to W8A8/W4A8 with
a quantization alternative; warns 8-13B. Pre-quantized checkpoints keep their
format. Loud memory-bandwidth advisory on every rec.
- CPUKVCacheSpaceGiB exported so the manifest (§3) sets VLLM_CPU_KVCACHE_SPACE to
the exact value the recommender budgeted.
Fixes the ÷0 the §2 audit flagged: the recommender dispatch (handlers.go:1338,
handlers_estimate.go:215, seeder.go:221) was `if neuron else GPU`, so a CPU
instance fell into the GPU Recommend and divided AcceleratorMemoryGiB by
AcceleratorCount=0. All three now have a `cpu` branch → RecommendCPU. Also guard
handleMemoryBreakdown (the GPU-VRAM breakdown, which ÷0'd at line 125) to 400 for
CPU instances — the VRAM breakdown is a GPU-shaped framing that doesn't apply.
Tests: fit on r8g.16xlarge, oversized-unquantized refusal + quant alternative,
large INT4 MoE fits the 1.5 TiB box, doesn't-fit small host, embedding rejection,
Graviton generation mapping, KV-cache-space clamping.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Jeremy Cowan <jicowan@hotmail.com>
…§3, §8) Add a third accelerator branch (cpu) to BOTH the orchestrator's live template (model-deployment.yaml.tmpl) AND the export template (handlers_export.go), so a CPU run deploys — and re-exports — correctly instead of falling into the neuron branch (wrong device resource, neuron toleration, 540 startup threshold). CPU branch in both: - tolerations: accelbench.io/cpu (no device-plugin taint); nodeSelector adds kubernetes.io/arch: arm64. - env: VLLM_CPU_KVCACHE_SPACE (absolute GiB, from recommend.CPUKVCacheSpaceGiB sized off host RAM) + VLLM_CPU_OMP_THREADS_BIND=auto. - resources: request/limit cpu + memory only, NO nvidia.com/gpu or aws.amazon.com/neuron device resource; shm volume for the streamer/TP staging. - securityContext: SYS_NICE cap + seccomp Unconfined (NUMA thread pinning, per vLLM CPU docs). - startupProbe: GPU-style threshold 120 (weight-load dominated, no neuron 540 compile wait). Refactored the probe conditional to `if neuron 540 else 120` so cpu joins gpu — gpu=120 and neuron=540 unchanged. - serve args: reuse VLLMcpu.BuildArgs via the RuntimeArgs path (orchestrator already; export now routes cpu through BuildArgs like sglang) so the exact deployed flags (forced bfloat16, TP, W8A8/W4A8, streamer w/ distributed gated off) are reproduced, not re-encoded in the template. Plumbing: ModelDeploymentParams.CPUKVCacheSpaceGiB + manifestData.RuntimeImage/ CPUKVCacheSpaceGiB; orchestrator cpuKVCacheSpaceGiB() helper (0 for non-cpu); export resolves the vllm-openai-cpu image (VLLM_CPU_IMAGE override → composed from VLLMCPUVersion+dockerhub PTC) and adds RunExportDetails.VLLMCPUVersion (empty until §10 wires the tool_versions column; runtime falls back to DefaultVLLMCPUVersion). Tests: CPU render (arm64/cpu-resources/kvcache-env/sys_nice/bfloat16, and NO device resource / NCCL / gpu-mem-util / 540) + CPU export (same, NOT neuron branch); gpu/neuron render + single-node export regression tests unchanged. NOTE: version source — lifecycle.go still passes FrameworkVersion to the image resolver; the CPU-specific VLLMCPUVersion source is wired in §10. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Jeremy Cowan <jicowan@hotmail.com>
Gated behind install_cpu_nodepool (default off, root + module vars). When on: - data.aws_ssm_parameter.cpu_arm64_ami: the standard EKS-optimized AL2023 ARM64 AMI (no accelerator variant), pinned via SSM like the gpu AMI. Ships containerd 2.2 + soci-snapshotter, so the FastImagePull userData works. - cpu EC2NodeClass: arm64 AMI, 500Gi gp3 root (large quantized MoE + streamer cache + image layers), and the shared soci_user_data (FastImagePull gate). - cpu NodePool: Graviton3/4/5 families (r8g/m8g/c8g, r7g/m7g/c7g, m9g/c9g/r9g), arch arm64, reserved+on-demand, WhenEmpty/1m disruption. NO device plugin. Taints (accelbench.io/cpu + accelbench.io/dedicated) are load-bearing beyond isolation: they keep the GPU/Neuron DaemonSets OFF arm64 nodes. Verified against the live cluster — nvidia-device-plugin, dcgm-exporter, neuron-device-plugin tolerate nvidia.com/gpu / aws.amazon.com/neuron + accelbench.io/dedicated but NOT accelbench.io/cpu, and their nodeAffinity requires accelerator labels a CPU node lacks, so they're double-blocked. The critical system DaemonSets (aws-node, kube-proxy, eks-pod-identity-agent, ebs-csi-node) tolerate all taints and are AWS multi-arch, so they run on arm64. Net: no NVIDIA driver / DCGM on CPU nodes. Applied live (install_cpu_nodepool=true): cpu EC2NodeClass Ready=True (arm64 AMI resolved for k8s 1.36), cpu NodePool Ready (0 nodes, provisions on demand), terraform plan clean. (The DCGM *scraper* orchestrator path is skipped for CPU separately in §7.) Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Jeremy Cowan <jicowan@hotmail.com>
Metrics: the DCGM/GPU scraper is ALREADY gpu-gated in the orchestrator (lifecycle.go: `if AcceleratorType == "gpu"`), so a CPU run skips it by construction — no change needed. The accelerator-agnostic host-memory scraper + vLLM /metrics (throughput/TTFT/TPOT) already cover the CPU-relevant signal. Added TestDeployModel_CPUInstance asserting a CPU run deploys the arm64 Deployment (VLLM_CPU_KVCACHE_SPACE env, arm64 nodeSelector, NO nvidia.com/gpu / aws.amazon.com/neuron device resource) and thus skips DCGM. Frontend: - FilterBar: add "cpu" to the accelerator filter chips. - Run.tsx: add a "CPU (ARM/Graviton)" optgroup to the instance dropdown (shows vCPU/memory, not count×name, since count=0); force framework vllm-cpu when a cpu instance is selected — derived from the instance's accelerator_type (robust across Graviton families) rather than a name regex, and reset off vllm-cpu/vllm-neuron when switching back to GPU. - Estimate.tsx: add "CPU (ARM) only" filter option. - ResultDetail.tsx: acceleratorNoun renders "CPU" for cpu runs; per-accelerator throughput already guards count>0 → blank (no ÷0), consistent with the backend compare.go guard. go build/vet/test + frontend tsc + build all green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Jeremy Cowan <jicowan@hotmail.com>
…ce (PRD-67 §10) Add vllm_cpu_version to tool_versions — the ARM/CPU image tag (vllm/vllm-openai-cpu:<ver>-arm64), DISTINCT from framework_version (different repo, own arm64 cadence), mirroring PRD-66's llmd_version/pd_vllm_version. - Migration 044: ADD COLUMN vllm_cpu_version DEFAULT 'v0.11.2' (= DefaultVLLMCPUVersion). - DB: ToolVersions.VLLMCPUVersion + GetToolVersions/PutToolVersions SQL + mock default. - Config API: GET/PUT round-trip the field (+ VLLM_CPU_IMAGE env-override status, which does NOT fall back to VLLM_IMAGE — the CPU runtime reads only VLLM_CPU_IMAGE), "keep existing" on empty, and audit line. - Configuration.tsx: "vLLM CPU Version (ARM/Graviton)" input in the Tool Versions panel; types.ts + api.ts request/response types. Wire the version SOURCE (the loose end from §3): CreateRun's version resolution (handlers.go) now passes the FULL tool_versions projection into rt.ResolveVersion, so vllm-cpu resolves tv.VLLMCPUVersion (and llm-d resolves LLMDVersion) instead of only framework_version/sglang_version. A CPU run's Request.FrameworkVersion is thus the vllm-openai-cpu tag, which lifecycle.go's DefaultImage composes correctly — no lifecycle change needed. Export path: injectMultinodeImageVersions also injects VLLMCPUVersion so exports name the configured CPU image. Applied migration 044 live (column defaults v0.11.2). go build/vet/test + frontend tsc/build green. Live API round-trip validated after the §10 redeploy. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Jeremy Cowan <jicowan@hotmail.com>
…en model name (PRD-67) First end-to-end live r8g CPU run surfaced three integration bugs (all green after these fixes: 600/600 requests, 586 tps, TTFT p50 217ms on r8g.16xlarge): 1. Image tag didn't exist. DefaultVLLMCPUVersion was v0.11.2 (the prebuilt-Arm- *wheels* "since" figure, misapplied to the image) → ImagePullBackOff "not found". Docker Hub vllm/vllm-openai-cpu arm64 images start at v0.18.0; pin v0.27.0. Updated migration 044 default + mock + tests. 2. Streamer not bundled in the CPU image. The prebuilt vllm/vllm-openai-cpu:*-arm64 image does NOT ship runai-model-streamer (crash: ModuleNotFoundError), unlike the GPU image — resolving PRD-67 §5b's open item. Gate the streamer off for cpu in resolveS3Model → HF-download fallback (forgoes S3 cache until a streamer-inclusive CPU image / init step; future work). 3. Loadgen model-name drift → 404 on every request. launchLoadgen re-derived the served model name from the model cache independently and set it to the S3 URI, but the CPU server (HF-download) registered the HF id → 404s. Now derives from the same resolveS3Model source of truth (honors the CPU gate). Eliminates the deploy/loadgen drift for all accelerators. Live-verified: vLLM V1 on device_config=cpu, forced bfloat16, VLLM_CPU_OMP_THREADS_ BIND=auto binding threads, NUMA MEMBIND, TP=1 — exactly as designed. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Jeremy Cowan <jicowan@hotmail.com>
Two complementary mechanisms so a user (UI) or API caller can't set a GPU-only knob on a CPU run. §11b/c server guards (CreateRun, handlers.go) — when instance is cpu: - kv_cache_dtype=fp8 → 400 (fp8 KV is CUDA/ROCm only, confirmed). - deployment_mode distributed/disaggregated → 400 (CPU is single-node). - tensor_parallel_degree > NUMA-node count → 400 (recommend.CPUNumaNodes, =1 on single-socket Graviton; TP maps to NUMA nodes, parallelism is OMP thread-bind). - §11c: oversized DENSE UNQUANTIZED model → 400 steering to W8A8/W4A8, using the SAME ceiling as RecommendCPU (recommend.CPUUnquantizedCeilingParams, ~13B). Fetches model config only for CPU runs (no HF call added to the GPU hot path); best-effort. Small bf16 models still allowed. (gpu_memory_utilization is not a request field — nothing to reject.) §11a UI gating (Run.tsx): derive isCPU from the selected instance's accelerator_type; hide the KV Cache Dtype (fp8) field on CPU AND clear it from the payload on switch-to-CPU so a stale value can't trip the server guard. A CPU fit failure is treated as host-mem/bandwidth (overridable), not a GPU-VRAM hard block. (The distributed/multi-node section lives on Distributed.tsx, not the Run page, so nothing to hide there.) Tests: table-driven CreateRun guards (fp8→400, distributed/disaggregated→400, TP>NUMA→400) + the mirror proving the SAME knobs still pass on a GPU instance (CPU-scoped, not a global regression). go vet + full backend suite + frontend tsc/build green. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Jeremy Cowan <jicowan@hotmail.com>
Reports & cross-tier comparison need no code change — the machinery was already CPU-safe by construction: - perGPU() guards AcceleratorCount<=0 → blank throughput_per_gpu_tps (no ÷0). - DCGM columns (sm/tensor/dram active, accel memory) are *float64 pointers, nil on a CPU run (never scraped) → fmtPtr renders them BLANK, not 0. - cost + aggregate throughput + TTFT depend only on hourly price + vLLM /metrics (accelerator-agnostic) → populate for CPU. - ResultDetail acceleratorNoun already renders "CPU" (§7). Added TestGenerateCompareCSV_CPUvsGPU: a CPU row (count 0, nil DCGM) vs a GPU row — asserts throughput_tps/ttft/cost_per_1m_tokens populate for CPU while throughput_per_gpu_tps + sm_active render blank, and accelerator_count=0. Pricing: cmd/pricingrefresh enumerates ALL instance types (ListInstanceTypes, now incl. the seeded Graviton rows) and queries the AWS Pricing API by name with NO architecture filter — so Graviton prices populate once the pricing CronJob runs. The live "no pricing row" was just the CronJob not having run since seeding, not a code gap; triggered a one-off refresh. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Signed-off-by: Jeremy Cowan <jicowan@hotmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Adds a CPU (ARM/Graviton) accelerator tier alongside GPU and Neuron (PRD-67):
vllm-cpuruntime (internal/runtime/vllm_cpu.go) targeting thevllm/vllm-openai-cpu:*-arm64images, with NUMA-aware TP andVLLM_CPU_KVCACHE_SPACEsizing.cpuKarpenter NodePool with theaccelbench.io/cputaint, and a settablevllm_cpu_versionin tool versions.Live-validated on
accelbench-eks: Qwen2.5-1.5B-Instruct on r8g.16xlarge, 600/600 requests (PRD-67 §1–§12).Test plan
go test ./... && go vet ./...cd frontend && npx tsc --noEmit && npm run build🤖 Generated with Claude Code