Skip to content

feat(energy): GPU energy meter with pluggable power models, validated on ML.ENERGY H100 data - #2

Open
therealansh wants to merge 1 commit into
sbu-fsl:mainfrom
therealansh:energy-model
Open

therealansh wants to merge 1 commit into
sbu-fsl:mainfrom
therealansh:energy-model

Conversation

@therealansh

Copy link
Copy Markdown

Summary

This PR adds GPU energy accounting to blis run and blis replay. A power model converts every step the engine prices into watts per GPU, and the meter integrates those watts over time. I compared two power models against measured H100 power from the ML.ENERGY benchmark. As per the gate fixed before any prediction, model A (utilization curve, from Vidur-Energy) is selected.

Design

  • sim.PowerModel has one method, StepPower(StepSample) float64, which returns watts per GPU.
  • StepSample carries the step time, prefill tokens, decode tokens, batch size and the per-resource busy time (when the engine gives it).
  • sim.EnergyMeter sits in sim.EngineDriver.timeBatch and records every batch priced by StepTime. Time between steps is charged at idle_w.
  • sim/energy holds the models and a strict YAML loader. A new model needs only one type and one case.
  • New flags: --energy-config, --energy-report (JSON) and --energy-trace (one CSV row per step).
  • With energy off, nothing changes. With energy on, all Result metrics stay the same; only input.options records the new flags.

Energy (J) = Σsteps P × tstep × GPUs + idle_w × GPUs × tidle. Facility energy = PUE × GPU energy, and gCO2 = kWh × grid intensity.

Design comparison

A: utilization B: resource-share TDP baseline Constant baseline
Power per GPU P_floor + (P_max − P_floor)·min(1, MFU/MFU_sat)^γ Σ_r (t_r / Σt) · W_r 700 W mean of calibration set
Inputs tokens, step time, active params, peak FLOPS kernel busy time per resource (SM, HBM, NVLink, NIC, PCIe, copy, host) none none
Source Vidur-Energy, Eq. 1–2 [1] constants from [2], [3] H100 datasheet —
Fitted params 4 3 0 1
Held-out power MAPE 13.1% (study 1), 13.2% (study 2) 33.3% (study 1) 35.8% 22.4%
Runs on the blis-vllm 0.32 engine Yes No: the engine does not expose per-resource busy time Yes Yes

B's weights sum to one over resources, so it cannot show compute and memory both busy in the same step. Even with weights bounded to 0–1000 W (post-hoc, not part of the gate), it needs W_sm = 1000 W, which is above TDP, to follow the measured rise in power.

Method

  • Ground truth: ML.ENERGY leaderboard, lm-arena-chat, H100 SXM, vLLM, BF16, steady state [4], [5]. The data is pinned to commit 2bdd007 with sha256 checks, downloaded at run time and not committed.
  • Points: 36 configurations. 11 are for calibration (Llama-3.1-8B, TP1). 25 are held out: Qwen3-14B (TP1), Llama-3.1-70B (TP4) and Qwen3-30B-A3B MoE (TP1).
  • Simulation: each point uses its measured max_num_seqs and output-length histogram, with a burst arrival. The steady-state window is steps with batch ≥ 0.9 × the measured average batch. Prompt length is 512 tokens, and 256 and 1024 are checked for sensitivity.
  • Gate (fixed before data):
    • the lowest calibrated held-out power MAPE wins;
    • within 2 points, fewer parameters wins;
    • the winner must beat both baselines.
  • Study 1 ran on inference-sim 29f783c7 (kernel adapter v0.3.0), where B can be scored. Study 2 ran on this branch (blis-vllm 0.32 engine) and covers A and the baselines.

Results

Regime Model Held-out power MAPE Held-out J/token MAPE Rank ρ
Zero-shot TDP 700 W 35.8% 73.6% 0.998
Zero-shot A 39.1% 39.5% 0.993
Calibrated Constant 22.4% 56.0% 0.998
Calibrated A 13.2% 46.1% 0.998

These are study 2 numbers; study 1 is in experiments/energy/findings.md.

The J/token error comes mostly from step time, not power:

Quantity (study 2) MAPE
A power 13.2%
Simulated ITL vs measured 24.6%
Simulated throughput vs measured 19.1%
A J/token 46.1%

Fitted A for H100: P_floor 436 W, P_max 689 W, MFU_sat 0.15, γ 0.75.

Over-provisioning (E3): Llama-3.1-8B, 20 req/s, 4,000 requests, PUE 1.2, 400 gCO2/kWh.

Instances Facility Wh J/request gCO2 p99 TTFT
1 45.0 33.7 18.0 688 ms
2 77.7 58.3 31.1 277 ms
4 139.5 104.6 55.8 214 ms
8 259.9 194.9 103.9 210 ms

Going from 1 to 8 instances uses 5.8× the energy for a 3.3× better p99 TTFT. Idle gaps stay below 0.2% of energy, because a GPU decoding small batches still draws about 65–70% of TDP.

Parallelism (E4): Llama-3.1-70B, 1,024-request burst.

Layout J/output token J/request Mean ITL
TP2 1.835 704.7 60.6 ms
TP4 0.734 281.9 63.6 ms
TP8 1.006 386.4 42.4 ms

E2 (workload shape) and the sensitivity tables are in experiments/energy/findings.md.

Not supported yet (refused, not under-counted)

  • data-parallel groups (dp > 1);
  • P/D disaggregation;
  • pipeline parallelism (pp > 1);
  • resource_share on an engine without sim.StepBreakdown.

Only GPU-board energy is counted. Host CPU, DRAM and network are not counted, and asynchronous KV offload transfers are charged at idle power.

Testing

  • go test ./..., go vet and gofmt pass. Unit tests cover the energy laws (additive, idle clamp), prefill/decode split, refusals and YAML validation. ./sim/energy is added to the CI test matrix as a new entry.
  • With energy off, the Result document matches an unmodified build, apart from the wall-clock runtime block. With energy on, every metric section matches.
  • The energy report and trace are deterministic across runs. The Python harness matches the Go meter to 5.5e-7.
  • python3 experiments/energy/run.py gate reproduces the tables in about 30 s.

References

  1. Özcan et al., Quantifying the Energy Consumption and Carbon Emissions of LLM Inference via Simulations, arXiv:2507.11417, 2025.
  2. Patel et al., Characterizing Power Management Opportunities for LLMs in the Cloud, ASPLOS 2024.
  3. Wilkins, Kazhamiaka, Rajagopal, From Servers to Sites: Compositional Power Trace Generation of LLM Inference, arXiv:2603.18383, 2026.
  4. Chung et al., The ML.ENERGY Benchmark, NeurIPS 2025 D&B, arXiv:2505.06371.
  5. ML.ENERGY leaderboard data, github.com/ml-energy/leaderboard @ 2bdd0070186956688e1f842dc28077980fe64013.

Kindly review.

🤖 Generated with Claude Code

Adds GPU energy accounting to run and replay. A one-method sim.PowerModel prices each batch
sim.EngineDriver times with engine.Engine.StepTime, in watts per GPU, from the step time,
prefill/decode tokens, batch size and (when the engine exposes sim.StepBreakdown) per-resource
busy time. Time between steps is charged at idle_w. With energy off nothing changes; with it
on every Result metric section is unchanged and only input.options records the flags.

Models (sim/energy, strict YAML): tdp, utilization (Vidur-Energy, arXiv:2507.11417 Eq. 1-2),
resource_share (time-weighted per-resource power; refused on an engine without a breakdown,
which the blis-vllm 0.32 engine is). Flags: --energy-config, --energy-report, --energy-trace.
Data-parallel groups, P/D and pipeline parallelism are refused rather than under-counted (R1).

experiments/energy/run.py scores the models against the ML.ENERGY leaderboard
(github.com/ml-energy/leaderboard @ 2bdd0070186956688e1f842dc28077980fe64013, sha256-pinned,
fetched at run time, not committed): calibrated on Llama-3.1-8B, scored on Qwen3-14B,
Llama-3.1-70B TP4 and Qwen3-30B-A3B. Held-out power MAPE on this base: utilization 13.2%,
constant mean 22.4%, TDP 35.8%. The A-vs-B comparison (resource_share 33.3%) ran on the
kernel-adapter base 29f783c; both are in experiments/energy/findings.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant