This folder is for people who want only a model endpoint: an OpenAI-compatible HTTP server on
Beverin compute nodes (AMD MI300A gfx942 APUs, 4 GPUs per node, Slurm, CSCS Container Engine). No
benchmark, no grading, no agents. Numbers here do not carry to discrete GPUs, and several do not
carry to MI300X.
| Page | Covers |
|---|---|
qwen38.md, kimi27sglang.md, glm53.md, oss120b.md |
one model each: configuration, DO / DO NOT, the numbers behind them |
known-issues.md |
problems we hit on Beverin (RCCL pipeline crash, fuse-overlayfs freeze, garbled output), with evidence and status |
knobs.md |
cross-model: APU memory model, KV pool threshold, aiter derate, HiCache, fabric, Slurm shape |
private-endpoint.md |
a keyed Qwen3.8 server only you can use, from your laptop or your own Daint jobs |
mi200-endpoint.md |
the same server in BF16 on an mi200 node, and the one-command ping test that checks it from anywhere |
extending-private-inference.md |
contributors: the private launcher's security contract, new presets, access paths, engines |
The authoritative launch line per model is the render of experiment:<model>
(hpcagent_bench/cluster/env_layers.sh render experiment:<model>). If a page here and the render disagree, the
render wins.
Once per account, register the EDFs (container definitions) and set your Slurm account:
cd "$REPO"
containers/images/install_edfs.sh # renders into ~/.edf
sbatch containers/images/registry.sbatch pull <role> # only if install_edfs.sh reports a missing image
export SBATCH_ACCOUNT=<project>; . hpcagent_bench/cluster/env.sh # Beverin rejects jobs without an accountThen:
S=hpcagent_bench/cluster/serve-only.sbatch
SUBMIT=0 $S # print what would be submitted
$S # Qwen3.8 (default MODEL)
MODEL=kimi27sglang $S # any experiments/layers/model-<MODEL>.env: qwen38, kimi27sglang, glm53, oss120b
SERVE_TIME=08:00:00 $S # longer than the 4 h default (SBATCH_TIMELIMIT is overridden)serve-only.sbatch renders experiment:<MODEL> (SERVE_BASE_ENV overrides), reads the node count from it, submits itself with
that --nodes, starts the server, polls /v1/models, then prints:
===== endpoint is live =====
base URL: http://<node>:8000/v1
model name: hpcagent-bench-vllm
...
The endpoint takes no API key. It stays up until this job ends; scancel <jobid> to stop it.
followed by a ready-to-paste curl. The job output is serve-only-<jobid>.out; server logs are
$SCRATCH/inference-server/<jobid>/server-<rank>.log, and serve.env there is the merged env that
actually ran.
serve-only.sbatch launches the inference role's derived EDF, as an experiment does: a single-node or replica
server (one inference node, or INFERENCE_MODE=replicas) runs without the cxi / aws-ofi-nccl hooks and the forced
NCCL_NET, with which its tensor-parallel group fails at init (Failed to initialize any NET plugin); a
pipeline-parallel server keeps them. See experiments/README.md.
An EDF is a TOML file in ~/.edf naming the image, bind mounts, environment and CE hooks. The CE does
not reliably keep the image's own ENV, so the EDFs re-declare PATH, LD_LIBRARY_PATH and cache
dirs. The fabric comes from three pinned hooks (netstack, cxi, aws_ofi_nccl); without the RCCL
plugin, multi-node RCCL silently falls back to TCP. The netstack hook also writes CSCS's NCCL and libfabric
defaults into the container after the step environment (NCCL_NET_GDR_LEVEL=PHB, NCCL_MIN_NCHANNELS=112,
FI_CXI_RDZV_*, ...), so a value we need different is set on the server command in run_cluster.sh, not in a
layer alone (known issue 5).
| EDF | Engine | Models | Rendered by install_edfs.sh |
|---|---|---|---|
hpcagent-bench-sglang-mi300-latest |
SGLang 0.5.20 | Qwen3.8, Kimi K2.7, GLM-5.3 | yes |
hpcagent-bench-vllm-mi300-latest |
vLLM 0.28.0 | gpt-oss-120b | yes |
hpcagent-bench-vllm-mi200-latest |
vLLM 0.28.0 (same image) | Qwen3.8 on mi200 | yes |
serve-only.sbatch already sets all of this. Copy it if you write your own launcher.
| Setting | Why |
|---|---|
--partition=mi300 |
every recipe here is MI300A-only |
no -A |
Slurm reads SBATCH_ACCOUNT; naming one yourself splits identical jobs across accounts |
--mem=0 |
otherwise the step's memory cgroup follows its CPU share and the server dies in weight load |
--gpus-per-node=4, --ntasks-per-node=1 |
every recipe is tp=4 inside a node |
--cpus-per-task="${SLURM_CPUS_ON_NODE}" on the server step |
see below |
ulimit -c 0 |
machine-global core_pattern drops multi-GB core files in the CWD; helpers/scripts/checks/check_core_dumps.py enforces it |
The CPU trap. --exclusive gives the job the node, not the step its CPUs. A step without
--cpus-per-task gets one core plus its SMT sibling (2 of 192). A starved server does not crash, it
degrades with load: 147 s per decode step after half an hour, against 88-91 tok/s for the same model
with --cpus-per-task=32. It can also hang in Triton JIT until the 600 s RCCL watchdog kills every
rank. Give a client or probe running alongside one socket: --cpus-per-task=24 --hint=nomultithread.
The server binds 0.0.0.0:8000 on the first node of the allocation. No gateway, no API key:
every Alps user can reach it. For a private server use private-endpoint.md.
squeue -u "$USER" -n serve-only -o '%i %T %N'
BASE=http://<node>:8000
curl -s "$BASE/v1/models"
curl -s "$BASE/v1/chat/completions" -H 'Content-Type: application/json' \
-d '{"model":"hpcagent-bench-vllm","max_tokens":128,"messages":[{"role":"user","content":"Say hi."}]}'
curl -s "$BASE/metrics" # Prometheus: throughput, running/waiting requests, cache hitsThe served name is hpcagent-bench-vllm for every model, not the HuggingFace id.
- Multi-node (Kimi K2.7, GLM-5.3,
pp=4): only rank 0 binds the port; talk to the first node. - Single-node models on several nodes: independent replicas, one per hostname, each with its own KV cache. The client spreads load itself.
| Model | MODEL= |
Engine | Nodes | Page |
|---|---|---|---|---|
Qwen/Qwen3.8-27B-FP8 |
qwen38 |
SGLang | 1 | qwen38.md |
moonshotai/Kimi-K2.7-Code |
kimi27sglang |
SGLang | 4 (pp=4) |
kimi27sglang.md |
zai-org/GLM-5.3 |
glm53 |
SGLang | 4 (pp=4) |
glm53.md |
openai/gpt-oss-120b |
oss120b |
vLLM | 1 | oss120b.md |
The engine is per model and partition: on mi300, Qwen3.8 on vLLM (ROCM_AITER_FA) garbles its output under
agent load and Kimi K2.7 on vLLM collapses above concurrency 1. On mi200 (MI250X, BF16) the SGLang base has no
kernels, and Qwen3.8 serves on the AMD vLLM image (layers/hardware-mi200-qwen38.env, 421 tok/s at
16 concurrent requests).
Readiness. serve-only.sbatch waits AGENT_READY_TIMEOUT_SECONDS, else
VLLM_READY_TIMEOUT_SECONDS, else 7200 s, the precedence the experiment's agent driver uses. Each model's layer
chain sets AGENT_READY_TIMEOUT_SECONDS (layers/replicas.env: 2400 s, with VLLM_READY_TIMEOUT_SECONDS 2400 s;
model-kimi27sglang.env: 7200 s; model-glm53.env: 10800 s). The rendered layers are sourced after the command
line, so a value on the command line is overwritten. To override, copy experiments/serve-only.env, add the key,
and pass SERVE_ENV_FILE=<copy> (sourced last, so it wins). The same applies to
VLLM_SERVED_MODEL.
SGLang startup, in order:
grep -aE "Load weight (begin|end)|Cache is allocated|max_total_num_tokens|Capture|fired up" server-0.log| Line | Meaning |
|---|---|
Load weight begin. avail mem=... |
free memory before weights (host memory on an APU) |
Load weight end. elapsed=... mem usage=... |
this rank's weight share |
Mamba Cache is allocated. max_mamba_cache_size: N |
Qwen3.8 state slots; below planned concurrency is a problem |
KV Cache is allocated. #tokens: N / max_total_num_tokens=N |
the whole prefix-cache budget: the number that matters |
Capture ... CUDA graph |
graph capture, takes memory after KV sizing |
The server is fired up |
answering HTTP |
Under load, a prefix-cache hit rate that falls as conversations grow means the KV pool is
thrashing (knobs.md). For vLLM, grep
Loading|KV cache|Capturing|Application startup complete; Available KV cache memory is the pool.
| Symptom | Cause | Fix |
|---|---|---|
log stops after Load weight begin, no traceback |
host OOM; KV cache is host memory | lower --mem-fraction-static, or more nodes |
minimum viable = ... at startup |
weights alone exceed the fraction at this node count | more nodes, not a bigger fraction |
| API never answers, log ends mid-JIT | CPU trap | --cpus-per-task |
| tool calls arrive as prose, or a 400 logged as success | only one parser named | pass --reasoning-parser and --tool-call-parser |
| RCCL watchdog abort after ~600 s | peer blocked: CPU trap or fabric fallback | check CPUs, then Using network |
wrong numbers across nodes, no error; or regattr. RC: -22 at the first cross-node send |
GPU-direct RDMA on this fabric | NCCL_NET_GDR_LEVEL=LOC on the server command (known issue 5) |
| multi-node throughput several times low | RCCL on TCP | grep -a "Using network" server-0.log must say AWS Libfabric |
Absent key vs empty key. run_cluster.sh reads ${SGLANG_ATTENTION_BACKEND-aiter} (dash, not
colon-dash):
| Env file says | ${VAR-default} |
${VAR:-default} |
|---|---|---|
| nothing | default |
default |
VAR= |
empty: flag omitted | default |
VAR=x |
x |
x |
Deleting a key turns the default on. To omit a flag, assign it empty (GLM-5.3 does this). When templating a flag, use the dash form.
Layers, last assignment wins. layers/common.env < setups.yaml experiment < layers/model-<m>.env < setups.yaml models.<m>
(Env layers); a layer can override a key, never unset it.
Render one with hpcagent_bench/cluster/env_layers.sh render experiment:<m>. serve-only.sbatch sources the render, then
serve-only.env (zero judge and agent nodes, RUN_ROOT), under set -a.
Setup .env.<setup> files are renders. Fix the layer that owns a key, never the render.
Mounts. An experiment job narrows the inference container's mounts; serve-only.sbatch uses the
registered EDF as-is. A model that serves here and fails in an experiment run: suspect the mounts first.
hpcagent_bench/cluster/serve-only.sbatch,experiments/serve-only.env: the launcher on this page.experiments/layers/model-<m>.env,experiments/setups.yaml: per-model launch lines with inline reasons.containers/inference/:accuracy-gate.py,verify-tools-reasoning.py(gates against a live server).
Node-to-node spread is about 30%. Re-measure a flag change on one node, back to back.