Skip to content

[kernrec] Keep the ROCm recorder off the AMD TurboQuant LM Eval steps - #912

Merged
AndreasKaratzas merged 1 commit into
vllm-project:mainfrom
AndreasKaratzas:kernrec-amd-skip-turboquant
Nov 14, 2026
Merged

AndreasKaratzas merged 1 commit into
vllm-project:mainfrom
AndreasKaratzas:kernrec-amd-skip-turboquant

Conversation

@AndreasKaratzas

Copy link
Copy Markdown
Member

Summary

When kernrec is armed on AMD, the TurboQuant LM Eval server sometimes segfaults inside rocprofiler-sdk. Kernrec's KERNEL_DISPATCH buffer tracing makes the SDK swap each dispatch's completion signal for a pooled one. On the image's SDK (1.3.5), the AsyncSignalHandler can hand that signal back while HIP is still using it:

SIGSEGV in rocprofiler::common::container::pool<signal_t>::release <- Queue::release_signal <- AsyncSignalHandler

Both hits came right after the first mid-serving Triton JIT of _tq_full_dequant_kv, and both passed on auto-retry:

Over the last 7 days the segfault showed up 2 times in about 58 TurboQuant runs on the 14 kernrec-armed main builds. It showed up 0 times in about 228 runs on unarmed main builds and 0 times in 688 PR-branch runs. Across all 2,717 failed AMD job logs since 11-06, this signature appears only in these two jobs.

The upstream fix is ROCm/rocm-systems#14385 ("fix(rocprofiler-sdk): preserve dispatch timing signal lifetime", ROCM-32928). It is still open.

Change

  • New KERNREC_AMD_SKIP_STEP_PREFIXES next to KERNREC_SKIP_STEPS. Its one entry is lm-eval-turboquant-, and _kernrec_applies checks it only for the amd setup profile.
  • The AMD mirrors of lm-eval-turboquant-{k8v4,t4nc,k3v4nc,t3nc} now run without the recorder: no setup, no .kernrec artifacts, no recording timeout margin, and the collect step does not wait for them. Their CUDA steps keep recording. As with KERNREC_SKIP_STEPS, a step with no recording keeps being selected, so the only cost is lost savings on those 4 steps.
  • Scope is TurboQuant only, because that is the only place the segfault has been seen. Other long AMD eval-server steps can be added with a one-line prefix if it shows up there.
  • Remove the entry once the vLLM ROCm image ships a rocprofiler-sdk that contains #14385. That bump also needs kernrec's 1.3.2/1.3.5 version gate updated.

Testing

  • uv run pytest buildkite/tests/pipeline_generator: 566 passed, 2 skipped. The new test test_amd_skip_prefix_unarms_only_the_amd_mirror fails without the change.
  • pre-commit run --files on both files: everything passes except ruff-format. That hook reformats four lines that already fail it on main and that this PR does not touch, so those lines are left out of this diff.

AI assistance (Claude Code) was used to analyze the CI logs and to draft this change.

kernrec's KERNEL_DISPATCH tracing makes rocprofiler-sdk swap every
dispatch's completion signal for a pooled one. On the image's SDK
(1.3.5) the AsyncSignalHandler can release that signal while HIP still
uses it, and the TurboQuant eval server segfaults in
pool<signal_t>::release right after its first mid-serving Triton JIT
(vllm/ci #101909, #102057). That is 2 failures in about 58 recorded
TurboQuant runs on main and 0 in about 900 unrecorded runs (main and
PR builds) over 7 days. Upstream fix: ROCm/rocm-systems#14385.

Add KERNREC_AMD_SKIP_STEP_PREFIXES so the AMD mirrors of the
lm-eval-turboquant-* steps run unrecorded, while their CUDA steps keep
recording. As with KERNREC_SKIP_STEPS, a step with no recording just
keeps being selected.

Co-Authored-By: Claude <noreply@anthropic.com>
Signed-off-by: Andreas Karatzas <akaratza@amd.com>
@AndreasKaratzas
AndreasKaratzas marked this pull request as ready for review November 14, 2026 00:21
@AndreasKaratzas
AndreasKaratzas merged commit 604b87e into vllm-project:main Nov 14, 2026
3 checks passed
@AndreasKaratzas
AndreasKaratzas deleted the kernrec-amd-skip-turboquant branch November 14, 2026 00:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant