Repository navigation
[kernrec] Keep the ROCm recorder off the AMD TurboQuant LM Eval steps - #912
Merged
AndreasKaratzas merged 1 commit intoNov 14, 2026
Merged
AndreasKaratzas merged 1 commit into
AndreasKaratzas merged 1 commit into
Conversation
kernrec's KERNEL_DISPATCH tracing makes rocprofiler-sdk swap every dispatch's completion signal for a pooled one. On the image's SDK (1.3.5) the AsyncSignalHandler can release that signal while HIP still uses it, and the TurboQuant eval server segfaults in pool<signal_t>::release right after its first mid-serving Triton JIT (vllm/ci #101909, #102057). That is 2 failures in about 58 recorded TurboQuant runs on main and 0 in about 900 unrecorded runs (main and PR builds) over 7 days. Upstream fix: ROCm/rocm-systems#14385. Add KERNREC_AMD_SKIP_STEP_PREFIXES so the AMD mirrors of the lm-eval-turboquant-* steps run unrecorded, while their CUDA steps keep recording. As with KERNREC_SKIP_STEPS, a step with no recording just keeps being selected. Co-Authored-By: Claude <noreply@anthropic.com> Signed-off-by: Andreas Karatzas <akaratza@amd.com>
AndreasKaratzas
marked this pull request as ready for review
November 14, 2026 00:21
Merged
6 tasks done
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
When kernrec is armed on AMD, the TurboQuant LM Eval server sometimes segfaults inside rocprofiler-sdk. Kernrec's
KERNEL_DISPATCHbuffer tracing makes the SDK swap each dispatch's completion signal for a pooled one. On the image's SDK (1.3.5), theAsyncSignalHandlercan hand that signal back while HIP is still using it:SIGSEGV in rocprofiler::common::container::pool<signal_t>::release <- Queue::release_signal <- AsyncSignalHandlerBoth hits came right after the first mid-serving Triton JIT of
_tq_full_dequant_kv, and both passed on auto-retry:Over the last 7 days the segfault showed up 2 times in about 58 TurboQuant runs on the 14 kernrec-armed main builds. It showed up 0 times in about 228 runs on unarmed main builds and 0 times in 688 PR-branch runs. Across all 2,717 failed AMD job logs since 11-06, this signature appears only in these two jobs.
The upstream fix is ROCm/rocm-systems#14385 ("fix(rocprofiler-sdk): preserve dispatch timing signal lifetime", ROCM-32928). It is still open.
Change
KERNREC_AMD_SKIP_STEP_PREFIXESnext toKERNREC_SKIP_STEPS. Its one entry islm-eval-turboquant-, and_kernrec_applieschecks it only for theamdsetup profile.lm-eval-turboquant-{k8v4,t4nc,k3v4nc,t3nc}now run without the recorder: no setup, no.kernrecartifacts, no recording timeout margin, and the collect step does not wait for them. Their CUDA steps keep recording. As withKERNREC_SKIP_STEPS, a step with no recording keeps being selected, so the only cost is lost savings on those 4 steps.Testing
uv run pytest buildkite/tests/pipeline_generator: 566 passed, 2 skipped. The new testtest_amd_skip_prefix_unarms_only_the_amd_mirrorfails without the change.pre-commit run --fileson both files: everything passes exceptruff-format. That hook reformats four lines that already fail it on main and that this PR does not touch, so those lines are left out of this diff.AI assistance (Claude Code) was used to analyze the CI logs and to draft this change.