Skip to content

[ci-selector] Write CUDA kernel names at launch so recordings no longer need the exit footer - #914

Merged
khluu merged 2 commits into
mainfrom
kernrec-launch-time-recording
Nov 16, 2026
Merged

khluu merged 2 commits into
mainfrom
kernrec-launch-time-recording

Conversation

@tahsintunan

Copy link
Copy Markdown
Collaborator

Summary

  • A CUDA recording only counts as complete when its process writes the # end footer at normal exit. Many processes never get there: vLLM kills the engine process 5 s into shutdown, and forked children leave through os._exit, which skips the code that writes the footer. One such process makes its whole step unusable, so only about half the CUDA rows can be used (130 of 250 on nightly #102399).
  • The recorder now writes each kernel's name at the moment it is launched or added to a CUDA graph, before the kernel runs. The file is complete at any moment, so a killed process still leaves a complete recording. The header says sync=1.
  • kernel_table.py accepts a sync=1 file without a footer, as long as it has no error lines. Older files and ROCm files keep the footer rule.
  • Full CUDA run on this branch (#102908): 229 of 252 rows usable, against 138 under the footer rule. No missed kernels in 7,380 processes, no slowdown, and no failures caused by the recorder.

…o footer

Signed-off-by: Tahsin Tunan <tahsintunan@gmail.com>
…r check

Signed-off-by: Tahsin Tunan <tahsintunan@gmail.com>
@tahsintunan
tahsintunan requested review from AndreasKaratzas and khluu and removed request for khluu November 15, 2026 23:46
@khluu
khluu merged commit ec20a87 into main Nov 16, 2026
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants