Repository navigation
馃悰 fix(ci): dump stacks from a thread that holds the GIL - #3377
Merged
Merged
Conversation
faulthandler.dump_traceback_later walks other threads' frames without the GIL. When a frame's memory is freed mid-walk, the watchdog faults; on macOS it loops instead of dying, so the process hangs at exit while cancel_dump_traceback_later waits for it. The self-tests dump every 50ms and hit this in about 1 of 50 runs; CI at 60s hits it rarely. Dump from a daemon thread with faulthandler.dump_traceback, which holds the GIL on default builds and stops the world on free-threaded ones, so frames stay valid while it reads them. Keep the native timer as a stall detector: the thread re-arms it at ten intervals after each dump, so it fires only when a thread holds the GIL that long. Serialize the plugin's faulthandler calls, and stop and join the thread at exit. Claude-Session: https://claude.ai/code/session_018yHsaqnsodWtkwrqQ4kKPr
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
faulthandler.dump_traceback_laterreads other threads' frames without the GIL, so a dump that races a thread starting, exiting or shrinking its stack can fault in the watchdog. On macOS the faulting thread loops instead of dying, and the process hangs at exit incancel_dump_traceback_later; #3362 caught the same fault in_PyFrame_SafeGetCodeunder LLDB. CPython documents the frame checks as heuristics and has no fix in 3.14.8 or later.faulthandler.dump_traceback, which holds the GIL (stops the world on free-threaded builds)cancel_dump_traceback_latercould wait foreverThe self-tests count the new
DUMPrecords, wait for each process's ownpytest_runtestloopframe, and gain a test that holds the GIL with a backtracking regex until the stall timer fires. The diagnostics self-test file passed 200 of 200 runs on 3.14, 75 of 75 on 3.13t and 50 of 50 on 3.14t; before, it failed about once in 50 runs on 3.14.