Skip to content

feat: bound pipeline work items with a per-item timeout - #60

Merged
lollllcat merged 1 commit into
mainfrom
pipeline-timeout
Jul 27, 2026
Merged

feat: bound pipeline work items with a per-item timeout#60
lollllcat merged 1 commit into
mainfrom
pipeline-timeout

Conversation

@lollllcat

Copy link
Copy Markdown
Collaborator

Summary

  • Add WORKITEM_TIMEOUT_SEC (default 120min, override via CRISP_WORKITEM_TIMEOUT_SEC) to crisp/pipeline.py: a work item running longer than this gets its worker process force-killed (terminate(), escalating to kill() if needed) and its Config marked failed, instead of blocking the pipeline stage forever.
  • pipelineWorkerReal's wait loop now bounds mp.connection.wait(...) by the soonest deadline and reaps expired workers each time it times out. Only applies to non-serialize (multiprocess) phases.
  • Extracted the existing out-of-order completion flush logic into a shared _flushReadyItems() helper.
  • Added tests for the timeout/kill-escalation/reaping logic, plus an end-to-end test confirming a hung work item gets reaped instead of hanging the stage.

Test plan

  • pytest full suite passes (647 passed)
  • bazel test //... passes (33/33 targets)

@lollllcat
lollllcat merged commit 8b6a9d8 into main Jul 27, 2026
3 checks passed
@lollllcat
lollllcat deleted the pipeline-timeout branch July 27, 2026 21:17
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant