Repository navigation
Conversation
sduenas
force-pushed
the
fix-maintenance-tasks
branch
4 times, most recently
from
November 13, 2026 14:09
f4d676e to
e2691d2
Compare
added 5 commits
November 13, 2026 16:22
The maintenance process considered that every running job was stopped. RQ 2.x stores each execution of a job in the started registry as '<job_id>:<execution_id>', so looking up the expiration time with the job id always failed. As a result, running jobs were canceled and a duplicate job was scheduled for the same task. The check now looks for the executions of the job. Workers extend the expiration time of each execution on every heartbeat, so the job is considered stopped only when none of its executions can be fetched from Redis. The maintenance process also crashed when a task to re-schedule didn't have a scheduled time. In that case, the current time is used. Signed-off-by: Santiago Dueñas <sduenas@grimoirelab.org> Assisted-by: Claude:claude-opus-5.5
GrimoireLabTestCase only flushed its own Redis mock connection, but RQ uses a different one that doesn't share data with it. Jobs created by a test were still in the queues when the next tests ran, so a worker could run jobs from other tests. Both connections are now flushed before and after each test. Signed-off-by: Santiago Dueñas <sduenas@grimoirelab.org> Assisted-by: Claude:claude-opus-5.5
Rescheduling a running task didn't stop the job that was running. If the job was alive, the task wasn't rescheduled. Otherwise, a new job was scheduled, but the old one was left in RQ, where it could still run. The running job is now marked as canceled, stopped and removed from RQ before scheduling a new job. The job is canceled first, so it is already canceled when its callbacks run. Signed-off-by: Santiago Dueñas <sduenas@grimoirelab.org> Assisted-by: Claude:claude-opus-5.5
Jobs canceled while they were running could still schedule a new job for their task when their callbacks ran, creating a duplicate. For example, when the maintenance process cancels an abandoned job, it schedules a new one. Later, RQ cleans up the abandoned job from the started registry and runs its failure callback, which marked the job as failed and scheduled another job. The same happens when a job finishes right before it is stopped by a reschedule. The success and failure callbacks now ignore jobs that are no longer active. Signed-off-by: Santiago Dueñas <sduenas@grimoirelab.org> Assisted-by: Claude:claude-opus-5.5
Canceling a task with a job in the started status but not executed by any worker raised an InvalidJobOperation error. The task was left half canceled, with some of its jobs canceled and others not. Also, jobs were marked as canceled after sending the stop command. If a job finished before being stopped, its callbacks ran while the job was still active and scheduled a new job for the task that was being canceled. Jobs are now marked as canceled before they are stopped and removed from RQ, the same way they are when rescheduling a running task. Jobs that no longer exist in RQ are also marked as canceled, instead of being left active. Signed-off-by: Santiago Dueñas <sduenas@grimoirelab.org> Assisted-by: Claude:claude-opus-5.5
sduenas
force-pushed
the
fix-maintenance-tasks
branch
from
November 13, 2026 19:30
e2691d2 to
b05d61a
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The maintenance process considered that every running job was stopped. RQ 2.x stores each execution of a job in the started registry as '<job_id>:<execution_id>', so looking up the expiration time with the job id always failed. As a result, running jobs were canceled and a duplicate job was scheduled for the same task.
The check now looks for the executions of the job in the started registry. The job is considered stopped only when none of its executions is still alive.
The maintenance process also crashed when a task to re-schedule didn't have a scheduled time. In that case, the current time is used.
Assisted-by: Claude:claude-opus-5.5