Skip to content

fix: update local state after runner assignment - #191

Merged
fridim merged 1 commit into
mainfrom
fix/assign-runner-pod-race
May 20, 2026
Merged

fridim merged 1 commit into
mainfrom
fix/assign-runner-pod-race

Conversation

@fridim

@fridim fridim commented May 20, 2026

Copy link
Copy Markdown
Contributor

assign_runner_pod uses deepcopy + replace_namespaced_custom_object to set the runner label on an AnarchyRun, but never updates self.definition with the result. This means runner_state (which reads from self.definition['metadata']['labels']) returns the stale pre-assignment value until the K8s watch fires and calls update_definition.

The Ansible runner never hit this because it processes runs in seconds, giving the watch time to catch up. The Go runner processes runs in ~200ms, consistently beating the watch. This causes two failures:

  1. POST /run/{name} (postResult) fails with 400 because the API checks anarchy_run.runner_state != anarchy_runner_pod.name — stale label.
  2. PATCH /run/subject/{name} (SubjectUpdate) fails with 400 because runner_assignments (populated by update_definition via watch) is empty — the watch hasn't fired yet.

The fix is one line: await self.update_definition(definition) after the K8s replace. This updates self.definition immediately (fixing runner_state) and calls update_runner_assignment (fixing runner_assignments). When the watch eventually fires with the same resourceVersion, handle_watch_found short-circuits (line 37).

This mirrors what AnarchyObject.replace() already does — assign_runner_pod was the only codepath that inlined the K8s replace without updating local state.

assign_runner_pod uses deepcopy + replace_namespaced_custom_object to
set the runner label on an AnarchyRun, but never updates self.definition
with the result. This means runner_state (which reads from
self.definition['metadata']['labels']) returns the stale pre-assignment
value until the K8s watch fires and calls update_definition.

The Ansible runner never hit this because it processes runs in seconds,
giving the watch time to catch up. The Go runner processes runs in
~200ms, consistently beating the watch. This causes two failures:

1. POST /run/{name} (postResult) fails with 400 because the API checks
   anarchy_run.runner_state \!= anarchy_runner_pod.name — stale label.
2. PATCH /run/subject/{name} (SubjectUpdate) fails with 400 because
   runner_assignments (populated by update_definition via watch) is
   empty — the watch hasn't fired yet.

The fix is one line: await self.update_definition(definition) after the
K8s replace. This updates self.definition immediately (fixing
runner_state) and calls update_runner_assignment (fixing
runner_assignments). When the watch eventually fires with the same
resourceVersion, handle_watch_found short-circuits (line 37).

This mirrors what AnarchyObject.replace() already does — assign_runner_pod
was the only codepath that inlined the K8s replace without updating local
state.
@fridim
fridim requested a review from jkupferer May 20, 2026 15:33
@fridim
fridim merged commit 02142e6 into main May 20, 2026
1 of 2 checks passed
@fridim
fridim deleted the fix/assign-runner-pod-race branch May 20, 2026 17:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants