Give executor pods an ownerReference to the Deployment - #73
Merged
Conversation
Executor and session pods were created with no ownerReferences, because the service creates them directly through the Kubernetes client rather than through a controller. Two consequences: - Monitoring cannot tell these short-lived pods apart from long-lived workloads. The usual "pod has an owner" test does not apply to them, so they raise alerts that ephemeral pods should not raise. - Nothing garbage-collects pods the service fails to clean up. One-shot pods run "sleep 3600" with restartPolicy Never and no deadline, so a service crash mid-execution leaves a pod for up to an hour, then a Completed pod until the cluster prunes terminated pods. Resolve the owning Deployment once at startup and set it as the owner of every pod the service creates. The lookup returns None, and pods are created exactly as before, when the feature is unconfigured, when the read is not permitted, or when the executor namespace differs from the service namespace. Kubernetes does not honour ownerReferences across namespaces: it would read the owner as deleted and collect the pod immediately. Catch every exception around the lookup, not only ApiException. This runs in __init__, and /health builds the executor through get_executor(), so a raising constructor would turn a graceful health error into a 500 and let the liveness probe restart the pod. blockOwnerDeletion stays false; setting it needs "update" on the owner's finalizers subresource, which the service does not have. The chart gates the env plumbing and the new apps/deployments rule on codeInterpreter.kubernetesExecutor.setOwnerReferences, default true.
Danelegend
approved these changes
Aug 17, 2026
2 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Executor and session pods carried no
ownerReferences. The service creates themdirectly through the Kubernetes client rather than through a controller, and
_create_pod_manifestnever set the field. Two consequences:distinguishes ephemeral pods from long-lived workloads does not apply, so
code-exec-*pods raise alerts that ephemeral pods should not raise. This wasreported by a self-hosted deployment whose alerting excludes owned pods.
sleep 3600withrestartPolicy: Neverand no deadline, and cleanup only happens in theservice's
finallyblock. A crash mid-execution leaves the pod for up to anhour, then a
Completedpod until the cluster prunes terminated pods.This resolves the owning Deployment once at startup and sets it as the owner of
every pod the service creates. Both
code-exec-*andcode-session-*pods gothrough
_create_pod_manifest, so both are covered.Behavioral changes
ownerReferenceto the chart's Deployment(
controller: true,blockOwnerDeletion: false).getonapps/deploymentsand passes the service's ownnamespace and Deployment name via the downward API. Both are gated on
codeInterpreter.kubernetesExecutor.setOwnerReferences, defaulttrue.intended cleanup.
Design notes
Why the Deployment, not the service pod. Owning by the service pod would be
cheaper — the downward API exposes
metadata.uidwith no extra RBAC and no APIcall. But a rollout would then garbage-collect live session pods. The Deployment
is stable across rollouts.
Why the lookup never raises. It runs in
__init__, and/healthbuilds theexecutor through
get_executor()without a guard. A raising constructor wouldturn a graceful health error into a 500 and let the liveness probe restart the
pod, so the lookup catches every exception, not only
ApiException. It returnsNoneand pods are created exactly as before when the feature is unconfigured,when the read is not permitted, or when the executor namespace differs from the
service namespace — Kubernetes does not honour
ownerReferencesacrossnamespaces and would read the owner as deleted, collecting the pod immediately.
Why
blockOwnerDeletion: false. Setting it true requiresupdateon theowner's
finalizerssubresource, which this service does not have and shouldnot need.
Known corner case. The UID is resolved once. Deleting the Deployment with
--cascade=orphanand recreating it would leave a stale UID, and new executorpods would be collected immediately. Normal deletion takes the service pod with
it and forces re-resolution on restart.
Versions
Bumps
code-interpreterand the Helm chart 0.4.5 -> 0.4.6, matching the releaseconvention from #63/#64.
executor/is unchanged and stays at 0.4.5.Note that
docker-build-push.ymltriggers onv*tags while this repo'srelease tags are
code-interpreter-X.Y.Z, so the image does not buildautomatically. The image needs a
workflow_dispatchrun after merge to publish:0.4.6and:latest. The chart publishes automatically on merge; if it landsfirst, the older image simply ignores the new environment variables.
Testing
tests/integration_tests/test_kubernetes_owner_references.py: ownerresolves correctly; returns
Nonewhen unconfigured, across namespaces (anddoes not call the API), when the read is forbidden, and when the API is
unreachable; the manifest carries the reference and omits it when unresolved.
without a Docker daemon and are identical before and after this change.
mypy .,ruff check .,ruff format --check .clean.helm lintclean, andthe chart renders correctly with the flag on, off, and with a separate
executor namespace.
ownerReferenceshape.kubectl auth can-i get deployments.appsfor theservice account returns
notoday, confirming the new rule is required andthat the fallback path is reachable during a partial upgrade.
Not yet verified end to end against a live cluster with a built image, which
needs the image published first.
Follow-ups (not in this PR)
also pulls in 0.4.4/0.4.5, including the always-on deny-all-egress executor
NetworkPolicy, so it deserves its own review.
internal/executor/kubernetes.go.v*tag filter indocker-build-push.ymldoes not match the repo'sactual release tags.