Skip to content

feat(probes): liveness probes for worker and ui pods, overridable per component - #28

Open
Azerothian wants to merge 1 commit into
mainfrom
devbot/approved-temporal-operator-17
Open

Azerothian wants to merge 1 commit into
mainfrom
devbot/approved-temporal-operator-17

Conversation

@Azerothian

Copy link
Copy Markdown

Closes #17

What

Adds the missing probes and makes every generated probe overridable.

Before this PR the worker pod had no liveness probe on purpose (the comment in the builder read "worker has no grpc endpoint so omit liveness probe") and the UI had none at all, so a wedged worker or UI process would hang forever without a restart.

Component Default liveness probe
frontend, history, matching, internalFrontend TCP on the rpc port (unchanged)
worker new — TCP on the membership port
ui new — HTTP GET /healthz on the http port
admintools exec: ls / (unchanged, now overridable)

The worker serves no gRPC, so probing rpc would always fail — the membership ring is the cheapest real liveness signal. Its thresholds are deliberately conservative (150 s initial delay, 30 s period, failure threshold 5 → ~2.5 minutes of a dead listener before a restart), because a slow worker is not a broken one. The UI probe hits /healthz, which the UI answers without reaching out to the frontend, so a UI that can't reach the cluster isn't restarted — only one whose HTTP server is gone.

Override story: a new spec.services.<name>.livenessProbe, spec.ui.livenessProbe and spec.adminTools.livenessProbe (LivenessProbeSpec) accept either probe (replace the default with any core/v1.Probe) or disabled: true (drop it). Unset keeps the operator default; disabled wins over probe. Resolve is nil-safe, so clusters that never set the field are unaffected.

Files

  • api/v1beta1/temporalcluster_types.goLivenessProbeSpec.Resolve + the three new fields
  • internal/resource/base/deployment_builder.godefaultLivenessProbe() (+ worker membership probe, override hook)
  • internal/resource/ui/deployment_builder.go, internal/resource/admintools/deployment_builder.go — defaults + overrides
  • regenerated zz_generated.deepcopy.go, config/crd/bases, chart CRD, docs/api/v1beta1.md
  • tests: api/v1beta1/livenessprobe_test.go, internal/resource/base/deployment_builder_test.go
  • docs/features/liveness-probes.md + nav + CHANGELOG

Verification

  • go build ./... ✓ / go vet ./... ✓ / gofmt clean ✓
  • make manifests + make api-docs regenerated (repo's yq CRD post-processing included), chart CRD rebuilt via kustomize
  • go test ./api/... ./internal/... ./controllers/...all ok, including:
    • TestDefaultLivenessProbe — frontend probes rpc, worker probes membership (asserts the port name, and that the worker's thresholds stay conservative)
    • TestLivenessProbeOverride — custom probe replaces, disabled removes
    • TestLivenessProbeSpecResolve — unset/empty/disabled/custom precedence, nil-safe
  • full envtest controller suite passes (7.4 s)

Live-cluster verification on bne1-cluster1 was not possible for this run: the host's container runtime is down (containerd panics on a corrupt bbolt metadata DB, docker cannot connect), so the operator image could not be built/loaded. The probes are exercised by unit tests against the real builders, and the change is additive (new optional fields).

…onent

The worker pod had no liveness probe (it serves no gRPC, so the probe the
other services use would fail against it) and the ui had none at all, so a
wedged process in either would hang indefinitely. The worker now probes its
membership port with conservative thresholds, the ui probes /healthz on its
http port.

Every generated probe can now be replaced or removed through
spec.services.<name>.livenessProbe, spec.ui.livenessProbe and
spec.adminTools.livenessProbe.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Implementation proposal: liveness probes for worker and UI pods

1 participant