feat(operator): hosting-migrate — a Roll's database migration runs as its own Job before the image moves by rbuergi · Pull Request #5148 · Systemorph/MeshWeaver

rbuergi · 2026-09-21T18:16:16Z

Why

A Roll Hosting/InstanceAction was only kubectl set image. When the target image expects a newer db_version, the new pod dies on DbVersionGate while the old pods keep answering 200. Measured on memex-cloud 2026-09-19, rolling 3.0.0-ci.8411 → 8955: CrashLoopBackOff, and no migration Job in the namespace.

Two paths already run the migration as a separate Job: helm upgrade (memex-migration-<revision>) and the in-pod self-updater (memex-migration-su-<tag>). The control lane's Roll has neither, and it is the routine path: an instance that reports to a control instance hands its updates over instead of patching itself.

Maintainer rule (2026-09-21, Systemorph/Memex#460): a schema change ships as a new image, and every roll runs its migration first, outside the portal, as a separate Job. Context: Systemorph/Memex#458. This is option (b1) in Doc/Architecture/SelfUpdateSchemaWall.

What

hosting-migrate --namespace <ns> --release <release> --image <registry>/…/memex-migration:<tag>

Reads the migration Job the release itself rendered: helm get manifest, then a kubectl create --dry-run=client -o json to get JSON, then the Job labelled app.kubernetes.io/component=memex-migration. Everything the chart puts on that Job is kept: wait-for-postgres, the rehearsal init container, activeDeadlineSeconds, envFrom (config, secrets and any Key Vault synced Secret) and the pull Secret.
Moves only the containers that run the migration image (rehearsal and run) to --image.
Runs it as memex-migration-roll-<tag> and waits. Exit 0 only when the Job reports succeeded ≥ 1.

Behaviour in each case:

Situation	Result
This tag's Job already succeeded	Reported, not run again
This tag's Job failed before	Deleted and run again
This tag's Job is still running	Waited on
Forbidden on the Job read	REFUSED, not treated as absent; nothing is created
Release renders no migration Job	Refusal; the image must not move without one
Failed Job, vanished Job, or Job past its own budget + grace	Non-zero exit naming the Job, with its log tail
`--image` names the portal image	Refused

It uses only bash, jq, kubectl and helm, because the operator image (Azure Linux 3) has no awk and no python.

Tests

13 new cases in run-tests.sh (stubs/migrate, fixtures/migrate). 537 passed, 0 failed on macOS bash 5.3 and in ubuntu:24.04. check-stderr-discarded and check-rbac-coverage are clean; pods/log and batch/jobs are already granted.

What else is needed

Plugins: the Roll plan calls this before set image. That PR is opened as a draft until an operator image with this command is on hosting-operator:main.
Once published, pearl rolls through it (Memex#458).
Merging this changes nothing that runs until the Plugins plan calls it. Nothing to recycle.

🤖 Generated with Claude Code

… its own Job before the image moves A Roll was `kubectl set image` and nothing else, so a target expecting a newer db_version died on DbVersionGate behind old pods answering 200 (memex-cloud 2026-09-19, 8411 → 8955). helm renders a migration Job only on upgrade and the self-updater mints its own; the control lane's Roll — the routine path for every reporting instance — had neither (Systemorph/Memex#458, #460; Doc/Architecture/SelfUpdateSchemaWall option b1). hosting-migrate --namespace <ns> --release <release> --image <…/memex-migration:tag> reads the migration Job the RELEASE rendered (helm get manifest → kubectl client dry run → JSON), moves only the containers running the migration image (rehearsal + run) to the target, runs it as Job memex-migration-roll-<tag> outside the portal, and exits 0 only when the Job reports succeeded. Idempotent per tag (succeeded → reported; failed → deleted and re-run; running → waited on). Forbidden is REFUSED, never absent; a release with no migration Job, a failed Job, a vanished Job or one past its own activeDeadlineSeconds (+grace) is a non-zero exit naming the Job with its log tail. Pure bash + jq + kubectl + helm: the operator image has no awk and no python. 13 new cases in run-tests.sh (stubs/migrate, fixtures/migrate): 537/537 on macOS bash 5.3 and in ubuntu:24.04; check-stderr-discarded and check-rbac-coverage clean (pods/log and batch/jobs already granted). The Roll plan calling this is the Plugins half, opened as a draft until an operator image carrying this command is on hosting-operator:main. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Copilot

Copilot review overview

🟡 Changes recommended

New documentation and script header comments hard-code a decision date/attribution that should be removed to avoid policy text rotting over time.

Get a fresh assessment by requesting another Copilot review.

Review effort: Lite
Findings: 1

Open (1)

Remove hard-coded decision date from operator command documentation · New

What changed in this PR

This PR adds an AKS operator-side step (hosting-migrate) to run a release-rendered database migration Job (as a standalone Job) before a Roll moves the portal image, and documents the operator/control-plane gap this closes.

Changes:

Add deploy/aks/operator/bin/hosting-migrate to render the release’s migration Job via helm get manifest, retarget only the migration containers to the target migration image, run it as memex-migration-roll-<tag>, and gate success on status.succeeded >= 1.
Add a focused bash test harness with helm/kubectl stubs + fixtures to cover success, failure, forbidden, missing-Job, and budget-expiry scenarios.
Update SelfUpdateSchemaWall.md to describe the new operator-side command as the missing half of the Roll plan.

File	Description
src/MeshWeaver.Documentation/Data/Architecture/SelfUpdateSchemaWall.md	Documents `hosting-migrate` as the operator-side mechanism to ensure migration precedes image move in a Roll plan.
deploy/aks/operator/bin/hosting-migrate	New operator command to run the chart-rendered migration Job out-of-band and block image movement until it succeeds.
deploy/aks/operator/test/run-tests.sh	Adds end-to-end stubbed tests for `hosting-migrate` behavior and failure modes.
deploy/aks/operator/test/stubs/migrate/kubectl	Kubectl stub to simulate Job lifecycle and capture created Job JSON for assertions.
deploy/aks/operator/test/stubs/migrate/helm	Helm stub to provide a manifest for `helm get manifest` and simulate missing release behavior.
deploy/aks/operator/test/fixtures/migrate/rendered.json	Fixture JSON representing the parsed/“rendered” Job used to validate retargeting and carry-over fields.
deploy/aks/operator/test/fixtures/migrate/manifest.yaml	Fixture manifest placeholder used by the helm stub.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

rbuergi · 2026-09-22T06:20:38Z

  operator route has the same ordering the in-pod route has had since the seam landed. Until it does,
  a `Roll` across a schema bump is `Roll` + `Reconcile`, in that order, by hand.
+
+  **The operator half is `hosting-migrate`** (`deploy/aks/operator/bin/`, 2026-09-21): it reads the


Agreed — that date is a policy marker, not evidence: it is the day the rule "a schema change ships as a new image, and every roll runs its migration first, outside the portal, as its own Job before the image moves" came into force, and it would have to change if the decision changed. That is policy policy-not-prose: the date moves into the register, the prose cites the id.

Fix pushed after this reply: a new register row roll-migrates-first in Doc/Architecture/PolicyNotProse carries the value, the in-force date and who set it; the flagged line now cites roll-migrates-first and links to the register instead of carrying the date. Nothing is deleted — the information moved.

github-actions · 2026-09-21T18:21:08Z