diff --git a/.github/workflows/hosting-operator.yml b/.github/workflows/hosting-operator.yml index 34a4a9a57b..415789e172 100644 --- a/.github/workflows/hosting-operator.yml +++ b/.github/workflows/hosting-operator.yml @@ -65,8 +65,8 @@ jobs: run: | set -euo pipefail tracked=$(git ls-files deploy/aks/operator/bin/ | wc -l | tr -d ' ') - if [ "$tracked" -lt 29 ]; then - echo "::error::only ${tracked} file(s) tracked under deploy/aks/operator/bin/ — expected 29 (26 commands + run.sh + _common.sh + _audit.jq). The floor is the REAL count — raise it with every command added, or a lost file passes." + if [ "$tracked" -lt 31 ]; then + echo "::error::only ${tracked} file(s) tracked under deploy/aks/operator/bin/ — expected 31 (28 commands + run.sh + _common.sh + _audit.jq). The floor is the REAL count — raise it with every command added, or a lost file passes." echo "::error::Almost certainly .gitignore matching this bin/ again. `git add` on an ignored path exits 0 and does NOTHING, so the loss is silent until a fresh checkout builds an image with no scripts in it." git check-ignore -v deploy/aks/operator/bin/run.sh || true exit 1 @@ -179,6 +179,14 @@ jobs: # reached the cluster through Systemorph/Memex's helm-release lane, since a Job never # widens its own RBAC. The line moves into the plan list above in that same change. hosting-db-release #4436 — ships and is behaviour-tested, but no plan step emits it yet: the plan half is draft Plugins#1937 + # + # hosting-migrate is the same shape (policy `roll-migrates-first`): it ships from + # MeshWeaver#5148 and the behaviour suite drives it against test/stubs/migrate, so deleting + # it reds this gate. The Roll plan step that emits it is draft Plugins#2219, held until an + # operator image carrying this command is on hosting-operator:main — a plan that emits it + # against an older image stops at "command not found" BEFORE the image moves. The line + # moves up into the plan list in the change that merges that plan step. + hosting-migrate #5148 — ships and is behaviour-tested, but no plan step emits it yet: the plan half is draft Plugins#2219 EOF unclaimed=() for path in deploy/aks/operator/bin/hosting-*; do diff --git a/deploy/aks/operator/bin/hosting-migrate b/deploy/aks/operator/bin/hosting-migrate new file mode 100755 index 0000000000..ee384a62a2 --- /dev/null +++ b/deploy/aks/operator/bin/hosting-migrate @@ -0,0 +1,178 @@ +#!/usr/bin/env bash +# hosting-migrate --namespace --release --image //memex-migration: +# +# Run the database migration for a TARGET image as its own run-once Job, OUTSIDE the portal, and +# wait for it to finish — the step a Roll runs BEFORE it moves the portal image. The rule +# (maintainer, 2026-09-21; Systemorph/Memex docs/delivery-model.md): a schema change ships as a new +# image, and every path that moves the image runs that tag's migration first, as a separate Job. +# +# 🚨 WHY THIS EXISTS. A Roll was `kubectl set image` and nothing else. A target that expects a newer +# db_version then dies on DbVersionGate ~3 s after start, behind old pods that keep answering 200 — +# measured on memex-cloud 2026-09-19 rolling 3.0.0-ci.8411 → 8955: CrashLoopBackOff, no migration +# Job in the namespace, and a record that said the roll was made. helm renders its migration Job +# only on `helm upgrade` (`memex-migration-`) and the in-pod self-updater mints its own +# (`memex-migration-su-`); the control lane's Roll — the routine path for every instance that +# reports to a control instance — had neither (Doc/Architecture/SelfUpdateSchemaWall, option b1). +# +# THE JOB IS THE RELEASE'S OWN. It is read from `helm get manifest` — the migration Job the chart +# rendered for this release, with its wait-for-postgres gate, its rehearsal init container, its +# budget (activeDeadlineSeconds), its envFrom (memex-migration-config / -secrets and any Key Vault +# synced Secret) and its pull Secret — and ONLY the image of the containers that run the migration +# image is moved to --image. Nothing is re-derived here, so a chart change to how the migration runs +# reaches this step without an edit. A release that renders no migration Job is a REFUSAL: a roll +# that cannot migrate must not move the image. +# +# IDEMPOTENT per tag. The Job is `memex-migration-roll-`: succeeded → reported, not re-run; +# failed → deleted and run again (a retried roll must not read the last failure as its verdict); +# still running → waited on. +# +# Output contract (the ::hosting:: lines the mesh reads): +# migration_job= the Job this run created or found +# migration=completed the Job reported succeeded ≥ 1 — the ONLY success +# A failed Job, a vanished Job or one still running past its own budget exits non-zero, naming the +# Job and quoting the tail of its log. + +source "$(dirname "$(readlink -f "${BASH_SOURCE[0]}")")/_common.sh" +# shellcheck disable=SC2034 # read by hosting::die in _common.sh, which shellcheck does not follow here +HOSTING_CMD="hosting-migrate" + +namespace="" release="" image="" +interval="${HOSTING_MIGRATE_INTERVAL:-5}" grace="${HOSTING_MIGRATE_GRACE:-120}" +while [ $# -gt 0 ]; do + case "$1" in + --namespace) namespace="${2:-}"; shift 2 ;; + --release) release="${2:-}"; shift 2 ;; + --image) image="${2:-}"; shift 2 ;; + *) hosting::die "unknown argument '$1'" ;; + esac +done + +hosting::need_flag namespace "$namespace" +hosting::need_flag release "$release" +hosting::need_flag image "$image" +hosting::safe_name namespace "$namespace" +hosting::safe_name release "$release" +# A plain reference to the MIGRATION image, never the portal's: it lands in a Job spec, and a portal +# image there would run the portal as a Job and report whatever that did as the migration. +[[ "$image" =~ ^[A-Za-z0-9][A-Za-z0-9./_-]*/memex-migration:[A-Za-z0-9][A-Za-z0-9._-]*$ ]] \ + || hosting::die "--image '${image}' is not a plain memex-migration image reference (registry/…/memex-migration:tag) — refusing" +command -v jq >/dev/null 2>&1 \ + || hosting::die "jq is not installed in this image — the release's manifest and the Job's status are JSON" + +# The Job's name: DNS-1123, at most 63 characters, one per tag. +tag="${image##*:}" +suffix="$(printf '%s' "$tag" | tr '[:upper:]' '[:lower:]' | tr -c 'a-z0-9-' '-')" +job="memex-migration-roll-${suffix}" +job="${job:0:63}" +while [[ "$job" == *- ]]; do job="${job%-}"; done + +hosting::log "image ${image}" +hosting::log "release ${release} → namespace ${namespace}" +hosting::log "job ${job}" + +# ── the release's own migration Job ───────────────────────────────────────────────────────────── +hosting::probe output helm get manifest "$release" --namespace "$namespace" +case $? in + 2) hosting::die_refused "release ${release}'s manifest in ${namespace} (helm reads it from the release Secrets), which holds the migration Job this step runs" ;; + 1) hosting::die "release ${release} in ${namespace} has no readable manifest — there is no migration Job to run, so the image must not move. helm said: ${HOSTING_PROBE_ERR}" ;; +esac +manifest="$HOSTING_PROBE_OUT" + +# YAML → JSON through kubectl itself (the image carries no YAML parser), then pick the Job the chart +# labels as the migration. +# A here-string, never a pipe: a pipe runs hosting::probe in a subshell and HOSTING_PROBE_OUT dies with it. +hosting::probe output kubectl -n "$namespace" create --dry-run=client -o json -f - <<< "$manifest" +case $? in + 2) hosting::die_refused "the kinds in release ${release}'s manifest (a client-side dry run still resolves them against the API)" ;; + 1) hosting::die "release ${release}'s manifest could not be parsed: ${HOSTING_PROBE_ERR}" ;; +esac +rendered="$(printf '%s' "$HOSTING_PROBE_OUT" | jq -c ' + (if .kind == "List" then .items[] else . end) + | select(.kind == "Job" and .metadata.labels["app.kubernetes.io/component"] == "memex-migration")' | head -1)" +[ -n "$rendered" ] \ + || hosting::die "release ${release} in ${namespace} renders no migration Job (kind Job, label app.kubernetes.io/component=memex-migration) — nothing establishes the schema, so the image must not move" + +# Rename, and move ONLY the containers that run the migration image. wait-for-postgres (busybox) +# and anything else keep their images; the rehearsal init container runs the migration image and +# must rehearse the SAME build it then executes. +body="$(printf '%s' "$rendered" | jq -c --arg name "$job" --arg img "$image" ' + def retarget: if (.image // "" | test("/memex-migration:")) then .image = $img else . end; + .metadata = { name: $name, labels: (.metadata.labels // {}) } + | .spec.template.spec.containers |= map(retarget) + | if .spec.template.spec.initContainers then .spec.template.spec.initContainers |= map(retarget) else . end + | del(.status)')" +moved="$(printf '%s' "$body" | jq --arg img "$image" '[.spec.template.spec.containers[]?, .spec.template.spec.initContainers[]? | select(.image == $img)] | length')" +[ "${moved:-0}" -ge 1 ] \ + || hosting::die "release ${release}'s migration Job runs no memex-migration image — nothing to retarget, refusing to run something else as the migration" +budget="$(printf '%s' "$body" | jq -r '.spec.activeDeadlineSeconds // 660')" +hosting::log "budget ${budget}s (the Job's activeDeadlineSeconds) + ${grace}s to schedule and pull" + +# ── PRESENT / ABSENT / REFUSED for this tag's Job ────────────────────────────────────────────── +job_state="" +read_job_state() { + hosting::probe any kubectl -n "$namespace" get job "$job" -o json + case $? in + 2) hosting::die_refused "job/${job} in ${namespace}, which says whether this tag's migration already ran" ;; + 1) job_state="absent"; return 0 ;; + esac + job_state="$(printf '%s' "$HOSTING_PROBE_OUT" | jq -r ' + if (.status.succeeded // 0) >= 1 then "succeeded" + elif ((.status.conditions // []) | any(.type == "Failed" and .status == "True")) then "failed" + else "active" end')" +} + +log_tail() { + hosting::probe any kubectl -n "$namespace" logs "job/${job}" --all-containers --tail=20 + case $? in + 0) printf '%s' "$HOSTING_PROBE_OUT" ;; + 2) printf '(the log is not readable by this operator: %s)' "$HOSTING_PROBE_ERR" ;; + *) printf '(no log: %s)' "$HOSTING_PROBE_ERR" ;; + esac +} + +read_job_state +case "$job_state" in + succeeded) + hosting::log "migration ${job} already SUCCEEDED for ${tag} — not run again" + hosting::say migration_job "$job" + hosting::say migration completed + exit 0 ;; + failed) + hosting::log "migration ${job} FAILED before — deleting it and running it again" + hosting::do kubectl -n "$namespace" delete job "$job" --wait=true >&2 ;; + active) + hosting::log "migration ${job} is already running — waiting on it" ;; +esac +if [ "$job_state" != "active" ]; then + hosting::do kubectl -n "$namespace" create -f - <<< "$body" >&2 \ + || hosting::die "could not create ${job} in ${namespace}" +fi +hosting::say migration_job "$job" + +if hosting::dry; then + hosting::log "dry run: the Job was not created, so there is nothing to wait for" + exit 0 +fi + +# ── wait: succeeded is the ONLY success ───────────────────────────────────────────────────────── +limit=$(( budget + grace )) waited=0 +while :; do + read_job_state + case "$job_state" in + succeeded) break ;; + failed) + hosting::die "migration ${job} FAILED for ${image} — the schema did not move, so the portal image must not either. Last lines: $(log_tail)" ;; + absent) + hosting::die "migration ${job} disappeared while it was being waited on (deleted, or reaped) — nothing establishes that the schema moved, so the image must not" ;; + esac + [ "$waited" -ge "$limit" ] \ + && hosting::die "migration ${job} has not completed after ${waited}s (its budget is ${budget}s) — refusing to move the image. Last lines: $(log_tail)" + sleep "$interval" + waited=$(( waited + interval )) + [ "$interval" -gt 0 ] || waited=$(( waited + 1 )) +done + +hosting::log "migration ${job} SUCCEEDED after ~${waited}s" +done_line="$(log_tail | grep -m1 'Database migration completed' || true)" +[ -n "$done_line" ] && hosting::log "$done_line" +hosting::say migration completed diff --git a/deploy/aks/operator/test/fixtures/migrate/manifest.yaml b/deploy/aks/operator/test/fixtures/migrate/manifest.yaml new file mode 100644 index 0000000000..39da874936 --- /dev/null +++ b/deploy/aks/operator/test/fixtures/migrate/manifest.yaml @@ -0,0 +1,2 @@ +kind: Job +# the stub parses nothing; rendered.json is the parse diff --git a/deploy/aks/operator/test/fixtures/migrate/rendered.json b/deploy/aks/operator/test/fixtures/migrate/rendered.json new file mode 100644 index 0000000000..5b3bc10386 --- /dev/null +++ b/deploy/aks/operator/test/fixtures/migrate/rendered.json @@ -0,0 +1,9 @@ +{"kind":"List","apiVersion":"v1","items":[ + {"kind":"ConfigMap","apiVersion":"v1","metadata":{"name":"memex-migration-config"}}, + {"kind":"Job","apiVersion":"batch/v1","metadata":{"name":"memex-migration-41","labels":{"app.kubernetes.io/component":"memex-migration","app.kubernetes.io/instance":"pearl"}}, + "spec":{"backoffLimit":6,"ttlSecondsAfterFinished":86400,"activeDeadlineSeconds":660,"template":{"metadata":{"labels":{"app.kubernetes.io/component":"memex-migration"}},"spec":{"restartPolicy":"Never", + "imagePullSecrets":[{"name":"registry-pull"}], + "initContainers":[{"name":"wait-for-postgres","image":"busybox:1.36"}, + {"name":"memex-migration-rehearsal","image":"cr.example.test/memex-migration:3.0.0-rc13","envFrom":[{"configMapRef":{"name":"memex-migration-config"}}]}], + "containers":[{"name":"memex-migration","image":"cr.example.test/memex-migration:3.0.0-rc13","envFrom":[{"configMapRef":{"name":"memex-migration-config"}},{"secretRef":{"name":"memex-migration-secrets"}}]}]}}}} +]} diff --git a/deploy/aks/operator/test/run-tests.sh b/deploy/aks/operator/test/run-tests.sh index f3d4825498..14d94cc28f 100755 --- a/deploy/aks/operator/test/run-tests.sh +++ b/deploy/aks/operator/test/run-tests.sh @@ -1960,6 +1960,151 @@ else bad "every kubectl read in bin/ that discards stderr is declared" "$sd_out" fi +# ── hosting-migrate: the Roll's migration runs as its OWN Job, and nothing moves until it succeeds ── +# 🚨 Systemorph/Memex#458/#460. A Roll was `kubectl set image` alone; across a db_version bump the new +# pod died on DbVersionGate behind old pods answering 200 (memex-cloud 2026-09-19, 8411 → 8955). The +# step runs the release's OWN migration Job (read from `helm get manifest`) with only the migration +# image moved, and its exit code is the gate the Roll's set-image sits behind. These cases pin: the +# Job is the release's (pull Secret, budget, envFrom, wait-for-postgres kept), BOTH the rehearsal and +# the migration container move to the target, succeeded is the only success, a failure / a vanished +# Job / a Job past its budget is a non-zero exit naming it, a Forbidden is REFUSED not absent, a +# release with no migration Job refuses, and a re-run per tag is idempotent. +echo +echo "── hosting-migrate: the migration runs as its own Job before the image moves ──" +MG_STUBS="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/stubs/migrate" && pwd)" +MG_FIXTURES="$(cd -- "$(dirname -- "${BASH_SOURCE[0]}")/fixtures/migrate" && pwd)" +_mg_img="cr.example.test/memex-migration:3.0.0-ci.9101" +_mg_job="memex-migration-roll-3-0-0-ci-9101" +_mg_new() { _mg_dir="$(mktemp -d)"; cp -R "$MG_FIXTURES/." "$_mg_dir/"; _mg_log="$_mg_dir/calls.log"; : > "$_mg_log"; } +_mg_run() { env PATH="$MG_STUBS:$PATH" HOSTING_MIGRATE_FIXTURE="$_mg_dir" HOSTING_MIGRATE_STUB_LOG="$_mg_log" \ + HOSTING_MIGRATE_INTERVAL=0 HOSTING_MIGRATE_GRACE=0 "$@" \ + hosting-migrate --namespace pearl --release pearl --image "$_mg_img" 2>&1; } + +# absent → created from the release's Job, retargeted, waited on, completed +_mg_new; echo 2 > "$_mg_dir/polls"; echo succeeded > "$_mg_dir/outcome" +_mg_out="$(_mg_run env)"; _mg_rc=$? +if [ "$_mg_rc" -eq 0 ] && printf '%s' "$_mg_out" | grep -q "::hosting:: migration_job=${_mg_job}" \ + && printf '%s' "$_mg_out" | grep -q '::hosting:: migration=completed' \ + && printf '%s' "$_mg_out" | grep -q 'Database migration completed. Version: 57'; then + ok "an absent Job is created, waited on, and only then reported completed" +else + bad "an absent Job is created, waited on and reported completed" "rc=${_mg_rc} out: ${_mg_out}" +fi +_mg_c="$_mg_dir/created.json" +if [ -f "$_mg_c" ] \ + && [ "$(jq -r '.metadata.name' "$_mg_c")" = "$_mg_job" ] \ + && [ "$(jq -r '.spec.template.spec.containers[0].image' "$_mg_c")" = "$_mg_img" ] \ + && [ "$(jq -r '.spec.template.spec.initContainers[] | select(.name=="memex-migration-rehearsal") | .image' "$_mg_c")" = "$_mg_img" ] \ + && [ "$(jq -r '.spec.template.spec.initContainers[] | select(.name=="wait-for-postgres") | .image' "$_mg_c")" = "busybox:1.36" ]; then + ok "BOTH migration containers (rehearsal + run) move to the target; wait-for-postgres keeps its image" +else + bad "the migration containers move to the target and nothing else does" "$(cat "$_mg_c" 2>/dev/null)" +fi +if [ "$(jq -r '.spec.template.spec.imagePullSecrets[0].name' "$_mg_c")" = "registry-pull" ] \ + && [ "$(jq -r '.spec.activeDeadlineSeconds' "$_mg_c")" = "660" ] \ + && [ "$(jq -r '.spec.template.spec.containers[0].envFrom[1].secretRef.name' "$_mg_c")" = "memex-migration-secrets" ] \ + && [ "$(jq -r '.metadata.labels["app.kubernetes.io/component"]' "$_mg_c")" = "memex-migration" ]; then + ok "the Job is the release's own: pull Secret, budget, envFrom and labels carried over" +else + bad "the Job is the release's own" "$(cat "$_mg_c")" +fi +_mg_get="$(grep -n '^kubectl -n pearl get job' "$_mg_log" | tail -1 | cut -d: -f1)" +_mg_create="$(grep -n '^kubectl -n pearl create -f -' "$_mg_log" | head -1 | cut -d: -f1)" +[ -n "$_mg_create" ] && [ -n "$_mg_get" ] && [ "$_mg_get" -gt "$_mg_create" ] \ + && ok "the Job's status is read AFTER it was created (the wait is real)" \ + || bad "the Job's status is read after it was created" "$(cat "$_mg_log")" +rm -rf "$_mg_dir" + +# already succeeded for this tag → reported, never re-run +_mg_new; echo succeeded > "$_mg_dir/job-state" +_mg_out="$(_mg_run env)"; _mg_rc=$? +if [ "$_mg_rc" -eq 0 ] && printf '%s' "$_mg_out" | grep -q '::hosting:: migration=completed' && ! grep -q ' create -f -' "$_mg_log"; then + ok "a Job that already SUCCEEDED for this tag is reported, not run again" +else + bad "an already-succeeded Job is not run again" "rc=${_mg_rc} log: $(cat "$_mg_log")" +fi +rm -rf "$_mg_dir" + +# failed before → deleted and run again +_mg_new; echo failed > "$_mg_dir/job-state"; echo succeeded > "$_mg_dir/outcome" +_mg_out="$(_mg_run env)"; _mg_rc=$? +_mg_del="$(grep -n "delete job ${_mg_job}" "$_mg_log" | head -1 | cut -d: -f1)" +_mg_create="$(grep -n ' create -f -' "$_mg_log" | head -1 | cut -d: -f1)" +if [ "$_mg_rc" -eq 0 ] && [ -n "$_mg_del" ] && [ -n "$_mg_create" ] && [ "$_mg_del" -lt "$_mg_create" ]; then + ok "a Job that FAILED before is deleted and run again — the last failure is not this run's verdict" +else + bad "a previously failed Job is deleted and re-run" "rc=${_mg_rc} log: $(cat "$_mg_log")" +fi +rm -rf "$_mg_dir" + +# the Job fails → non-zero, names the Job, says the image must not move, never "completed" +_mg_new; echo 1 > "$_mg_dir/polls"; echo failed > "$_mg_dir/outcome" +_mg_out="$(_mg_run env)"; _mg_rc=$? +if [ "$_mg_rc" -ne 0 ] && printf '%s' "$_mg_out" | grep -q "${_mg_job} FAILED" \ + && printf '%s' "$_mg_out" | grep -q 'must not either' && ! printf '%s' "$_mg_out" | grep -q 'migration=completed'; then + ok "a FAILED migration exits non-zero, names the Job, and never reports completed" +else + bad "a failed migration is a failed step" "rc=${_mg_rc} out: ${_mg_out}" +fi +rm -rf "$_mg_dir" + +# never finishes → stops at the Job's own budget, non-zero +_mg_new; echo hang > "$_mg_dir/outcome" +jq '(.items[] | select(.kind=="Job") | .spec.activeDeadlineSeconds) = 3' "$MG_FIXTURES/rendered.json" > "$_mg_dir/rendered.json" +_mg_out="$(_mg_run env)"; _mg_rc=$? +if [ "$_mg_rc" -ne 0 ] && printf '%s' "$_mg_out" | grep -q 'has not completed after' && ! printf '%s' "$_mg_out" | grep -q 'migration=completed'; then + ok "a migration still running past its own budget is a failed step, not a pass" +else + bad "a migration past its budget fails" "rc=${_mg_rc} out: ${_mg_out}" +fi +rm -rf "$_mg_dir" + +# REFUSED is not ABSENT: a Forbidden on the Job read stops before anything is created +_mg_new; echo forbidden > "$_mg_dir/job-state" +_mg_out="$(_mg_run env)"; _mg_rc=$? +if [ "$_mg_rc" -ne 0 ] && printf '%s' "$_mg_out" | grep -q 'REFUSED, not absent' && ! grep -q ' create -f -' "$_mg_log"; then + ok "a Forbidden on the Job read is REFUSED (not absent) and nothing is created" +else + bad "a Forbidden job read is refused" "rc=${_mg_rc} out: ${_mg_out} log: $(cat "$_mg_log")" +fi +rm -rf "$_mg_dir" + +# a release that renders no migration Job → refusal, nothing created +_mg_new; jq '.items |= map(select(.kind != "Job"))' "$MG_FIXTURES/rendered.json" > "$_mg_dir/rendered.json" +_mg_out="$(_mg_run env)"; _mg_rc=$? +if [ "$_mg_rc" -ne 0 ] && printf '%s' "$_mg_out" | grep -q 'renders no migration Job' && ! grep -q ' create -f -' "$_mg_log"; then + ok "a release with no migration Job is a refusal — the image must not move without one" +else + bad "a release with no migration Job refuses" "rc=${_mg_rc} out: ${_mg_out}" +fi +rm -rf "$_mg_dir" + +# no release at all → refusal naming helm's answer +_mg_new; rm -f "$_mg_dir/manifest.yaml" +_mg_out="$(_mg_run env)"; _mg_rc=$? +[ "$_mg_rc" -ne 0 ] && printf '%s' "$_mg_out" | grep -q 'has no readable manifest' \ + && ok "a release helm cannot find is a refusal naming helm's answer" \ + || bad "a missing release refuses" "rc=${_mg_rc} out: ${_mg_out}" +rm -rf "$_mg_dir" + +# dry run: reads, narrates the create, creates nothing, waits for nothing +_mg_new +_mg_out="$(_mg_run env HOSTING_DRY_RUN=true)"; _mg_rc=$? +if [ "$_mg_rc" -eq 0 ] && printf '%s' "$_mg_out" | grep -q 'DRY-RUN would run: kubectl -n pearl create -f -' \ + && [ ! -f "$_mg_dir/created.json" ] && ! printf '%s' "$_mg_out" | grep -q 'migration=completed'; then + ok "a dry run narrates the Job, creates nothing and claims no migration" +else + bad "a dry run creates nothing" "rc=${_mg_rc} out: ${_mg_out}" +fi +rm -rf "$_mg_dir" + +refuses_hard "hosting-migrate refuses a PORTAL image — only memex-migration runs as the migration" "not a plain memex-migration image reference" \ + env HOSTING_DRY_RUN=true hosting-migrate --namespace pearl --release pearl --image cr.example.test/memex-portal-ai:3.0.0-ci.9101 +refuses_hard "hosting-migrate refuses an image reference with a metacharacter" "not a plain memex-migration image reference" \ + env HOSTING_DRY_RUN=true hosting-migrate --namespace pearl --release pearl --image 'cr.example.test/memex-migration:1;rm -rf /' +refuses_hard "hosting-migrate needs --release" "missing required flag --release" \ + env HOSTING_DRY_RUN=true hosting-migrate --namespace pearl --image "$_mg_img" + # ── every kubectl verb+resource in bin/ is GRANTED by the operator's ClusterRole ───────────────── # The manifest lives three directories away from the scripts and is reviewed separately; twice a # script reached main without its grant (storageclasses for pv-resize — failed the first Reconcile diff --git a/deploy/aks/operator/test/stubs/migrate/helm b/deploy/aks/operator/test/stubs/migrate/helm new file mode 100755 index 0000000000..17d8ba5c51 --- /dev/null +++ b/deploy/aks/operator/test/stubs/migrate/helm @@ -0,0 +1,11 @@ +#!/usr/bin/env bash +# A stand-in `helm` for hosting-migrate's tests. Only `get manifest` is answered: from +# $HOSTING_MIGRATE_FIXTURE/manifest.yaml when it exists, else the real CLI's "release: not found". +# Every invocation is recorded in $HOSTING_MIGRATE_STUB_LOG. It asserts nothing about helm itself. +set -uo pipefail +log="${HOSTING_MIGRATE_STUB_LOG:?the helm stub needs HOSTING_MIGRATE_STUB_LOG}" +fixture="${HOSTING_MIGRATE_FIXTURE:?the helm stub needs HOSTING_MIGRATE_FIXTURE}" +printf 'helm %s\n' "$*" >> "$log" +[ "${1:-} ${2:-}" = "get manifest" ] || { echo "helm stub: unexpected invocation: $*" >&2; exit 1; } +[ -f "$fixture/manifest.yaml" ] || { echo "Error: release: not found" >&2; exit 1; } +cat "$fixture/manifest.yaml" diff --git a/deploy/aks/operator/test/stubs/migrate/kubectl b/deploy/aks/operator/test/stubs/migrate/kubectl new file mode 100755 index 0000000000..c52059d763 --- /dev/null +++ b/deploy/aks/operator/test/stubs/migrate/kubectl @@ -0,0 +1,45 @@ +#!/usr/bin/env bash +# A stand-in `kubectl` for hosting-migrate's tests. Records every invocation (and the body of every +# `create -f -`) in $HOSTING_MIGRATE_STUB_LOG / $FIXTURE/created.json. The Job's life is played from +# files in $HOSTING_MIGRATE_FIXTURE: +# rendered.json what `create --dry-run=client -o json -f -` answers (kubectl's own YAML parsing is +# not under test — the manifest's text is ignored) +# job-state absent | succeeded | failed | active | forbidden (the Job as `get job` sees it) +# outcome what a CREATED Job becomes after `polls` more reads: succeeded | failed | hang +# polls how many reads a created Job stays active before it reaches its outcome +set -uo pipefail +log="${HOSTING_MIGRATE_STUB_LOG:?the kubectl stub needs HOSTING_MIGRATE_STUB_LOG}" +fixture="${HOSTING_MIGRATE_FIXTURE:?the kubectl stub needs HOSTING_MIGRATE_FIXTURE}" +printf 'kubectl %s\n' "$*" >> "$log" +if [ "${1:-}" = "-n" ]; then shift 2; fi +state() { cat "$fixture/job-state" 2>/dev/null || echo absent; } +job_json() { + case "$1" in + succeeded) echo '{"kind":"Job","status":{"succeeded":1}}' ;; + failed) echo '{"kind":"Job","status":{"failed":7,"conditions":[{"type":"Failed","status":"True"}]}}' ;; + *) echo '{"kind":"Job","status":{"active":1}}' ;; + esac +} +case "${1:-}" in + create) + if [ "${2:-}" = "--dry-run=client" ]; then cat >/dev/null; cat "$fixture/rendered.json"; exit 0; fi + cat > "$fixture/created.json" + echo active > "$fixture/job-state" + echo "job.batch/created" ;; + get) + s="$(state)" + case "$s" in + absent) echo "Error from server (NotFound): jobs.batch \"${3:-}\" not found" >&2; exit 1 ;; + forbidden) echo "Error from server (Forbidden): jobs.batch \"${3:-}\" is forbidden: User cannot get resource" >&2; exit 1 ;; + active) + n="$(cat "$fixture/polls" 2>/dev/null || echo 0)" + if [ "$n" -gt 0 ]; then echo $((n - 1)) > "$fixture/polls"; job_json active; exit 0; fi + o="$(cat "$fixture/outcome" 2>/dev/null || echo succeeded)" + [ "$o" = "hang" ] && { job_json active; exit 0; } + echo "$o" > "$fixture/job-state"; job_json "$o" ;; + *) job_json "$s" ;; + esac ;; + delete) echo absent > "$fixture/job-state"; echo "job.batch deleted" ;; + logs) printf 'Rehearsal: 2 pending migrations\nDatabase migration completed. Version: 57\n' ;; + *) echo "kubectl stub: unexpected invocation: $*" >&2; exit 1 ;; +esac diff --git a/src/MeshWeaver.Documentation/Data/Architecture/PolicyNotProse.md b/src/MeshWeaver.Documentation/Data/Architecture/PolicyNotProse.md index 83ccc1b3eb..f95db32543 100644 --- a/src/MeshWeaver.Documentation/Data/Architecture/PolicyNotProse.md +++ b/src/MeshWeaver.Documentation/Data/Architecture/PolicyNotProse.md @@ -136,8 +136,10 @@ written. | `query-fanin-stall-terminal` | A query provider that neither emits, completes nor errors TERMINATES its merged query with a named error instead of hanging: the fan-in's Initial gate is a bound, not a wait. Every consumer that turns a mesh read into a decision reads that error as an availability failure — an access-control read fails CLOSED and reports that it could not be established, never `denied` and never `granted`. | in force | 2026-09-21 | maintainer | | `open-vocabulary-string-constants` | A vocabulary that is persisted, serialised, or extended by a module is a `static class` of `const string` named exactly as the enum would have been — never a C# `enum`. It stays OPEN: any other party may add its own values from its own constants class, so the platform's set is never treated as exhaustive. Consumers compare against constants and always carry a branch for a value this build does not know. | in force | 2026-09-21 | maintainer | | `thread-graceful-error` | Wherever user code is executed, innermost in a thread, it must gracefully error: every failure path ends in a stamped terminal state on the node. A failure the thread's own hub cannot stamp — because it is the hub that died — is observed, cleaned up and relaunched under a bound, and what a relaunch cannot fix is filed into bug triage. The relaunch/dispatch side is bounded by a configurable pool cap (`maxConcurrentAgents`, default 50) whose queue is a page, not a log. | in force | 2026-09-20 | maintainer | +| `roll-migrates-first` | A schema change ships as a new image, and every roll runs that image's database migration FIRST — outside the portal, as its own run-once Job that must report succeeded — before the image moves. The operator half is `hosting-migrate`; the `Roll` plan calling it is the Plugins half. | in force | 2026-09-21 | maintainer | Cited by: [Release Process](../ReleaseProcess) · +[The Self-Update Schema Wall](../SelfUpdateSchemaWall) · [Issue Taxonomy and the Release Readiness Gate](../IssueTaxonomy) · [Adding a Data Sync Needs a Global Admin](../DataSyncApproval) · [Thread Supervision](../ThreadSupervision) · diff --git a/src/MeshWeaver.Documentation/Data/Architecture/SelfUpdateSchemaWall.md b/src/MeshWeaver.Documentation/Data/Architecture/SelfUpdateSchemaWall.md index 7c9b3781e1..8f7224cad6 100644 --- a/src/MeshWeaver.Documentation/Data/Architecture/SelfUpdateSchemaWall.md +++ b/src/MeshWeaver.Documentation/Data/Architecture/SelfUpdateSchemaWall.md @@ -233,6 +233,17 @@ Plugins/control-plane half, and it has a precise shape: the target tag, wait for `Database migration completed. Version: N`, then set the image — so the operator route has the same ordering the in-pod route has had since the seam landed. Until it does, a `Roll` across a schema bump is `Roll` + `Reconcile`, in that order, by hand. + + **The operator half is `hosting-migrate`** (`deploy/aks/operator/bin/`; policy `roll-migrates-first` + in the [register](../PolicyNotProse)): it reads the + migration Job the release itself rendered (`helm get manifest` — wait-for-postgres, rehearsal, + budget, envFrom and pull Secret all kept), moves only the migration containers to the target tag, + runs it as its own Job `memex-migration-roll-`, and exits non-zero unless the Job + **succeeded** — so the `Roll` plan's `set image` sits behind a migration that demonstrably ran, + outside the portal. Idempotent per tag: a succeeded Job is reported, a failed one is deleted and + run again. The `Roll` plan calling it is the Plugins half; until an operator image carrying the + command is on `hosting-operator:main`, that plan step would stop with *command not found* — before + the image moves, which is the safe side. - **(b2) The `Deployments/` record must carry the verdict.** The in-pod path reports on `Admin/UpdatePolicy`; an operator `Roll` reports on the instance record, and a plan step that patched an image the new pods then refuse must land there as a failure rather than as a completed