fix(migrate): fail the job when migrations remain pending (ENG-565) - #497
Merged
Conversation
bobodread876
force-pushed
the
fix/migrate-job-fail-loudly
branch
from
August 26, 2026 17:45
70e7a38 to
c3b719a
Compare
Every Mongo-touching workload waits on this Job via the
wait-for-mongodb-migrate initContainer, so the gate is only worth
anything if the Job can fail. It could not.
The script was three unchecked commands with no set -e:
migrate-mongo status
migrate-mongo up # aborts the whole run on the first throwing
# migration; exit code discarded
migrate-mongo status # always succeeds -> script exits 0
So prod applied NO migration between 2026-03-28 and 2026-08-26 while
every deploy reported success. 20260330161656 threw on an empty
NOTIFICATION_TOPICS, migrate-mongo stopped, and the two migrations
behind it never ran. It surfaced only when an api pod crash-looped
building the npub unique index whose dedupe migration had never
executed -- five months later, in a different PR.
Now: set -eu, so a throwing migration fails the Job directly; and after
the up step, if status still reports PENDING the script exits 1 rather
than letting pods start against a database that is not at the schema
this release expects.
The same hole was hiding a failure in CI. `make test-migrate` runs the
migrations against a throwaway mongo with no NOTIFICATION_TOPICS, so
20260317125624 has been failing there too -- and the swallowed exit code
made the job report green. docker-compose.yml now sets the variable, so
the clean-migration test actually exercises the full chain. The first
run of the stricter script is what turned that long-standing false green
red.
The prod/test config value is fixed in deployments#195.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01NEoz7nBtdtsHyuYG5wNPQV
bobodread876
force-pushed
the
fix/migrate-job-fail-loudly
branch
from
August 26, 2026 17:54
3ea0b3d to
fc61cd6
Compare
Surfaced by the stricter migrate script in this PR: `make test-migrate` runs the whole chain against a throwaway mongo, and this migration died there with "ns does not exist: galoy.bridgevirtualaccounts". `col.indexes()` throws when the namespace has never been created, unlike `createIndex` further down, which creates it implicitly. So the migration only worked on a database where the app had already written a virtual account. Both clusters have the collection (prod 16 docs, test 0), so this changes nothing for them -- it is fresh databases that could not replay the chain, which defeats the purpose of the clean migration test. Treating "no namespace" as "no indexes" is the correct reading, not merely a tolerant one: there cannot be a stale plain index on a collection that does not exist. Verified: `make test-migrate` now runs every migration from empty to "All migrations applied." and exits 0. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01NEoz7nBtdtsHyuYG5wNPQV
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The other half of ENG-565. Config value: lnflash/deployments#195.
The defect
Every Mongo-touching workload gates on this Job through the
wait-for-mongodb-migrateinitContainer. That gate is worthless if the Job cannot fail — and it couldn't:No
set -e. So prod applied no migration between 2026-03-28 and 2026-08-26 while every deploy reported success.20260330161656threw on an emptyNOTIFICATION_TOPICS,migrate-mongostopped, and the two migrations behind it never ran.It surfaced five months later, in an unrelated PR, as an api pod crash-looping while building the
npubunique index whose dedupe migration had never executed.The fix
set -eu— a throwing migration now fails the Job directly.up, re-readstatus; if anything is still PENDING, exit 1.migrate-mongoreporting success while leaving work undone is exactly the case the first change doesn't cover.A green Job that applied nothing was the real defect. The crash-loop was just the first symptom loud enough to notice.
Notes
grepis present in thenode:24-alpinebase (/bin/grep) — verified, not assumed.sh -n.