solver: avoid mutating exec args for QEMU - #7092
Merged
Merged
Conversation
Use per-execution arguments when adding the QEMU emulator. This keeps the stored ExecOp unchanged when a canceled operation is executed again. Fixes moby#7089. Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
scttbnsn
added a commit
to CodesWhat/drydock
that referenced
this pull request
Aug 29, 2026
The multi-arch smoke build failed once on 2026-08-28 with .buildkit_qemu_emulator: /bin/sh: Invalid ELF image for this architecture 0.078s into the RUN, with no apk output at all. Two theories were tested and both are wrong. It is not rotted Alpine pins: the identical apk add succeeded natively on amd64 in the same run. It is not a moved tonistiigi/binfmt:latest: that tag has resolved to the same digest since 2026-06-08 and the qemu-aarch64 inside it has a correct ELF header. It is moby/buildkit#7089, open, filed the day before. ExecOp mutates its own Meta.Args in place to prepend the emulator path, so a canceled and re-executed arm64 op gets it prepended twice and QEMU execs itself. The fix is unmerged in moby/buildkit#7092. Retry rather than a config change, because retry is mechanism-agnostic: each attempt is a fresh Solve() that reparses the LLB graph, so a mutated Meta.Args cannot carry over. Two config workarounds were considered and rejected. A separate cache scope does not hold, since a cache hit skips Exec() entirely and the re-execution is intra-solve. Dropping cache-to to mode=min would stop exporting the arm64 npm ci and build layers, making cold emulated builds slower, with no confirmed effect on the race. release-cut.yml needs the heavier treatment because its existing recovery cannot reach this failure. The manifest retry requires a digest already in a registry and hard errors with "cannot perform manifest-only retry" without one, and this race pushes nothing anywhere. So a full-build retry is gated on exactly that shape, the manifest retry is narrowed to the case it can actually handle, and the digest fallback chain gains a tier. It reuses build-push-action rather than a hand-rolled CLI so the digest stays computed the way the steps downstream already depend on. Exposure is these two files only. security-grype.yml and e2e-playwright.yml build single-platform with no QEMU setup, so they are structurally immune rather than merely less likely. In release-cut.yml the build is gated on is_prerelease, so this is every RC cut and no GA cut. Drop both once buildkit ships the fix.
crazy-max
approved these changes
Aug 30, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Use per-execution arguments when adding the QEMU emulator. This keeps the stored ExecOp unchanged when a canceled operation is executed again.
Fixes #7089.