Skip to content

ci: image builds pull through a mirror instead of rate-limited Docker Hub - #5377

Merged
miguel-heygen merged 2 commits into
mainfrom
ci/docker-hub-mirror
Oct 9, 2026
Merged

miguel-heygen merged 2 commits into
mainfrom
ci/docker-hub-mirror

Conversation

@miguel-heygen

@miguel-heygen miguel-heygen commented Oct 9, 2026 •

Copy link
Copy Markdown
Collaborator

What changes

The regression shards, the BeginFrame image contract and the fast video validation build their Docker images with buildx on GitHub-hosted runners. Those runners pull moby/buildkit and node:22.23.3-bookworm-slim from Docker Hub anonymously, and Docker Hub has started answering with rate limits:

ERROR: failed to build: failed to solve: node:22.23.3-bookworm-slim: failed to resolve source metadata for docker.io/library/node:22.23.3-bookworm-slim: unexpected status from HEAD request to https://registry-1.docker.io/v2/library/node/manifests/22.23.3-bookworm-slim: 429 Too Many Requests

That turns shards red on main and on every PR, at random.

The three docker/setup-buildx-action steps now:

  • start BuildKit from mirror.gcr.io/moby/buildkit:buildx-stable-1 instead of Docker Hub;
  • tell BuildKit to pull docker.io images through mirror.gcr.io, Google's public Docker Hub mirror. BuildKit falls back to Docker Hub if the mirror misses.

The regression workflow's change filter now also matches .github/workflows/regression.yml, as ci.yml already lists itself for the BeginFrame contract job. Before, an edit to the regression workflow skipped the very shards it changed.

The Dockerfiles are unchanged, so anyone building them locally pulls exactly as before. The two docker run steps run the image the job just built and pull nothing.

Checked

  • mirror.gcr.io serves both images: docker buildx imagetools inspect resolves mirror.gcr.io/moby/buildkit:buildx-stable-1 and mirror.gcr.io/library/node:22.23.3-bookworm-slim.
  • The pinned setup-buildx-action accepts both driver-opts and buildkitd-config-inline.
  • actionlint reports the same findings on the three workflows before and after.
  • This PR's regression shards start BuildKit from mirror.gcr.io/moby/buildkit:buildx-stable-1 (the job log shows the pull) with the mirror config loaded, and no shard hit a 429. BuildKit logs image names as docker.io/... whichever host answers, so the routing was checked on a Linux box with a debug builder and this exact config, building FROM node:22.23.3-bookworm-slim with --no-cache: the manifest request and every layer went to mirror.gcr.io/v2/library/node/..., and Docker Hub got no requests. With the mirror pointed at a bad path, BuildKit logged trying next host after status: 404 Not Found, pulled from Docker Hub and the build succeeded.
  • Docker Hub and the mirror return the same digests for both images, so build cache keys are unchanged.
  • A manual dispatch of the fast video validation workflow on this branch built its image the same way and passed.

Small on purpose: one CI setting in three places, so nothing else can carry it.

@miguel-heygen
miguel-heygen marked this pull request as ready for review October 9, 2026 22:33
@github-actions

github-actions Bot commented Oct 9, 2026 •

Copy link
Copy Markdown
Contributor

Edit accuracy: accurate 2061 (base branch 2061), smooth 1472 of those

The gate passes.
Smoothness is reported in the artifact, not gated. A case fails only if it fails 2 of 3 runs.

Quarantined, measured but not gated (0)

@jrusso1020 jrusso1020 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approve at 301304ad. The mirror only affects how CI pulls Docker Hub images while building test images. Nothing is pushed or published through it, no credentials are involved, and nothing that was pinned before is pinned any less now.

Checked

  • Scope. These three setup-buildx-action steps are the only buildx setups in .github/workflows. All three build CI-only images:
    • ci.yml uses push: false.
    • regression.yml and fast-video-validation.yml use load: true and then docker run the local tag.
    • No workflow logs in to Docker Hub, so no credentials can reach the mirror. Release and publish workflows are untouched.
  • What gets mirrored. Only the docker.io registry. Every FROM in the repo pulls node from Docker Hub (Dockerfile.test, Dockerfile.render, gcp-cloud-run/Dockerfile), and nothing pulls from another registry. When the mirror misses, BuildKit falls back to Docker Hub.
  • Pinning. No base image was pinned by digest before this PR. Both node:22.23.3-bookworm-slim and the BuildKit image were tag references, and moby/buildkit:buildx-stable-1 was already the action's default. Moving that same tag to mirror.gcr.io adds no floating reference.
  • Digests match. I queried both registries directly at review time, and each tag resolves to the same manifest digest on both:
    • library/node:22.23.3-bookworm-slim → sha256:c3de60bf…978392;
    • moby/buildkit:buildx-stable-1 → sha256:cec9f139…2e3dea.
    • Layers are content-addressed against that manifest, so the mirror can serve a given digest's content only as it is.
  • Trust change. Tag resolution now also trusts Google's documented Docker Hub cache, alongside Docker Hub itself. That's a reasonable trade for CI-only, unpublished images.
  • The CI run uses it. The shard-10 log shows buildx create … --driver-opt image=mirror.gcr.io/moby/buildkit:buildx-stable-1, the pull from mirror.gcr.io, and mirrors = ['mirror.gcr.io'] in the loaded buildkitd.toml. All 10 shards passed in the first regression run at this head. The extra buildkitd entitlement flags in that log are the action's defaults and are the same as on main.
  • Self-filter. Adding .github/workflows/regression.yml to the regression change filter is right. Before, a PR that edited the shards' own workflow skipped those shards.

Nits (none blocking)

  • buildx-stable-1 is a floating tag on either registry. Pinning it as mirror.gcr.io/moby/buildkit:buildx-stable-1@sha256:… would make the builder reproducible. That's separate hardening, not something this PR loosened.
  • A cache can lag a tag that gets re-pushed. Official node images are rebuilt under the same tag for Debian updates, so the mirror may briefly serve the previous digest. The same digest stays reachable from Docker Hub, and the GHA build cache keys follow whichever digest comes back, so it's harmless.

This approval waits for the required checks at this head.

— Rames

@miguel-heygen
miguel-heygen added this pull request to the merge queue Oct 9, 2026
Merged via the queue into main with commit 70c873a Oct 9, 2026
179 checks passed
@miguel-heygen
miguel-heygen deleted the ci/docker-hub-mirror branch October 9, 2026 23:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants