publish.yml Tag + push is not concurrency-safe: a merge during the run orphans tags and fails the job after npm already published

[khaliqgant]

What happens

The Tag + push step (.github/workflows/publish.yml:608-617) runs after npm publish and does:

for entry in ${{ steps.bump.outputs.versions }}; do
  git tag -a "$pkg-v$version" -m "$NPM_NAME@$version"
done
git push origin HEAD --follow-tags

It never syncs with main between checkout and push. If any PR merges to main while the job is running, the branch ref push is a non-fast-forward and is rejected — but --follow-tags pushes tag refs and the branch ref independently, so the new tag refs land while HEAD -> main is rejected. Net result:

• ✅ all packages published to npm
• ✅ all *-v<version> tags pushed
• ❌ the version-bump commit never reaches main
• ❌ the job reports failure — after the irreversible npm publish already succeeded

Observed (run 29573660080, the 4.1.27 release)

 * [new tag]  runtime-v4.1.27 -> runtime-v4.1.27
 ...
 ! [rejected] HEAD -> main (fetch first)
error: failed to push some refs

PR #283 merged mid-run. npm went to 4.1.27; main's manifests stayed 4.1.26; twelve *-v4.1.27 tags now point at orphaned commit 7fe85f78, which is not an ancestor of main. Given release cadence here (main advanced 4.1.26 → 4.1.32 within an hour), this is a when, not an if.

Why it currently 'self-heals' but shouldn't be relied on

The Heal local versions to lockstep baseline step (publish.yml:120+) already absorbs failure mode #1 — a prior run that published to npm but failed at Tag + push — by pulling local versions up to the highest npm latest before the next bump. That's why the 4.1.27 drift disappeared once 4.1.28/4.1.29 ran.

But that's recovery on the next release, not correctness of the current one. The failing run still:

1. reports failure despite a fully successful publish (misleading; invites a re-run that would then collide at the "Verify new versions are not yet published" gate),
2. leaves orphaned tags pointing at a commit not on main,
3. defers repo consistency to whenever someone next publishes.

Fix

Make Tag + push rebase-and-retry so the release completes on its own run:

# create tags first (local), then:
for i in 1 2 3; do
  git fetch origin main
  git rebase origin/main || { git rebase --abort; continue; }
  if git push origin HEAD --follow-tags; then exit 0; fi
done
echo 'Tag + push failed after retries' >&2; exit 1

Rebasing the bump commit onto concurrent merges is always safe here — it only touches package.json versions + CHANGELOGs, which the heal step already reconciles to a lockstep baseline. Alternatively, push the commit before the tags so a rejected branch doesn't leave dangling tags, but rebase-retry is the more complete fix.

Cleanup for the existing orphans

The twelve *-v4.1.27 tags still point at 7fe85f78 (not on main). Harmless but misleading — delete if tags are meant to mark release commits in history.

Context

Surfaced while shipping workforce#277 ([[NO_REPLY]]); the 4.1.27 publish I drove is the run above. Consumer-side pin follow-up: AgentWorkforce/cloud#2718.