Skip to content

development.md

Latest commit

 

History

History
598 lines (540 loc) · 35.7 KB

development.md

File metadata and controls

598 lines (540 loc) · 35.7 KB
Raw

Development

For system-level changes, start with docs/design/system.md and docs/design/roadmap.md, and state which agent decision gets cheaper or more accurate, which layer owns the change, and which durable artifact lets the next worker continue. Read docs/design/ before changing a public surface or revisiting a design decision, and docs/spec/ when a change touches the record-stream contract (record shapes, attribution, the session graph, the cursor). When working on an adapter, read its runtime page (native API inputs and results, resume semantics, current OAR mapping, evidence gaps) and update it when the mapping changes.

Detail lives next to the code: packages/oar/src/README.md (source layout, import rules, ownership), sea-trial/README.md (behavior-suite layout and harness), experiments/README.md (live probes and their conclusions).

Ad-hoc runs while developing

Run first, test second: while shaping a change, run the real thing and look. Drive the CLI (pnpm oar ... builds and runs the in-repo oar): oar run <runtime> "<prompt>" shows a turn's progress live, --record <file> keeps the run as an oar-voyage/3 JSONL log, and oar list, oar installation <id>, oar usage <id> and oar models <id> cover the observation surfaces. Or point a scratch pnpm tsx script at the public Session API, or reuse an experiment. This is how you find out what correct looks like before pinning it.

When the check you need has no tool (a CLI flag, a script that replays a recorded run), build it or extend the oar CLI as part of the change: a command someone else can rerun turns "trust me, I checked" into "run this and see".

  • Ad-hoc evidence never closes a change. Once the behavior is understood, encode it at the cheapest test layer that can catch a regression (below); the commit gate stays the arbiter.
  • Leave the lever behind. A scratch script that earned its keep becomes a test, or an experiment with its conclusion recorded, not a loose file. A tool you built or extended for the run is polished and committed with the change.
  • Leave a handoff. For work that spans turns or owners, record the runtime identity, cursor or artifact, current conclusion, and next action. A status message without a continuation is not an accumulated result.
  • Public evidence must be readable without a login. Repository docs must not rely on private Artifact pages as evidence. Commit the report, reproduction script and selected, redacted results under experiments/ and link them with repository-relative paths. Keep raw credentials, account identifiers and machine-local paths out of published results.
  • Real logins cost quota. Repeatable ad-hoc runs belong on the mock or aimock backends; use a real installation deliberately, as with OAR_TEST=<real id>.

How to validate the changes

Setup: Node.js 24+, pnpm install.

pnpm run typecheck and both packages' builds use tsrs, pinned exactly to 0.20.0-ts7.1.0-beta in the root devDependencies (TypeScript 7.1 beta semantics, not TypeScript 6), locally and in CI. On every tsrs upgrade keep the pin exact and compare the emitted files, especially declarations and source maps, against the previous tsrs and the matching official TypeScript 7 build (migration evidence: timings, comparisons, incremental-cache checks). Native binaries exist for Linux x64/arm64 (GNU and musl), macOS x64/arm64 and Windows x64; elsewhere set TSRS_BINARY to a locally built compiler or rely on the required CI checks (there is no automatic tsc fallback). Incremental build caches live inside dist and are excluded from the npm packages; removing dist forces a complete rebuild.

The official typescript dependency also parses source for the spawn-boundary check in tests/spawn-boundary.test.ts. tests/clean-install.ts checks that a TypeScript 6 host can consume the packed, tsrs-built declarations. An editor on that workspace TypeScript 6 can report different diagnostics from CI; tsrs has its own language server (editor setup). Type-aware lint uses oxlint/oxlint-tsgolint; tsx, esbuild and vitest are not replaced by tsrs.

The tests let whoever makes a change, human or coding agent, verify it end to end without a person as the bottleneck: a change is done when the tests prove it, not when someone has eyeballed it. The suite is also the proof that the contracts hold on every runtime, so never weaken an assertion to get past a failure: a red test means the work is not done.

Each layer below validates a different part of a change at a different cost. Run the cheapest one that can catch your likely mistake, escalate as the change gets riskier, and finish with the commit gate.

Unit: is the pure logic right?

tests/**/*.test.ts (per-runtime subdirectories included), run by pnpm test (vitest). Seconds, no vendor binaries. The inner loop for shared mechanisms: folds, resolvers, process plumbing. Use virtual time via vi.useFakeTimers (mock Date too when clocks matter).

On Windows, Vitest global setup first runs one real CIM process-table query (60-second limit), recording total and query duration in the job log and oar-trial-run/windows-process-table-preparation.jsonl; a failed read or missing observer identity fails setup. This keeps first-use work outside the unit tests' timing windows. The standalone behavior entry does not prewarm CIM, so it exercises product preparation and the cold cleanup path under the 15-second disposal assertion. Product preparation and its separate capture budget: record contract; cold first-use timing: Windows CIM experiment.

The Windows watchdog regression keeps launcher, sampler/CIM, nested test and process-result diagnostics under oar-trial-run/pi-rpc-diagnostics, outside its disposable fixture directory. The Windows check job uploads them even when the unit suite fails. Startup records distinguish an entered sampler, returned initial resource queries, a launched child and an entered test; a missing checkpoint is an evidence gap, not a cause. The diagnostics change no time limit or termination assertion.

Behavior: does the public contract still hold?

One suite in sea-trial/cases/, run against interchangeable backends that OAR_TEST selects (sea-trial/harness/backends.ts; unset means mock, an unavailable runtime skips locally; OAR_TEST_REQUIRE_AVAILABLE=1 makes an unavailable installation fail). Write a case for a contract promise every runtime must honor: it asserts only through the public Session API and must pass on the mock, the aimock backends and real logins alike, race-honest where runtimes may legitimately differ.

  • pnpm sea-trial: the in-process mock fixture. Fast, deterministic, no binaries or network; the default validation for any change to contracts or session behavior.
  • OAR_TEST=<runtime>-aimock pnpm sea-trial (claude-aimock, codex-aimock, pi-aimock, pi-rpc-aimock, pi-durable-aimock, qwen-aimock, grok-aimock, kimi-aimock, opencode-aimock, codebuddy-aimock, dsh-aimock, antigravity-aimock; Qoder uses qoder-native): the real vendor binary and adapter with only the model provider scripted, no login. Use it when you touched a runtime's adapter: it catches real-process integration mistakes the mock cannot.
  • OAR_TEST=scripted pnpm sea-trial: the public scriptedRuntime (@botiverse/oar/testing) held to the same contract.
  • pnpm tsx sea-trial/all.ts [backend...]: several backends, by default mock, mock-shared, mock-isolated, every aimock backend except antigravity-aimock, and Qoder native replay. Run it before pushing a change to shared runtime machinery. It runs at most max(1, min(4, floor(availableParallelism() / 2))) backends at once, leaving room for native CLIs' own threads and children, and every selected backend runs even after another fails. The bound is local only (CI runs backends on separate VMs) and changes no product or individual test deadline (local overload evidence). Pin local native binaries to the CI versions before comparing results (for example, OAR_CODEBUDDY_BIN=/absolute/path/to/2.179.3/codebuddy): an old installation found on PATH is not evidence about the pinned version.
  • OAR_TEST=<real id> (antigravity, claude, codebuddy, codex, cursor, dsh, grok, kimi, opencode, pi, qoder, qwen): your local installation and login. The final word when vendor reality is in doubt; it costs quota, so never run it by default.

Vendor: is the runtime-specific integration right?

sea-trial/vendor/*.vendor.test.ts, run by OAR_TEST=<runtime>-aimock pnpm vitest run sea-trial/vendor. The ACP runtimes' vendor backends (opencode-aimock, kimi-aimock, grok-aimock, antigravity-aimock, sea-trial/harness/aimock-acp.ts; codebuddy-aimock, sea-trial/harness/codebuddy-aimock.ts) need the installed CLI; which CI job runs each is listed under the commit gate. Vendor tests see the scripted provider's side: request contents, error edges, scripted tool rounds. Run them when a change affects what a runtime sends upstream or how it handles vendor-specific behavior; write one for anything that needs the provider's view or a vendor-specific trigger. Each is gated with skipIf(process.env.OAR_TEST !== ...); helpers live in sea-trial/vendor/support/.

The manual OpenCode snapshot comparison retains existing native logs before HOME cleanup under OAR_TRIAL_RUN_DIR/native-opencode/. Retention is bounded to four files and 1 MiB per file, with explicit head/tail gaps and omitted file counts. Normal fixtures do not copy these logs. Raw logs can contain session IDs and local paths; publish selected evidence only. Diagnostic retention decisions explain which other observations remain active and why.

The Antigravity MCP name-clash vendor case retains failure evidence under OAR_TRIAL_RUN_DIR/agy-mcp-failure/ (inside the existing agy-acp-* CI artifact). Before deleting its isolated HOME, it reads up to four native antigravity-cli/log/*.log files, at most 512 KiB from each tail. It also keeps the echo servers' bounded startup and handshake metadata, without RPC payloads. Paths, known credentials and credential-bearing native entries are removed before writing; the manifest identifies missing files and truncation. Passing cases discard the temporary metadata. Missing startup records mean startup was not observed, not that no process was launched. This changes neither the assertion nor the deadline or retry policy. Capture validation and replay.

Experiment: what does the real runtime actually do?

experiments/, run by pnpm tsx experiments/<name>.ts. A live probe that answers one question about a real runtime, kept as evidence with its conclusion indexed in the experiments README. These are not part of the normal CI change-validation gate; reach for one when the vendor's actual behavior is the open question. The manual shared-runtime evidence workflow runs the account-free experiments/sharing probes to collect platform observations, including actual MCP, hook/plugin and tool child cwd and environment. It preserves failures, retains differences as failures, and does not admit a runtime to sharing or alter the pull-request gate.

Live: does the strong claim hold on the real runtime?

The shared cases assert the minimum every backend must honor. The strong claims (a steer lands in the same turn, a queued input runs as its own turn, an abort ends with the runtime's own report, a killed process shows up as an exited response, sub-agent frames attribute) need a real login: experiments/live-contract.ts <runtime> writes one voyage log per scenario plus a report.json of observed facts under oar-trial-run/live-<runtime>-*/. It burns tokens; run it when an adapter changes what the stream carries, and record the observation in the runtime's page and the experiments README.

The commit gate

pnpm run check must be green before every commit: typecheck, lint, pnpm test and the mock behavior suite. Vendor tests are collected by pnpm test but skip unless OAR_TEST names their backend, so run them for the backend you touched. Rebase on origin/main before pushing.

CI (.github/workflows/ci.yml) skips a push or pull request whose changes are all Markdown files or under docs/ or assets/. Otherwise it runs these jobs:

  • check: pnpm run check and pnpm run build on Linux, macOS and Windows, then native checks: CodeBuddy's official npm installation in an isolated HOME and global prefix with its bin directory off PATH (installed version and an idempotent second install); OpenCode 2.2.7 isolated native tool cleanup (readiness, surviving-peer and final residual-process assertions); and Kimi 2.4.1's native login tests against a local OAuth/model server, including browser suppression and credential writes. The agy CLI 1.5.0 (Antigravity) is installed from official platform downloads with pinned SHA-512 digests (install-antigravity.mjs) and runs the antigravity-aimock behavior and vendor suites against a local Gemini provider in an isolated home (concurrency 2, a missing installation fails). Each suite requires a successful install; an earlier unrelated check failure does not skip installation. CodeBuddy, OpenCode, Kimi and Antigravity upload diagnostic artifacts, failures included. CodeBuddy records discovery, install, verification and cleanup phases under oar-trial-run/codebuddy-install, enables npm verbose and timing logs in its isolated environment, and saves the latest ten debug/timing files before deleting that HOME (last 1 MiB per file, with truncation marked). A killed npm process may not finish its timing file; the partial debug log and phase records remain useful. The install limit is 180 seconds and the enclosing child limit 240 seconds. Evidence. The browser-entry unit test bundles the complete portable entry for a browser and refuses Node imports or unresolved externals. Synthetic sharing routing and public placement-refusal regressions stay in these required jobs.
  • behavior: pnpm run sea-trial plus the backend's vendor tests on the same three systems for claude-aimock, codex-aimock, pi-aimock, pi-rpc-aimock (not on Windows, below), pi-durable-aimock, qwen-aimock, dsh-aimock (DSH pinned to DSH_MINIMUM_VERSION 0.2.1-rc.5, a missing installation fails) and qoder-native (Qoder pinned to 1.1.81 because native replay stops driving inference from 1.1.82), the last two at shared-case concurrency 2.
  • pi-rpc-minimum: Ubuntu behavior and vendor suites with a separately installed Pi CLI 1.4.0 selected through OAR_PI_RPC_COMMAND. This guards the explicit-command floor while OAR keeps its current bundled dependency. Any floor increase must update this CI pin and PI_RPC_MINIMUM_VERSION together in the same pull request.
  • pi-rpc-windows (behavior) and pi-rpc-windows (vendor): Windows Pi RPC on separate VMs, so each phase keeps its outcome if the other loses its runner (job ceiling 20 minutes). Both run under the resource sampler, with phase-suffixed artifacts and separate vendor group directories and resource files. The vendor job runs four named steps in sequence (RPC-specific, native Pi, MCP and remaining shared files) that together cover every vendor file exactly once, new files included; each starts Vitest separately with the usual worker settings, and an ordinary failure does not skip later groups. Step names and start/end times stay readable through the GitHub jobs API when logs are not; they identify a group, not a test or why it stopped. Vendor cases keep their 30 to 120 second Vitest limits. An external monitor also stops the test process tree when a case (hooks included) stays active for three minutes, naming it in the resource log; each vendor group has a five-minute command ceiling inside a six-minute step ceiling. Its exit 124 means the diagnostic monitor reached its deadline, not a native test result, and does not prove descendant cleanup (residual processes remain the runner's responsibility). Tree termination runs in the background within a ten-second budget: an unreturned call records termination_unreturned, a thrown one termination_error, and either records stage_exit and exits 124 without joining the worker or taking another resource sample; a returned call waits up to five seconds of that budget for root exit. Output handles stay inherited, so native test output reaches the job log. The always-uploaded artifact holds incremental case events, the last active cases, combined Vitest output and redacted Pi RPC stderr tails. None of this changes an assertion, concurrency or native deadline, or establishes the cause of the intermittent Windows hangs.
  • acp-behavior: Kimi 2.4.1, Grok 1.0.70, OpenCode v1 1.19.5 and CodeBuddy 2.179.3 on Linux and macOS: shared behavior and vendor tests against native programs with local scripted providers, isolated configuration, synthetic keys and npm download caches. Shared-case concurrency and Vitest worker count are both 2; CodeBuddy serializes shared cases in its isolated fixture HOME. OpenCode prepares each behavior fixture's database with one serial native open and exit before concurrent cases begin; a preparation failure stops the suite without retries (the independent cold-start observation retains the native database race). A missing installation fails; individual unsupported capabilities remain visible skips. Ceilings are 20 minutes per job and 12 per suite step, changing no case assertion or native deadline. Logs and traces upload even on failure. Evidence and deferred runtimes.
  • browser-demo: pnpm durable-demo:smoke in headless Chromium on Linux. The real Pi Durable Harness and native fetch provider run in the browser; provider requests go to local scripted responses using a synthetic key.
  • clean-install: pnpm run clean-install (tests/clean-install.ts) on Linux. Both packages are packed and npm installed into an empty project: without the optional peers, the root entry and the browser, forms, observe, kernel, brands, testing, agents and agents/report entries must load and type-check, OAR must probe every built-in runtime, and the line that adds cursor must fail the host's compile; with the SDK added, cursor loads. With the Pi Durable peers added, a pi-durable session opens and its entry bundles for a browser. The CLI, installed after the host's SDK is removed again, lists cursor's models through the SDK it depends on itself. It needs the npm registry.

Outside ci.yml and its pull-request and release gates:

  • acp-windows.yml runs the Kimi, Grok and OpenCode v1 ACP backends on Windows after main pushes. All three had intermittent Windows startup or MCP timeouts whose causes remain unknown. The jobs still fail, without continue-on-error or retries, and retain logs and traces; daily inspection summarizes them. Grok's launch comparison runs after its tests, so it cannot prewarm them. It retains bootstrap read/decode/write/copy/spawn phases for the explicit Node path and repeats that path and the OAR-resolved executable in the same HOME after either success or failure. The seven launches keep their 15-second initialize and five-second cleanup limits; the diagnostic step remains three minutes. Measurement scope.
  • osv-scanner.yml scans dependencies daily and reports findings in its job log without failing; only a scanner error fails it.
  • OpenCode and Grok shared placement is not admitted (explicit shared is refused), so their shared session behavior suites and native shared/isolated controls run only in the manual opencode-sharing-experiment.yml (with a configured plugin child-environment comparison) and grok-sharing-experiment.yml workflows, against experimental adapters outside the package.

The Antigravity, Grok, Kimi and OpenCode synthetic ACP fixtures retain incremental provider requests, matcher selections and streamed response chunks under acp-provider/ in the existing run artifacts. They survive a failed assertion or unfinished turn and fixture HOME cleanup; entries and files have explicit size limits. Headers, URL queries and environment variables are omitted. This is test evidence only and changes no matcher, assertion or native timeout (November 6 failures and boundaries).

Windows codex-aimock, every qoder-native and dsh-aimock behavior job, and the Windows ACP observations, forward child stderr to the job log with OAR_CHILD_STDERR=inherit, including failures before a session trace exists (a login's or logout's process is never forwarded: its output may carry secrets). Artifact upload warns when no trace directory was created; the failed test or startup step still fails the job. Codex RPC exit errors also retain a bounded stderr tail without this setting, so use that evidence before changing a timeout or assertion to address an intermittent startup failure. Installation readiness and --version execution failures retain the native error, signal, timeout, exit code and stderr tail in their exception too. On a one-shot command timeout, diagnostics also snapshot whether Node had observed the root process's exit and the output streams' close, with the root's exit time, elapsed duration, code and signal when available. The snapshot is taken before cleanup; a later cleanup-induced exit cannot replace it. This distinguishes an exited root with open pipes from an exit not yet observed without changing deadlines, pipe draining or process cleanup.

The Windows Codex comparison (nine default/two/one file-worker samples, including a cleanup timeout in the single-worker group) does not establish a worker-limit remedy; CI keeps its per-case concurrency, assertions and deadlines.

How to add a new runtime

  1. Probe reality first. Write an experiment that answers how the vendor actually behaves (session lifecycle, event stream, error shapes) and index its conclusion in the experiments README. Build the adapter on that evidence, not on the vendor's docs, and against the runtime's own interface, never another host's adapter for it (decision).

  2. Implement it in packages/oar/src/runtimes/<id>/. index.ts declares the runtime with defineRuntime({ id, ... }), listing only the capabilities the runtime honestly supports: an absent capability is correct, a faked one is not. That index.ts is also the public @botiverse/oar/runtimes/<id> entry, so it imports no SDK statically: load SDK-bound code on first call (onFirstCall, shared/lazy.ts) and add the id to tests/runtime-entries.test.ts. The session adapter feeds a createSessionKernel():

    • every native frame becomes exactly one Frame record via kernel.frame() (type, native verbatim, events holding the RuntimeEventBodys oar reads from it), never gated on turn state, never dropped, never synthesized; the shared eventsOf derives the flat Session.events() surface from those records, so an adapter never builds it;
    • control calls go through kernel.control() so request and response are records;
    • the process exit is an exited response;
    • a whole operation the runtime cannot do is a member the session lacks: no steer on a runtime that cannot inject, never a steer that always rejects; capabilities declares queue durability, the attribution tier and image input as the runtime actually exposes them;
    • an option the runtime cannot honor is refused at open with an UnsupportedOptionError, never dropped; one it refuses whenever given is declared in refusedSessionOptions and checked by refuseSessionOptions, so the declaration and the refusal agree.

    Keep the projection a pure fold (frame → commands) so replay tests (tests/replay/, fixtures captured by pnpm sea-trial:record) can pin it. Keep parsing, compatibility policy and protocol details in that directory; reuse shared/ mechanisms freely but never add runtime identity to shared/ (source layout has the import rules).

  3. Cover the whole runtime in the first version, not just sessions. Hosts need the account side on day one: installation and version (installation, install/installPlan, checkUpdate/upgrade), sign-in (login, logout, authStatus, providers), listModels, accountUsage, and limit resets where the runtime has them. For each, find the runtime's own interface for it first (a CLI subcommand, a protocol method, a read-only slash command) and implement it in the same change. Where the runtime has none, leave the member absent or return its typed unsupported with the reason, and say on the runtime page what was checked, so the gap is a finding, not a deferral. A usage or rate limit the runtime reports on a turn is classified as a limit from the start (why).

  4. Register it in src/index.ts, the only composition root: import, add to the built-in defaultRuntimes registry, re-export. Give it a runtimeBrands entry in src/brands.ts with its SVG in packages/oar/assets/brands/ (attribution in NOTICE.md). A runtime that needs something the host installs (cursor's SDK) is not built in: export its constructor instead, and add it to allRuntimes in sea-trial/harness/runtimes.ts and to the CLI's registry in packages/cli/src/runtimes.ts (capabilities). A runtime with host-owned execution/storage, such as pi-durable, stays out of the default and CLI registries; test it with its configured fixture.

  5. Make the behavior suite pass unchanged. OAR_TEST=<id> pnpm sea-trial against your real local installation. The cases in sea-trial/cases/ are the contract: make the runtime pass them, don't loosen them to fit. A case that needs a capability skips on its absence (a session without steer), never on a runtime name.

  6. Add an aimock backend if feasible (sea-trial/harness/aimock.ts, backends.ts, and the behavior or acp-behavior matrix in .github/workflows/ci.yml) so the contract stays verifiable without a login, and add vendor tests for the provider-side specifics worth pinning.

  7. Update the docs in the same commit: the runtime lists in the root and packages/oar READMEs, a runtime page and its index entry, and the experiments README for any probes you added. The runtime page starts with the vendor's programming interface and concepts, then maps each capability to OAR or names the missing support and evidence.

How to fix a runtime bug

  1. Locate the layer; that picks the test. Wrong behavior visible through the public Session API: a behavior case (if every runtime must honor the promise). A wrong request sent to the vendor, or a mishandled vendor-specific edge: a vendor test. A pure-logic mistake in a fold or resolver: a unit test.
  2. If the vendor's actual behavior is the open question, probe it first with an experiment and record the conclusion; don't derive the fix from documentation guesses.
  3. Write the failing test before the fix, at the cheapest layer that can express it. OAR_TEST=<runtime>-aimock reproduces most integration bugs without a login.
  4. Fix at the right level. Runtime-specific policy belongs in runtimes/<id>/; touch shared/ only when the mistake is genuinely runtime-independent, and then run pnpm tsx sea-trial/all.ts, because every backend is affected.
  5. Validate up the layers and finish with pnpm run check. The reproducing test stays in the suite as the regression proof.
  6. Update the runtime page when a bug changes the documented native behavior, OAR mapping or verification status. Distinguish the vendor's behavior from loss or policy introduced by OAR.

Testing conventions

  • Assert, don't if + throw. Checks use node:assert/strict (or vitest expect), inside behavior cases and vendor tests too. Shared narrowing helpers (promptTurn, expectAvailable) live in sea-trial/vendor/support/asserts.ts.
  • Snapshot value shapes. When asserting what a value looks like (an outcome object, an event sequence, a fold result), pin the whole thing with toMatchInlineSnapshot; no substring checks like reason.includes("400"). Snapshots update on local runs and are enforced when CI is set (vitest.config.ts: update none). Keep plain assertions for logic and invariants (ordering, ranges, idempotence).
  • Test layer = assertion channel. What the public Session API shows goes in a behavior case that runs on every backend; anything that needs the scripted provider's view (request contents, error edges, vendor fingerprints) or a vendor-specific trigger goes in a vendor test.
  • Failures must self-diagnose. Include the actual value in the assertion message; long-running suites write traces and artifacts under oar-trial-run/.

How to release

Both packages carry one version. A release is a pull request (chore: release vX.Y.Z) that only sets version in packages/oar/package.json and packages/cli/package.json; its body is the release notes. Squash merge it, then start .github/workflows/release.yml on main:

stamp workflow run release.yml --repo botiverse/oar -f version=X.Y.Z

From the release commit's merge until its tag exists, do not merge a pull request that changes .github/workflows/. A workflow change on main can make GitHub reject the older commit's new tag as a workflow update, which the release job's token cannot authorize. Cindy owns this hold and lifts it once the tag is confirmed at the release commit.

Dispatch right after the release pull request merges: the workflow releases the head of main only after ci.yml succeeded on that exact commit (a docs-only head has no run and fails the gate). It refuses a version that already has a tag or differs from either package version, runs pnpm run check and pnpm run build, and publishes both packages with pnpm -r publish, which rewrites the CLI's workspace:* dependency to the real version. A separate release job, which runs no project code and is the only one with write access, then creates the annotated vX.Y.Z tag and the GitHub release, whose notes are the release pull request's body. For a transient failure after publishing, "Re-run failed jobs" retries only the failed release job. A workflow-permission refusal needs the recovery below; retrying does not grant the missing permission. If ci.yml flaked on the release commit, start a fresh run with stamp workflow run ci.yml --repo botiverse/oar and dispatch again once it passes. npm trusted publishing (OIDC) authenticates the publish job; the repo holds no npm token. Pushing a vX.Y.Z tag by hand still publishes that commit, without the CI gate or the GitHub release.

Recover after npm publication

If tag creation fails with without workflows permission, keep the original run's commit as the release target. This can happen when main gains workflow changes while the release waits for CI or publishes: the old commit's workflow files now differ from main. It does not mean every release that contains a workflow change will fail. The 0.108.1 investigation records this ordering and the successful recovery.

Changing the command to gh release create --target <sha> does not solve the permission problem: GitHub documents the same workflow-write requirement for automatic tag creation by the releases API, which GITHUB_TOKEN cannot acquire. The create-reference API also lists workflow-write permission; it is not a verified bypass. Keep publication on the original commit rather than tagging a newer main or adding a more privileged credential to the job.

Only Cindy performs this recovery:

  1. Save the failed run and its full commit SHA. Confirm the publish job succeeded and both exact package versions are present on the public npm registry (npm view @botiverse/oar@X.Y.Z version and npm view @botiverse/oar-cli@X.Y.Z version, both with --registry=https://registry.npmjs.org). Verify those names and versions against the manifests at the original commit. Missing or inconsistent publication evidence needs investigation, not this recovery.

  2. Read the remote tag, including the peeled commit for an annotated tag: git ls-remote origin refs/tags/vX.Y.Z 'refs/tags/vX.Y.Z^{}'. If it already points to the original commit, skip tag creation. If it points elsewhere, stop; do not move or force-push it.

  3. Only if the remote tag is absent, create it at the verified original commit and push through Stamp from a checkout configured with stamp repo use .. Replace the placeholders before running:

    git tag -a vX.Y.Z ORIGINAL_RELEASE_COMMIT_SHA -m vX.Y.Z
stamp push origin refs/tags/vX.Y.Z

    An existing local tag must be checked against that commit too; do not overwrite it. Check Stamp's publication receipt and re-read the remote tag before continuing. Stamp has a different authorization path from the workflow's GITHUB_TOKEN; if it refuses the push, keep the failure and resolve that permission with the repository owner.

  4. An external tag push triggers the legacy tag-push publish workflow. Record and inspect that run before dispatching recovery. It is not suppressed like a push made with GITHUB_TOKEN: 0.108.1 reran the publish command, which found no new packages. Do not promise that a manual tag push runs only the release job, and do not start another normal publish dispatch or rerun all jobs after npm publication succeeded.

If npm publication succeeded and the correct tag already exists but the GitHub release is missing, dispatch the recovery path explicitly from main (also when recovering an older version; do not use the tag as --ref):

stamp workflow run release.yml --repo botiverse/oar --ref main -f version=X.Y.Z -f release_only=true

Recovery does not publish packages, create or move tags, run package code, or require the historical version to remain npm's latest. It requires both exact package versions on the public npm registry, matching package names and versions in the tagged commit, and exactly one merged release pull request for that commit. It takes notes from that pull request rather than current main, verifies the tag has not moved, and uses gh release create --verify-tag. Missing or inconsistent evidence, API errors and an already existing GitHub release fail the recovery. Normal publication keeps its existing gates.