fix(checkpoint): report save failures instead of dropping or crashing on them

[Agnik47]

Fixes #71

The bug

save() is declared void but does asynchronous work, and nobody observed the resulting promise:

// src/orchestrator/checkpoint.ts:64-74
save(checkpoint: RunCheckpoint): void {
  const currentQueue = this.saveLock.get(checkpoint.runId) || Promise.resolve()
  const nextQueue = currentQueue.then(() => this._performSave(checkpoint))
  this.saveLock.set(checkpoint.runId, nextQueue)

  nextQueue.finally(() => { ... })      // return value discarded
}

_performSave ends with throw lastError when the retries are exhausted or the error is non-retriable — ENOSPC, EACCES, a read-only volume, a bad path. That rejection had no handler: nextQueue was never caught, and nextQueue.finally(...) created a second unhandled rejected promise. Under Bun/Node that becomes an unhandledRejection, which by default terminates the process mid-run, with nothing indicating a checkpoint write rather than a provider was the cause.

Three consequences, all reproduced in the tests:

1. The process dies on an opaque rejection.
2. Chained saves are skipped. A save() issued while a rejected promise sat in saveLock chained onto it with .then(), so _performSave was never called — those updates were silently lost.
3. flush() inherited the rejection, rethrowing into Orchestrator.run (index.ts:311) after every phase had completed, so runBenchmark wrote status: "failed" over a run whose work was entirely done.

Plus the queue promised a snapshot it never delivered: JSON.stringify ran inside _performSave, i.e. at write time, while the checkpoint object is mutated concurrently by in-flight tasks (updatePhase does Object.assign).

The fix

Terminate the chain with a .catch that records the failure and logs it. The promise stored in saveLock therefore never rejects — which fixes points 1 and 2 together: there is no unhandled rejection, and a failed write no longer prevents the saves queued behind it from running. .finally is likewise safe, since the promise it observes cannot reject.

flush() now surfaces failures deliberately rather than by accident: it reports accumulated errors naming checkpoint persistence and the affected run, then consumes them so a later flush reports new failures instead of repeating one already surfaced. Added hasSaveError(runId) for a non-throwing check, and delete(runId) clears any pending error so a removed run cannot fail a later flush.

On whether flush() should still throw — I kept it throwing. If the checkpoint cannot be written then updateStatus(checkpoint, "completed") cannot be written either, so the on-disk record is stale no matter what; claiming completion would be false. The defect in point 3 was the opacity, not the failure itself, and the message now names the cause. Both call sites (orchestrator/index.ts:311, routes/runs.ts:269) already sit inside a catch, so this is the same control flow with a legible reason.

Serialise in save() rather than _performSave, so a queued write persists the state as of the save() call instead of whatever concurrent tasks have since mutated it into. updatedAt moves with it, which is the more accurate timestamp anyway.

Verification

• bun test — 11 new tests in src/orchestrator/checkpoint.test.ts pass.
• Against unfixed main, 9 of the 11 fail, including the unhandled-rejection test, which catches the real rejection via a process.on("unhandledRejection") listener.
• Failure injection is real, not mocked: writes are broken by rooting the run directory under a regular file, so mkdir and the write both fail with ENOTDIR — the same code path as ENOSPC/EACCES. Each test gets its own mkdtemp directory, cleaned up in afterEach.
• Covered: successful persistence, call-time snapshotting, no unhandled rejection, later saves still running after a failure, quiet flush on success, the error naming persistence and the run, error consumption, all-runs flush, delete clearing pending errors, and write ordering.
• tsc --noEmit clean.

src/orchestrator/checkpoint.ts already fails prettier --check on main; I left that alone and confirmed every line I added is within the configured printWidth.