Repository navigation
Roadmap v0.41.0: Durable execution, recovery, and aggregation #755
Description
Summary
Make externally scheduled Waza runs safe to retry, resume, inspect, and aggregate without corrupting evidence or recomputing completed work. This is the second of three sequential roadmap releases for production-scale external execution.
Release sequence:
- v0.40.0: Versioned, exact runner contract
- v0.41.0: Durable execution, recovery, and aggregation
- v0.42.0: Verifiable production integration
Current signals
- #752 and #753 improve reviewable conversation evidence while preserving missing-data and first-repetition limitations.
- This release extends the workflow reliability epic #657 with durable externally scheduled execution; it does not replace that work.
Goals
Make externally scheduled runs safe to retry, resume, inspect, and aggregate without corrupting evidence or recomputing completed work.
Scoped deliverables
- Define immutable, content-addressed remote artifact manifests and idempotent create/finalize semantics.
- Separate logical IDs from individual attempts while preventing two writers from silently overwriting the same artifact.
- Add resumability/checkpointing at documented safe boundaries, including compatibility checks before resume.
- Provide standalone aggregation that consumes completed result/artifact manifests without starting agents or requiring the original worker.
- Specify failure propagation, partial completion, retry eligibility, cancellation acknowledgement, deadlines, and orphaned-attempt handling.
- Preserve conversation, tool, grader, reliability, usage, and provenance evidence with explicit unavailable/incomplete states.
Cross-cutting completion expectations
Every feature in this release is complete only when durable storage, retry, resume, lifecycle, aggregation, and evidence behavior is implemented and validated at its documented failure boundaries. Automated coverage must include focused unit tests, storage and process integration tests, cross-feature conformance fixtures, and kill/retry/resume end-to-end scenarios as relevant.
User and developer documentation must define the manifest, attempt, checkpoint, aggregation, lifecycle, and evidence contracts. The documentation site must connect those contracts through operational examples and troubleshooting guidance. Release and migration notes must identify schema or behavior changes, compatibility limits, and safe adoption steps.
Lab dashboard coverage
All six features add user-observable operational state and therefore require waza dashboard --lab additions:
- #763: manifest upload/finalization state and integrity diagnostics.
- #764: logical-work and attempt lineage, duplicate handling, and result reuse.
- #765: checkpoint availability, compatibility, and resume progress.
- #766: aggregate inputs, conflicts, omissions, and completion state.
- #767: retry eligibility, cancellation, deadline, partial-completion, and orphan status.
- #768: evidence availability, completeness, redaction, conflict, and lineage.
Explicit non-goals
- No scheduler-owned retry policy, global transaction coordinator, mutable shared result directory, or exactly-once infrastructure claim.
- No aggregation that upgrades missing evidence into a pass.
- No implicit resume across incompatible runner, schema, model, tool, or policy versions.
Dependencies
- Depends on v0.40.0: Versioned, exact runner contract, including its deterministic identity model.
- This release is a prerequisite for v0.42.0: Verifiable production integration.
- Existing snapshot/replay, reliability, evidence, and controlled-comparison formats tracked through #657.
- Storage backends must implement the Waza artifact contract; Waza does not own their replication or availability service-level commitment.
Acceptance criteria
- Concurrent duplicate submissions either reuse a verified immutable result or create distinct attempts; they never overwrite evidence.
- Kill-and-resume tests at every supported checkpoint produce a valid final manifest without rerunning completed units.
- Cancellation and timeout tests prove bounded shutdown and distinguish requested, acknowledged, and forced termination.
- Standalone aggregation is deterministic for the same manifest set and reports missing, conflicting, cancelled, and failed inputs explicitly.
- Artifact corruption, stale leases, incompatible checkpoints, and partial uploads fail closed with actionable diagnostics.
- All six feature issues meet their tailored implementation, automated-test, user/developer documentation, documentation-site, release-note, and Lab dashboard criteria.
- The documentation site provides a connected operational path from immutable publication through retries, checkpoints, aggregation, lifecycle handling, and evidence review.
- Lab views remain derived from public manifests and lifecycle contracts and do not introduce mutable state or scheduler responsibilities.
Risks
- Storage consistency models may make idempotency appear stronger than it is.
- Checkpoints can capture secrets or non-portable runtime state.
- Aggregation compatibility may become coupled to internal result layouts.
🤖 Posted by Henry (AI Agent)
Activity
spboyer commentedon Oct 28, 2026
User-directed release scope addition (October 28, 2026): added #790 as a sub-issue targeting v0.41.0. Keep Waza as the execution/grading/evidence harness and define a versioned external optimizer adapter boundary, with a fake conformance adapter and one verified reference integration. Depends on the v0.40.0 runner contracts and schema-interoperability foundation #789; preserve held-out qualification boundaries, explicit budgets/consent, and evidence provenance. No new optimization algorithm or blanket SkillOpt/GEPA/WikiSkill compatibility claim. Completion includes offline failure tests, an opt-in walkthrough, public docs/site guidance, release notes, and Lab coverage for new saved result state.
docs: reconcile merged v0.41.0 release scope #755
2 remaining items
spboyer commentedon Nov 11, 2026
Status as of 2026-11-11T23:36:54Z (scope: release:v0.41.0)
Slots/work in flight: 3 item PRs (feat: captured checkpoint CLI and HTTP source preview #959-feat: add protected checkpoint recovery to Lab (#765) #961); configured worker cap unavailable. chore: prepare v0.41.0 release (publication pending) #951 release preparation remains protected and draft with review required. No new ready issues admitted. Protected work: chore: prepare v0.41.0 release (publication pending) #951, feat: recover protected evidence capture libraries (partial #768) #958-feat: add protected checkpoint recovery to Lab (#765) #961.
Blockers: feat: captured checkpoint CLI and HTTP source preview #959 current-head Go run is in progress and review threads remain open; feat: execution-aware standalone aggregation core #766 (partial) #960 has a dashboard failure and Go run in progress; feat: add protected checkpoint recovery to Lab (#765) #961 needs refreshed-stack reconciliation; Feature: Add compatible resumability and checkpoints #765 waits on Feature: Separate logical work identities from execution attempts #764; Feature: Aggregate completed manifests independently #766 waits on Feature: Preserve complete execution evidence states #768; Feature: Define distributed execution lifecycle semantics #767 waits on Feature: Separate logical work identities from execution attempts #764/Feature: Add compatible resumability and checkpoints #765; Feature: Preserve complete execution evidence states #768 waits on Epic: trustworthy regression testing for agent workflows #657; Feature: Add an optimizer adapter contract over the Waza evaluation harness #790 waits on Feature: Add versioned external eval schema interoperability #789 and API/license research.
Owner decisions waiting on Shayne Boyer (@spboyer): none.
Pace: forecast 240 active runs/day (planner 36, worker 144, verifier 60) against budget 240; observed daily consumption unavailable in this execution. Workarounds to remove: none tracked.
spboyer commentedon Nov 11, 2026
merged #935 9843e94
spboyer commentedon Nov 11, 2026
merged #856 fadaed1
spboyer commentedon Nov 11, 2026
merged #944 82bffdf
spboyer commentedon Nov 11, 2026
Release coordination snapshot — 2026-11-11:
Branches are owned by one repair lane each; downstream stack updates will follow bottom-up to avoid duplicate builds and concurrent writes.
spboyer commentedon Nov 11, 2026
merged #943 444cd77
spboyer commentedon Nov 11, 2026
Status update — 2026-11-11
444cd77f; this is not yet onmain. PR feat: recover logical execution-attempt core (partial #764) #942 now carries the combined attempt-lineage and checkpoint-core change, with its Go/native integration checks running.feat: recover logical execution-attempt core (partial #764) (#942)
spboyer commentedon Nov 11, 2026
Status update — 2026-11-11
mainasd9631f7a. Required checks were green and review threads resolved at merge. This lands the logical-attempt and exact-unit checkpoint core; it does not close the roadmap acceptance criteria.main. Conflict resolution and fresh CI are underway.No feature issue or acceptance checkbox is closed by this partial integration.
spboyer commentedon Nov 11, 2026
merged #942 d9631f7
feat: recover protected evidence capture libraries (partial #768) (#958)
spboyer commentedon Nov 11, 2026
spboyer commentedon Nov 11, 2026
Status update - 2026-11-11
mainas9cde557e; all required checks passed and review threads were resolved.main, and rerunning CI before resolving the four operation-error threads.No feature issue or roadmap acceptance criterion is closed by this partial integration.
spboyer commentedon Nov 11, 2026
Status update - 2026-11-11
7c933284. Terra confirmed the stale webserver assertion is corrected; the Windows small-write timeout is now 5 seconds. Fresh CI is running. Four operation-error review threads remain open pending the new checks and final reply/resolution.0960d77a; the direct review thread is resolved. Dashboard build, runner container, offline producer, pinned runtimes and Squad tools are passing; the remaining CI checks are running. The inherited feat: captured checkpoint CLI and HTTP source preview #959 thread remains open until its dependency is integrated.No feature or roadmap acceptance criterion is closed by this partial integration.
spboyer commentedon Nov 11, 2026
Status update - 2026-11-11
2b216c07; localgolangci-lint runand the targeted CLI regression pass. Fresh CI has no failures so far, with six required checks still running. The PR remains draft with 10 unresolved review threads, so it is not ready to transition or merge.9b297a19). All 18 aggregate-client Playwright tests andnpm run lintpass locally; fresh CI is running. The branch is behindmain, and its inherited feat: captured checkpoint CLI and HTTP source preview #959 review thread remains unresolved.d1712fb2, now mergeable with no unresolved review threads. Local web lint/build, 64 checkpoint-operation Playwright tests, targeted Go webapi/webserver tests, and site documentation tests/build pass; CI is running.No feature or roadmap acceptance criterion is closed by this partial integration; no PR is being merged while CI or review feedback is pending.