diff --git a/CHANGELOG.md b/CHANGELOG.md
index 92d139435..2044a58f1 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,12 +9,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
+- **Standalone manifest aggregation (unreleased v0.41 source preview, #766)** -
+ Aggregate explicit public completed manifest/evidence references without
+ agents, credential helpers or private worker ownership. Aggregate schema 1.0
+ retains separate member/manifest/execution-record populations, repeated
+ executions, compatible views, original failed/cancelled outcomes and explicit
+ omissions/conflicts. Local CLI/API/Lab composition and canonical JSON exports
+ distinguish admission completeness from pass/attestation and withhold partial
+ or cached historical usage. No automatic migration invents execution records,
+ evidence or billing. See [ordering, compatibility and migration](docs/MANIFEST_AGGREGATION.md).
+
- **Checkpoint embedding API (unreleased partial #765)** - Recover immutable
before-unit/finalized-unit checkpoints and compatibility-checked restore atop
the attempt core. Finalized restore reuses the original terminal; before-unit
restore rejects a recorded dispatch even without completion, preventing
- uncertain execution from silently rerunning. Public CLI/API/Lab and full
- production-CLI/platform acceptance remain pending. See
+ uncertain execution from silently rerunning. Source commands `runner checkpoint
+ create` and `runner checkpoint resume` emit capture-approved results. Protected
+ HTTP operations and the source Lab preserve original outcomes, unavailable
+ progress, and exact JSON exports without raw source inspection. Linux clean-CLI
+ boundary-kill/fresh-process controls cover offline mock work; full roadmap,
+ native-platform acceptance, and independent approval remain pending. See
[checkpoint contracts and migration limits](docs/EXECUTION_CHECKPOINTS.md).
- **Logical execution-attempt embedding API (unreleased partial #764)** - Recover
diff --git a/README.md b/README.md
index 6b72fb34a..3edd1d832 100644
--- a/README.md
+++ b/README.md
@@ -30,7 +30,10 @@ adds immutable reservations and explicit verified reuse without changing
`runner run`. Public submission commands and Lab lineage integration remain
pending; this is not available in installed v0.40.1 artifacts.
The [checkpoint embedding preview](docs/EXECUTION_CHECKPOINTS.md) adds compatible
-whole-unit recovery; its public CLI/API/Lab integration is also pending.
+whole-unit recovery. Its unreleased source CLI adds `runner checkpoint create`
+and `runner checkpoint resume`, with protected HTTP create/resume operations.
+The source Lab can create/resume checkpoints and export protected results.
+Ordered evaluation-level lineage and full platform acceptance remain pending.
**v0.40.0 adds** bounded
runner/text execution, selected AgentV assertion conversion, task-level MCP
@@ -291,6 +294,26 @@ canonical workflow, target-specific validation steps, and a reusable delegation
## Commands
+### `waza aggregate` (unreleased v0.41 source preview)
+
+Aggregate explicit completed public manifests from an existing local immutable
+store without starting agents or requiring the original worker:
+
+```bash
+waza aggregate --store .waza-artifacts "$DIGEST" -o aggregate.json
+waza aggregate --store .waza-artifacts --inputs inputs.json -o qualified.json
+```
+
+Input reference pairs, verified manifests and verified execution records are
+separate populations. Repeated executions and compatible views are retained;
+missing/conflicting inputs remain in emitted reports and exit nonzero.
+Complete admission is not a passing task or execution attestation. Historical
+usage is unavailable rather than a partial or cached subtotal. Lab's
+**Runner** view's **Aggregate completed public manifests** panel exposes composition,
+diagnostics and the exact JSON download.
+See [worked examples](site/src/content/docs/guides/manifest-aggregation.mdx) and
+[ordering, compatibility and migration](docs/MANIFEST_AGGREGATION.md).
+
### `waza artifact` (unreleased v0.41 source preview)
Publish immutable artifact manifests without executing agents. The opt-in store
@@ -349,7 +372,7 @@ waza runner run < examples/runner/unit.json
The v0.40.0 release implements `negotiation`, `exact-unit`, `inline.mock-text`,
`terminal-result` and opt-in `inline.copilot-text`.
-Only `runner run` starts an engine; each dispatch runs its selected task/trial/
+In v0.40.0, only `runner run` starts an engine; each dispatch runs its selected task/trial/
attempt once, without discovery, cache or retries. Admitted receipts include
versioned [logical identities](site/src/content/docs/guides/runner-identities.mdx)
derived from the original captured input, not proof of execution or deduplication.
@@ -392,6 +415,13 @@ The task entry must be eval-relative, and successful adaptation passes shared
offline admission within the complete 4 MiB emitted-byte limit.
See the [compatibility table and before/after examples](site/src/content/docs/guides/runner-compatibility.mdx).
+**Unreleased checkpoint source preview:** `waza runner checkpoint create --store
+.waza-artifacts < submission.json` publishes whole-unit checkpoints;
+`waza runner checkpoint resume CHECKPOINT_SHA256 --store .waza-artifacts <
+submission.json` admits compatible recovery. These commands require a clean
+VCS-stamped CLI and a captured submission wrapper, not a bare runner request.
+See the [checkpoint workflow and limits](site/src/content/docs/guides/immutable-artifacts.mdx#checkpoint-public-source-preview).
+
The [local container reference](site/src/content/docs/guides/runner-container.mdx)
keeps the default scratch image mock-only and provides an explicit
`native-reference` target for the exact-unit native Copilot text profile. The
diff --git a/cmd/waza/cmd_aggregate.go b/cmd/waza/cmd_aggregate.go
new file mode 100644
index 000000000..6a030249e
--- /dev/null
+++ b/cmd/waza/cmd_aggregate.go
@@ -0,0 +1,94 @@
+package main
+
+import (
+ "errors"
+ "fmt"
+ "io"
+ "os"
+ "path/filepath"
+
+ "github.com/microsoft/waza/internal/artifactmanifest"
+ "github.com/microsoft/waza/internal/manifestaggregate"
+ "github.com/spf13/cobra"
+)
+
+func newAggregateCommand() *cobra.Command {
+ var directory, inputPath, outputPath string
+ cmd := &cobra.Command{
+ Use: "aggregate [MANIFEST_SHA256...]",
+ Short: "Aggregate public completed manifests without agents or worker state",
+ SilenceUsage: true,
+ Long: "Aggregate explicit references from an existing local immutable store. Optional evidence sidecars qualify recorded categories and execution records. Admission completeness is not a pass verdict or execution attestation. Recorded historical usage is not new SDK billing.",
+ RunE: func(cmd *cobra.Command, args []string) error {
+ inputs := make([]manifestaggregate.Input, 0, len(args))
+ if inputPath != "" {
+ if len(args) != 0 {
+ return fmt.Errorf("%w: --inputs and positional digests are mutually exclusive", manifestaggregate.ErrInvalid)
+ }
+ data, err := readArtifactFile(inputPath, artifactmanifest.MaxManifestBytes)
+ if err != nil {
+ return err
+ }
+ inputs, err = manifestaggregate.ParseInputs(data)
+ if err != nil {
+ return err
+ }
+ } else {
+ for _, digest := range args {
+ inputs = append(inputs, manifestaggregate.Input{Manifest: digest})
+ }
+ }
+ inputs, err := manifestaggregate.Normalize(inputs)
+ if err != nil {
+ return err
+ }
+ sources, closeSources, err := manifestaggregate.OpenLocal(directory)
+ if err != nil {
+ return err
+ }
+ report, buildErr := manifestaggregate.Build(cmd.Context(), sources, inputs, manifestaggregate.DefaultLimits())
+ closeErr := closeSources()
+ if closeErr != nil {
+ // Do not emit a success-shaped report after a local close failure.
+ return errors.Join(buildErr, fmt.Errorf("closing aggregate sources: %w", closeErr))
+ }
+ if report == nil {
+ return buildErr
+ }
+ wire, err := manifestaggregate.Encode(*report)
+ if err != nil {
+ return errors.Join(buildErr, err)
+ }
+ if outputPath != "" {
+ if err := writeAggregateOutput(outputPath, wire); err != nil {
+ return errors.Join(buildErr, err)
+ }
+ }
+ _, emitErr := cmd.OutOrStdout().Write(append(wire, '\n'))
+ return errors.Join(buildErr, emitErr)
+ },
+ }
+ cmd.Flags().StringVar(&directory, "store", ".waza-artifacts", "Existing local immutable store (never reads ambient configuration or starts credentials)")
+ cmd.Flags().StringVar(&inputPath, "inputs", "", "JSON array of manifest_sha256 and optional evidence_sha256 reference pairs")
+ cmd.Flags().StringVarP(&outputPath, "output", "o", "", "Create a new canonical aggregate JSON file; existing files are never replaced")
+ return cmd
+}
+
+func writeAggregateOutput(path string, wire []byte) error {
+ root, err := os.OpenRoot(filepath.Dir(path))
+ if err != nil {
+ return fmt.Errorf("opening aggregate output parent: %w", err)
+ }
+ file, err := root.OpenFile(filepath.Base(path), os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o600)
+ if err != nil {
+ return errors.Join(fmt.Errorf("creating aggregate output: %w", err), root.Close())
+ }
+ written, writeErr := file.Write(wire)
+ if writeErr == nil && written != len(wire) {
+ writeErr = io.ErrShortWrite
+ }
+ if err := errors.Join(writeErr, file.Close(), root.Close()); err != nil {
+ return fmt.Errorf("aggregate output may be partial at %q: %w", path, err)
+ }
+ return nil
+}
diff --git a/cmd/waza/cmd_aggregate_offline_test.go b/cmd/waza/cmd_aggregate_offline_test.go
new file mode 100644
index 000000000..1b05b69df
--- /dev/null
+++ b/cmd/waza/cmd_aggregate_offline_test.go
@@ -0,0 +1,110 @@
+package main
+
+import (
+ "bytes"
+ "errors"
+ "net/http"
+ "os"
+ "path/filepath"
+ "sync/atomic"
+ "testing"
+ "time"
+
+ "github.com/microsoft/waza/internal/artifactmanifest"
+ "github.com/microsoft/waza/internal/manifestaggregate"
+ "github.com/microsoft/waza/internal/testutil/manifestfixture"
+ "github.com/spf13/cobra"
+ "github.com/stretchr/testify/require"
+)
+
+type aggregateNetworkGuard struct {
+ calls atomic.Int64
+ contacted chan struct{}
+}
+
+func (g *aggregateNetworkGuard) RoundTrip(*http.Request) (*http.Response, error) {
+ g.calls.Add(1)
+ select {
+ case g.contacted <- struct{}{}:
+ default:
+ }
+ return nil, errors.New("network forbidden in aggregate offline control")
+}
+
+func TestAggregateUpdateCheckOnlyExcludesExactCommandAndAncestors(t *testing.T) {
+ t.Setenv("WAZA_NO_UPDATE_CHECK", "")
+ root := newRootCommand()
+ for _, control := range []struct {
+ name string
+ want bool
+ }{{"aggregate", false}, {"models", true}, {"results", true}, {"artifact", true}, {"runner", false}, {"preflight", false}} {
+ command, _, err := root.Find([]string{control.name})
+ require.NoError(t, err)
+ require.Equal(t, control.name, command.Name())
+ require.Equal(t, control.want, shouldRunUpdateCheck(command, false), control.name)
+ }
+ aggregate, _, err := root.Find([]string{"aggregate"})
+ require.NoError(t, err)
+ child := &cobra.Command{Use: "child"}
+ aggregate.AddCommand(child)
+ require.False(t, shouldRunUpdateCheck(child, false))
+ require.True(t, shouldRunUpdateCheck(&cobra.Command{Use: "aggregate-other"}, false))
+ require.True(t, shouldRunUpdateCheck(root, false))
+}
+
+func TestAggregateRegisteredRootIsOfflineWithoutUpdateOptOut(t *testing.T) {
+ directory := t.TempDir()
+ store := filepath.Join(directory, "store")
+ require.NoError(t, os.Mkdir(store, 0o700))
+ backend, err := artifactmanifest.OpenLocal(store)
+ require.NoError(t, err)
+ source := manifestfixture.Publish(t, backend, "../../examples/runner/unit.json", nil, manifestfixture.RecordedNonzero(1))
+ require.NoError(t, backend.Close())
+ t.Chdir(directory)
+ require.NoError(t, os.WriteFile(".waza.yaml", []byte("invalid: [ambient configuration must not be read"), 0o600))
+ home := t.TempDir()
+ t.Setenv("HOME", home)
+ t.Setenv("USERPROFILE", home)
+ t.Setenv("WAZA_NO_UPDATE_CHECK", "")
+ oldVersion := version
+ version = "0.1.0"
+ t.Cleanup(func() { version = oldVersion })
+ guard := &aggregateNetworkGuard{contacted: make(chan struct{}, 1)}
+ transport := http.DefaultTransport
+ http.DefaultTransport = guard
+ t.Cleanup(func() { http.DefaultTransport = transport })
+ output := filepath.Join(directory, "saved.json")
+ root := newRootCommand()
+ var stdout, stderr bytes.Buffer
+ root.SetOut(&stdout)
+ root.SetErr(&stderr)
+ root.SetArgs([]string{"aggregate", "--store", store, source.Manifest, "-o", output})
+ require.NoError(t, root.Execute())
+ require.False(t, root.PersistentFlags().Lookup("no-update-check").Changed)
+ require.Equal(t, "false", root.PersistentFlags().Lookup("no-update-check").Value.String())
+ saved, err := os.ReadFile(output)
+ require.NoError(t, err)
+ require.Equal(t, append(bytes.Clone(saved), '\n'), stdout.Bytes())
+ report, err := manifestaggregate.Parse(saved)
+ require.NoError(t, err)
+ require.Equal(t, "complete", report.State)
+ require.Equal(t, 1, report.Counts.VerifiedManifests)
+ require.Equal(t, source.Manifest, report.Members[0].Manifest)
+ require.Empty(t, stderr.String())
+ require.Zero(t, guard.calls.Load(), "registered root must not start its update checker")
+ require.NoDirExists(t, filepath.Join(home, ".waza"))
+
+ // Prove the same unsuppressed root hook can reach the rejecting transport.
+ control := newRootCommand()
+ models, _, err := control.Find([]string{"models"})
+ require.NoError(t, err)
+ require.True(t, shouldRunUpdateCheck(models, false))
+ control.PersistentPreRun(models, nil)
+ select {
+ case <-guard.contacted:
+ case <-time.After(5 * time.Second):
+ t.Fatal("released-version, fresh-home update-check positive control did not reach transport")
+ }
+ control.PersistentPostRun(models, nil)
+ require.EqualValues(t, 1, guard.calls.Load())
+}
diff --git a/cmd/waza/cmd_aggregate_test.go b/cmd/waza/cmd_aggregate_test.go
new file mode 100644
index 000000000..b2b7c828e
--- /dev/null
+++ b/cmd/waza/cmd_aggregate_test.go
@@ -0,0 +1,178 @@
+package main
+
+import (
+ "bytes"
+ "encoding/json"
+ "fmt"
+ "net/http"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "regexp"
+ "strings"
+ "testing"
+
+ "github.com/microsoft/waza/internal/artifactmanifest"
+ "github.com/microsoft/waza/internal/manifestaggregate"
+ "github.com/microsoft/waza/internal/testutil/manifestfixture"
+ "github.com/stretchr/testify/require"
+)
+
+func TestAggregateSavedCLIToPublicAPIReport(t *testing.T) {
+ directory := t.TempDir()
+ storeDir := filepath.Join(directory, ".waza-artifacts")
+ require.NoError(t, os.Mkdir(storeDir, 0o700))
+ backend, err := artifactmanifest.OpenLocal(storeDir)
+ require.NoError(t, err)
+ source := manifestfixture.Publish(t, backend, "../../examples/runner/unit.json", nil, manifestfixture.RecordedZero)
+ require.NoError(t, backend.Close())
+ good := manifestaggregate.Input{Manifest: source.Manifest, Evidence: source.Evidence}
+ missing := manifestaggregate.Input{Manifest: strings.Repeat("a", 64)}
+ for _, control := range []struct {
+ name string
+ inputs []manifestaggregate.Input
+ incomplete bool
+ }{
+ {"complete", []manifestaggregate.Input{good}, false},
+ {"missing neighbor", []manifestaggregate.Input{missing, good}, true},
+ } {
+ t.Run(control.name, func(t *testing.T) {
+ wire, err := json.Marshal(control.inputs)
+ require.NoError(t, err)
+ inputFile := filepath.Join(t.TempDir(), "inputs.json")
+ outputFile := filepath.Join(t.TempDir(), "aggregate.json")
+ require.NoError(t, os.WriteFile(inputFile, wire, 0o600))
+ command := newAggregateCommand()
+ var stdout bytes.Buffer
+ command.SetOut(&stdout)
+ command.SetErr(&bytes.Buffer{})
+ command.SetArgs([]string{"--store", storeDir, "--inputs", inputFile, "-o", outputFile})
+ err = command.Execute()
+ if control.incomplete {
+ require.ErrorIs(t, err, manifestaggregate.ErrIncomplete)
+ } else {
+ require.NoError(t, err)
+ }
+ saved, err := os.ReadFile(outputFile)
+ require.NoError(t, err, "evaluated incomplete reports must also be persisted")
+ report, err := manifestaggregate.Parse(saved)
+ require.NoError(t, err)
+ require.Equal(t, len(control.inputs), report.Counts.Members)
+ require.Equal(t, append(bytes.Clone(saved), '\n'), stdout.Bytes())
+ require.Equal(t, !control.incomplete, report.Usage.Recorded != nil)
+ status, body := postAggregateAPI(t, storeDir, control.inputs)
+ expectedStatus := http.StatusOK
+ if control.incomplete {
+ expectedStatus = http.StatusConflict
+ }
+ require.Equal(t, expectedStatus, status)
+ require.Equal(t, saved, body, "actual CLI saved JSON and API canonical bytes must agree")
+ retry := newAggregateCommand()
+ retry.SetOut(&bytes.Buffer{})
+ retry.SetErr(&bytes.Buffer{})
+ retry.SetArgs([]string{"--store", storeDir, "--inputs", inputFile, "-o", outputFile})
+ require.Error(t, retry.Execute(), "aggregate output must not replace an occupied file")
+ preserved, err := os.ReadFile(outputFile)
+ require.NoError(t, err)
+ require.Equal(t, saved, preserved)
+ })
+ }
+}
+
+func TestAggregateInputAdmissionPrecedesStoreOpen(t *testing.T) {
+ command := newAggregateCommand()
+ command.SetOut(&bytes.Buffer{})
+ command.SetErr(&bytes.Buffer{})
+ command.SetArgs([]string{"--store", filepath.Join(t.TempDir(), "must-not-be-opened"), "invalid"})
+ require.ErrorIs(t, command.Execute(), manifestaggregate.ErrInvalid)
+}
+
+func TestAggregateOutputRejectsLeafSymlinkAndKeepsExistingBytes(t *testing.T) {
+ directory := t.TempDir()
+ target := filepath.Join(directory, "user-owned")
+ require.NoError(t, os.WriteFile(target, []byte("unchanged"), 0o600))
+ link := filepath.Join(directory, "aggregate.json")
+ if err := os.Symlink(target, link); err != nil {
+ t.Skipf("symlink creation unavailable: %v", err)
+ }
+
+ require.Error(t, writeAggregateOutput(link, []byte("new")))
+ preserved, err := os.ReadFile(target)
+ require.NoError(t, err)
+ require.Equal(t, "unchanged", string(preserved))
+}
+
+func TestAggregateDocumentedInputBuilderUsesActualPublicReferences(t *testing.T) {
+ directory := t.TempDir()
+ backend, err := artifactmanifest.OpenLocal(directory)
+ require.NoError(t, err)
+ source := manifestfixture.Publish(t, backend, "../../examples/runner/unit.json", nil, nil)
+ require.NoError(t, backend.Close())
+ document, err := os.ReadFile("../../site/src/content/docs/guides/manifest-aggregation.mdx")
+ require.NoError(t, err)
+ t.Logf("native checkout newline bytes: LF=%d CRLF=%d", bytes.Count(document, []byte("\n")), bytes.Count(document, []byte("\r\n")))
+ normalized := bytes.ReplaceAll(document, []byte("\r\n"), []byte("\n"))
+ for _, ending := range []string{"\n", "\r\n"} {
+ t.Run(fmt.Sprintf("line-ending-%q", ending), func(t *testing.T) {
+ fixture := bytes.ReplaceAll(normalized, []byte("\n"), []byte(ending))
+ require.Equal(t, 1, bytes.Count(fixture, []byte("node --input-type=module <<'JS'"+ending)))
+ require.Equal(t, 1, bytes.Count(fixture, []byte(ending+"JS"+ending)))
+ snippet, err := aggregateInputBuilderScript(string(fixture))
+ require.NoError(t, err)
+ builderDir := t.TempDir()
+ builder := exec.Command("node", "--input-type=module", "-e", snippet)
+ builder.Dir = builderDir
+ builder.Env = append(os.Environ(), "MANIFEST_SHA256="+source.Manifest, "EVIDENCE_SHA256="+source.Evidence)
+ output, err := builder.CombinedOutput()
+ require.NoError(t, err, string(output))
+ created, err := os.ReadFile(filepath.Join(builderDir, "aggregate-inputs.json"))
+ require.NoError(t, err)
+ inputs, err := manifestaggregate.ParseInputs(created)
+ require.NoError(t, err)
+ require.Equal(t, []manifestaggregate.Input{{Manifest: source.Manifest, Evidence: source.Evidence}}, inputs)
+ builder = exec.Command("node", "--input-type=module", "-e", snippet)
+ builder.Dir = builderDir
+ builder.Env = append(os.Environ(), "MANIFEST_SHA256="+source.Manifest, "EVIDENCE_SHA256="+source.Evidence)
+ _, err = builder.CombinedOutput()
+ require.Error(t, err, "documented builder must not replace an existing input set")
+ command := newAggregateCommand()
+ command.SetOut(&bytes.Buffer{})
+ command.SetErr(&bytes.Buffer{})
+ command.SetArgs([]string{"--store", directory, "--inputs", filepath.Join(builderDir, "aggregate-inputs.json")})
+ require.NoError(t, command.Execute(), "copied documentation must work through the actual aggregate command")
+ })
+ }
+}
+
+func aggregateInputBuilderScript(document string) (string, error) {
+ openers := regexp.MustCompile(`(?m)^node --input-type=module <<'JS'\r?\n`).FindAllStringIndex(document, -1)
+ if len(openers) != 1 {
+ return "", fmt.Errorf("expected exactly one documented builder opener")
+ }
+ snippet := document[openers[0][1]:]
+ closing := regexp.MustCompile(`\r?\nJS\r?\n`).FindStringIndex(snippet)
+ if closing == nil {
+ return "", fmt.Errorf("missing documented builder terminator")
+ }
+ return snippet[:closing[0]], nil
+}
+
+func TestAggregateDocumentedInputBuilderBoundaries(t *testing.T) {
+ const opener = "node --input-type=module <<'JS'"
+ for _, ending := range []string{"\n", "\r\n"} {
+ script := "console.log('unchanged');" + ending + "// exact script bytes"
+ document := opener + ending + script + ending + "JS" + ending
+ actual, err := aggregateInputBuilderScript(document)
+ require.NoError(t, err)
+ require.Equal(t, script, actual, "only delimiters may be removed")
+ for _, invalid := range []string{
+ "missing opener" + ending + script + ending + "JS" + ending,
+ document + document,
+ opener + "near-miss" + ending + script + ending + "JS" + ending,
+ opener + ending + script,
+ } {
+ _, err := aggregateInputBuilderScript(invalid)
+ require.Error(t, err)
+ }
+ }
+}
diff --git a/cmd/waza/cmd_aggregate_usage_test.go b/cmd/waza/cmd_aggregate_usage_test.go
new file mode 100644
index 000000000..138f9eee4
--- /dev/null
+++ b/cmd/waza/cmd_aggregate_usage_test.go
@@ -0,0 +1,170 @@
+package main
+
+import (
+ "bytes"
+ "encoding/json"
+ "io"
+ "net/http"
+ "net/http/httptest"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+
+ "github.com/microsoft/waza/internal/artifactmanifest"
+ "github.com/microsoft/waza/internal/evidencestate"
+ "github.com/microsoft/waza/internal/manifestaggregate"
+ "github.com/microsoft/waza/internal/models"
+ "github.com/microsoft/waza/internal/runnerunit"
+ "github.com/microsoft/waza/internal/testutil/manifestfixture"
+ "github.com/microsoft/waza/internal/webapi"
+ "github.com/stretchr/testify/require"
+)
+
+func callAggregateCLI(t *testing.T, directory string, inputs []manifestaggregate.Input) []byte {
+ t.Helper()
+ wire, err := json.Marshal(inputs)
+ require.NoError(t, err)
+ input := filepath.Join(t.TempDir(), "inputs.json")
+ output := filepath.Join(t.TempDir(), "aggregate.json")
+ require.NoError(t, os.WriteFile(input, wire, 0o600))
+ command := newAggregateCommand()
+ var stdout bytes.Buffer
+ command.SetOut(&stdout)
+ command.SetErr(&bytes.Buffer{})
+ command.SetArgs([]string{"--store", directory, "--inputs", input, "-o", output})
+ require.NoError(t, command.Execute())
+ saved, err := os.ReadFile(output)
+ require.NoError(t, err, "evaluated membership must be persisted")
+ require.Equal(t, append(bytes.Clone(saved), '\n'), stdout.Bytes())
+ return saved
+}
+
+func callAggregateAPI(t *testing.T, directory string, inputs []manifestaggregate.Input) []byte {
+ t.Helper()
+ status, body := postAggregateAPI(t, directory, inputs)
+ require.Equal(t, http.StatusOK, status, string(body))
+ return body
+}
+
+func postAggregateAPI(t *testing.T, directory string, inputs []manifestaggregate.Input) (int, []byte) {
+ t.Helper()
+ wire, err := json.Marshal(inputs)
+ require.NoError(t, err)
+ mux := http.NewServeMux()
+ webapi.RegisterAggregateRoutesWithSources(mux, func() (manifestaggregate.Sources, func() error, error) {
+ return manifestaggregate.OpenLocal(directory)
+ })
+ server := httptest.NewServer(mux)
+ defer server.Close()
+ response, err := server.Client().Post(server.URL+"/api/v1/aggregates", "application/json", bytes.NewReader(wire))
+ require.NoError(t, err)
+ defer func() { require.NoError(t, response.Body.Close()) }()
+ body, err := io.ReadAll(response.Body)
+ require.NoError(t, err)
+ return response.StatusCode, body
+}
+
+func TestAggregateNonzeroUsageDeduplicatesExecutionsAndCompatibleViewsInSavedReport(t *testing.T) {
+ directory := t.TempDir()
+ backend, err := artifactmanifest.OpenLocal(directory)
+ require.NoError(t, err)
+ first := manifestfixture.Publish(t, backend, "../../examples/runner/unit.json", nil, manifestfixture.RecordedNonzero(1))
+ second := manifestfixture.Publish(t, backend, "../../examples/runner/unit.json", nil, manifestfixture.RecordedNonzero(2))
+ require.NotEqual(t, first.Execution, second.Execution)
+ inspection, err := evidencestate.Inspect(t.Context(), backend, first.Evidence)
+ require.NoError(t, err)
+ view := *inspection.Document
+ view.Lineage = []evidencestate.Lineage{{Manifest: second.Manifest, Relationship: evidencestate.Aggregation}}
+ otherView, err := evidencestate.Publish(t.Context(), backend, view)
+ require.NoError(t, err)
+ require.NotEqual(t, first.Evidence, otherView.EvidenceSHA256)
+ require.NoError(t, backend.Close())
+ firstInput := manifestaggregate.Input{Manifest: first.Manifest, Evidence: first.Evidence}
+ inputs := []manifestaggregate.Input{
+ {Manifest: first.Manifest, Evidence: otherView.EvidenceSHA256},
+ {Manifest: second.Manifest, Evidence: second.Evidence},
+ firstInput, firstInput,
+ }
+ saved := callAggregateCLI(t, directory, inputs)
+ require.Equal(t, saved, callAggregateAPI(t, directory, inputs))
+ report, err := manifestaggregate.Parse(saved)
+ require.NoError(t, err)
+ require.Equal(t, manifestaggregate.Counts{Members: 3, VerifiedManifests: 2, VerifiedExecutions: 2,
+ MemberStates: map[string]int{"admitted": 3}, MemberOutcomes: map[string]int{"success": 3}}, report.Counts)
+ require.Equal(t, "complete", report.Usage.State)
+ require.Equal(t, 2, report.Usage.Executions, "denominator is distinct executions, not pairs/views")
+ credits := 3.75
+ require.Equal(t, &models.UsageStats{
+ Turns: 3, InputTokens: 33, OutputTokens: 21, CacheReadTokens: 9, CacheWriteTokens: 6,
+ PremiumRequests: 1.5, AICredits: &credits,
+ ModelMetrics: map[string]models.ModelUsage{"fixture-model": {
+ InputTokens: 33, OutputTokens: 21, CacheReadTokens: 9, CacheWriteTokens: 6,
+ RequestCount: 6, RequestCost: .75, AICredits: &credits,
+ }},
+ }, report.Usage.Recorded, "a missing execution or double-counted compatible view must change the exact total")
+}
+
+func TestAggregateOversizedUsagePreservesSavedCLIAndAPIReport(t *testing.T) {
+ directory := t.TempDir()
+ backend, err := artifactmanifest.OpenLocal(directory)
+ require.NoError(t, err)
+ model := "escaped-model-" + strings.Repeat("\x01", (manifestaggregate.MaxReportBytes-512)/6)
+ var largeUsage, smallUsage *models.UsageStats
+ large := manifestfixture.Publish(t, backend, "../../examples/runner/unit.json", nil, func(report *runnerunit.Report) {
+ manifestfixture.RecordedNonzero(1)(report)
+ usage := report.Observation.Usage
+ usage.ModelMetrics = map[string]models.ModelUsage{model: usage.ModelMetrics["fixture-model"]}
+ wire, err := json.Marshal(usage)
+ require.NoError(t, err)
+ require.Less(t, len(model), manifestaggregate.MaxReportBytes, "character count is not a serialized-byte bound")
+ require.Less(t, len(wire), manifestaggregate.MaxReportBytes, "usage alone fits; full report overhead must be measured")
+ require.LessOrEqual(t, len(wire), artifactmanifest.MaxArtifactBytes, "native source remains within its supported profile")
+ largeUsage = usage
+ t.Logf("admitted recorded source usage: %d Go-escaped bytes", len(wire))
+ })
+ small := manifestfixture.Publish(t, backend, "../../examples/runner/unit.json", nil, func(report *runnerunit.Report) {
+ manifestfixture.RecordedNonzero(2)(report)
+ smallUsage = report.Observation.Usage
+ })
+ require.NoError(t, backend.Close())
+ inputs := []manifestaggregate.Input{
+ {Manifest: large.Manifest, Evidence: large.Evidence},
+ {Manifest: small.Manifest, Evidence: small.Evidence},
+ }
+ var cli []byte
+ for _, caller := range []struct {
+ name string
+ call func(*testing.T, string, []manifestaggregate.Input) []byte
+ }{{"saved CLI", callAggregateCLI}, {"public API", callAggregateAPI}} {
+ t.Run(caller.name, func(t *testing.T) {
+ saved := caller.call(t, directory, inputs)
+ report, err := manifestaggregate.Parse(saved)
+ require.NoError(t, err)
+ require.Equal(t, "complete", report.State, "all public sources were admitted; usage availability is separate")
+ require.Equal(t, manifestaggregate.Counts{Members: 2, VerifiedManifests: 2, VerifiedExecutions: 2,
+ MemberStates: map[string]int{"admitted": 2}, MemberOutcomes: map[string]int{"success": 2}}, report.Counts)
+ require.Equal(t, "unavailable", report.Usage.State)
+ require.Equal(t, 2, report.Usage.Executions, "withholding must not erase the eligible population")
+ require.Nil(t, report.Usage.Recorded, "even the small neighbor must not become an unlabeled subset total")
+ require.Equal(t, []string{"recorded_usage_report_size_unavailable"}, report.Usage.Diagnostics)
+ require.NotContains(t, string(saved), "escaped-model-")
+ require.LessOrEqual(t, len(saved), manifestaggregate.MaxReportBytes)
+ candidate := *report
+ candidate.Usage = manifestaggregate.Usage{
+ State: "complete", Population: report.Usage.Population, Executions: 2, Diagnostics: []string{},
+ Recorded: models.AggregateUsageStats([]*models.UsageStats{largeUsage, smallUsage}),
+ }
+ oversized, err := json.Marshal(candidate)
+ require.NoError(t, err)
+ require.Greater(t, len(oversized), manifestaggregate.MaxReportBytes,
+ "actual full Go-escaped report, not usage/string size alone, exceeds the wire limit")
+ t.Logf("full candidate report: %d bytes; retained report: %d bytes", len(oversized), len(saved))
+ if caller.name == "saved CLI" {
+ cli = saved
+ } else if cli != nil {
+ require.Equal(t, cli, saved)
+ }
+ })
+ }
+}
diff --git a/cmd/waza/cmd_runner.go b/cmd/waza/cmd_runner.go
index 59c52fcdc..d96b81857 100644
--- a/cmd/waza/cmd_runner.go
+++ b/cmd/waza/cmd_runner.go
@@ -27,6 +27,7 @@ func newRunnerCommandWithService(service *runnerunit.Service) *cobra.Command {
SilenceUsage: true,
}
cmd.AddCommand(newRunnerAdaptCommand())
+ cmd.AddCommand(newRunnerCheckpointCommand(service))
cmd.AddCommand(&cobra.Command{
Use: "capabilities", Short: "Print supported runner versions and capabilities as JSON",
Args: cobra.NoArgs,
@@ -118,8 +119,15 @@ func newRunnerCommandWithService(service *runnerunit.Service) *cobra.Command {
// Supplied readers remain caller-owned. Inherited stdin needs a separate path
// because it can be a blocking pipe without deadline support.
func readRunnerInput(ctx context.Context, input io.Reader) (data []byte, resultErr error) {
+ return readRunnerInputLimit(ctx, input, runnercontract.MaxRequestBytes)
+}
+
+func readRunnerInputLimit(ctx context.Context, input io.Reader, maxBytes int64) (data []byte, resultErr error) {
if input == os.Stdin {
- return readInheritedRunnerInput(ctx)
+ if maxBytes == runnercontract.MaxRequestBytes {
+ return readInheritedRunnerInput(ctx)
+ }
+ return readInheritedRunnerInputLimit(ctx, maxBytes)
}
if file, ok := input.(*os.File); ok {
if err := file.SetReadDeadline(time.Time{}); err == nil {
@@ -146,7 +154,7 @@ func readRunnerInput(ctx context.Context, input io.Reader) (data []byte, resultE
if err := ctx.Err(); err != nil {
return nil, fmt.Errorf("runner input interrupted: %w", err)
}
- data, err := io.ReadAll(io.LimitReader(input, runnercontract.MaxRequestBytes+1))
+ data, err := io.ReadAll(io.LimitReader(input, maxBytes+1))
if cause := ctx.Err(); cause != nil {
return nil, fmt.Errorf("runner input interrupted: %w", cause)
}
diff --git a/cmd/waza/cmd_runner_checkpoint.go b/cmd/waza/cmd_runner_checkpoint.go
new file mode 100644
index 000000000..ec4d58f1c
--- /dev/null
+++ b/cmd/waza/cmd_runner_checkpoint.go
@@ -0,0 +1,105 @@
+package main
+
+import (
+ "context"
+ "errors"
+ "fmt"
+ "os"
+ "os/signal"
+ "syscall"
+
+ "github.com/microsoft/waza/internal/artifactmanifest"
+ "github.com/microsoft/waza/internal/executioncheckpoint"
+ "github.com/microsoft/waza/internal/runnercheckpoint"
+ "github.com/microsoft/waza/internal/runnerunit"
+ "github.com/spf13/cobra"
+)
+
+func newRunnerCheckpointCommand(service *runnerunit.Service) *cobra.Command {
+ return newRunnerCheckpointCommandWithRevision(service, executioncheckpoint.CurrentRunnerRevision)
+}
+
+func newRunnerCheckpointCommandWithRevision(service *runnerunit.Service, revision func() (string, error)) *cobra.Command {
+ var directory, azureURL string
+ cmd := &cobra.Command{
+ Use: "checkpoint", Short: "Create or restore immutable exact-unit checkpoints from captured submission JSON",
+ SilenceUsage: true,
+ }
+ cmd.PersistentFlags().StringVar(&directory, "store", ".waza-artifacts", "Existing private local immutable store directory")
+ cmd.PersistentFlags().StringVar(&azureURL, "azure-container-url", "", "Explicit HTTPS Azure container URL using existing credentials")
+ emit := func(cmd *cobra.Command, result runnercheckpoint.Result) error {
+ data, err := runnercheckpoint.EncodeResult(result)
+ if err != nil {
+ return &ExitCodeError{Code: 2, Err: fmt.Errorf("validating checkpoint operation output: %w", err)}
+ }
+ return emitRunnerOperationJSON(cmd, data, result.ExitCode, result.Status)
+ }
+ for _, action := range []string{"create", "resume"} {
+ unit := &cobra.Command{
+ Use: action, Short: "Read one captured submission from stdin; emit one protected checkpoint result",
+ SilenceErrors: true,
+ RunE: func(cmd *cobra.Command, args []string) error {
+ if cmd.Flags().Changed("store") && cmd.Flags().Changed("azure-container-url") {
+ return emit(cmd, runnercheckpoint.RejectPublic("checkpoint_flags_conflict"))
+ }
+ if (action == "create" && len(args) != 0) || (action == "resume" && len(args) != 1) {
+ return emit(cmd, runnercheckpoint.RejectPublic("checkpoint_arguments_invalid"))
+ }
+ if action == "resume" {
+ if err := executioncheckpoint.ValidateDigest(args[0]); err != nil {
+ return emit(cmd, runnercheckpoint.RejectPublic("checkpoint_digest_invalid"))
+ }
+ }
+ ctx, stop := signal.NotifyContext(cmd.Context(), os.Interrupt, syscall.SIGTERM)
+ defer stop()
+ data, err := readRunnerInputLimit(ctx, cmd.InOrStdin(), runnerunit.MaxSubmissionBytes)
+ if err != nil {
+ return emit(cmd, runnercheckpoint.RejectPublic("checkpoint_input_unavailable"))
+ }
+ if _, err := runnerunit.ParseSubmission(data); err != nil {
+ return emit(cmd, runnercheckpoint.RejectPublic("checkpoint_input_invalid"))
+ }
+ buildRevision, err := revision()
+ if err != nil {
+ _, diagnosticErr := fmt.Fprintf(cmd.ErrOrStderr(), "checkpoint build admission: %v\n", err)
+ return errors.Join(emit(cmd, runnercheckpoint.RejectPublic("checkpoint_build_incompatible")), diagnosticErr)
+ }
+ checkpoints, err := runnercheckpoint.NewService(service, buildRevision)
+ if err != nil {
+ return &ExitCodeError{Code: 2, Err: err}
+ }
+ backend, closeBackend, err := artifactmanifest.OpenBackend(directory, azureURL)
+ if err != nil {
+ return &ExitCodeError{Code: 2, Err: fmt.Errorf("opening checkpoint store: %w", err)}
+ }
+ observers := runnercheckpoint.Hooks{Published: func(_ context.Context, reference runnercheckpoint.Reference) error {
+ _, err := fmt.Fprintf(cmd.ErrOrStderr(), "checkpoint %s %s\n", reference.Boundary, reference.Digest)
+ return err
+ }}
+ var result runnercheckpoint.Result
+ if action == "resume" {
+ result, err = checkpoints.ResumeCaptured(ctx, data, backend, args[0], observers)
+ } else {
+ result, err = checkpoints.SubmitCaptured(ctx, data, backend, observers)
+ }
+ if closeErr := closeBackend(); closeErr != nil {
+ return &ExitCodeError{Code: 2, Err: errors.Join(err, fmt.Errorf("closing checkpoint store: %w", closeErr))}
+ }
+ var diagnosticErr error
+ if err != nil {
+ _, diagnosticErr = fmt.Fprintf(cmd.ErrOrStderr(), "checkpoint operation: %v\n", err)
+ }
+ return errors.Join(emit(cmd, result), diagnosticErr)
+ },
+ }
+ if action == "resume" {
+ unit.Use = "resume CHECKPOINT_SHA256"
+ }
+ unit.SetFlagErrorFunc(func(cmd *cobra.Command, err error) error {
+ _, diagnosticErr := fmt.Fprintf(cmd.ErrOrStderr(), "checkpoint flags: %v\n", err)
+ return errors.Join(emit(cmd, runnercheckpoint.RejectPublic("checkpoint_flags_invalid")), diagnosticErr)
+ })
+ cmd.AddCommand(unit)
+ }
+ return cmd
+}
diff --git a/cmd/waza/cmd_runner_checkpoint_privacy_test.go b/cmd/waza/cmd_runner_checkpoint_privacy_test.go
new file mode 100644
index 000000000..c596f5d9e
--- /dev/null
+++ b/cmd/waza/cmd_runner_checkpoint_privacy_test.go
@@ -0,0 +1,84 @@
+package main
+
+import (
+ "bytes"
+ "encoding/json"
+ "strings"
+ "testing"
+
+ "github.com/microsoft/waza/internal/artifactmanifest"
+ "github.com/microsoft/waza/internal/execution"
+ "github.com/microsoft/waza/internal/executionattempt"
+ "github.com/microsoft/waza/internal/executionledger"
+ "github.com/microsoft/waza/internal/models"
+ "github.com/microsoft/waza/internal/runnercheckpoint"
+ "github.com/microsoft/waza/internal/runnerunit"
+ "github.com/stretchr/testify/require"
+)
+
+func TestCheckpointPublicCLIWithholdsUnapprovedSourceAndPreservesOriginal(t *testing.T) {
+ directory := t.TempDir()
+ metadata := "retained-checkpoint-metadata-control"
+ wire := checkpointCLIWire(t)
+ changed := bytes.Replace(wire, []byte("id: selected"), []byte("id: "+metadata), 1)
+ require.NotEqual(t, wire, changed)
+ wire = changed
+ changed = bytes.Replace(wire, []byte(`"task":"selected"`), []byte(`"task":"`+metadata+`"`), 1)
+ require.NotEqual(t, wire, changed)
+ wire = changed
+ factories := 0
+ core := runnerunit.NewService(func(config models.Config) (execution.AgentEngine, error) {
+ factories++
+ return execution.NewMockEngine(config.ModelID), nil
+ })
+ revision := func() (string, error) { return strings.Repeat("c", 40), nil }
+ created, _ := runCheckpointCLI(t, core, revision, directory, wire, "create")
+ require.Equal(t, metadata, created.Submission.Reservation.Binding.Selection.Task)
+ incompatibleRevision := func() (string, error) { return strings.Repeat("d", 40), nil }
+ rejected, _ := runCheckpointCLI(t, core, incompatibleRevision, directory, wire, "resume", created.FinalizedDigest)
+ require.True(t, rejected.SourceWithheld)
+ require.Equal(t, []string{"runner_revision"}, rejected.CompatibilityDifferences)
+ public, err := json.Marshal(rejected)
+ require.NoError(t, err)
+ require.NotContains(t, string(public), metadata)
+ require.Equal(t, 1, factories)
+
+ failedWire := bytes.Replace(wire, []byte("- hello"), []byte("- original-private-terminal-control"), 1)
+ require.NotEqual(t, wire, failedWire)
+ backend, closeBackend, err := artifactmanifest.OpenBackend(directory, "")
+ require.NoError(t, err)
+ legacy, err := runnercheckpoint.NewService(core, strings.Repeat("c", 40))
+ require.NoError(t, err)
+ original, err := legacy.Submit(t.Context(), failedWire, backend, runnercheckpoint.Hooks{})
+ require.NoError(t, err)
+ require.NotEmpty(t, original.Submission.Terminal.Diagnostics)
+ require.NotEqual(t, "[REDACTED]", original.Submission.Terminal.Diagnostics[0].Message)
+ owner, err := executionledger.New(backend)
+ require.NoError(t, err)
+ source := executionattempt.Source{
+ ExecutionID: original.Submission.Reservation.ExecutionID, ManifestDigest: original.Submission.Completion.ManifestDigest,
+ }
+ before, err := owner.ReadCompleted(t.Context(), source)
+ require.NoError(t, err)
+ terminalBefore := bytes.Clone(before.Publication.ArtifactBytes["terminal.json"])
+ require.NoError(t, closeBackend())
+ denied, _ := runCheckpointCLI(t, core, revision, directory, failedWire, "resume", original.FinalizedDigest)
+ require.True(t, denied.SourceWithheld)
+ require.Nil(t, denied.Source)
+ require.Nil(t, denied.Submission.Terminal)
+ require.Nil(t, denied.Submission.Completion)
+ require.Nil(t, denied.RestoredUnits)
+ require.Equal(t, 2, factories)
+ public, err = json.Marshal(denied)
+ require.NoError(t, err)
+ require.NotContains(t, string(public), metadata)
+ require.NotContains(t, string(public), "original-private-terminal-control")
+ backend, closeBackend, err = artifactmanifest.OpenBackend(directory, "")
+ require.NoError(t, err)
+ owner, err = executionledger.New(backend)
+ require.NoError(t, err)
+ after, err := owner.ReadCompleted(t.Context(), source)
+ require.NoError(t, err)
+ require.Equal(t, terminalBefore, after.Publication.ArtifactBytes["terminal.json"])
+ require.NoError(t, closeBackend())
+}
diff --git a/cmd/waza/cmd_runner_checkpoint_process_linux_test.go b/cmd/waza/cmd_runner_checkpoint_process_linux_test.go
new file mode 100644
index 000000000..4f8901715
--- /dev/null
+++ b/cmd/waza/cmd_runner_checkpoint_process_linux_test.go
@@ -0,0 +1,360 @@
+package main
+
+import (
+ "bytes"
+ "context"
+ "debug/buildinfo"
+ "errors"
+ "fmt"
+ "os"
+ "os/exec"
+ "path/filepath"
+ "runtime"
+ "strings"
+ "syscall"
+ "testing"
+ "time"
+
+ "github.com/microsoft/waza/internal/artifactmanifest"
+ "github.com/microsoft/waza/internal/executioncheckpoint"
+ "github.com/microsoft/waza/internal/executionledger"
+ "github.com/microsoft/waza/internal/runnercheckpoint"
+ "github.com/microsoft/waza/internal/runneridentity"
+ "github.com/microsoft/waza/internal/runnerterminal"
+ "github.com/microsoft/waza/internal/runnerunit"
+ "github.com/stretchr/testify/require"
+ "golang.org/x/sys/unix"
+)
+
+func TestCheckpointProductionProcessRecovery(t *testing.T) {
+ binary := buildCheckpointRecoveryCLI(t)
+ info, err := buildinfo.ReadFile(binary)
+ require.NoError(t, err)
+ require.Equal(t, "github.com/microsoft/waza/cmd/waza", info.Path)
+ settings := map[string]string{}
+ for _, setting := range info.Settings {
+ settings[setting.Key] = setting.Value
+ }
+ require.Equal(t, "false", settings["vcs.modified"], "recovery requires an actual clean VCS-stamped CLI; commit source before running this process fixture")
+ require.Len(t, settings["vcs.revision"], 40)
+
+ for _, action := range []string{"create", "resume"} {
+ t.Run(action+"-flag-diagnostics", func(t *testing.T) {
+ directory := filepath.Join(t.TempDir(), "must-not-open")
+ args := []string{"runner", "checkpoint", action}
+ if action == "resume" {
+ args = append(args, strings.Repeat("e", 64))
+ }
+ const flag = "private-unknown-flag-control"
+ args = append(args, "--store", directory, "--"+flag)
+ command := exec.CommandContext(t.Context(), binary, args...)
+ command.Stdin = bytes.NewReader(checkpointCLIWire(t))
+ var output, diagnostic bytes.Buffer
+ command.Stdout, command.Stderr = &output, &diagnostic
+ err := command.Run()
+ exit, ok := errors.AsType[*exec.ExitError](err)
+ require.True(t, ok, "%v", err)
+ require.Equal(t, 6, exit.ExitCode())
+ result, err := runnercheckpoint.ParseResult(output.Bytes())
+ require.NoError(t, err, "%s", output.Bytes())
+ require.Equal(t, exit.ExitCode(), result.ExitCode)
+ require.Equal(t, "checkpoint_flags_invalid", result.Diagnostics[0].Code)
+ require.Equal(t, 1, bytes.Count(output.Bytes(), []byte{'\n'}))
+ require.NotContains(t, output.String(), flag)
+ require.Contains(t, diagnostic.String(), "checkpoint flags: unknown flag: --"+flag)
+ require.NoDirExists(t, directory)
+ })
+ }
+
+ for _, control := range []struct {
+ name, boundary string
+ failedChecks bool
+ }{
+ {"before-dispatch", executioncheckpoint.BeforeUnit, false},
+ {"finalized-success", executioncheckpoint.FinalizedUnit, false},
+ {"finalized-failed-checks", executioncheckpoint.FinalizedUnit, true},
+ } {
+ t.Run(control.name, func(t *testing.T) {
+ directory := t.TempDir()
+ wire := checkpointCLIWire(t)
+ if control.failedChecks {
+ changed := bytes.Replace(wire, []byte("- hello"), []byte("- missing-process-control"), 1)
+ require.NotEqual(t, wire, changed)
+ wire = changed
+ }
+ digest, source := killCheckpointProcessAtBoundary(t, binary, directory, wire, control.boundary)
+ require.Equal(t, settings["vcs.revision"], source.Compatibility.RunnerRevision)
+ original, err := os.ReadFile(filepath.Join(directory, "blob-"+digest))
+ require.NoError(t, err)
+ backend, err := artifactmanifest.OpenLocal(directory)
+ require.NoError(t, err)
+ defer backend.Close() //nolint:errcheck
+ ledger, err := executionledger.New(backend)
+ require.NoError(t, err)
+ var terminal []byte
+ if control.boundary == executioncheckpoint.BeforeUnit {
+ _, err := executioncheckpoint.ReadBefore(t.Context(), ledger, *source, source.Compatibility)
+ require.NoError(t, err, "joined original must have no dispatch or completion")
+ } else {
+ completed, err := executioncheckpoint.ReadStoredFinalized(t.Context(), ledger, *source, source.Compatibility)
+ require.NoError(t, err)
+ terminal = completed.Publication.ArtifactBytes[completed.Publication.Manifest.TerminalArtifact]
+ require.NotEmpty(t, terminal)
+ }
+
+ destination := bytes.Replace(wire, []byte(`"attempt":2`), []byte(`"attempt":3`), 1)
+ require.NotEqual(t, wire, destination)
+ request, err := runnerunit.ParseSubmission(destination)
+ require.NoError(t, err)
+ caller, err := runneridentity.DeriveJSON(request.Request)
+ require.NoError(t, err)
+ code := 0
+ if control.failedChecks {
+ code = 1
+ }
+ result := resumeCheckpointProcess(t, binary, directory, destination, digest, code)
+ require.Equal(t, "compatible", result.CompatibilityResult)
+ require.NotNil(t, result.Source)
+ require.Equal(t, source.ExecutionID, result.Source.ExecutionID)
+ require.NotNil(t, result.Submission)
+ require.Equal(t, caller, result.Submission.Caller)
+ require.Empty(t, result.Submission.Report)
+ require.NotNil(t, result.Submission.Terminal)
+ require.NotNil(t, result.RemainingWork)
+ require.Zero(t, *result.RemainingWork)
+ if control.boundary == executioncheckpoint.BeforeUnit {
+ require.Equal(t, "completed", result.Status)
+ require.Equal(t, 1, result.Submission.EngineCalls)
+ require.NotEqual(t, source.ExecutionID, result.Submission.Reservation.ExecutionID)
+ require.NotNil(t, result.Submission.Reservation.Parent)
+ require.Equal(t, source.ExecutionID, result.Submission.Reservation.Parent.ExecutionID)
+ require.Equal(t, "retry", result.Submission.Reservation.Parent.Relationship)
+ require.NotNil(t, result.RestoredUnits)
+ require.Zero(t, *result.RestoredUnits)
+ } else {
+ require.Equal(t, "reused", result.Status)
+ require.Zero(t, result.Submission.EngineCalls, "fresh compiled child must not execute finalized work")
+ require.Equal(t, source.ExecutionID, result.Submission.Reservation.ExecutionID)
+ require.Equal(t, 2, result.Submission.Terminal.Selection.Attempt)
+ require.NotNil(t, result.RestoredUnits)
+ require.Equal(t, 1, *result.RestoredUnits)
+ actual, err := runnerterminal.Encode(*result.Submission.Terminal, result.Submission.Reservation.Binding.TerminalBinding())
+ require.NoError(t, err)
+ require.Equal(t, terminal, actual, "restore must preserve exact original terminal bytes and failure")
+ }
+ preserved, err := os.ReadFile(filepath.Join(directory, "blob-"+digest))
+ require.NoError(t, err)
+ require.Equal(t, original, preserved)
+ })
+ }
+
+ t.Run("dispatched-without-completion", func(t *testing.T) {
+ directory := t.TempDir()
+ wire := checkpointCLIWire(t)
+ backend, err := artifactmanifest.OpenLocal(directory)
+ require.NoError(t, err)
+ defer backend.Close() //nolint:errcheck
+ checkpoints, err := runnercheckpoint.NewService(runnerunit.NewService(nil), settings["vcs.revision"])
+ require.NoError(t, err)
+ failure := errors.New("stop publication after actual dispatch and terminal observation")
+ created, err := checkpoints.SubmitCaptured(t.Context(), wire, backend, runnercheckpoint.Hooks{
+ Submission: runnerunit.SubmissionHooks{Terminal: func(context.Context, *executionledger.Ledger,
+ executionledger.Creator, executionledger.Reservation, runnerterminal.Envelope,
+ ) error {
+ return failure
+ }},
+ })
+ require.ErrorIs(t, err, failure)
+ require.NotEmpty(t, created.BeforeDigest)
+ require.Equal(t, 1, created.Submission.EngineCalls)
+ require.Nil(t, created.Submission.Completion)
+ ledger, err := executionledger.New(backend)
+ require.NoError(t, err)
+ dispatched, err := ledger.HasDistinctDecision(t.Context(), created.Submission.Reservation.ExecutionID)
+ require.NoError(t, err)
+ require.True(t, dispatched)
+ before, err := os.ReadDir(directory)
+ require.NoError(t, err)
+ result := resumeCheckpointProcess(t, binary, directory, wire, created.BeforeDigest, 6)
+ require.Equal(t, "rejected", result.Status)
+ require.Nil(t, result.Submission)
+ require.True(t, result.SourceWithheld)
+ require.Nil(t, result.Source)
+ require.Nil(t, result.RestoredUnits)
+ require.Nil(t, result.RemainingWork)
+ after, err := os.ReadDir(directory)
+ require.NoError(t, err)
+ require.Len(t, after, len(before), "stale restore must not create another reservation or decision")
+ })
+}
+
+func buildCheckpointRecoveryCLI(t *testing.T) string {
+ t.Helper()
+ root, err := exec.CommandContext(t.Context(), "git", "rev-parse", "--show-toplevel").Output()
+ require.NoError(t, err)
+ directory := strings.TrimSpace(string(root))
+ diff := exec.CommandContext(t.Context(), "git", "diff", "--exit-code", "HEAD", "--",
+ "*.go", "cmd", "internal", "schemas", "web", "docs/graders", "go.mod", "go.sum")
+ diff.Dir = directory
+ log, err := diff.CombinedOutput()
+ require.NoError(t, err, "commit changed build inputs before the clean revision fixture: %s", log)
+ untracked := exec.CommandContext(t.Context(), "git", "ls-files", "--others", "--exclude-standard", "--",
+ "*.go", "go.mod", "go.sum", "schemas/", "docs/graders/")
+ untracked.Dir = directory
+ log, err = untracked.CombinedOutput()
+ require.NoError(t, err)
+ require.Empty(t, log, "commit new Go/schema build inputs rather than testing an older HEAD")
+
+ snapshot := filepath.Join(t.TempDir(), "source")
+ commands := [][]string{
+ {"clone", "--quiet", "--shared", "--single-branch", "--no-tags", "--no-checkout", directory, snapshot},
+ {"-C", snapshot, "sparse-checkout", "set", "--cone", "cmd", "internal", "schemas", "web", "docs/graders"},
+ {"-C", snapshot, "checkout", "--quiet", "--detach", "HEAD"},
+ }
+ for _, args := range commands {
+ command := exec.CommandContext(t.Context(), "git", append([]string{"-c", "core.hooksPath=/dev/null"}, args...)...)
+ command.Env = append(os.Environ(), "GIT_LFS_SKIP_SMUDGE=1", "GIT_TERMINAL_PROMPT=0")
+ log, err := command.CombinedOutput()
+ require.NoError(t, err, "preparing private clean committed source: %s", log)
+ }
+ return buildRecoveryCLIFromDirectory(t, filepath.Join(snapshot, "cmd", "waza"))
+}
+
+func killCheckpointProcessAtBoundary(t *testing.T, binary, directory string, wire []byte, boundary string) (string, *executioncheckpoint.Checkpoint) {
+ t.Helper()
+ // Trace the unchanged binary's syscalls, not its bounded diagnostic writer.
+ // The publishing thread cannot return from the immutable link syscall until
+ // this tracer resumes it, so no wall-clock diagnostic timeout is a barrier.
+ runtime.LockOSThread()
+ defer runtime.UnlockOSThread()
+ ctx, cancel := context.WithTimeout(t.Context(), 30*time.Second)
+ defer cancel()
+ child := exec.CommandContext(ctx, binary, "runner", "checkpoint", "create", "--store", directory)
+ child.SysProcAttr = &syscall.SysProcAttr{Ptrace: true, Setpgid: true}
+ child.Stdin = bytes.NewReader(wire)
+ var output, diagnostic bytes.Buffer
+ child.Stdout, child.Stderr = &output, &diagnostic
+ require.NoError(t, child.Start())
+ reap := func() (syscall.WaitStatus, error) {
+ var final syscall.WaitStatus
+ for {
+ var status syscall.WaitStatus
+ pid, err := syscall.Wait4(-child.Process.Pid, &status, unix.WALL, nil)
+ if errors.Is(err, syscall.ECHILD) {
+ return final, nil
+ }
+ if err != nil {
+ return final, err
+ }
+ if pid == child.Process.Pid && (status.Exited() || status.Signaled()) {
+ final = status
+ }
+ }
+ }
+ joined := false
+ defer func() {
+ if !joined {
+ _ = child.Process.Kill()
+ _, _ = reap()
+ _ = child.Wait()
+ }
+ }()
+ var initial syscall.WaitStatus
+ _, err := syscall.Wait4(child.Process.Pid, &initial, 0, nil)
+ require.NoError(t, err)
+ require.True(t, initial.Stopped())
+ require.NoError(t, syscall.PtraceSetOptions(child.Process.Pid, unix.PTRACE_O_TRACESYSGOOD|unix.PTRACE_O_TRACECLONE))
+ require.NoError(t, syscall.PtraceSyscall(child.Process.Pid, 0))
+ var digest string
+ var source *executioncheckpoint.Checkpoint
+ for source == nil {
+ var status syscall.WaitStatus
+ tid, err := syscall.Wait4(-child.Process.Pid, &status, unix.WALL, nil)
+ require.NoError(t, err)
+ require.NoError(t, ctx.Err())
+ if !status.Stopped() {
+ require.NotEqual(t, child.Process.Pid, tid, "CLI exited before the requested boundary")
+ continue
+ }
+ entries, err := os.ReadDir(directory)
+ require.NoError(t, err)
+ for _, entry := range entries {
+ if !strings.HasPrefix(entry.Name(), "blob-") {
+ continue
+ }
+ data, err := os.ReadFile(filepath.Join(directory, entry.Name()))
+ if err != nil || !bytes.Contains(data, []byte(`"kind":"waza.execution-checkpoint"`)) {
+ continue
+ }
+ checkpoint, err := executioncheckpoint.Parse(data)
+ if err == nil && checkpoint.Boundary == boundary {
+ digest, source = artifactmanifest.Hash(data), checkpoint
+ }
+ }
+ if source != nil {
+ break
+ }
+ signal := int(status.StopSignal())
+ if signal == int(syscall.SIGTRAP) || signal == int(syscall.SIGTRAP)|0x80 || signal == int(syscall.SIGSTOP) {
+ signal = 0
+ }
+ require.NoError(t, syscall.PtraceSyscall(tid, signal))
+ }
+ require.NoError(t, child.Process.Kill())
+ status, err := reap()
+ require.NoError(t, err)
+ waitErr := child.Wait() // Wait4 reaped the process; Wait still joins owned output-copy goroutines.
+ joined = true
+ require.ErrorIs(t, waitErr, syscall.ECHILD)
+ require.Equal(t, syscall.SIGKILL, status.Signal())
+ require.Empty(t, output.Bytes(), "killed operation must not have delivered a terminal record")
+ if boundary == executioncheckpoint.BeforeUnit {
+ require.Empty(t, diagnostic.Bytes())
+ } else {
+ checkpointDigest := checkpointDigestAtBoundary(t, directory, executioncheckpoint.BeforeUnit)
+ require.Equal(t, fmt.Sprintf("checkpoint %s %s\n", executioncheckpoint.BeforeUnit, checkpointDigest), diagnostic.String())
+ }
+ return digest, source
+}
+
+func checkpointDigestAtBoundary(t *testing.T, directory, boundary string) string {
+ t.Helper()
+ entries, err := os.ReadDir(directory)
+ require.NoError(t, err)
+ for _, entry := range entries {
+ if !strings.HasPrefix(entry.Name(), "blob-") {
+ continue
+ }
+ data, err := os.ReadFile(filepath.Join(directory, entry.Name()))
+ require.NoError(t, err)
+ checkpoint, err := executioncheckpoint.Parse(data)
+ if err == nil && checkpoint.Boundary == boundary {
+ return artifactmanifest.Hash(data)
+ }
+ }
+ t.Fatalf("no immutable %s checkpoint", boundary)
+ return ""
+}
+
+func resumeCheckpointProcess(t *testing.T, binary, directory string, wire []byte, digest string, code int) runnercheckpoint.Result {
+ t.Helper()
+ ctx, cancel := context.WithTimeout(t.Context(), 30*time.Second)
+ defer cancel()
+ child := exec.CommandContext(ctx, binary, "runner", "checkpoint", "resume", digest, "--store", directory)
+ child.Stdin = bytes.NewReader(wire)
+ var output, diagnostic bytes.Buffer
+ child.Stdout, child.Stderr = &output, &diagnostic
+ err := child.Run()
+ if code == 0 {
+ require.NoError(t, err, "%s", diagnostic.String())
+ } else {
+ var exit *exec.ExitError
+ require.ErrorAs(t, err, &exit, "%s", diagnostic.String())
+ require.Equal(t, code, exit.ExitCode(), "%s", diagnostic.String())
+ }
+ result, err := runnercheckpoint.ParseResult(output.Bytes())
+ require.NoError(t, err, "%s", output.Bytes())
+ require.Equal(t, code, result.ExitCode)
+ require.Equal(t, 1, bytes.Count(output.Bytes(), []byte{'\n'}))
+ return *result
+}
diff --git a/cmd/waza/cmd_runner_checkpoint_test.go b/cmd/waza/cmd_runner_checkpoint_test.go
new file mode 100644
index 000000000..2b9084bd4
--- /dev/null
+++ b/cmd/waza/cmd_runner_checkpoint_test.go
@@ -0,0 +1,201 @@
+package main
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "errors"
+ "os"
+ "strings"
+ "testing"
+
+ "github.com/microsoft/waza/internal/execution"
+ "github.com/microsoft/waza/internal/models"
+ "github.com/microsoft/waza/internal/runnercheckpoint"
+ "github.com/microsoft/waza/internal/runnerunit"
+ "github.com/stretchr/testify/require"
+)
+
+func checkpointCLIWire(t *testing.T) []byte {
+ t.Helper()
+ raw, err := os.ReadFile("../../examples/runner/unit.json")
+ require.NoError(t, err)
+ data, err := json.Marshal(runnerunit.SubmissionRequest{
+ Kind: "waza.execution-submission", SchemaVersion: "1.0", Request: raw,
+ })
+ require.NoError(t, err)
+ return data
+}
+
+func runCheckpointCLI(t *testing.T, service *runnerunit.Service, revision func() (string, error),
+ directory string, wire []byte, args ...string,
+) (runnercheckpoint.Result, string) {
+ t.Helper()
+ command := newRunnerCheckpointCommandWithRevision(service, revision)
+ command.SetArgs(append(args, "--store", directory))
+ command.SetIn(bytes.NewReader(wire))
+ var output, progress bytes.Buffer
+ command.SetOut(&output)
+ command.SetErr(&progress)
+ err := command.ExecuteContext(t.Context())
+ result, decodeErr := runnercheckpoint.ParseResult(output.Bytes())
+ require.NoError(t, decodeErr, "%s", output.Bytes())
+ require.Equal(t, 1, bytes.Count(output.Bytes(), []byte{'\n'}))
+ if result.ExitCode == 0 {
+ require.NoError(t, err)
+ } else {
+ exit, ok := errors.AsType[*ExitCodeError](err)
+ require.True(t, ok, "%v", err)
+ require.Equal(t, result.ExitCode, exit.Code)
+ }
+ return *result, progress.String()
+}
+
+func TestCheckpointPublicCLIOriginalOutcomeAndStaleResume(t *testing.T) {
+ for _, control := range []string{"success", "failed-checks", "canceled"} {
+ t.Run(control, func(t *testing.T) {
+ directory := t.TempDir()
+ factories := 0
+ service := runnerunit.NewService(func(config models.Config) (execution.AgentEngine, error) {
+ factories++
+ if control == "canceled" {
+ return nil, context.Canceled
+ }
+ return execution.NewMockEngine(config.ModelID), nil
+ })
+ wire := checkpointCLIWire(t)
+ wantCode := 0
+ if control == "failed-checks" {
+ changed := bytes.Replace(wire, []byte("- hello"), []byte("- missing-output-control"), 1)
+ require.NotEqual(t, wire, changed)
+ wire, wantCode = changed, 1
+ }
+ if control == "canceled" {
+ wantCode = 3
+ }
+ revision := func() (string, error) { return strings.Repeat("c", 40), nil }
+ created, progress := runCheckpointCLI(t, service, revision, directory, wire, "create")
+ require.Equal(t, wantCode, created.ExitCode)
+ require.Contains(t, progress, "checkpoint before-unit "+created.BeforeDigest+"\n")
+ require.Contains(t, progress, "checkpoint finalized-unit "+created.FinalizedDigest+"\n")
+ require.NotNil(t, created.Submission.Completion)
+ require.Empty(t, created.Submission.Report)
+ original, err := json.Marshal(created.Submission.Terminal)
+ require.NoError(t, err)
+ destination := bytes.Replace(wire, []byte(`"attempt":2`), []byte(`"attempt":3`), 1)
+ require.NotEqual(t, wire, destination)
+ reused, progress := runCheckpointCLI(t, service, revision, directory, destination, "resume", created.FinalizedDigest)
+ require.Empty(t, progress)
+ require.Equal(t, wantCode, reused.ExitCode)
+ require.Zero(t, reused.Submission.EngineCalls)
+ require.Equal(t, 1, factories)
+ require.Equal(t, 2, reused.Submission.Terminal.Selection.Attempt)
+ terminal, err := json.Marshal(reused.Submission.Terminal)
+ require.NoError(t, err)
+ require.Equal(t, original, terminal)
+ rejected, _ := runCheckpointCLI(t, service, revision, directory, wire, "resume", created.BeforeDigest)
+ require.Equal(t, "rejected", rejected.Status)
+ require.True(t, rejected.SourceWithheld)
+ require.Nil(t, rejected.Source)
+ require.Nil(t, rejected.RestoredUnits)
+ require.Equal(t, 1, factories)
+ })
+ }
+}
+
+func TestCheckpointPublicCLIRejectsBeforeProvider(t *testing.T) {
+ for _, control := range []string{"invalid-input", "over-limit", "missing-revision", "arguments", "invalid-digest", "output-marker-input", "conflicting-store-flags"} {
+ t.Run(control, func(t *testing.T) {
+ wire := checkpointCLIWire(t)
+ revisions, factories := 0, 0
+ service := runnerunit.NewService(func(models.Config) (execution.AgentEngine, error) {
+ factories++
+ return nil, errors.New("must not construct")
+ })
+ revision := func() (string, error) {
+ revisions++
+ return "", errors.New("private-build-metadata-control")
+ }
+ args := []string{"create"}
+ switch control {
+ case "invalid-input":
+ wire = []byte("{}")
+ case "over-limit":
+ wire = append(wire, bytes.Repeat([]byte(" "), runnerunit.MaxSubmissionBytes)...)
+ case "output-marker-input":
+ changed := bytes.Replace(wire, []byte(`"schema_version":"1.0"`), []byte(`"source_withheld":true,"schema_version":"1.0"`), 1)
+ require.NotEqual(t, wire, changed)
+ wire = changed
+ case "arguments":
+ args = append(args, "extra")
+ case "invalid-digest":
+ args = []string{"resume", strings.Repeat("A", 64)}
+ case "conflicting-store-flags":
+ args = append(args, "--azure-container-url", "https://fixture.invalid/container")
+ }
+
+ directory := t.TempDir() + "/must-not-open"
+ result, diagnostic := runCheckpointCLI(t, service, revision, directory, wire, args...)
+ require.Equal(t, "rejected", result.Status)
+ require.Nil(t, result.Submission)
+ require.Zero(t, factories)
+ if control == "missing-revision" {
+ require.Equal(t, 1, revisions)
+ require.Equal(t, "checkpoint_build_incompatible", result.Diagnostics[0].Code)
+ require.Contains(t, diagnostic, "checkpoint build admission: private-build-metadata-control")
+ encoded, err := runnercheckpoint.EncodeResult(result)
+ require.NoError(t, err)
+ require.NotContains(t, string(encoded), "private-build-metadata-control")
+ } else {
+ require.Zero(t, revisions)
+ }
+ _, err := os.Stat(directory)
+ require.True(t, os.IsNotExist(err))
+ })
+ }
+}
+
+func TestCheckpointPublicCLIRetainsPrivateOperationCauseOnlyInDiagnostics(t *testing.T) {
+ digest := strings.Repeat("e", 64)
+ result, diagnostic := runCheckpointCLI(t, runnerunit.NewService(nil),
+ func() (string, error) { return strings.Repeat("c", 40), nil },
+ t.TempDir(), checkpointCLIWire(t), "resume", digest)
+ require.Equal(t, "rejected", result.Status)
+ require.Equal(t, 6, result.ExitCode)
+ require.Contains(t, diagnostic, "checkpoint operation:")
+ require.Contains(t, diagnostic, "reading checkpoint blob "+digest+": missing immutable object")
+ data, err := runnercheckpoint.EncodeResult(result)
+ require.NoError(t, err)
+ require.NotContains(t, string(data), digest)
+}
+
+func TestCheckpointPublicCLIRetainsFlagFailureOnlyInOperatorDiagnostics(t *testing.T) {
+ for _, action := range []string{"create", "resume"} {
+ t.Run(action, func(t *testing.T) {
+ earlyCalls := 0
+ service := runnerunit.NewService(func(models.Config) (execution.AgentEngine, error) {
+ earlyCalls++
+ return nil, errors.New("must not construct")
+ })
+ revision := func() (string, error) {
+ earlyCalls++
+ return strings.Repeat("c", 40), nil
+ }
+ args := []string{action}
+ if action == "resume" {
+ args = append(args, strings.Repeat("e", 64))
+ }
+ const flag = "private-unknown-flag-control"
+ args = append(args, "--"+flag)
+ directory := t.TempDir() + "/must-not-open"
+ result, diagnostic := runCheckpointCLI(t, service, revision, directory, checkpointCLIWire(t), args...)
+ require.Equal(t, "checkpoint_flags_invalid", result.Diagnostics[0].Code)
+ require.Contains(t, diagnostic, "checkpoint flags: unknown flag: --"+flag)
+ require.Zero(t, earlyCalls)
+ require.NoDirExists(t, directory)
+ public, err := runnercheckpoint.EncodeResult(result)
+ require.NoError(t, err)
+ require.NotContains(t, string(public), flag)
+ })
+ }
+}
diff --git a/cmd/waza/cmd_runner_read_progress_test.go b/cmd/waza/cmd_runner_read_progress_test.go
index b41bc9b28..0ddb9d8f1 100644
--- a/cmd/waza/cmd_runner_read_progress_test.go
+++ b/cmd/waza/cmd_runner_read_progress_test.go
@@ -30,7 +30,7 @@ import (
const runnerReadProgressSize = (2 << 20) + 1
const runnerReadProgressPrefix = "WAZA_TEST_STDIN_PROGRESS/"
-const runnerReadProgressOriginalSHA256 = "62934ad12d6e4c4751f4bda278b6c1d0ee2e75e9823dba409aea2584bdd4920b"
+const runnerReadProgressOriginalSHA256 = "71f2229c29a9e5f6a485cedf1505e1532b2fffd3896e554a5ec9b652b8de6413"
// Sharing OS-independent process ownership lets portable fixtures run
// the same sole-Wait/copy/cleanup path as the native Windows controller.
@@ -423,7 +423,7 @@ func runnerReadProgressOverlay(source []byte, nonce string) ([]byte, error) {
}
var body *ast.BlockStmt
for _, decl := range file.Decls {
- if fn, ok := decl.(*ast.FuncDecl); ok && fn.Name.Name == "readInheritedRunnerInput" {
+ if fn, ok := decl.(*ast.FuncDecl); ok && fn.Name.Name == "readInheritedRunnerInputLimit" {
if body != nil {
return nil, errors.New("duplicate inherited reader")
}
@@ -449,7 +449,7 @@ func runnerReadProgressOverlay(source []byte, nonce string) ([]byte, error) {
return nil, errors.New("unexpected initial deadline expression")
}
read, ok := body.List[2].(*ast.AssignStmt)
- if !ok || len(read.Rhs) != 1 || slice(read.Rhs[0]) != "io.ReadAll(io.LimitReader(os.Stdin, runnercontract.MaxRequestBytes+1))" {
+ if !ok || len(read.Rhs) != 1 || slice(read.Rhs[0]) != "io.ReadAll(io.LimitReader(os.Stdin, maxBytes+1))" {
return nil, errors.New("unexpected inherited ReadAll/LimitReader expression")
}
var encoded bytes.Buffer
@@ -458,14 +458,14 @@ func runnerReadProgressOverlay(source []byte, nonce string) ([]byte, error) {
}
text := encoded.String()
declaration := fmt.Sprintf("\n\tprogress := &runnerObservedStdin{file: os.Stdin, nonce: %q, digest: sha256.New()}", nonce)
- signature := "func readInheritedRunnerInput(ctx context.Context) (data []byte, resultErr error) {"
+ signature := "func readInheritedRunnerInputLimit(ctx context.Context, maxBytes int64) (data []byte, resultErr error) {"
if strings.Count(text, signature) != 1 {
return nil, errors.New("unexpected inherited-reader signature")
}
text = strings.Replace(text, signature, signature+declaration, 1)
text = strings.Replace(text, "err := os.Stdin.SetReadDeadline(time.Time{})", "err := progress.deadline(os.Stdin.SetReadDeadline(time.Time{}))", 1)
- text = strings.Replace(text, "io.ReadAll(io.LimitReader(os.Stdin, runnercontract.MaxRequestBytes+1))",
- "io.ReadAll(io.LimitReader(progress, runnercontract.MaxRequestBytes+1))", 1)
+ text = strings.Replace(text, "io.ReadAll(io.LimitReader(os.Stdin, maxBytes+1))",
+ "io.ReadAll(io.LimitReader(progress, maxBytes+1))", 1)
text = strings.Replace(text, "import (", "import (\n\"crypto/sha256\"\n\"encoding/json\"\n\"hash\"", 1)
return goformat.Source([]byte(text + runnerObservedStdinSource))
}
@@ -497,7 +497,7 @@ func validateRunnerReadProgressOverlay(original, overlay []byte, nonce string) (
for _, undo := range []struct{ from, to string }{
{fmt.Sprintf("\n\tprogress := &runnerObservedStdin{file: os.Stdin, nonce: %q, digest: sha256.New()}", nonce), ""},
{"progress.deadline(os.Stdin.SetReadDeadline(time.Time{}))", "os.Stdin.SetReadDeadline(time.Time{})"},
- {"io.ReadAll(io.LimitReader(progress, runnercontract.MaxRequestBytes+1))", "io.ReadAll(io.LimitReader(os.Stdin, runnercontract.MaxRequestBytes+1))"},
+ {"io.ReadAll(io.LimitReader(progress, maxBytes+1))", "io.ReadAll(io.LimitReader(os.Stdin, maxBytes+1))"},
{"\t\"crypto/sha256\"\n", ""}, {"\t\"encoding/json\"\n", ""}, {"\t\"hash\"\n", ""},
} {
if strings.Count(restored, undo.from) != 1 {
@@ -699,7 +699,7 @@ func TestRunnerTerminalReadProgressCheckout(t *testing.T) {
}
require.Equal(t, expected, physical)
if autocrlf {
- require.Equal(t, "0b65da8b09adae5c8a6a8b586c05eb1959512cb8d0287e9426c71f875a2d4272",
+ require.Equal(t, "d1bc94a3abfdde69d87cd503b2b82d1258816a0c829a8abf615d36252327765a",
fmt.Sprintf("%x", sha256.Sum256(physical)))
}
t.Chdir(directory)
@@ -799,7 +799,7 @@ func TestRunnerTerminalReadProgressOverlay(t *testing.T) {
{"r.file.Read(p)", "0, io.EOF"},
{"r.total == 2097153", "r.total > 0"},
{"os.Stdin.Close()", "nil"},
- {"runnercontract.MaxRequestBytes+1", "runnercontract.MaxRequestBytes"},
+ {"maxBytes+1", "maxBytes"},
{"progress.deadline(os.Stdin.SetReadDeadline(time.Time{}))", "os.Stdin.SetReadDeadline(time.Time{})"},
{"func (r *runnerObservedStdin) Read", "//go:nosplit\nfunc (r *runnerObservedStdin) Read"},
} {
diff --git a/cmd/waza/recovery_process_test.go b/cmd/waza/recovery_process_test.go
index 277d1a812..b7e748f0d 100644
--- a/cmd/waza/recovery_process_test.go
+++ b/cmd/waza/recovery_process_test.go
@@ -26,13 +26,20 @@ import (
)
func buildRecoveryCLI(t *testing.T) string {
+ t.Helper()
+ return buildRecoveryCLIFromDirectory(t, ".")
+}
+
+func buildRecoveryCLIFromDirectory(t *testing.T, directory string) string {
t.Helper()
binary := filepath.Join(t.TempDir(), "waza")
if runtime.GOOS == "windows" {
binary += ".exe"
}
started := time.Now()
- log, err := exec.CommandContext(t.Context(), "go", "build", "-o", binary, ".").CombinedOutput()
+ command := exec.CommandContext(t.Context(), "go", "build", "-o", binary, ".")
+ command.Dir = directory
+ log, err := command.CombinedOutput()
require.NoError(t, err, "%s", log)
info, err := os.Stat(binary)
require.NoError(t, err)
diff --git a/cmd/waza/root.go b/cmd/waza/root.go
index 295682abe..ababddaa0 100644
--- a/cmd/waza/root.go
+++ b/cmd/waza/root.go
@@ -85,6 +85,7 @@ performance against predefined test cases.`,
cmd.AddCommand(newRegistryCommand())
cmd.AddCommand(newRunnerCommand())
cmd.AddCommand(newArtifactCommand())
+ cmd.AddCommand(newAggregateCommand())
cmd.AddCommand(newInteropCommand())
return cmd
@@ -101,7 +102,7 @@ func shouldRunUpdateCheck(cmd *cobra.Command, noUpdateCheck bool) bool {
return false
}
for c := cmd; c != nil; c = c.Parent() {
- if c.Name() == "update" || c.Name() == "preflight" || c.Name() == "compare-plan" || c.Name() == "compare-assurance-plan" || c.Name() == "compare-collect" || c.Name() == "assure" || c.Name() == "runner" || c.Name() == "interop" {
+ if c.Name() == "update" || c.Name() == "preflight" || c.Name() == "compare-plan" || c.Name() == "compare-assurance-plan" || c.Name() == "compare-collect" || c.Name() == "assure" || c.Name() == "runner" || c.Name() == "interop" || c.Name() == "aggregate" {
return false
}
}
diff --git a/cmd/waza/runner_operation_unix.go b/cmd/waza/runner_operation_unix.go
new file mode 100644
index 000000000..da23b5b40
--- /dev/null
+++ b/cmd/waza/runner_operation_unix.go
@@ -0,0 +1,39 @@
+//go:build unix
+
+package main
+
+import (
+ "bytes"
+ "errors"
+ "fmt"
+ "io"
+ "os"
+ "time"
+
+ "golang.org/x/sys/unix"
+)
+
+func emitInheritedRunnerJSON(data []byte) error {
+ fd, err := unix.Dup(int(os.Stdout.Fd()))
+ if err != nil {
+ return fmt.Errorf("duplicating runner operation output: %w", err)
+ }
+ output := os.NewFile(uintptr(fd), "runner-operation-output")
+ payload := bytes.Clone(data)
+ result := make(chan error, 1)
+ go func() {
+ written, writeErr := output.Write(payload)
+ if writeErr == nil && written != len(payload) {
+ writeErr = io.ErrShortWrite
+ }
+ result <- errors.Join(writeErr, output.Close())
+ }()
+ timer := time.NewTimer(5 * time.Second)
+ defer timer.Stop()
+ select {
+ case err := <-result:
+ return err
+ case <-timer.C:
+ return fmt.Errorf("writing runner operation output: %w", os.ErrDeadlineExceeded)
+ }
+}
diff --git a/cmd/waza/runner_operation_windows.go b/cmd/waza/runner_operation_windows.go
new file mode 100644
index 000000000..0f98097a0
--- /dev/null
+++ b/cmd/waza/runner_operation_windows.go
@@ -0,0 +1,30 @@
+package main
+
+import (
+ "bytes"
+ "fmt"
+ "io"
+ "os"
+ "time"
+)
+
+func emitInheritedRunnerJSON(data []byte) error {
+ payload := bytes.Clone(data)
+ output := os.Stdout
+ done := make(chan error, 1)
+ timer := time.NewTimer(5 * time.Second)
+ defer timer.Stop()
+ go func() {
+ written, err := output.Write(payload)
+ if err == nil && written != len(payload) {
+ err = io.ErrShortWrite
+ }
+ done <- err
+ }()
+ select {
+ case err := <-done:
+ return err
+ case <-timer.C:
+ return fmt.Errorf("writing runner operation output: %w", os.ErrDeadlineExceeded)
+ }
+}
diff --git a/cmd/waza/runner_stdin_unix.go b/cmd/waza/runner_stdin_unix.go
index 4b432c012..9f8d0a498 100644
--- a/cmd/waza/runner_stdin_unix.go
+++ b/cmd/waza/runner_stdin_unix.go
@@ -29,9 +29,13 @@ type runnerOutputResult struct {
// Inherited descriptors may share their open file description with a parent or
// peer. Keep their flags untouched; the CLI exits after either result wins.
func readInheritedRunnerInput(ctx context.Context) ([]byte, error) {
+ return readInheritedRunnerInputLimit(ctx, runnercontract.MaxRequestBytes)
+}
+
+func readInheritedRunnerInputLimit(ctx context.Context, maxBytes int64) ([]byte, error) {
result := make(chan runnerInputResult, 1)
go func() {
- data, err := readInheritedRunnerRequest()
+ data, err := readInheritedRunnerRequest(maxBytes)
result <- runnerInputResult{data: data, err: err}
}()
select {
@@ -51,10 +55,10 @@ func readInheritedRunnerInput(ctx context.Context) ([]byte, error) {
// Signals can interrupt a blocking Unix read before signal.NotifyContext has
// scheduled its cancellation. Retrying EINTR lets that cancellation win instead
// of converting SIGINT/SIGTERM into an infrastructure failure.
-func readInheritedRunnerRequest() ([]byte, error) {
+func readInheritedRunnerRequest(maxBytes int64) ([]byte, error) {
var data bytes.Buffer
buffer := make([]byte, 32<<10)
- limit := int64(runnercontract.MaxRequestBytes + 1)
+ limit := maxBytes + 1
for int64(data.Len()) < limit {
remaining := limit - int64(data.Len())
readBuffer := buffer
diff --git a/cmd/waza/runner_stdin_windows.go b/cmd/waza/runner_stdin_windows.go
index 29283f568..a22dcda5b 100644
--- a/cmd/waza/runner_stdin_windows.go
+++ b/cmd/waza/runner_stdin_windows.go
@@ -13,6 +13,10 @@ import (
)
func readInheritedRunnerInput(ctx context.Context) (data []byte, resultErr error) {
+ return readInheritedRunnerInputLimit(ctx, runnercontract.MaxRequestBytes)
+}
+
+func readInheritedRunnerInputLimit(ctx context.Context, maxBytes int64) (data []byte, resultErr error) {
if err := os.Stdin.SetReadDeadline(time.Time{}); err == nil {
finished := make(chan struct{})
var interruptErr error
@@ -50,7 +54,7 @@ func readInheritedRunnerInput(ctx context.Context) (data []byte, resultErr error
if err := ctx.Err(); err != nil {
return nil, fmt.Errorf("runner input interrupted: %w", err)
}
- data, err := io.ReadAll(io.LimitReader(os.Stdin, runnercontract.MaxRequestBytes+1))
+ data, err := io.ReadAll(io.LimitReader(os.Stdin, maxBytes+1))
if cause := ctx.Err(); cause != nil {
return nil, fmt.Errorf("runner input interrupted: %w", cause)
}
diff --git a/cmd/waza/runner_submission_transport.go b/cmd/waza/runner_submission_transport.go
new file mode 100644
index 000000000..e285fb21b
--- /dev/null
+++ b/cmd/waza/runner_submission_transport.go
@@ -0,0 +1,36 @@
+package main
+
+import (
+ "bytes"
+ "encoding/json"
+ "fmt"
+ "io"
+ "os"
+
+ "github.com/spf13/cobra"
+)
+
+// Callers validate the strict result profile and apply capture policy first.
+func emitRunnerOperationJSON(cmd *cobra.Command, data []byte, exitCode int, status string) error {
+ if !json.Valid(data) {
+ return &ExitCodeError{Code: 2, Err: fmt.Errorf("runner operation output is not valid JSON")}
+ }
+ payload := append(bytes.Clone(bytes.TrimRight(data, " \t\r\n")), '\n')
+ var err error
+ if cmd.OutOrStdout() == os.Stdout {
+ err = emitInheritedRunnerJSON(payload)
+ } else {
+ var written int
+ written, err = cmd.OutOrStdout().Write(payload)
+ if err == nil && written != len(payload) {
+ err = io.ErrShortWrite
+ }
+ }
+ if err != nil {
+ return &ExitCodeError{Code: 2, Err: fmt.Errorf("writing runner operation: %w", err)}
+ }
+ if exitCode != 0 {
+ return &ExitCodeError{Code: exitCode, Err: fmt.Errorf("runner operation: %s", status)}
+ }
+ return nil
+}
diff --git a/cmd/waza/runner_submission_transport_test.go b/cmd/waza/runner_submission_transport_test.go
new file mode 100644
index 000000000..61d502b17
--- /dev/null
+++ b/cmd/waza/runner_submission_transport_test.go
@@ -0,0 +1,200 @@
+package main
+
+import (
+ "bytes"
+ "context"
+ "crypto/sha256"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "io"
+ "os"
+ "os/exec"
+ "strings"
+ "testing"
+
+ "github.com/microsoft/waza/internal/runnercontract"
+ "github.com/microsoft/waza/internal/runnerunit"
+ "github.com/spf13/cobra"
+ "github.com/stretchr/testify/require"
+)
+
+func TestRunnerSubmissionInputRetainsWrapperLimit(t *testing.T) {
+ raw, err := os.ReadFile("../../examples/runner/unit.json")
+ require.NoError(t, err)
+ wrapper, err := json.Marshal(runnerunit.SubmissionRequest{
+ Kind: "waza.execution-submission", SchemaVersion: "1.0", Request: raw,
+ })
+ require.NoError(t, err)
+ padded := append(bytes.Clone(wrapper), bytes.Repeat([]byte(" "), runnerunit.MaxSubmissionBytes-len(wrapper))...)
+ require.Len(t, padded, runnerunit.MaxSubmissionBytes)
+ for _, control := range []struct {
+ name string
+ data []byte
+ size int
+ }{
+ {"at-limit", padded, runnerunit.MaxSubmissionBytes},
+ {"over-limit", append(bytes.Clone(padded), ' ', ' '), runnerunit.MaxSubmissionBytes + 1},
+ } {
+ t.Run(control.name, func(t *testing.T) {
+ data, err := readRunnerInputLimit(t.Context(), bytes.NewReader(control.data), runnerunit.MaxSubmissionBytes)
+ require.NoError(t, err)
+ require.Equal(t, control.data[:control.size], data)
+ _, err = runnerunit.ParseSubmission(data)
+ if control.name == "at-limit" {
+ require.NoError(t, err)
+ } else {
+ require.Error(t, err)
+ }
+ })
+ }
+ legacy, err := readRunnerInput(t.Context(), bytes.NewReader(padded))
+ require.NoError(t, err)
+ require.Equal(t, padded[:runnercontract.MaxRequestBytes+1], legacy)
+}
+
+func TestRunnerSubmissionInheritedInputChild(t *testing.T) {
+ if os.Getenv("WAZA_TEST_SUBMISSION_INPUT") != "1" {
+ return
+ }
+ data, err := readRunnerInputLimit(t.Context(), os.Stdin, runnerunit.MaxSubmissionBytes)
+ require.NoError(t, err)
+ _, parseErr := runnerunit.ParseSubmission(data)
+ require.NoError(t, json.NewEncoder(os.Stdout).Encode(struct {
+ Bytes int `json:"bytes"`
+ SHA256 string `json:"sha256"`
+ Admitted bool `json:"admitted"`
+ }{len(data), fmt.Sprintf("%x", sha256.Sum256(data)), parseErr == nil}))
+}
+
+func TestRunnerSubmissionInheritedInputRetainsWrapperLimit(t *testing.T) {
+ raw, err := os.ReadFile("../../examples/runner/unit.json")
+ require.NoError(t, err)
+ wrapper, err := json.Marshal(runnerunit.SubmissionRequest{
+ Kind: "waza.execution-submission", SchemaVersion: "1.0", Request: raw,
+ })
+ require.NoError(t, err)
+ padded := append(bytes.Clone(wrapper), bytes.Repeat([]byte(" "), runnerunit.MaxSubmissionBytes-len(wrapper))...)
+ for _, overLimit := range []bool{false, true} {
+ t.Run(fmt.Sprint(overLimit), func(t *testing.T) {
+ payload := bytes.Clone(padded)
+ expectedSize := len(payload)
+ if overLimit {
+ payload = append(payload, ' ', ' ')
+ expectedSize++
+ }
+ child := exec.CommandContext(t.Context(), os.Args[0], "-test.run=^TestRunnerSubmissionInheritedInputChild$", "-test.count=1")
+ for _, entry := range os.Environ() {
+ if !strings.HasPrefix(entry, "WAZA_TEST_SUBMISSION_INPUT=") {
+ child.Env = append(child.Env, entry)
+ }
+ }
+ child.Env = append(child.Env, "WAZA_TEST_SUBMISSION_INPUT=1")
+ child.Stdin = bytes.NewReader(payload)
+ var stderr bytes.Buffer
+ child.Stderr = &stderr
+ output, err := child.Output()
+ require.NoError(t, err, "%s", stderr.String())
+ var observed struct {
+ Bytes int `json:"bytes"`
+ SHA256 string `json:"sha256"`
+ Admitted bool `json:"admitted"`
+ }
+ require.NoError(t, json.NewDecoder(bytes.NewReader(output)).Decode(&observed), "%s", output)
+ require.Equal(t, expectedSize, observed.Bytes)
+ require.Equal(t, fmt.Sprintf("%x", sha256.Sum256(payload[:expectedSize])), observed.SHA256)
+ require.Equal(t, !overLimit, observed.Admitted)
+ })
+ }
+}
+
+func TestRunnerSubmissionInputCancellationAndReadFailure(t *testing.T) {
+ ctx, cancel := context.WithCancel(t.Context())
+ cancel()
+ reader := &submissionInputFailure{}
+ data, err := readRunnerInputLimit(ctx, reader, runnerunit.MaxSubmissionBytes)
+ require.ErrorIs(t, err, context.Canceled)
+ require.Nil(t, data)
+ require.Zero(t, reader.calls, "canceled admission must not start reading")
+
+ data, err = readRunnerInputLimit(t.Context(), reader, runnerunit.MaxSubmissionBytes)
+ require.ErrorIs(t, err, errSubmissionInput)
+ require.Nil(t, data)
+ require.Equal(t, 1, reader.calls)
+}
+
+var errSubmissionInput = errors.New("controlled submission input failure")
+
+type submissionInputFailure struct{ calls int }
+
+func (r *submissionInputFailure) Read([]byte) (int, error) {
+ r.calls++
+ return 0, errSubmissionInput
+}
+
+func TestRunnerOperationOutputFramingAndFailure(t *testing.T) {
+ command := &cobra.Command{}
+ for _, suffix := range []string{"", "\n", "\n\n", "\r\n \t\n"} {
+ payload := []byte("{\"recorded\":0}" + suffix)
+ original := bytes.Clone(payload)
+ var output bytes.Buffer
+ command.SetOut(&output)
+ require.NoError(t, emitRunnerOperationJSON(command, payload, 0, "completed"))
+ require.Equal(t, "{\"recorded\":0}\n", output.String())
+ require.Equal(t, original, payload, "framing must not modify caller-owned source bytes")
+ }
+ for _, invalid := range []string{"{", "{}{}", ""} {
+ var output bytes.Buffer
+ command.SetOut(&output)
+ err := emitRunnerOperationJSON(command, []byte(invalid), 0, "completed")
+ require.Error(t, err)
+ require.Empty(t, output.Bytes())
+ }
+ for _, control := range []struct {
+ name string
+ err error
+ short bool
+ }{
+ {name: "write-error", err: io.ErrClosedPipe},
+ {name: "short-write", short: true},
+ {name: "complete-write"},
+ } {
+ t.Run(control.name, func(t *testing.T) {
+ writer := &submissionOutputWriter{err: control.err, short: control.short}
+ command.SetOut(writer)
+ err := emitRunnerOperationJSON(command, []byte(`{"recorded":0}`), 1, "completed")
+ exit, ok := errors.AsType[*ExitCodeError](err)
+ require.True(t, ok)
+ require.Equal(t, 1, writer.calls, "output failure must never trigger a fallback write")
+ require.Equal(t, "{\"recorded\":0}\n", string(writer.data))
+ if control.err != nil {
+ require.Equal(t, 2, exit.Code)
+ require.ErrorIs(t, err, control.err)
+ } else if control.short {
+ require.Equal(t, 2, exit.Code)
+ require.ErrorIs(t, err, io.ErrShortWrite)
+ } else {
+ require.Equal(t, 1, exit.Code)
+ }
+ })
+ }
+}
+
+type submissionOutputWriter struct {
+ calls int
+ data []byte
+ err error
+ short bool
+}
+
+func (w *submissionOutputWriter) Write(data []byte) (int, error) {
+ w.calls++
+ w.data = bytes.Clone(data)
+ if w.err != nil {
+ return 0, w.err
+ }
+ if w.short {
+ return len(data) - 1, nil
+ }
+ return len(data), nil
+}
diff --git a/docs/DEMO-GUIDE.md b/docs/DEMO-GUIDE.md
index f0b2ae6bf..3c422eb6e 100644
--- a/docs/DEMO-GUIDE.md
+++ b/docs/DEMO-GUIDE.md
@@ -21,6 +21,19 @@ the [terminal outcome and diagnostics](../site/src/content/docs/guides/runner-te
with an exact terminal JSON export. This demo remains mock-only; opted-in exact
Copilot/text CLI execution has its own paid-call and disabled-tool/skill boundaries.
+For the unreleased checkpoint source preview, build a clean versioned CLI and
+open **Runner contract → Create or resume an exact-unit checkpoint**. Use an
+existing private `.waza-artifacts` store under the served results directory.
+Create the captured mock example, then resume its finalized digest: new calls
+must be zero while the original terminal outcome and ordinal remain unchanged.
+Failed originals remain failed grading. Unknown progress is unavailable, not
+zero; an incompatible source is withheld. Download the exact protected JSON.
+This is not a whole-evaluation or ordered-lineage demonstration. Stop and join
+the original worker before before-unit recovery; stopping a browser request
+does not prove worker termination. See the [checkpoint workflow](../site/src/content/docs/guides/immutable-artifacts.mdx#checkpoint-public-source-preview).
+
+
+
For an offline controlled-comparison demo, follow
[the source-bound planning and collection example](CONTROLLED-COMPARISONS.md).
Its one-cluster mock design intentionally remains inconclusive (selected gate
diff --git a/docs/EXECUTION_CHECKPOINTS.md b/docs/EXECUTION_CHECKPOINTS.md
index c01647250..cf0bf1d69 100644
--- a/docs/EXECUTION_CHECKPOINTS.md
+++ b/docs/EXECUTION_CHECKPOINTS.md
@@ -2,9 +2,11 @@
This unreleased, partial #765 layer builds on the
[logical-attempt core](EXECUTION_ATTEMPTS.md) and immutable manifest v1. It
-provides an embedding service, not released CLI commands, protected HTTP
-operations, or a Lab checkpoint view. Those public surfaces and full #755
-acceptance remain pending.
+provides an embedding service and unreleased source commands
+`runner checkpoint create` and `runner checkpoint resume`. Installed v0.40.1
+artifacts do not contain these commands. Protected HTTP create/resume operations
+are registered and consumed by the source Lab. Broader native-platform evidence
+and full #755 acceptance remain pending.
## Supported boundaries
@@ -101,6 +103,52 @@ are integration controls, not evidence that a clean production CLI was run.
Working-directory Git, caller JSON and version-shaped fallback strings are not
accepted sources of build identity.
+The shared CLI input reader accepts an explicit bound for submission wrappers
+(`runnerunit.MaxSubmissionBytes`), including inherited stdin. It retains one
+extra byte so the strict parser can reject oversized input rather than accepting
+a truncated document. Legacy `runner run` and `runner validate` retain their
+original `runnercontract.MaxRequestBytes` bound and stdin cancellation behavior.
+Public commands use this wrapper bound, not the bare runner-request bound.
+
+The CLI accepts the existing private store through `--store` or an explicit Azure
+container through `--azure-container-url`; those flags are mutually exclusive.
+It writes one capture-approved `waza.runner-checkpoint-result` record to stdout
+and boundary/digest progress to stderr. Save stdout even when the exit is nonzero:
+reused failure, cancellation, and timeout preserve their original exit codes.
+An output or storage-close failure exits 2 and is not successful delivery.
+See the [operator workflow](../site/src/content/docs/guides/immutable-artifacts.mdx#checkpoint-public-source-preview).
+
+The dashboard server registers `POST /api/v1/checkpoints` and
+`POST /api/v1/checkpoints/{digest}/resume`. Both accept the same submission JSON
+and return the protected checkpoint-result record with `X-Waza-Exit-Code`.
+They require a loopback peer, literal loopback Host, same-origin browser metadata,
+JSON content type, and bounded request/response transport. The server selects
+the existing `RESULTS_DIR/.waza-artifacts` store or its configured Azure provider;
+request JSON cannot select host paths or provider locations. Unsupported active
+providers fail closed. No raw checkpoint GET endpoint is exposed.
+
+The source Lab's **Runner contract** page explicitly creates or resumes one unit,
+shows compatibility, source withholding, original terminal outcome/ordinal, retry
+parent or reuse source, and separate new/original engine-call counts. Unknown
+progress stays unavailable; recorded zero stays zero. It downloads the exact
+protected response, not a reconstruction or raw report. Edits, new requests and
+reload failures clear old results; demo mode has no operation inputs.
+Contradictory original terminal exits or new engine-call counts are rejected,
+not displayed or exported as successful results.
+Error observations may retain partial or empty grader verdicts; they are not
+reclassified as successfully graded work.
+Completion declarations must match the original execution ID and canonical
+reservation digest before display or download. This checks declaration linkage,
+not storage integrity or execution authority.
+Resume source declarations must also hash to the requested checkpoint digest,
+independent of response object-key order, before display or download.
+A completed resume, including a partial publication failure, without a source is
+rejected unless the failure explicitly withholds it. Source-withheld failures remain
+available with their recorded unavailable progress.
+Create results cannot contain resume-only source or reuse state.
+Stopping a browser request is not proof of worker termination. This panel is
+not an ordered evaluation-wide attempt history or a whole-evaluation resume UI.
+
`Put` can return a digest **and an error** if a blob may remain after failure.
Preserve that reference and inspect it; never treat it as verified publication.
Finalized checkpoint publication can fail after the original execution completed.
@@ -109,8 +157,13 @@ submission's terminal facts. There is no cross-object transaction, automatic
rollback, inferred crash durability, or deletion of partial evidence.
Raw core results can contain original source metadata. They are not approved
-public disclosure responses. Public endpoints must compose the evidence owner's
-capture policy before encoding them; that integration remains separately owned.
+public disclosure responses. `SubmitCaptured` and `ResumeCaptured` compose the
+evidence owner's shared `ownedsubmission.CaptureHooks` before projection and
+encoding. An incompatible resume can return a source before submission hooks run;
+the wrapper withholds that unapproved source rather than encoding the raw result.
+Capture-denied finalized reuse returns a protected submission error (HTTP 500,
+exit 2) without new engine calls. The HTTP regression checks withheld source and
+terminal data and unchanged original artifact, manifest, sidecar and checkpoint bytes.
The checkpoint result writer validates the original nested submission, including
its reservation, decision and terminal, before marshaling. Invalid UTF-8 is
rejected rather than normalized into apparently valid text. Original
@@ -126,6 +179,10 @@ work separately from the retained-reservation and provider-validation phases;
phase timings are diagnostics, not execution or recovery evidence. Publication
budgets are unchanged. A timeout remains a publication failure: preserve the
original completion and any reported unverified blob rather than retrying.
+The CLI sends underlying operation and flag-parsing errors to bounded operator stderr while
+keeping the public JSON diagnostic sanitized. Conflicting store flags also use
+the single-record JSON rejection path. Build-revision admission failures retain
+their actionable cause and rebuild guidance on operator stderr as well.
## Migration and validation limits
@@ -142,5 +199,19 @@ they are not production CLI/clean-build, live-inference, native Windows, or Lab
evidence. The root test process skips two Unix permission-denial cases; those
same compiled cases are separately executed under a non-root identity.
-Protected CLI/API/Lab integration, compiled clean-CLI kill/resume cases, all
-current-head broad/platform gates and independent review remain release blockers.
+The additional Linux production-process fixture builds the ordinary owning CLI
+from an isolated clean checkout of committed build inputs. A Linux syscall tracer
+stops the unchanged binary when the actual immutable boundary blob becomes visible,
+kills and reaps it, and joins its output-copy goroutines before starting a fresh
+CLI. It does not use a bounded diagnostic write or a wall-clock pause as a barrier.
+Before restore creates a fresh retry child; finalized success and failed-check
+restore preserve original terminal bytes with zero new calls. A separate actual
+service producer leaves a dispatch without completion; the compiled consumer
+refuses it without new records. That control does not kill a CLI mid-dispatch.
+These are offline mock-execution controls, not native runtime, LFS installation,
+live inference, dashboard delivery, or Windows execution evidence.
+
+Ordered evaluation-level lineage, generic protected submissions, full current-head
+broad/platform gates and independent review remain release blockers. Broad test
+failures and timed-out lint runs are not passing checks even when focused recovery
+controls pass.
diff --git a/docs/GUIDE.md b/docs/GUIDE.md
index a1fe5f4a3..68bd1c835 100644
--- a/docs/GUIDE.md
+++ b/docs/GUIDE.md
@@ -16,6 +16,11 @@ Explicit opt-in and platform limits still apply; see
## What is Waza?
+For the opt-in v0.41 source preview of standalone aggregation over completed
+public manifests, see [commands, populations and migration](MANIFEST_AGGREGATION.md).
+It does not start agents or recover worker ownership; admission completeness,
+terminal outcomes and recorded historical usage remain separate.
+
Waza helps you:
- **Define agent skills** with comprehensive documentation and behavioral requirements
@@ -514,6 +519,14 @@ unchanged. See [outcomes, exits and migration](../site/src/content/docs/guides/r
[identity guide](../site/src/content/docs/guides/runner-identities.mdx) and
[canonical contract and migration guide](../site/src/content/docs/guides/runner-contract.mdx).
+The unreleased source checkpoint commands accept a captured submission wrapper:
+`waza runner checkpoint create --store .waza-artifacts < submission.json` and
+`waza runner checkpoint resume CHECKPOINT_SHA256 --store .waza-artifacts <
+submission.json`. They require a clean VCS-stamped owning CLI and preserve
+original terminal failures on finalized reuse. See the
+[checkpoint workflow, boundaries and troubleshooting](../site/src/content/docs/guides/immutable-artifacts.mdx#checkpoint-public-source-preview);
+these commands are not in installed v0.40.1 artifacts.
+
### `waza init [directory]`
Initialize a waza project with directory structure.
diff --git a/docs/MANIFEST_AGGREGATION.md b/docs/MANIFEST_AGGREGATION.md
new file mode 100644
index 000000000..448d35a16
--- /dev/null
+++ b/docs/MANIFEST_AGGREGATION.md
@@ -0,0 +1,204 @@
+# Standalone completed-manifest aggregation
+
+The opt-in, unreleased v0.41 aggregate profile (#766) reads **public completed
+manifests**, not mutable `results.json`, worker state or creator capabilities.
+It does not construct an engine, invoke a subprocess, read ambient project
+configuration or obtain credentials. The command accepts an explicitly selected
+existing **local** store. Azure aggregation is not supported by this profile:
+the default credential chain may invoke external credential helpers.
+
+## Commands and interpretation
+
+```bash
+# DIGEST is a full lowercase manifest SHA-256 from artifact finalize.
+waza aggregate --store .waza-artifacts "$DIGEST" -o aggregate.json
+
+# Reference pairs additionally qualify categories and execution records.
+waza aggregate --store .waza-artifacts --inputs inputs.json -o qualified.json
+```
+
+`inputs.json` is a nonempty array of at most 128 objects. Each object has
+`manifest_sha256` and optionally `evidence_sha256`. Explicit null/empty evidence
+references, unknown fields, duplicate keys and abbreviated digests are invalid.
+See the [site worked example](../site/src/content/docs/guides/manifest-aggregation.mdx).
+
+The command emits the evaluated report to stdout and, when selected, exclusively
+creates `--output`. Existing output files and symlinks are never replaced.
+An incomplete/conflicting evaluated report is **saved and emitted before a
+nonzero exit**. Input admission, unavailable store, output write/close and stdout
+errors also fail. An output failure may leave a partial invocation-created file;
+the error identifies it and no competing/user-owned file is deleted. This is
+exclusive file creation, not a transactional or crash-durable publication.
+
+The read-only `POST /api/v1/aggregates` route accepts the same reference array
+against the server's explicitly configured results store. Evaluated reports use
+HTTP 200 (complete) or 409 (incomplete/conflicting), preserving report bodies.
+Malformed requests use 400; unavailable/unsupported providers use 503. Requests
+cannot choose paths, credentials, providers or budgets. Lab retains evaluated
+409 reports, exposes composition/category states/omissions/compatibility and
+downloads the exact API JSON. Source links inspect manifest receipts, not
+restricted artifact payloads.
+
+HTTP body admission retains the 1 MiB request limit and has a five-second read
+deadline, before any store is opened. A stalled upload returns 408 with
+`aggregate_request_timeout`; cancellation stops admission (408 when delivery is
+still possible). Partial uploads close the connection rather
+than draining indefinitely. Deadline setup/reset failure returns 500 with
+`aggregate_transport_unavailable`. Unsupported bounded delivery aborts the
+transport without an implicit successful response. Successful input admission
+clears its read deadline before the separate 30-second evaluation budget.
+Response delivery is bounded to five seconds and preserves canonical report
+bytes unchanged; a failed/partial delivery is not a report receipt.
+The aggregate client allows 50 seconds: these three phases plus ten seconds of
+headroom. Caller cancellation remains active. This is not a universal server
+lifetime guarantee: source open/close and synchronous filesystem I/O have their
+own constraints. Aggregate diagnostics use the existing bounded reporter;
+a blocked operator sink cannot retain one new logging goroutine per request.
+
+Open `waza serve --lab --results-dir RESULTS_DIR`, select **Runner**, then use
+**Aggregate completed public manifests**. The store is
+`RESULTS_DIR/.waza-artifacts`, independent of the server's working directory.
+Aggregation is disabled in the synthetic demo and active non-local providers
+are rejected without attempting credential discovery.
+
+
+
+The example's usage is authored mock-source data, not observed SDK billing.
+
+**Complete means aggregate admission completed, not that any task passed.**
+Original failed, cancelled and timed-out terminal outcomes remain visible.
+Optional absent sidecars, supported metadata-only manifests and legitimate
+privacy-redacted categories do not make admission incomplete. Supplied invalid
+sidecars or unverified execution claims do.
+
+## Population and conflict rules
+
+Three explicitly named populations must not be interchanged:
+
+1. `input_members`: unique exact manifest/evidence reference pairs, including
+ omissions and multiple views. State and outcome counts use this denominator.
+2. `unique_verified_manifests`: distinct manifests whose current complete bytes,
+ terminal binding and final marker were verified.
+3. `unique_verified_execution_records`: distinct source execution IDs whose
+ original canonical reservation/completion records match the verified manifest.
+ Record-context verification is **not execution attestation**.
+
+The sidecar-selected attempt descriptor must match the original reservation's
+digest, byte length and JSON media type even when it uses a custom artifact
+name. Matching execution ID and binding alone do not establish this association.
+
+All source objects and the terminal are verified by `Store.ReadCompleted`, even
+when evidence-sidecar selection does not retain terminal bytes. Aggregation never
+infers a unique execution from a legacy attempt ordinal or a digest.
+
+Members are display-grouped by the entire admitted identity, selection, mode and
+sorted grader profile. Different evals/runs/tasks/trials/attempts/modes/profiles
+remain separate. Distinct actual execution records in the same profile are
+valid repeated executions: retain them all, without a latest winner or pooled
+scores. Multiple compatible sidecars/views of one record remain separate members
+but contribute at most one execution. Do not union category disclosure or select
+the most permissive view. Actual incompatible origin/binding/record/usage claims
+produce explicit conflicts; no arbitrary winner is selected.
+
+### Historical usage, never aggregation billing
+
+Aggregation itself incurs no SDK billing. `recorded_historical_usage` is available
+only for a nonempty population of uniquely admitted native execution records
+whose supplied views all provide compatible, present recorded native usage.
+Failed/cancelled executions are eligible when their usage is genuinely complete.
+Cached lineage or cached categories, unknown legacy identity, missing usage,
+withheld views or conflicting usage withhold the **whole total**, not a partial
+subtotal. Missing credits/model attribution remain unavailable. Explicit recorded
+zero credits and native zero counters remain zero, not absence.
+Saved-report and client admission also reject a complete usage claim when any
+supplied view lacks execution identity or a verified, present single recorded
+usage observation, or declares cached categories/lineage exclusion. A compatible
+eligible view cannot override another view's exclusion. Complete metadata-only
+admission still retains unavailable/null usage. These checks do not reverify
+saved evidence or require live-only provenance. Optional aggregate credits and
+model attribution may remain absent; absence is unavailable, not recorded zero.
+
+Totals use `models.AggregateUsageStats` in sorted execution-ID order. Mock,
+authored, cached and unknown source labels remain distinguishable per view.
+Recorded content and shape do not prove SDK execution, billing authenticity,
+independent witnessing or assurance.
+
+## Canonical schema, compatibility and migration
+
+Aggregate `kind: waza.manifest-aggregate`, `schema_version: "1.0"` is independent
+of manifest, evidence, attempt, terminal and legacy result versions. Canonical
+bytes contain no timestamp, self-hash, filesystem path, provider URL, captured
+payload or free-text diagnostic. Reference pairs, groups, category records,
+diagnostic codes, map keys and summation inputs have deterministic ordering.
+Identical requested sets **and observed source state** produce identical bytes;
+an earlier successful read is never a cache of current verification.
+
+| Input | v1 interpretation |
+|---|---|
+| Completed manifest 1.0, terminal 1.0, no sidecar | Complete metadata-only admission; execution/usage unknown |
+| Evidence 1.0, capture 1.0 with original attempt/completion 1.0 | Verify current bytes, category shape and original record context separately |
+| Optional sidecar omitted | Not a failed input; no inferred category/usage/ownership |
+| Supplied sidecar invalid, missing, mismatched or unsupported | Explicit incomplete/conflicting member |
+| Missing/corrupt source or absent/conflicting final marker | Explicit omission/conflict; independently valid neighbors retained |
+| Mixed supported legacy/native sources | Preserve per-member profile and unavailable usage, never upgrade legacy evidence |
+| Unknown major/profile version | Explicit unsupported/invalid profile; no repair or downgrade |
+| Historical mutable result files | Not aggregate inputs; remain readable through existing result readers |
+
+No automatic migration synthesizes execution records, scores, billing or evidence.
+Retain the original immutable store, select exact digests and add a real public
+sidecar only when original evidence exists. Existing readers/writer versions and
+ordinary run/regrade/cache behavior are unchanged.
+
+Developer entrypoints are `manifestaggregate.Build(ctx, Sources, []Input, Limits)`,
+`ParseInputs`, `Encode` and `Parse`. Sources explicitly carry a read-only object
+capability and optional scoped `NamespaceReader`; no backend-derived namespace
+discovery or creator hydration occurs. One per-build budget bounds object count
+and total object/namespace reads, reserving the full permitted read before I/O.
+Errors retain their reservation because consumed bytes may be unknown. Default
+limits are 128 input references, 4,096 reads, 512 MiB cumulative source reads and
+32 MiB report bytes (including worst-case escaped identifiers), in addition to
+each existing source profile's limits. The aggregate client enforces the same
+32 MiB UTF-8 byte limit and saved-report population/state/group accounting.
+Unrelated JSON clients retain their existing parser limit. Numeric sum overflow
+withholds historical usage with an explicit diagnostic rather than emitting
+wrapped counters.
+Recorded usage totals must also stay within JavaScript's exact integer range
+(`0..9007199254740991`) for the JSON/API/Lab contract. An input or sum outside
+that supported range, non-finite arithmetic, or addition that loses an entire
+positive floating contribution withholds the total. Native floating fields
+retain their existing IEEE-754 arithmetic; no exact-decimal or derived billing
+claim is added. An empty eligible population and unknown identity/usage remain
+unavailable/null, never an available zero or an unlabeled subset total.
+
+The full serialized Go-escaped UTF-8 report is measured after accounting. If
+recorded usage would push it above 32 MiB, the entire total is withheld with
+`recorded_usage_report_size_unavailable`; all members, diagnostics and the unique
+execution denominator remain unchanged. No model/provider strings are truncated
+and no smaller execution subset becomes the total. If membership without usage
+cannot fit, aggregation returns an explicit size-limit error rather than dropping
+members. Native source limits and the report cap are unchanged.
+
+`Build` returns report plus classified error for evaluated per-input failures.
+`Parse` validates saved wire/accounting only; it is not current source inspection.
+Canonical report bytes and any future storage/publication receipts must remain
+separate; a stored hash alone does not establish current completeness.
+
+## Troubleshooting
+
+- `missing_source`: preserve the request set; restore its original declared
+ objects/finalization instead of silently dropping members.
+- `corrupt_source` / `conflicting_source`: preserve evidence and inspect original
+ public manifests and records. Never overwrite or choose an arbitrary winner.
+- `unsupported_or_invalid_profile`: inspect independent version declarations;
+ use a compatible reader, not invented conversion or legacy identity.
+- `execution_read_capability_unavailable`: provide the explicit public namespace
+ read capability. Do not recover private ownership or fall back to raw reads.
+- `read_budget_exhausted`: report remains incomplete. Reduce an explicitly
+ selected set or use an embedding caller with reviewed finite bounds; do not
+ describe a subset as the complete original population.
+- `complete_historical_usage_unavailable`: inspect each view's source/state and
+ cached lineage. An unavailable total is not zero or a passing quality verdict.
+
+This source preview is not a v0.41 publication announcement or full platform
+acceptance claim. Hosted checks, independent review and release delivery remain
+separate gates.
diff --git a/docs/PRD.md b/docs/PRD.md
index f74f60e7b..319dca158 100644
--- a/docs/PRD.md
+++ b/docs/PRD.md
@@ -85,6 +85,25 @@ and an explicit manifest; no billing, evidence or execution attestation is
invented. Attempt ownership remains the separate #764 contract. See
[publication and migration](IMMUTABLE_ARTIFACTS.md).
+The additive standalone aggregation profile (#766) consumes explicit completed
+public manifest/evidence references without agents or original worker state.
+It preserves repeated executions and compatible per-record views, distinguishes
+input-member/manifest/execution-record populations, and reports omissions and
+incompatible claims without selecting a winner or inventing usage. Lab exposes
+composition and downloads; supported legacy metadata remains explicitly
+unqualified. Aggregate schema 1.0 and complete historical usage are independent
+of terminal pass and execution attestation. This is unreleased source behavior,
+not publication or full current-head acceptance.
+See [aggregation contracts and migration](MANIFEST_AGGREGATION.md).
+
+The #765 source preview adds protected checkpoint create/resume in the CLI,
+HTTP API and Lab. Lab displays original outcomes, retry/reuse references and
+explicitly unavailable progress, and downloads the unchanged protected result.
+Finalized restore preserves the original terminal with zero new calls;
+before-unit restore requires a stopped original worker and a fresh retry child.
+This does not complete #764 ordered lineage, native-platform acceptance or the
+v0.41 release. See [checkpoint guarantees and limits](EXECUTION_CHECKPOINTS.md).
+
External-schema interoperability (#789) remains partial. The v0.40.0
adapter projects one explicitly selected, commit-pinned AgentV case-sensitive
`contains` assertion to a single native text grader and exports the same
diff --git a/docs/images/dashboard-lab-checkpoint.png b/docs/images/dashboard-lab-checkpoint.png
new file mode 100644
index 000000000..834cf5ffd
Binary files /dev/null and b/docs/images/dashboard-lab-checkpoint.png differ
diff --git a/docs/images/manifest-aggregate.png b/docs/images/manifest-aggregate.png
new file mode 100644
index 000000000..4daa39ed3
Binary files /dev/null and b/docs/images/manifest-aggregate.png differ
diff --git a/docs/images/site-cli-reference.png b/docs/images/site-cli-reference.png
index 57cb93e16..0d407240b 100644
Binary files a/docs/images/site-cli-reference.png and b/docs/images/site-cli-reference.png differ
diff --git a/internal/manifestaggregate/accounting_test.go b/internal/manifestaggregate/accounting_test.go
new file mode 100644
index 000000000..575dddb34
--- /dev/null
+++ b/internal/manifestaggregate/accounting_test.go
@@ -0,0 +1,60 @@
+package manifestaggregate
+
+import (
+ "encoding/json"
+ "strings"
+ "testing"
+
+ "github.com/microsoft/waza/internal/models"
+ "github.com/stretchr/testify/require"
+)
+
+func TestSavedReportAccountingAndWireBoundary(t *testing.T) {
+ _, sources, _ := setup(t)
+ report, err := Build(t.Context(), sources, []Input{{Manifest: strings.Repeat("a", 64)}}, DefaultLimits())
+ require.ErrorIs(t, err, ErrIncomplete)
+ canonical, err := Encode(*report)
+ require.NoError(t, err)
+ for _, control := range []struct {
+ name string
+ change func(*Report)
+ valid bool
+ }{
+ {"valid incomplete", func(*Report) {}, true},
+ {"omitted disguised complete", func(r *Report) { r.State = "complete" }, false},
+ {"usage execution count differs", func(r *Report) { r.Usage.Executions = 1 }, false},
+ {"conflicting priority retained", func(r *Report) {
+ r.State, r.Members[0].State = "conflicting", "conflicting"
+ r.Counts.MemberStates = map[string]int{"conflicting": 1}
+ }, true},
+ {"conflicting disguised incomplete", func(r *Report) {
+ r.Members[0].State = "conflicting"
+ r.Counts.MemberStates = map[string]int{"conflicting": 1}
+ }, false},
+ {"incomplete cannot claim complete usage", func(r *Report) {
+ r.Members[0].Execution = "exec-" + strings.Repeat("b", 64)
+ r.Counts.VerifiedExecutions, r.Usage.Executions = 1, 1
+ r.Usage.State, r.Usage.Recorded, r.Usage.Diagnostics = "complete", &models.UsageStats{}, []string{}
+ }, false},
+ } {
+ t.Run(control.name, func(t *testing.T) {
+ var changed Report
+ require.NoError(t, json.Unmarshal(canonical, &changed))
+ control.change(&changed)
+ wire, err := json.Marshal(changed)
+ require.NoError(t, err)
+ _, err = Parse(wire)
+ if control.valid {
+ require.NoError(t, err)
+ } else {
+ require.ErrorIs(t, err, ErrInvalid)
+ }
+ })
+ }
+ exact := append([]byte(strings.Repeat(" ", MaxReportBytes-len(canonical))), canonical...)
+ require.Len(t, exact, MaxReportBytes)
+ _, err = Parse(exact)
+ require.NoError(t, err)
+ _, err = Parse(append(exact, ' '))
+ require.ErrorIs(t, err, ErrInvalid)
+}
diff --git a/internal/manifestaggregate/aggregate.go b/internal/manifestaggregate/aggregate.go
new file mode 100644
index 000000000..125e80f60
--- /dev/null
+++ b/internal/manifestaggregate/aggregate.go
@@ -0,0 +1,436 @@
+// Package manifestaggregate aggregates public completed publications without
+// worker ownership, runtime construction, or execution attestation.
+package manifestaggregate
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "slices"
+ "strings"
+
+ "github.com/microsoft/waza/internal/artifactmanifest"
+ "github.com/microsoft/waza/internal/evidencestate"
+ "github.com/microsoft/waza/internal/executionattempt"
+ "github.com/microsoft/waza/internal/executionledger"
+ "github.com/microsoft/waza/internal/models"
+ "github.com/microsoft/waza/internal/runnerterminal"
+)
+
+const (
+ Version = "1.0"
+ MaxInputs = 128
+ MaxReportBytes = 32 << 20
+)
+
+var (
+ ErrInvalid = errors.New("invalid aggregate request")
+ ErrIncomplete = errors.New("aggregate admission is incomplete")
+ ErrConflicting = errors.New("aggregate contains conflicting claims")
+ ErrBudget = errors.New("aggregate read budget exhausted")
+)
+
+type Input struct {
+ Manifest string `json:"manifest_sha256"`
+ Evidence string `json:"evidence_sha256,omitempty"`
+}
+
+// Sources are explicit read-only capabilities. A missing namespace never causes
+// raw provider reads or hydration of a creator capability.
+type Sources struct {
+ Objects artifactmanifest.ReadBackend
+ Namespace *artifactmanifest.NamespaceReader
+}
+
+type Limits struct {
+ Objects int
+ Bytes int64
+}
+
+func DefaultLimits() Limits { return Limits{Objects: 4096, Bytes: 512 << 20} }
+
+type Category struct {
+ Category evidencestate.Category `json:"category"`
+ State evidencestate.State `json:"state"`
+ DeclaredState evidencestate.State `json:"declared_state,omitempty"`
+ Source evidencestate.Source `json:"source"`
+ Context evidencestate.Context `json:"context"`
+ Verification string `json:"verification"`
+ Reason string `json:"reason,omitempty"`
+ ObservedCount *int64 `json:"observed_count,omitempty"`
+}
+
+type Member struct {
+ Input
+ State string `json:"state"`
+ Binding *artifactmanifest.Binding `json:"binding,omitempty"`
+ Outcome runnerterminal.Outcome `json:"outcome,omitempty"`
+ Execution string `json:"execution_id,omitempty"`
+ Categories []Category `json:"categories"`
+ Diagnostics []string `json:"diagnostics"`
+}
+
+type Group struct {
+ Profile string `json:"profile_sha256"`
+ Members []Input `json:"members"`
+}
+
+type Counts struct {
+ Members int `json:"input_members"`
+ VerifiedManifests int `json:"unique_verified_manifests"`
+ VerifiedExecutions int `json:"unique_verified_execution_records"`
+ MemberStates map[string]int `json:"member_states"`
+ MemberOutcomes map[string]int `json:"member_outcomes"`
+}
+
+type Usage struct {
+ State string `json:"state"`
+ Population string `json:"population"`
+ Executions int `json:"executions"`
+ Recorded *models.UsageStats `json:"recorded_historical_usage"`
+ Diagnostics []string `json:"diagnostics"`
+}
+
+type Report struct {
+ Kind string `json:"kind"`
+ SchemaVersion string `json:"schema_version"`
+ State string `json:"state"`
+ Verification string `json:"verification"`
+ Members []Member `json:"members"`
+ Groups []Group `json:"groups"`
+ Counts Counts `json:"counts"`
+ Usage Usage `json:"usage"`
+}
+
+type view struct {
+ usage *models.UsageStats
+ eligible bool
+}
+
+// Normalize names the eligible requested population: unique exact reference
+// pairs, not catalog discovery, logical attempt ordinals, or successful results.
+func Normalize(inputs []Input) ([]Input, error) {
+ if len(inputs) == 0 || len(inputs) > MaxInputs {
+ return nil, fmt.Errorf("%w: require 1..%d input references", ErrInvalid, MaxInputs)
+ }
+ inputs = slices.Clone(inputs)
+ for _, input := range inputs {
+ if evidencestate.ValidateDigest(input.Manifest) != nil ||
+ (input.Evidence != "" && evidencestate.ValidateDigest(input.Evidence) != nil) {
+ return nil, fmt.Errorf("%w: expected full lowercase SHA-256 references", ErrInvalid)
+ }
+ }
+ slices.SortFunc(inputs, compareInput)
+ return slices.Compact(inputs), nil
+}
+
+func compareInput(a, b Input) int {
+ if order := strings.Compare(a.Manifest, b.Manifest); order != 0 {
+ return order
+ }
+ return strings.Compare(a.Evidence, b.Evidence)
+}
+
+// Build evaluates every independent requested member. Incomplete/conflicting
+// admissions return both the full evaluated report and a classified error.
+// Completeness is not a pass verdict: failed/canceled terminals, privacy
+// redaction and supported legacy metadata-only sources can be complete.
+func Build(ctx context.Context, sources Sources, inputs []Input, limits Limits) (*Report, error) {
+ inputs, err := Normalize(inputs)
+ if err != nil {
+ return nil, err
+ }
+ if sources.Objects == nil || limits.Objects <= 0 || limits.Bytes <= 0 {
+ return nil, fmt.Errorf("%w: readable store and positive read limits are required", ErrInvalid)
+ }
+ budget := &readBudget{remainingObjects: limits.Objects, remainingBytes: limits.Bytes}
+ backend := readOnlyBackend{source: sources.Objects, budget: budget}
+ store := artifactmanifest.NewStore(backend)
+ report := &Report{
+ Kind: "waza.manifest-aggregate", SchemaVersion: Version, State: "complete",
+ Verification: "current-content-and-original-record-context-only",
+ Members: []Member{}, Groups: []Group{},
+ Counts: Counts{Members: len(inputs), MemberStates: map[string]int{}, MemberOutcomes: map[string]int{}},
+ Usage: Usage{State: "unavailable", Population: "unique-verified-execution-records",
+ Diagnostics: []string{}, Recorded: nil},
+ }
+ manifests := map[string]bool{}
+ executions := map[string][]int{}
+ groups := map[string][]Input{}
+ views := make([]view, 0, len(inputs))
+ var failures []error
+ for _, input := range inputs {
+ member, usage, verified, memberErr := loadMember(ctx, sources, backend, store, budget, input)
+ if verified {
+ manifests[input.Manifest] = true
+ }
+ if member.Binding != nil {
+ profile, encodeErr := json.Marshal(member.Binding)
+ if encodeErr != nil {
+ return nil, encodeErr
+ }
+ key := artifactmanifest.Hash(profile)
+ groups[key] = append(groups[key], input)
+ }
+ if member.Execution != "" {
+ executions[member.Execution] = append(executions[member.Execution], len(report.Members))
+ }
+ report.Members = append(report.Members, member)
+ views = append(views, usage)
+ if memberErr != nil {
+ failures = append(failures, memberErr)
+ }
+ }
+ // Distinct executions in the same profile are valid repeats. Only actual
+ // incompatible views of one admitted record withhold that record's usage.
+ usageStats := []*models.UsageStats{}
+ usageComplete := len(executions) > 0
+ for _, id := range sortedKeys(executions) {
+ indices := executions[id]
+ var original *models.UsageStats
+ for _, index := range indices {
+ current := views[index]
+ if !current.eligible || current.usage == nil {
+ usageComplete = false
+ continue
+ }
+ if original == nil {
+ original = current.usage
+ } else if !equalUsage(original, current.usage) {
+ usageComplete = false
+ report.Usage.State = "conflicting"
+ report.Usage.Diagnostics = append(report.Usage.Diagnostics, "incompatible_recorded_usage")
+ for _, affected := range indices {
+ report.Members[affected].State = "conflicting"
+ report.Members[affected].Diagnostics = append(report.Members[affected].Diagnostics, "incompatible_recorded_usage")
+ }
+ failures = append(failures, ErrConflicting)
+ }
+ }
+ if original != nil {
+ usageStats = append(usageStats, original)
+ }
+ }
+ for _, member := range report.Members {
+ if member.Execution == "" || member.State != "admitted" {
+ usageComplete = false
+ }
+ }
+ report.Counts.VerifiedManifests, report.Counts.VerifiedExecutions = len(manifests), len(executions)
+ report.Usage.Executions = len(executions)
+ if usageComplete {
+ recorded, reason := sumRecordedUsage(usageStats)
+ if reason == "" {
+ report.Usage.State, report.Usage.Recorded = "complete", recorded
+ } else {
+ report.Usage.Diagnostics = append(report.Usage.Diagnostics, reason)
+ }
+ } else {
+ report.Usage.Diagnostics = append(report.Usage.Diagnostics, "complete_historical_usage_unavailable")
+ }
+ for _, key := range sortedKeys(groups) {
+ report.Groups = append(report.Groups, Group{Profile: key, Members: groups[key]})
+ }
+ for i := range report.Members {
+ member := &report.Members[i]
+ slices.Sort(member.Diagnostics)
+ member.Diagnostics = slices.Compact(member.Diagnostics)
+ report.Counts.MemberStates[member.State]++
+ if member.Outcome != "" {
+ report.Counts.MemberOutcomes[string(member.Outcome)]++
+ }
+ if member.State == "conflicting" {
+ report.State = "conflicting"
+ } else if member.State != "admitted" && report.State != "conflicting" {
+ report.State = "incomplete"
+ }
+ }
+ slices.Sort(report.Usage.Diagnostics)
+ report.Usage.Diagnostics = slices.Compact(report.Usage.Diagnostics)
+ for {
+ wire, err := json.Marshal(report)
+ if err != nil {
+ return report, fmt.Errorf("%w: measuring aggregate report: %w", ErrInvalid, err)
+ }
+ if len(wire) <= MaxReportBytes {
+ break
+ }
+ if report.Usage.Recorded == nil {
+ failures = append(failures, fmt.Errorf("%w: evaluated membership without recorded usage exceeds %d bytes",
+ ErrInvalid, MaxReportBytes))
+ break
+ }
+ report.Usage.State, report.Usage.Recorded = "unavailable", nil
+ report.Usage.Diagnostics = append(report.Usage.Diagnostics, "recorded_usage_report_size_unavailable")
+ slices.Sort(report.Usage.Diagnostics)
+ report.Usage.Diagnostics = slices.Compact(report.Usage.Diagnostics)
+ }
+ if report.State == "conflicting" {
+ failures = append(failures, ErrConflicting)
+ } else if report.State != "complete" {
+ failures = append(failures, ErrIncomplete)
+ }
+ return report, errors.Join(failures...)
+}
+
+func loadMember(ctx context.Context, sources Sources, backend readOnlyBackend, store *artifactmanifest.Store,
+ budget *readBudget, input Input) (Member, view, bool, error) {
+ member := Member{Input: input, State: "admitted", Categories: []Category{}, Diagnostics: []string{}}
+ fail := func(err error) (Member, view, bool, error) {
+ member.State = "omitted"
+ if errors.Is(err, artifactmanifest.ErrConflict) || errors.Is(err, executionledger.ErrContext) {
+ member.State = "conflicting"
+ }
+ member.Diagnostics = append(member.Diagnostics, failureCode(err))
+ if errors.Is(err, artifactmanifest.ErrInvalid) || errors.Is(err, evidencestate.ErrInvalid) {
+ member.Diagnostics = append(member.Diagnostics, "unsupported_or_invalid_profile")
+ }
+ return member, view{}, member.Binding != nil, err
+ }
+ publication, err := store.ReadCompleted(ctx, input.Manifest)
+ if err != nil {
+ return fail(err)
+ }
+ member.Binding = &publication.Manifest.Binding
+ terminal, err := runnerterminal.Validate(publication.ArtifactBytes[publication.Manifest.TerminalArtifact], publication.Manifest.Binding.TerminalBinding())
+ if err != nil {
+ return fail(err)
+ }
+ member.Outcome = terminal.Outcome
+ if input.Evidence == "" {
+ member.Diagnostics = append(member.Diagnostics, "execution_and_usage_unavailable_without_sidecar")
+ return member, view{}, true, nil
+ }
+ inspection, categories, inspectionErr := evidencestate.InspectCategories(ctx, backend, input.Evidence)
+ if inspection.Document == nil || inspection.Document.OriginManifest != input.Manifest {
+ if inspectionErr != nil {
+ return fail(inspectionErr)
+ }
+ return fail(artifactmanifest.ErrConflict)
+ }
+ for _, category := range categories {
+ var declared evidencestate.Record
+ for _, record := range inspection.Document.Categories {
+ if record.Category == category.Category {
+ declared = record
+ }
+ }
+ member.Categories = append(member.Categories, Category{
+ Category: category.Category, State: category.State, DeclaredState: category.DeclaredState,
+ Source: declared.Source, Context: declared.Context,
+ Verification: category.Verification, Reason: category.Reason, ObservedCount: category.ObservedCount,
+ })
+ }
+ slices.SortFunc(member.Categories, func(a, b Category) int {
+ return strings.Compare(string(a.Category), string(b.Category))
+ })
+ if inspectionErr != nil {
+ return fail(inspectionErr)
+ }
+ document := inspection.Document
+ if document.ExecutionSource == nil {
+ member.Diagnostics = append(member.Diagnostics, "execution_identity_unavailable")
+ return member, view{}, true, nil
+ }
+ if sources.Namespace == nil {
+ return fail(fmt.Errorf("%w: namespace read capability unavailable", executionledger.ErrCapability))
+ }
+ if err := verifyExecution(ctx, sources.Namespace, budget, *document.ExecutionSource,
+ publication.ManifestBytes, publication.Manifest.Artifacts, document.AttemptArtifact); err != nil {
+ return fail(err)
+ }
+ member.Execution = document.ExecutionSource.ExecutionID
+ result := view{}
+ cached := false
+ for _, link := range document.Lineage {
+ cached = cached || link.Relationship == evidencestate.Cache
+ }
+ for _, record := range document.Categories {
+ cached = cached || record.Source == evidencestate.Cached
+ }
+ for _, category := range categories {
+ if category.Category == evidencestate.Usage && category.State == evidencestate.Present &&
+ category.Usage != nil && !cached {
+ result.usage, result.eligible = category.Usage, true
+ }
+ }
+ if cached {
+ member.Diagnostics = append(member.Diagnostics, "cached_historical_usage_excluded")
+ }
+ return member, result, true, nil
+}
+
+func verifyExecution(ctx context.Context, namespace *artifactmanifest.NamespaceReader, budget *readBudget,
+ source executionattempt.Source, manifest []byte, artifacts []artifactmanifest.Artifact, selectedAttempt string) error {
+ read := func(key string) ([]byte, error) {
+ return budget.read(ctx, artifactmanifest.MaxNamespaceObjectBytes, func() ([]byte, error) {
+ return namespace.ReadUnverified(ctx, key)
+ })
+ }
+ reservation, err := read("reservation/" + source.ExecutionID)
+ if err != nil {
+ return err
+ }
+ selected := slices.IndexFunc(artifacts, func(artifact artifactmanifest.Artifact) bool {
+ return artifact.Name == selectedAttempt
+ })
+ if selected < 0 || artifacts[selected].SHA256 != artifactmanifest.Hash(reservation) ||
+ artifacts[selected].SizeBytes != int64(len(reservation)) || artifacts[selected].MediaType != "application/json" {
+ return executionledger.ErrContext
+ }
+ completionBytes, err := read("completion/" + source.ExecutionID)
+ if err != nil {
+ return err
+ }
+ completion, err := executionledger.ParseCompletion(completionBytes)
+ if err != nil {
+ return err
+ }
+ canonical, err := executionledger.EncodeCompletion(*completion)
+ if err != nil {
+ return err
+ }
+ if !bytes.Equal(canonical, completionBytes) || completion.ExecutionID != source.ExecutionID ||
+ completion.ManifestDigest != source.ManifestDigest {
+ return executionledger.ErrContext
+ }
+ return executionledger.ValidateContext(*completion, reservation, manifest)
+}
+
+func equalUsage(a, b *models.UsageStats) bool {
+ left, leftErr := json.Marshal(a)
+ right, rightErr := json.Marshal(b)
+ return leftErr == nil && rightErr == nil && bytes.Equal(left, right)
+}
+
+func sortedKeys[V any](values map[string]V) []string {
+ keys := make([]string, 0, len(values))
+ for key := range values {
+ keys = append(keys, key)
+ }
+ slices.Sort(keys)
+ return keys
+}
+
+func failureCode(err error) string {
+ switch {
+ case errors.Is(err, ErrBudget):
+ return "read_budget_exhausted"
+ case errors.Is(err, executionledger.ErrCapability):
+ return "execution_read_capability_unavailable"
+ case errors.Is(err, artifactmanifest.ErrConflict), errors.Is(err, executionledger.ErrContext):
+ return "conflicting_source"
+ case errors.Is(err, artifactmanifest.ErrCorrupt):
+ return "corrupt_source"
+ case errors.Is(err, artifactmanifest.ErrMissing):
+ return "missing_source"
+ case errors.Is(err, context.Canceled), errors.Is(err, context.DeadlineExceeded):
+ return "read_canceled"
+ case errors.Is(err, artifactmanifest.ErrInvalid), errors.Is(err, evidencestate.ErrInvalid), errors.Is(err, executionledger.ErrInvalid):
+ return "unsupported_or_invalid_profile"
+ default:
+ return "source_read_error"
+ }
+}
diff --git a/internal/manifestaggregate/aggregate_test.go b/internal/manifestaggregate/aggregate_test.go
new file mode 100644
index 000000000..a580bb80f
--- /dev/null
+++ b/internal/manifestaggregate/aggregate_test.go
@@ -0,0 +1,527 @@
+package manifestaggregate
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "errors"
+ "math"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+
+ "github.com/microsoft/waza/internal/artifactmanifest"
+ "github.com/microsoft/waza/internal/evidencestate"
+ "github.com/microsoft/waza/internal/models"
+ "github.com/microsoft/waza/internal/runnercontract"
+ "github.com/microsoft/waza/internal/runnerterminal"
+ "github.com/microsoft/waza/internal/runnerunit"
+ "github.com/microsoft/waza/internal/testutil/manifestfixture"
+ "github.com/stretchr/testify/require"
+)
+
+func setup(t *testing.T) (*artifactmanifest.Local, Sources, string) {
+ t.Helper()
+ directory := t.TempDir()
+ backend, err := artifactmanifest.OpenLocal(directory)
+ require.NoError(t, err)
+ t.Cleanup(func() { require.NoError(t, backend.Close()) })
+ namespace, err := artifactmanifest.NewNamespace(backend, artifactmanifest.ExecutionLedgerNamespaceV1)
+ require.NoError(t, err)
+ reader, err := namespace.Reader(func(read artifactmanifest.ReadBackend) artifactmanifest.ReadBackend { return read })
+ require.NoError(t, err)
+ return backend, Sources{Objects: backend, Namespace: reader}, directory
+}
+
+func publish(t *testing.T, backend *artifactmanifest.Local, change func(*runnerunit.Report)) Input {
+ t.Helper()
+ source := manifestfixture.Publish(t, backend, "../../examples/runner/unit.json", nil, change)
+ return Input{Manifest: source.Manifest, Evidence: source.Evidence}
+}
+
+func TestActualRepeatedExecutionsAndCompatibleViewsAreDeterministic(t *testing.T) {
+ backend, sources, _ := setup(t)
+ first := publish(t, backend, manifestfixture.RecordedNonzero(1))
+ second := publish(t, backend, manifestfixture.RecordedNonzero(2))
+ require.NotEqual(t, first, second, "fresh owners must actually reserve distinct executions")
+ inspection, err := evidencestate.Inspect(t.Context(), backend, first.Evidence)
+ require.NoError(t, err)
+ view := *inspection.Document
+ view.Lineage = []evidencestate.Lineage{{Manifest: second.Manifest, Relationship: evidencestate.Aggregation}}
+ otherView, err := evidencestate.Publish(t.Context(), backend, view)
+ require.NoError(t, err)
+ third := Input{Manifest: first.Manifest, Evidence: otherView.EvidenceSHA256}
+ inputs := []Input{third, second, first, first}
+ report, err := Build(t.Context(), sources, inputs, DefaultLimits())
+ require.NoError(t, err)
+ require.Equal(t, "complete", report.State)
+ require.Equal(t, Counts{Members: 3, VerifiedManifests: 2, VerifiedExecutions: 2,
+ MemberStates: map[string]int{"admitted": 3}, MemberOutcomes: map[string]int{"success": 3}}, report.Counts)
+ require.Len(t, report.Groups, 1, "same complete native binding has one display profile, not a conflict")
+ require.Len(t, report.Groups[0].Members, 3)
+ require.Equal(t, "complete", report.Usage.State)
+ require.NotNil(t, report.Usage.Recorded)
+ require.NotNil(t, report.Usage.Recorded.AICredits)
+ credits := 3.75
+ require.Equal(t, &models.UsageStats{
+ Turns: 3, InputTokens: 33, OutputTokens: 21, CacheReadTokens: 9, CacheWriteTokens: 6,
+ PremiumRequests: 1.5, AICredits: &credits,
+ ModelMetrics: map[string]models.ModelUsage{"fixture-model": {
+ InputTokens: 33, OutputTokens: 21, CacheReadTokens: 9, CacheWriteTokens: 6,
+ RequestCount: 6, RequestCost: .75, AICredits: &credits,
+ }},
+ }, report.Usage.Recorded)
+ require.Equal(t, 2, report.Usage.Executions)
+ wire, err := Encode(*report)
+ require.NoError(t, err)
+ require.NotContains(t, string(wire), `"recorded_historical_usage":null`)
+ require.NotContains(t, string(wire), "hello", "public aggregate must not copy captured content")
+ require.NotContains(t, string(wire), "execution-evidence-usage.json")
+ reordered, err := Build(t.Context(), sources, []Input{first, second, third}, DefaultLimits())
+ require.NoError(t, err)
+ again, err := Encode(*reordered)
+ require.NoError(t, err)
+ require.Equal(t, wire, again, "same admitted set and source state must produce identical canonical bytes")
+ parsed, err := Parse(wire)
+ require.NoError(t, err)
+ require.Equal(t, report, parsed)
+}
+
+func TestMissingCorruptAndUnsupportedSourcesRetainValidNeighbors(t *testing.T) {
+ for _, control := range []string{"missing", "corrupt", "mixed-version", "sidecar-missing", "sidecar-wrong-origin", "marker-conflict"} {
+ t.Run(control, func(t *testing.T) {
+ backend, sources, directory := setup(t)
+ good := publish(t, backend, nil)
+ bad := publish(t, backend, nil)
+ switch control {
+ case "missing":
+ bad.Manifest = strings.Repeat("a", 64)
+ case "corrupt":
+ target := filepath.Join(directory, "manifest-"+bad.Manifest)
+ require.NoError(t, os.Chmod(target, 0o600))
+ require.NoError(t, os.WriteFile(target, []byte("{}"), 0o600))
+ case "mixed-version":
+ wire, err := backend.Get(t.Context(), "manifest-"+bad.Manifest, artifactmanifest.MaxManifestBytes)
+ require.NoError(t, err)
+ wire = bytes.Replace(wire, []byte(`"schema_version":"1.0"`), []byte(`"schema_version":"2.0"`), 1)
+ bad.Manifest = artifactmanifest.Hash(wire)
+ created, err := backend.Create(t.Context(), "manifest-"+bad.Manifest, wire)
+ require.NoError(t, err)
+ require.True(t, created, "unsupported profile must actually be stored under its content hash")
+ case "sidecar-missing":
+ bad.Evidence = strings.Repeat("b", 64)
+ case "sidecar-wrong-origin":
+ bad.Evidence = good.Evidence
+ case "marker-conflict":
+ target := filepath.Join(directory, "final-"+bad.Manifest)
+ require.NoError(t, os.Chmod(target, 0o600))
+ require.NoError(t, os.WriteFile(target, []byte(strings.Repeat("f", 64)), 0o600))
+ }
+ report, err := Build(t.Context(), sources, []Input{bad, good}, DefaultLimits())
+ require.Error(t, err)
+ require.NotNil(t, report)
+ require.Len(t, report.Members, 2)
+ var valid, invalid *Member
+ for index := range report.Members {
+ member := &report.Members[index]
+ if member.Input == good {
+ valid = member
+ } else {
+ invalid = member
+ }
+ }
+ require.NotNil(t, valid)
+ require.Equal(t, "admitted", valid.State)
+ require.NotNil(t, invalid)
+ require.NotEqual(t, "admitted", invalid.State)
+ require.NotEmpty(t, invalid.Diagnostics)
+ if control == "mixed-version" {
+ require.Contains(t, invalid.Diagnostics, "unsupported_or_invalid_profile")
+ }
+ require.Nil(t, report.Usage.Recorded, "a partial historical subtotal is never authoritative")
+ _, encodeErr := Encode(*report)
+ require.NoError(t, encodeErr, "incomplete reports must remain serializable")
+ })
+ }
+}
+
+func TestLegitimateOmissionsLegacyMetadataAndFailedOutcomesAreComplete(t *testing.T) {
+ backend, sources, _ := setup(t)
+ success := publish(t, backend, nil)
+ failed := publish(t, backend, func(report *runnerunit.Report) {
+ report.Status = "execution_error"
+ report.Observation.Status = models.StatusError
+ report.Observation.ErrorMsg = "synthetic restricted diagnostic"
+ })
+ legacy := Input{Manifest: success.Manifest}
+ report, err := Build(t.Context(), sources, []Input{success, failed, legacy}, DefaultLimits())
+ require.NoError(t, err)
+ require.Equal(t, "complete", report.State, "admission completeness is not terminal success or optional sidecar presence")
+ require.Equal(t, 1, report.Counts.MemberOutcomes["infrastructure_failure"])
+ require.Equal(t, 2, report.Counts.MemberOutcomes["success"])
+ require.Equal(t, 3, report.Counts.Members)
+ require.Equal(t, 2, report.Counts.VerifiedExecutions)
+ require.Equal(t, "unavailable", report.Usage.State)
+ require.Nil(t, report.Usage.Recorded)
+ var redacted bool
+ for _, member := range report.Members {
+ for _, category := range member.Categories {
+ if category.Category == evidencestate.Conversation {
+ require.Equal(t, evidencestate.Redacted, category.State)
+ require.Nil(t, category.ObservedCount)
+ redacted = true
+ }
+ }
+ }
+ require.True(t, redacted, "the actual default privacy control must be exercised")
+ wire, err := Encode(*report)
+ require.NoError(t, err)
+ require.NotContains(t, string(wire), "synthetic restricted diagnostic")
+}
+
+func TestCachedViewsWithholdUsageWithoutInventingConflicts(t *testing.T) {
+ backend, sources, _ := setup(t)
+ first, second := publish(t, backend, manifestfixture.RecordedZero), publish(t, backend, nil)
+ inspection, err := evidencestate.Inspect(t.Context(), backend, first.Evidence)
+ require.NoError(t, err)
+ document := *inspection.Document
+ document.Lineage = []evidencestate.Lineage{{Manifest: second.Manifest, Relationship: evidencestate.Cache}}
+ cached, err := evidencestate.Publish(t.Context(), backend, document)
+ require.NoError(t, err)
+ report, err := Build(t.Context(), sources, []Input{first, {Manifest: first.Manifest, Evidence: cached.EvidenceSHA256}}, DefaultLimits())
+ require.NoError(t, err)
+ require.Equal(t, "complete", report.State)
+ require.Equal(t, 1, report.Counts.VerifiedExecutions)
+ require.Nil(t, report.Usage.Recorded)
+ require.Equal(t, "unavailable", report.Usage.State)
+ var found bool
+ for _, member := range report.Members {
+ if member.Evidence == cached.EvidenceSHA256 {
+ require.Contains(t, member.Diagnostics, "cached_historical_usage_excluded")
+ found = true
+ }
+ }
+ require.True(t, found, "the actual cached view must be retained")
+}
+
+func TestCapabilityAndCumulativeBudgetFailuresStayStructured(t *testing.T) {
+ backend, sources, _ := setup(t)
+ input := publish(t, backend, nil)
+ for _, control := range []struct {
+ name string
+ sources Sources
+ limits Limits
+ code string
+ }{
+ {"missing namespace", Sources{Objects: backend}, DefaultLimits(), "execution_read_capability_unavailable"},
+ {"object budget", sources, Limits{Objects: 1, Bytes: 512 << 20}, "read_budget_exhausted"},
+ {"byte budget", sources, Limits{Objects: 4096, Bytes: artifactmanifest.MaxManifestBytes}, "read_budget_exhausted"},
+ } {
+ t.Run(control.name, func(t *testing.T) {
+ report, err := Build(t.Context(), control.sources, []Input{input}, control.limits)
+ require.Error(t, err)
+ require.ErrorIs(t, err, ErrIncomplete)
+ require.NotNil(t, report)
+ require.Len(t, report.Members, 1)
+ require.Contains(t, report.Members[0].Diagnostics, control.code)
+ require.Empty(t, report.Members[0].Execution)
+ _, encodeErr := Encode(*report)
+ require.NoError(t, encodeErr)
+ })
+ }
+}
+
+func TestDifferentWorkProfilesAreNotPooledOrConflicted(t *testing.T) {
+ backend, sources, _ := setup(t)
+ first := publish(t, backend, nil)
+ second := manifestfixture.Publish(t, backend, "../../examples/runner/unit.json", func(request *runnercontract.Request) {
+ request.RunKey = "independent-eval-run"
+ }, nil)
+ report, err := Build(t.Context(), sources, []Input{first, {Manifest: second.Manifest, Evidence: second.Evidence}}, DefaultLimits())
+ require.NoError(t, err)
+ require.Equal(t, "complete", report.State)
+ require.Len(t, report.Groups, 2)
+ require.Equal(t, 2, report.Counts.VerifiedExecutions)
+}
+
+func TestIncompatibleExecutionClaimDoesNotSelectAWinningView(t *testing.T) {
+ backend, sources, _ := setup(t)
+ first, second := publish(t, backend, manifestfixture.RecordedZero), publish(t, backend, nil)
+ one, err := evidencestate.Inspect(t.Context(), backend, first.Evidence)
+ require.NoError(t, err)
+ two, err := evidencestate.Inspect(t.Context(), backend, second.Evidence)
+ require.NoError(t, err)
+ document := *one.Document
+ document.ExecutionSource.ExecutionID = two.Document.ExecutionSource.ExecutionID
+ wire, err := evidencestate.Encode(document)
+ require.NoError(t, err, "negative control must have a valid declaration profile")
+ digest := artifactmanifest.Hash(wire)
+ require.NoError(t, artifactmanifest.NewStore(backend).Put(t.Context(), artifactmanifest.Artifact{
+ Name: "incompatible-view", SHA256: digest, SizeBytes: int64(len(wire)), MediaType: "application/json",
+ }, wire), "model an actually stored incompatible public view, not a published verified source")
+ report, err := Build(t.Context(), sources, []Input{first, {Manifest: first.Manifest, Evidence: digest}}, DefaultLimits())
+ require.ErrorIs(t, err, ErrConflicting)
+ require.Equal(t, "conflicting", report.State)
+ require.Len(t, report.Members, 2)
+ require.Equal(t, 1, report.Counts.MemberStates["admitted"])
+ require.Equal(t, 1, report.Counts.MemberStates["conflicting"])
+ require.Equal(t, 1, report.Counts.VerifiedExecutions)
+ require.Nil(t, report.Usage.Recorded, "a good view must not hide an incompatible supplied view")
+ _, encodeErr := Encode(*report)
+ require.NoError(t, encodeErr)
+}
+
+func TestOriginalCancellationRemainsVisibleWithoutInvalidatingAdmission(t *testing.T) {
+ backend, sources, _ := setup(t)
+ input := publish(t, backend, nil)
+ store := artifactmanifest.NewStore(backend)
+ source, err := store.ReadCompleted(t.Context(), input.Manifest)
+ require.NoError(t, err)
+ manifest := source.Manifest
+ terminal, err := runnerterminal.Validate(source.ArtifactBytes[manifest.TerminalArtifact], manifest.Binding.TerminalBinding())
+ require.NoError(t, err)
+ terminal.Outcome = runnerterminal.Canceled
+ terminal.ExitCode, _ = terminal.Outcome.ExitCode()
+ terminal.FailureBoundary = "shutdown"
+ terminal.Diagnostics = []runnerterminal.Diagnostic{{Code: "cancellation", Message: "Synthetic original cancellation"}}
+ terminalBytes, err := runnerterminal.Encode(*terminal, manifest.Binding.TerminalBinding())
+ require.NoError(t, err)
+ // This authored terminal-only source deliberately has no execution sidecar;
+ // it must remain metadata-only rather than borrowing the original owner.
+ manifest.Artifacts = []artifactmanifest.Artifact{{
+ Name: manifest.TerminalArtifact, SHA256: artifactmanifest.Hash(terminalBytes),
+ SizeBytes: int64(len(terminalBytes)), MediaType: "application/json",
+ }}
+ require.NoError(t, store.Put(t.Context(), manifest.Artifacts[0], terminalBytes))
+ wire, err := artifactmanifest.Encode(manifest)
+ require.NoError(t, err)
+ digest, err := store.Create(t.Context(), wire)
+ require.NoError(t, err)
+ _, err = store.Finalize(t.Context(), digest)
+ require.NoError(t, err)
+ report, err := Build(t.Context(), sources, []Input{{Manifest: digest}, input}, DefaultLimits())
+ require.NoError(t, err)
+ require.Equal(t, "complete", report.State)
+ require.Equal(t, 1, report.Counts.MemberOutcomes[string(runnerterminal.Canceled)])
+ require.Equal(t, 1, report.Counts.MemberOutcomes["success"])
+ require.Equal(t, 1, report.Counts.VerifiedExecutions)
+ require.Nil(t, report.Usage.Recorded)
+ encoded, err := Encode(*report)
+ require.NoError(t, err)
+ require.NotContains(t, string(encoded), "Synthetic original cancellation")
+}
+
+func TestNamespaceDenialNeverFallsBackToProviderOrOrdinal(t *testing.T) {
+ backend, sources, _ := setup(t)
+ input := publish(t, backend, nil)
+ namespace, err := artifactmanifest.NewNamespace(backend, artifactmanifest.ExecutionLedgerNamespaceV1)
+ require.NoError(t, err)
+ sources.Namespace, err = namespace.Reader(func(artifactmanifest.ReadBackend) artifactmanifest.ReadBackend { return readFailure{} })
+ require.NoError(t, err)
+ report, err := Build(t.Context(), sources, []Input{input}, DefaultLimits())
+ require.ErrorIs(t, err, ErrIncomplete)
+ require.Empty(t, report.Members[0].Execution)
+ require.Zero(t, report.Counts.VerifiedExecutions)
+ require.Contains(t, report.Members[0].Diagnostics, "source_read_error")
+ require.Nil(t, report.Usage.Recorded)
+}
+
+func TestRecordedUsageOverflowWithholdsTotalThroughPublicSources(t *testing.T) {
+ backend, sources, _ := setup(t)
+ makeUsage := func(report *runnerunit.Report) {
+ manifestfixture.RecordedZero(report)
+ report.Observation.Usage.InputTokens = math.MaxInt
+ }
+ first, second := publish(t, backend, makeUsage), publish(t, backend, makeUsage)
+ report, err := Build(t.Context(), sources, []Input{first, second}, DefaultLimits())
+ require.NoError(t, err, "source admission remains complete independently of unavailable arithmetic")
+ require.Equal(t, "complete", report.State)
+ require.Equal(t, 2, report.Counts.VerifiedExecutions)
+ require.Equal(t, "unavailable", report.Usage.State)
+ require.Nil(t, report.Usage.Recorded)
+ require.Contains(t, report.Usage.Diagnostics, "recorded_usage_numeric_precision_unavailable")
+ _, err = Encode(*report)
+ require.NoError(t, err)
+}
+
+func TestNonemptyRecordedZeroIsNotAnAbsentPopulation(t *testing.T) {
+ usage, reason := sumRecordedUsage([]*models.UsageStats{{}})
+ require.Empty(t, reason)
+ require.NotNil(t, usage)
+ require.Nil(t, usage.AICredits, "missing credits are not invented as zero")
+ require.Zero(t, usage.InputTokens)
+ absent, reason := sumRecordedUsage(nil)
+ require.Equal(t, "complete_historical_usage_unavailable", reason)
+ require.Nil(t, absent)
+}
+
+func TestIndividuallyValidLargeUsageCannotBecomeRoundedUITotal(t *testing.T) {
+ largeCount := int64(1) << 52
+ if int64(int(largeCount)) != largeCount {
+ t.Skip("large native counter control requires a 64-bit int platform")
+ }
+ for _, control := range []struct {
+ name string
+ change func(*models.UsageStats, bool)
+ available bool
+ }{
+ {"safe input counters exceed safe total", func(usage *models.UsageStats, _ bool) {
+ usage.InputTokens = int(largeCount)
+ }, false},
+ {"model counters exceed safe total", func(usage *models.UsageStats, _ bool) {
+ model := usage.ModelMetrics["fixture-model"]
+ model.InputTokens = int(largeCount)
+ usage.ModelMetrics["fixture-model"] = model
+ }, false},
+ {"safe credit values exceed safe total", func(usage *models.UsageStats, _ bool) {
+ value := float64(1 << 52)
+ usage.AICredits = &value
+ }, false},
+ {"positive fractional usage disappears at large magnitude", func(usage *models.UsageStats, second bool) {
+ value := float64(1 << 52)
+ if second {
+ value = 0.25
+ }
+ usage.PremiumRequests = value
+ }, false},
+ {"model request values exceed safe total", func(usage *models.UsageStats, _ bool) {
+ model := usage.ModelMetrics["fixture-model"]
+ model.RequestCount = 1 << 52
+ usage.ModelMetrics["fixture-model"] = model
+ }, false},
+ {"small exact fractions remain supported", func(usage *models.UsageStats, _ bool) {
+ usage.PremiumRequests = 0.125
+ }, true},
+ } {
+ t.Run(control.name, func(t *testing.T) {
+ backend, sources, _ := setup(t)
+ inputs := []Input{}
+ for _, second := range []bool{false, true} {
+ input := publish(t, backend, func(report *runnerunit.Report) {
+ manifestfixture.RecordedZero(report)
+ control.change(report.Observation.Usage, second)
+ })
+ _, categories, err := evidencestate.InspectCategories(t.Context(), backend, input.Evidence)
+ require.NoError(t, err)
+ var checked bool
+ for _, category := range categories {
+ if category.Category == evidencestate.Usage {
+ require.Equal(t, evidencestate.Present, category.State)
+ require.NotNil(t, category.Usage)
+ require.LessOrEqual(t, int64(category.Usage.InputTokens), maxJSONInteger)
+ require.LessOrEqual(t, category.Usage.PremiumRequests, float64(maxJSONInteger))
+ require.NotNil(t, category.Usage.AICredits)
+ require.LessOrEqual(t, *category.Usage.AICredits, float64(maxJSONInteger))
+ checked = true
+ }
+ }
+ require.True(t, checked, "every original public record must actually admit present usage")
+ inputs = append(inputs, input)
+ }
+ report, err := Build(t.Context(), sources, inputs, DefaultLimits())
+ require.NoError(t, err, "usage unavailability must not erase otherwise valid input admissions")
+ require.Equal(t, "complete", report.State)
+ require.Equal(t, 2, report.Counts.VerifiedExecutions)
+ if control.available {
+ require.Equal(t, "complete", report.Usage.State)
+ require.NotNil(t, report.Usage.Recorded)
+ require.Equal(t, 0.25, report.Usage.Recorded.PremiumRequests)
+ } else {
+ require.Equal(t, "unavailable", report.Usage.State)
+ require.Nil(t, report.Usage.Recorded)
+ require.Contains(t, report.Usage.Diagnostics, "recorded_usage_numeric_precision_unavailable")
+ }
+ wire, err := Encode(*report)
+ require.NoError(t, err)
+ _, err = Parse(wire)
+ require.NoError(t, err, "the actual emitted report must honor the UI-safe numeric wire schema")
+ })
+ }
+}
+
+func TestZeroEligibleRecordsAndUnknownUsageDoNotCreateZeroTotal(t *testing.T) {
+ backend, sources, _ := setup(t)
+ zero := publish(t, backend, manifestfixture.RecordedZero)
+ absent := publish(t, backend, nil)
+ for _, control := range []struct {
+ name string
+ inputs []Input
+ records int
+ }{
+ {"metadata has no eligible record", []Input{{Manifest: zero.Manifest}}, 0},
+ {"known zero cannot exclude unknown usage", []Input{zero, absent}, 2},
+ {"known zero cannot exclude unknown identity view", []Input{zero, {Manifest: absent.Manifest}}, 1},
+ } {
+ t.Run(control.name, func(t *testing.T) {
+ report, err := Build(t.Context(), sources, control.inputs, DefaultLimits())
+ require.NoError(t, err)
+ require.Equal(t, len(control.inputs), report.Counts.Members)
+ require.Equal(t, control.records, report.Counts.VerifiedExecutions)
+ require.Equal(t, "unavailable", report.Usage.State)
+ require.Nil(t, report.Usage.Recorded)
+ require.Contains(t, report.Usage.Diagnostics, "complete_historical_usage_unavailable")
+ })
+ }
+}
+
+func TestStrictInputAndReportWirePresence(t *testing.T) {
+ digest := strings.Repeat("a", 64)
+ for _, input := range []string{
+ `null`, `[]`, `[null]`, `[{}]`,
+ `[{"manifest_sha256":"` + digest + `","evidence_sha256":null}]`,
+ `[{"manifest_sha256":"` + digest + `","evidence_sha256":""}]`,
+ `[{"manifest_sha256":"` + digest + `","extra":false}]`,
+ `[{"manifest_sha256":"` + digest + `","manifest_sha256":"` + digest + `"}]`,
+ } {
+ _, err := ParseInputs([]byte(input))
+ require.ErrorIs(t, err, ErrInvalid, input)
+ }
+ backend, sources, _ := setup(t)
+ good := publish(t, backend, nil)
+ report, err := Build(t.Context(), sources, []Input{good}, DefaultLimits())
+ require.NoError(t, err)
+ wire, err := Encode(*report)
+ require.NoError(t, err)
+ object := func(value map[string]any, key string) map[string]any {
+ result, ok := value[key].(map[string]any)
+ require.True(t, ok)
+ return result
+ }
+ for _, mutation := range []func(map[string]any){
+ func(value map[string]any) { value["schema_version"] = "2.0" },
+ func(value map[string]any) { object(value, "counts")["input_members"] = 0 },
+ func(value map[string]any) {
+ object(value, "usage")["recorded_historical_usage"] = map[string]any{}
+ },
+ func(value map[string]any) { object(value, "usage")["recorded_historical_usage"] = false },
+ func(value map[string]any) { delete(object(value, "usage"), "recorded_historical_usage") },
+ } {
+ var value map[string]any
+ require.NoError(t, json.Unmarshal(wire, &value))
+ mutation(value)
+ changed, err := json.Marshal(value)
+ require.NoError(t, err)
+ require.NotEqual(t, wire, changed)
+ _, err = Parse(changed)
+ require.Error(t, err)
+ }
+}
+
+type readFailure struct{}
+
+func (readFailure) Get(context.Context, string, int64) ([]byte, error) {
+ return nil, errors.New("/private/operator/path token=synthetic-secret")
+}
+
+func TestPrivateReadErrorsAreNotPublicDiagnostics(t *testing.T) {
+ report, err := Build(t.Context(), Sources{Objects: readFailure{}},
+ []Input{{Manifest: strings.Repeat("a", 64)}}, DefaultLimits())
+ require.Error(t, err)
+ require.ErrorContains(t, err, "/private/operator/path")
+ wire, encodeErr := Encode(*report)
+ require.NoError(t, encodeErr)
+ require.NotContains(t, string(wire), "/private")
+ require.NotContains(t, string(wire), "synthetic-secret")
+ require.Contains(t, report.Members[0].Diagnostics, "source_read_error")
+}
diff --git a/internal/manifestaggregate/alias_test.go b/internal/manifestaggregate/alias_test.go
new file mode 100644
index 000000000..1ced387b8
--- /dev/null
+++ b/internal/manifestaggregate/alias_test.go
@@ -0,0 +1,89 @@
+package manifestaggregate
+
+import (
+ "strings"
+ "testing"
+
+ "github.com/microsoft/waza/internal/artifactmanifest"
+ "github.com/microsoft/waza/internal/evidencestate"
+ "github.com/microsoft/waza/internal/executionattempt"
+ "github.com/microsoft/waza/internal/executionledger"
+ "github.com/stretchr/testify/require"
+)
+
+func TestSelectedAliasedAttemptMustMatchOriginalReservation(t *testing.T) {
+ for _, mismatch := range []bool{false, true} {
+ t.Run(map[bool]string{false: "alternative-valid alias", true: "different original reservation bytes"}[mismatch], func(t *testing.T) {
+ backend, sources, _ := setup(t)
+ original := publish(t, backend, nil)
+ store := artifactmanifest.NewStore(backend)
+ inspected, err := evidencestate.Inspect(t.Context(), backend, original.Evidence)
+ require.NoError(t, err)
+ declaration, err := store.ReadCompleted(t.Context(), original.Manifest)
+ require.NoError(t, err)
+ names := []string{}
+ for _, artifact := range declaration.Manifest.Artifacts {
+ names = append(names, artifact.Name)
+ }
+ publication, err := store.ReadCompleted(t.Context(), original.Manifest, names...)
+ require.NoError(t, err)
+ ledger, err := executionledger.New(backend)
+ require.NoError(t, err)
+ creator, err := ledger.Reserve(t.Context(), publication.Manifest.Binding, nil)
+ require.NoError(t, err)
+ reserved, err := ledger.Reservation(creator)
+ require.NoError(t, err)
+ attempt := reserved.Attempt
+ if mismatch {
+ attempt.Parent = &executionattempt.Parent{ExecutionID: "exec-" + strings.Repeat("f", 64), Relationship: "retry"}
+ }
+ aliasBytes, err := executionattempt.EncodeAttempt(attempt)
+ require.NoError(t, err)
+ require.Equal(t, reserved.Attempt.ExecutionID, attempt.ExecutionID)
+ require.Equal(t, reserved.Attempt.Binding, attempt.Binding)
+ require.Equal(t, !mismatch, string(reserved.Bytes) == string(aliasBytes),
+ "control must change canonical bytes without changing execution ID or binding")
+ const alias = "custom-attempt.json"
+ manifest := publication.Manifest
+ manifest.Artifacts = []artifactmanifest.Artifact{}
+ for _, artifact := range publication.Manifest.Artifacts {
+ data := publication.ArtifactBytes[artifact.Name]
+ if artifact.Name == inspected.Document.AttemptArtifact {
+ artifact.Name, data = alias, aliasBytes
+ artifact.SHA256, artifact.SizeBytes = artifactmanifest.Hash(data), int64(len(data))
+ }
+ require.NotEqual(t, "execution-attempt.json", artifact.Name, "literal dependency check must not handle this control")
+ require.NoError(t, store.Put(t.Context(), artifact, data))
+ manifest.Artifacts = append(manifest.Artifacts, artifact)
+ }
+ wire, err := artifactmanifest.Encode(manifest)
+ require.NoError(t, err)
+ digest, err := store.Create(t.Context(), wire)
+ require.NoError(t, err)
+ _, err = store.Finalize(t.Context(), digest)
+ require.NoError(t, err)
+ _, err = ledger.Complete(t.Context(), creator, digest)
+ require.NoError(t, err, "original dependency currently admits completion without checking a custom descriptor")
+ document := *inspected.Document
+ document.OriginManifest, document.AttemptArtifact = digest, alias
+ document.ExecutionSource = &executionattempt.Source{ExecutionID: attempt.ExecutionID, ManifestDigest: digest}
+ evidence, err := evidencestate.Publish(t.Context(), backend, document)
+ require.NoError(t, err, "sidecar source admission legitimately supports canonical custom artifact names")
+ report, err := Build(t.Context(), sources, []Input{{Manifest: digest, Evidence: evidence.EvidenceSHA256}}, DefaultLimits())
+ require.NotNil(t, report)
+ if mismatch {
+ require.ErrorIs(t, err, ErrConflicting, "selected descriptor must bind to actual reservation bytes, not just ID/binding")
+ require.Empty(t, report.Members[0].Execution)
+ require.Equal(t, "conflicting", report.Members[0].State)
+ } else {
+ require.NoError(t, err)
+ require.Equal(t, attempt.ExecutionID, report.Members[0].Execution)
+ require.Equal(t, "complete", report.State)
+ }
+ canonical, err := Encode(*report)
+ require.NoError(t, err)
+ _, err = Parse(canonical)
+ require.NoError(t, err)
+ })
+ }
+}
diff --git a/internal/manifestaggregate/browser_fixture_test.go b/internal/manifestaggregate/browser_fixture_test.go
new file mode 100644
index 000000000..4ba72adbe
--- /dev/null
+++ b/internal/manifestaggregate/browser_fixture_test.go
@@ -0,0 +1,89 @@
+package manifestaggregate
+
+import (
+ "flag"
+ "fmt"
+ "os"
+ "slices"
+ "strconv"
+ "strings"
+ "testing"
+
+ "github.com/microsoft/waza/internal/artifactmanifest"
+ "github.com/microsoft/waza/internal/runnerterminal"
+ "github.com/stretchr/testify/require"
+)
+
+var browserFixtureOutput = flag.String("aggregate-browser-fixture-output", "", "Exclusively save the actual large canonical report for the browser caller controls")
+
+func TestLargeCanonicalPublicReportFitsAggregateProfile(t *testing.T) {
+ backend, sources, _ := setup(t)
+ wire, err := os.ReadFile("../artifactmanifest/testdata/publication-manifest.json")
+ require.NoError(t, err)
+ manifest, err := artifactmanifest.Parse(wire)
+ require.NoError(t, err)
+ original, err := os.ReadFile("../artifactmanifest/testdata/publication-terminal.json")
+ require.NoError(t, err)
+ terminal, err := runnerterminal.Validate(original, manifest.Binding.TerminalBinding())
+ require.NoError(t, err)
+ manifest.Binding.GraderIDs = []string{}
+ terminal.Observation.Validations = []runnerterminal.Verdict{}
+ for index := range 128 {
+ prefix := fmt.Sprintf("grader-%03d-", index)
+ switch index {
+ case 0:
+ prefix = "grader-\U00010000<>&\u2028\u2029"
+ case 1:
+ prefix = "grader-\ue000"
+ }
+ name := prefix + strings.Repeat("\x01", 256-len(prefix))
+ require.Len(t, name, 256)
+ manifest.Binding.GraderIDs = append(manifest.Binding.GraderIDs, name)
+ terminal.Observation.Validations = append(terminal.Observation.Validations, runnerterminal.Verdict{Identifier: name, Passed: true})
+ }
+ slices.Sort(manifest.Binding.GraderIDs)
+ terminalBytes, err := runnerterminal.Encode(*terminal, manifest.Binding.TerminalBinding())
+ require.NoError(t, err)
+ store := artifactmanifest.NewStore(backend)
+ terminalArtifact := artifactmanifest.Artifact{
+ Name: manifest.TerminalArtifact, SHA256: artifactmanifest.Hash(terminalBytes),
+ SizeBytes: int64(len(terminalBytes)), MediaType: "application/json",
+ }
+ require.NoError(t, store.Put(t.Context(), terminalArtifact, terminalBytes))
+ inputs := []Input{}
+ for index := range MaxInputs {
+ content := []byte(strconv.Itoa(index))
+ nonce := artifactmanifest.Artifact{
+ Name: "population-member", SHA256: artifactmanifest.Hash(content),
+ SizeBytes: int64(len(content)), MediaType: "application/octet-stream",
+ }
+ require.NoError(t, store.Put(t.Context(), nonce, content))
+ manifest.Artifacts = []artifactmanifest.Artifact{terminalArtifact, nonce}
+ declaration, err := artifactmanifest.Encode(*manifest)
+ require.NoError(t, err)
+ digest, err := store.Create(t.Context(), declaration)
+ require.NoError(t, err)
+ _, err = store.Finalize(t.Context(), digest)
+ require.NoError(t, err)
+ inputs = append(inputs, Input{Manifest: digest})
+ }
+ report, err := Build(t.Context(), sources, inputs, DefaultLimits())
+ require.NoError(t, err)
+ require.Equal(t, MaxInputs, report.Counts.VerifiedManifests)
+ canonical, err := Encode(*report)
+ require.NoError(t, err)
+ require.Greater(t, len(canonical), 16<<20)
+ require.LessOrEqual(t, len(canonical), MaxReportBytes)
+ t.Logf("actual canonical public report: %d bytes", len(canonical))
+ _, err = Parse(canonical)
+ require.NoError(t, err)
+ if *browserFixtureOutput != "" {
+ file, err := os.OpenFile(*browserFixtureOutput, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o600)
+ require.NoError(t, err)
+ written, err := file.Write(canonical)
+ closeErr := file.Close()
+ require.NoError(t, err)
+ require.NoError(t, closeErr)
+ require.Equal(t, len(canonical), written)
+ }
+}
diff --git a/internal/manifestaggregate/budget.go b/internal/manifestaggregate/budget.go
new file mode 100644
index 000000000..1a3e1863c
--- /dev/null
+++ b/internal/manifestaggregate/budget.go
@@ -0,0 +1,49 @@
+package manifestaggregate
+
+import (
+ "context"
+ "fmt"
+
+ "github.com/microsoft/waza/internal/artifactmanifest"
+)
+
+type readBudget struct {
+ remainingObjects int
+ remainingBytes int64
+}
+
+func (b *readBudget) read(ctx context.Context, limit int64, read func() ([]byte, error)) ([]byte, error) {
+ if err := ctx.Err(); err != nil {
+ return nil, err
+ }
+ // Reserve the full bounded read, including excess detection, before I/O.
+ if limit < 0 || b.remainingObjects <= 0 || limit >= b.remainingBytes {
+ return nil, ErrBudget
+ }
+ b.remainingObjects--
+ b.remainingBytes -= limit + 1
+ data, err := read()
+ if int64(len(data)) > limit {
+ return nil, fmt.Errorf("%w: provider exceeded read bound", artifactmanifest.ErrCorrupt)
+ }
+ if err != nil {
+ // An error need not return the bytes already consumed by a provider.
+ // Keep its complete reservation instead of inventing a zero-byte read.
+ return nil, err
+ }
+ b.remainingBytes += limit + 1 - int64(len(data))
+ return data, nil
+}
+
+type readOnlyBackend struct {
+ source artifactmanifest.ReadBackend
+ budget *readBudget
+}
+
+func (b readOnlyBackend) Get(ctx context.Context, key string, limit int64) ([]byte, error) {
+ return b.budget.read(ctx, limit, func() ([]byte, error) { return b.source.Get(ctx, key, limit) })
+}
+
+func (readOnlyBackend) Create(context.Context, string, []byte) (bool, error) {
+ return false, fmt.Errorf("%w: aggregation has no object write capability", ErrInvalid)
+}
diff --git a/internal/manifestaggregate/codec.go b/internal/manifestaggregate/codec.go
new file mode 100644
index 000000000..8de219178
--- /dev/null
+++ b/internal/manifestaggregate/codec.go
@@ -0,0 +1,210 @@
+package manifestaggregate
+
+import (
+ "encoding/json"
+ "fmt"
+ "reflect"
+ "slices"
+ "strings"
+ "sync"
+ "unicode/utf8"
+
+ "github.com/microsoft/waza/internal/artifactmanifest"
+ "github.com/microsoft/waza/internal/evidencestate"
+ "github.com/microsoft/waza/internal/jsonutil"
+ "github.com/microsoft/waza/internal/schemaloader"
+ "github.com/microsoft/waza/schemas"
+ "github.com/santhosh-tekuri/jsonschema/v6"
+)
+
+var reportSchema = sync.OnceValues(func() (*jsonschema.Schema, error) {
+ compiler := jsonschema.NewCompiler()
+ compiler.UseLoader(schemaloader.Offline{})
+ for name, text := range map[string]string{
+ "manifest-aggregate-1.0.schema.json": schemas.ManifestAggregateSchemaJSON,
+ "artifact-manifest-1.0.schema.json": schemas.ArtifactManifestSchemaJSON,
+ "runner-terminal-1.0.schema.json": schemas.RunnerTerminalSchemaJSON,
+ } {
+ document, err := jsonschema.UnmarshalJSON(strings.NewReader(text))
+ if err != nil {
+ return nil, err
+ }
+ if err := compiler.AddResource("https://microsoft.github.io/waza/schemas/"+name, document); err != nil {
+ return nil, err
+ }
+ }
+ return compiler.Compile("https://microsoft.github.io/waza/schemas/manifest-aggregate-1.0.schema.json")
+})
+
+// Parse admits the report's wire profile and internal population accounting.
+// It does not reverify sources or turn a saved report into current evidence.
+func Parse(data []byte) (*Report, error) {
+ if len(data) > MaxReportBytes {
+ return nil, fmt.Errorf("%w: report exceeds byte limit", ErrInvalid)
+ }
+ value, err := jsonutil.Parse(data)
+ if err != nil {
+ return nil, fmt.Errorf("%w: report JSON: %w", ErrInvalid, err)
+ }
+ schema, err := reportSchema()
+ if err != nil {
+ return nil, err
+ }
+ if err := schema.Validate(value); err != nil {
+ return nil, fmt.Errorf("%w: report schema: %w", ErrInvalid, err)
+ }
+ var report Report
+ if err := json.Unmarshal(data, &report); err != nil {
+ return nil, err
+ }
+ if err := accounting(report); err != nil {
+ return nil, err
+ }
+ return &report, nil
+}
+
+func accounting(report Report) error {
+ fail := func() error { return fmt.Errorf("%w: inconsistent population accounting or ordering", ErrInvalid) }
+ counts := Counts{Members: len(report.Members), MemberStates: map[string]int{}, MemberOutcomes: map[string]int{}}
+ manifests, executions := map[string]bool{}, map[string]bool{}
+ profiles := map[string][]Input{}
+ state := "complete"
+ allUsageEligible := true
+ for index, member := range report.Members {
+ if index > 0 && compareInput(report.Members[index-1].Input, member.Input) >= 0 {
+ return fail()
+ }
+ if !slices.IsSorted(member.Diagnostics) {
+ return fail()
+ }
+ for i, category := range member.Categories {
+ if i > 0 && member.Categories[i-1].Category >= category.Category {
+ return fail()
+ }
+ }
+ counts.MemberStates[member.State]++
+ if member.Outcome != "" {
+ counts.MemberOutcomes[string(member.Outcome)]++
+ }
+ if member.Binding != nil {
+ if !slices.IsSorted(member.Binding.GraderIDs) {
+ return fail()
+ }
+ manifests[member.Manifest] = true
+ wire, err := json.Marshal(member.Binding)
+ if err != nil {
+ return err
+ }
+ key := artifactmanifest.Hash(wire)
+ profiles[key] = append(profiles[key], member.Input)
+ }
+ if member.Execution != "" {
+ executions[member.Execution] = true
+ }
+ allUsageEligible = allUsageEligible && memberUsageEligible(member)
+ if member.State == "conflicting" {
+ state = "conflicting"
+ } else if member.State != "admitted" && state != "conflicting" {
+ state = "incomplete"
+ }
+ }
+ counts.VerifiedManifests, counts.VerifiedExecutions = len(manifests), len(executions)
+ expectedGroups := []Group{}
+ for _, key := range sortedKeys(profiles) {
+ expectedGroups = append(expectedGroups, Group{Profile: key, Members: profiles[key]})
+ }
+ if !reflect.DeepEqual(counts, report.Counts) || !reflect.DeepEqual(expectedGroups, report.Groups) ||
+ state != report.State || report.Usage.Executions != len(executions) || !slices.IsSorted(report.Usage.Diagnostics) {
+ return fail()
+ }
+ if report.Usage.State == "complete" && (len(executions) == 0 || state != "complete" || !allUsageEligible) {
+ return fail()
+ }
+ return nil
+}
+
+func memberUsageEligible(member Member) bool {
+ if member.Execution == "" || slices.Contains(member.Diagnostics, "cached_historical_usage_excluded") {
+ return false
+ }
+ present := false
+ for _, category := range member.Categories {
+ if category.Source == evidencestate.Cached {
+ return false
+ }
+ if category.Category == evidencestate.Usage {
+ present = category.State == evidencestate.Present && category.DeclaredState == evidencestate.Present &&
+ category.Verification == "source-content-and-recorded-usage-shape-only" &&
+ category.ObservedCount != nil && *category.ObservedCount == 1
+ }
+ }
+ return present
+}
+
+// Encode emits typed field order, lexical reference/profile/code order inherited
+// from Build, sorted JSON map keys, and no newline or self-referential digest.
+func Encode(report Report) ([]byte, error) {
+ for _, member := range report.Members {
+ if member.Binding != nil {
+ for _, text := range append(slices.Clone(member.Binding.GraderIDs), member.Binding.Selection.Shard, member.Binding.Selection.Task) {
+ if !utf8.ValidString(text) {
+ return nil, fmt.Errorf("%w: binding is not UTF-8", ErrInvalid)
+ }
+ }
+ }
+ }
+ if usage := report.Usage.Recorded; usage != nil {
+ if !utf8.ValidString(usage.Provider) || !utf8.ValidString(usage.ProviderHost) {
+ return nil, fmt.Errorf("%w: usage metadata is not UTF-8", ErrInvalid)
+ }
+ for model := range usage.ModelMetrics {
+ if !utf8.ValidString(model) {
+ return nil, fmt.Errorf("%w: usage model is not UTF-8", ErrInvalid)
+ }
+ }
+ }
+ data, err := json.Marshal(report)
+ if err != nil {
+ return nil, err
+ }
+ if _, err := Parse(data); err != nil {
+ return nil, err
+ }
+ return data, nil
+}
+
+// ParseInputs rejects duplicate keys, unknown fields and explicit null/empty
+// evidence references before typed decoding can erase their presence.
+func ParseInputs(data []byte) ([]Input, error) {
+ if len(data) > artifactmanifest.MaxManifestBytes {
+ return nil, fmt.Errorf("%w: input set exceeds byte limit", ErrInvalid)
+ }
+ value, err := jsonutil.Parse(data)
+ if err != nil {
+ return nil, fmt.Errorf("%w: input JSON: %w", ErrInvalid, err)
+ }
+ array, ok := value.([]any)
+ if !ok {
+ return nil, fmt.Errorf("%w: expected reference array", ErrInvalid)
+ }
+ for _, value := range array {
+ object, ok := value.(map[string]any)
+ if !ok || len(object) < 1 || len(object) > 2 {
+ return nil, ErrInvalid
+ }
+ for key, field := range object {
+ text, ok := field.(string)
+ if !ok || text == "" || (key != "manifest_sha256" && key != "evidence_sha256") {
+ return nil, ErrInvalid
+ }
+ }
+ if _, ok := object["manifest_sha256"]; !ok {
+ return nil, ErrInvalid
+ }
+ }
+ var inputs []Input
+ if err := json.Unmarshal(data, &inputs); err != nil {
+ return nil, fmt.Errorf("%w: %w", ErrInvalid, err)
+ }
+ return Normalize(inputs)
+}
diff --git a/internal/manifestaggregate/local.go b/internal/manifestaggregate/local.go
new file mode 100644
index 000000000..bf872a9b6
--- /dev/null
+++ b/internal/manifestaggregate/local.go
@@ -0,0 +1,25 @@
+package manifestaggregate
+
+import (
+ "errors"
+
+ "github.com/microsoft/waza/internal/artifactmanifest"
+)
+
+// OpenLocal opens an explicitly selected existing public store. It does not
+// create directories, inspect ambient configuration, or construct credentials.
+func OpenLocal(directory string) (Sources, func() error, error) {
+ backend, err := artifactmanifest.OpenLocal(directory)
+ if err != nil {
+ return Sources{}, nil, err
+ }
+ namespace, err := artifactmanifest.NewNamespace(backend, artifactmanifest.ExecutionLedgerNamespaceV1)
+ if err != nil {
+ return Sources{}, nil, errors.Join(err, backend.Close())
+ }
+ reader, err := namespace.Reader(func(read artifactmanifest.ReadBackend) artifactmanifest.ReadBackend { return read })
+ if err != nil {
+ return Sources{}, nil, errors.Join(err, backend.Close())
+ }
+ return Sources{Objects: backend, Namespace: reader}, backend.Close, nil
+}
diff --git a/internal/manifestaggregate/mixed_identity_test.go b/internal/manifestaggregate/mixed_identity_test.go
new file mode 100644
index 000000000..16a440818
--- /dev/null
+++ b/internal/manifestaggregate/mixed_identity_test.go
@@ -0,0 +1,231 @@
+package manifestaggregate
+
+import (
+ "encoding/json"
+ "flag"
+ "os"
+ "testing"
+
+ "github.com/microsoft/waza/internal/evidencestate"
+ "github.com/microsoft/waza/internal/runnerunit"
+ "github.com/microsoft/waza/internal/testutil/manifestfixture"
+ "github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
+)
+
+var mixedIdentityOutput = flag.String("aggregate-mixed-identity-fixture-output", "", "Exclusively save producer-backed mixed identity caller controls")
+
+func mixedIdentityReport(t *testing.T) (*Report, Usage) {
+ t.Helper()
+ backend, sources, _ := setup(t)
+ qualified := publish(t, backend, manifestfixture.RecordedNonzero(1))
+ known, err := Build(t.Context(), sources, []Input{qualified}, DefaultLimits())
+ require.NoError(t, err)
+ require.Equal(t, "complete", known.Usage.State)
+ mixed, err := Build(t.Context(), sources, []Input{qualified, {Manifest: qualified.Manifest}}, DefaultLimits())
+ require.NoError(t, err)
+ require.Equal(t, "complete", mixed.State)
+ require.Equal(t, 2, mixed.Counts.Members)
+ require.Equal(t, 1, mixed.Counts.VerifiedExecutions)
+ require.Empty(t, mixed.Members[0].Execution)
+ require.NotEmpty(t, mixed.Members[1].Execution)
+ require.Equal(t, "unavailable", mixed.Usage.State)
+ require.Nil(t, mixed.Usage.Recorded)
+ return mixed, known.Usage
+}
+
+func TestMixedIdentityPublicReportFixture(t *testing.T) {
+ report, subtotal := mixedIdentityReport(t)
+ canonical, err := Encode(*report)
+ require.NoError(t, err)
+ parsed, err := Parse(canonical)
+ require.NoError(t, err)
+ require.Equal(t, report, parsed)
+ forged := *report
+ forged.Usage = subtotal
+ invalid, err := json.Marshal(forged)
+ require.NoError(t, err)
+ var value any
+ require.NoError(t, json.Unmarshal(invalid, &value))
+ schema, err := reportSchema()
+ require.NoError(t, err)
+ require.NoError(t, schema.Validate(value), "negative differs only in a schema-valid usage claim")
+ if *mixedIdentityOutput != "" {
+ fixture, err := json.Marshal(struct {
+ Inputs []Input `json:"inputs"`
+ Canonical string `json:"canonical"`
+ Forged string `json:"forged"`
+ Eligibility []eligibilityFixture `json:"eligibility"`
+ }{[]Input{report.Members[0].Input, report.Members[1].Input}, string(canonical), string(invalid), declaredUsageFixtures(t)})
+ require.NoError(t, err)
+ file, err := os.OpenFile(*mixedIdentityOutput, os.O_CREATE|os.O_EXCL|os.O_WRONLY, 0o600)
+ require.NoError(t, err)
+ written, writeErr := file.Write(fixture)
+ closeErr := file.Close()
+ require.NoError(t, writeErr)
+ require.NoError(t, closeErr)
+ require.Equal(t, len(fixture), written)
+ }
+}
+
+type eligibilityFixture struct {
+ Name string `json:"name"`
+ Inputs []Input `json:"inputs"`
+ Canonical string `json:"canonical"`
+ Forged string `json:"forged"`
+}
+
+func declaredUsageFixtures(t *testing.T) []eligibilityFixture {
+ t.Helper()
+ backend, sources, _ := setup(t)
+ known := publish(t, backend, manifestfixture.RecordedNonzero(1))
+ complete, err := Build(t.Context(), sources, []Input{known}, DefaultLimits())
+ require.NoError(t, err)
+ require.Equal(t, "complete", complete.Usage.State)
+ incomplete := publish(t, backend, func(report *runnerunit.Report) {
+ manifestfixture.RecordedNonzero(1)(report)
+ report.Observation.Usage.AICredits = nil
+ })
+ unavailable := publish(t, backend, nil)
+ inspection, err := evidencestate.Inspect(t.Context(), backend, known.Evidence)
+ require.NoError(t, err)
+ document := *inspection.Document
+ document.Lineage = []evidencestate.Lineage{{Manifest: unavailable.Manifest, Relationship: evidencestate.Cache}}
+ cached, err := evidencestate.Publish(t.Context(), backend, document)
+ require.NoError(t, err)
+ var fixtures []eligibilityFixture
+ for _, control := range []struct {
+ name string
+ inputs []Input
+ usageState evidencestate.State
+ }{
+ {"native incomplete", []Input{incomplete}, evidencestate.Incomplete},
+ {"native unavailable", []Input{unavailable}, evidencestate.Unavailable},
+ {"compatible present plus cached view", []Input{known, {Manifest: known.Manifest, Evidence: cached.EvidenceSHA256}}, evidencestate.Present},
+ } {
+ report, err := Build(t.Context(), sources, control.inputs, DefaultLimits())
+ require.NoError(t, err)
+ require.Equal(t, "complete", report.State)
+ require.Equal(t, 1, report.Counts.VerifiedExecutions)
+ for _, member := range report.Members {
+ require.NotEmpty(t, member.Execution, "this negative must pass the existing identity guard")
+ found := false
+ for _, category := range member.Categories {
+ if category.Category == evidencestate.Usage {
+ found = true
+ require.Equal(t, control.usageState, category.State)
+ }
+ }
+ require.True(t, found)
+ }
+ require.Equal(t, "unavailable", report.Usage.State)
+ require.Nil(t, report.Usage.Recorded)
+ canonical, err := Encode(*report)
+ require.NoError(t, err)
+ report.Usage = complete.Usage
+ forged, err := json.Marshal(report)
+ require.NoError(t, err)
+ var value any
+ require.NoError(t, json.Unmarshal(forged, &value))
+ schema, err := reportSchema()
+ require.NoError(t, err)
+ require.NoError(t, schema.Validate(value))
+ fixtures = append(fixtures, eligibilityFixture{control.name, control.inputs, string(canonical), string(forged)})
+ }
+ recorded := complete.Usage.Recorded
+ withoutCredits := *recorded
+ withoutCredits.AICredits = nil
+ complete.Usage.Recorded = &withoutCredits
+ canonical, err := Encode(*complete)
+ require.NoError(t, err)
+ fixtures = append(fixtures, eligibilityFixture{"optional aggregate credits absent", []Input{known}, string(canonical), ""})
+ withoutModels := *recorded
+ withoutModels.ModelMetrics = nil
+ complete.Usage.Recorded = &withoutModels
+ canonical, err = Encode(*complete)
+ require.NoError(t, err)
+ fixtures = append(fixtures, eligibilityFixture{"optional aggregate attribution absent", []Input{known}, string(canonical), ""})
+ zero := publish(t, backend, manifestfixture.RecordedZero)
+ zeroReport, err := Build(t.Context(), sources, []Input{zero}, DefaultLimits())
+ require.NoError(t, err)
+ require.NotNil(t, zeroReport.Usage.Recorded)
+ require.NotNil(t, zeroReport.Usage.Recorded.AICredits)
+ require.Zero(t, *zeroReport.Usage.Recorded.AICredits)
+ canonical, err = Encode(*zeroReport)
+ require.NoError(t, err)
+ fixtures = append(fixtures, eligibilityFixture{"recorded zero credits", []Input{zero}, string(canonical), ""})
+ return fixtures
+}
+
+func TestSavedReportRejectsDeclaredUsageExclusions(t *testing.T) {
+ for _, fixture := range declaredUsageFixtures(t) {
+ t.Run(fixture.Name, func(t *testing.T) {
+ _, err := Parse([]byte(fixture.Canonical))
+ require.NoError(t, err)
+ if fixture.Forged == "" {
+ return
+ }
+ _, err = Parse([]byte(fixture.Forged))
+ assert.ErrorIs(t, err, ErrInvalid)
+ var report Report
+ require.NoError(t, json.Unmarshal([]byte(fixture.Forged), &report))
+ _, err = Encode(report)
+ assert.ErrorIs(t, err, ErrInvalid)
+ })
+ }
+}
+
+func TestSavedReportRejectsUnqualifiedUsageCategories(t *testing.T) {
+ fixtures := declaredUsageFixtures(t)
+ require.Equal(t, "optional aggregate credits absent", fixtures[3].Name)
+ for _, change := range []string{"missing category", "cached category", "unverified", "empty count", "withheld declaration"} {
+ t.Run(change, func(t *testing.T) {
+ report, err := Parse([]byte(fixtures[3].Canonical))
+ require.NoError(t, err)
+ member := &report.Members[0]
+ switch change {
+ case "missing category":
+ member.Categories = []Category{}
+ case "cached category":
+ member.Categories[0].Source = evidencestate.Cached
+ default:
+ for index := range member.Categories {
+ category := &member.Categories[index]
+ if category.Category != evidencestate.Usage {
+ continue
+ }
+ switch change {
+ case "unverified":
+ category.Verification = "declared-state-only"
+ case "empty count":
+ category.ObservedCount = new(int64)
+ case "withheld declaration":
+ category.DeclaredState = evidencestate.Redacted
+ }
+ }
+ }
+ wire, err := json.Marshal(report)
+ require.NoError(t, err)
+ var value any
+ require.NoError(t, json.Unmarshal(wire, &value))
+ schema, err := reportSchema()
+ require.NoError(t, err)
+ require.NoError(t, schema.Validate(value))
+ _, err = Parse(wire)
+ assert.ErrorIs(t, err, ErrInvalid)
+ _, err = Encode(*report)
+ assert.ErrorIs(t, err, ErrInvalid)
+ })
+ }
+}
+
+func TestSavedReportRejectsCompleteUsageWithUnknownExecution(t *testing.T) {
+ report, subtotal := mixedIdentityReport(t)
+ report.Usage = subtotal
+ wire, err := json.Marshal(report)
+ require.NoError(t, err)
+ _, err = Parse(wire)
+ assert.ErrorIs(t, err, ErrInvalid, "saved reader must not admit a subset as the complete total")
+ _, err = Encode(*report)
+ assert.ErrorIs(t, err, ErrInvalid, "writer must apply the same admission rule")
+}
diff --git a/internal/manifestaggregate/usage.go b/internal/manifestaggregate/usage.go
new file mode 100644
index 000000000..313dd5d0f
--- /dev/null
+++ b/internal/manifestaggregate/usage.go
@@ -0,0 +1,84 @@
+package manifestaggregate
+
+import (
+ "math"
+
+ "github.com/microsoft/waza/internal/models"
+)
+
+const maxJSONInteger int64 = (1 << 53) - 1
+
+func sumRecordedUsage(stats []*models.UsageStats) (*models.UsageStats, string) {
+ if len(stats) == 0 {
+ return nil, "complete_historical_usage_unavailable"
+ }
+ totals := [5]int{}
+ modelsTotals := map[string][4]int{}
+ floatTotals := [2]float64{}
+ modelFloatTotals := map[string][3]float64{}
+ validFloat := func(value float64) bool {
+ return !math.IsNaN(value) && !math.IsInf(value, 0) && value >= 0 && value <= float64(maxJSONInteger)
+ }
+ validCredits := func(value *float64) bool { return value == nil || validFloat(*value) }
+ addFloat := func(total *float64, value float64) bool {
+ sum := *total + value
+ if !validFloat(value) || !validFloat(sum) ||
+ (*total > 0 && value > 0 && (sum == *total || sum == value)) {
+ return false
+ }
+ *total = sum
+ return true
+ }
+ for _, usage := range stats {
+ if usage == nil {
+ return nil, "complete_historical_usage_unavailable"
+ }
+ if !addFloat(&floatTotals[0], usage.PremiumRequests) ||
+ (usage.AICredits != nil && !addFloat(&floatTotals[1], *usage.AICredits)) {
+ return nil, "recorded_usage_numeric_precision_unavailable"
+ }
+ for i, value := range [5]int{usage.Turns, usage.InputTokens, usage.OutputTokens, usage.CacheReadTokens, usage.CacheWriteTokens} {
+ if value < 0 || totals[i] > math.MaxInt-value {
+ return nil, "recorded_usage_numeric_overflow"
+ }
+ totals[i] += value
+ if int64(totals[i]) > maxJSONInteger {
+ return nil, "recorded_usage_numeric_precision_unavailable"
+ }
+ }
+ for name, metrics := range usage.ModelMetrics {
+ floats := modelFloatTotals[name]
+ if !addFloat(&floats[0], metrics.RequestCount) || !addFloat(&floats[1], metrics.RequestCost) ||
+ (metrics.AICredits != nil && !addFloat(&floats[2], *metrics.AICredits)) {
+ return nil, "recorded_usage_numeric_precision_unavailable"
+ }
+ modelFloatTotals[name] = floats
+ total := modelsTotals[name]
+ for i, value := range [4]int{metrics.InputTokens, metrics.OutputTokens, metrics.CacheReadTokens, metrics.CacheWriteTokens} {
+ if value < 0 || total[i] > math.MaxInt-value {
+ return nil, "recorded_usage_numeric_overflow"
+ }
+ total[i] += value
+ if int64(total[i]) > maxJSONInteger {
+ return nil, "recorded_usage_numeric_precision_unavailable"
+ }
+ }
+ modelsTotals[name] = total
+ }
+ }
+ result := models.AggregateUsageStats(stats)
+ if result == nil {
+ // Native recorded zero counters do not disappear merely because optional
+ // credits/model attribution are unavailable.
+ result = &models.UsageStats{}
+ }
+ if !validFloat(result.PremiumRequests) || !validCredits(result.AICredits) {
+ return nil, "recorded_usage_numeric_precision_unavailable"
+ }
+ for _, metrics := range result.ModelMetrics {
+ if !validFloat(metrics.RequestCount) || !validFloat(metrics.RequestCost) || !validCredits(metrics.AICredits) {
+ return nil, "recorded_usage_numeric_precision_unavailable"
+ }
+ }
+ return result, ""
+}
diff --git a/internal/runnercheckpoint/captured.go b/internal/runnercheckpoint/captured.go
new file mode 100644
index 000000000..789cdb234
--- /dev/null
+++ b/internal/runnercheckpoint/captured.go
@@ -0,0 +1,103 @@
+package runnercheckpoint
+
+import (
+ "context"
+
+ "github.com/microsoft/waza/internal/artifactmanifest"
+ "github.com/microsoft/waza/internal/executionledger"
+ "github.com/microsoft/waza/internal/ownedsubmission"
+ "github.com/microsoft/waza/internal/runnercontract"
+ "github.com/microsoft/waza/internal/runnerunit"
+)
+
+// RejectPublic withholds native input, provider locations and operator causes.
+func RejectPublic(code string) Result {
+ result := newResult()
+ result.fail(code)
+ return result
+}
+
+func (s *Service) SubmitCaptured(ctx context.Context, data []byte, backend artifactmanifest.Backend, observers Hooks) (Result, error) {
+ return s.captured(ctx, data, backend, "", observers, false)
+}
+
+func (s *Service) ResumeCaptured(ctx context.Context, data []byte, backend artifactmanifest.Backend, digest string, observers Hooks) (Result, error) {
+ return s.captured(ctx, data, backend, digest, observers, true)
+}
+
+func (s *Service) captured(ctx context.Context, data []byte, backend artifactmanifest.Backend, digest string,
+ observers Hooks, resume bool,
+) (Result, error) {
+ hooks, err := ownedsubmission.CaptureHooks(observers.Submission)
+ if err != nil {
+ return RejectPublic("checkpoint_capture_policy_invalid"), err
+ }
+ approved, metadataAdmitted := false, false
+ admission := hooks.Admission
+ hooks.Admission = func(ctx context.Context, binding artifactmanifest.Binding) error {
+ if err := admission(ctx, binding); err != nil {
+ return err
+ }
+ metadataAdmitted = true
+ return nil
+ }
+ validate := hooks.ValidatePublication
+ hooks.ValidatePublication = func(ctx context.Context, original runnerunit.SubmissionSnapshot, publication runnerunit.SubmissionPublication) error {
+ if err := validate(ctx, original, publication); err != nil {
+ return err
+ }
+ approved, metadataAdmitted = true, true
+ return nil
+ }
+ reuse := hooks.ReuseAdmission
+ hooks.ReuseAdmission = func(ctx context.Context, source *executionledger.CompletedExecution) error {
+ if err := reuse(ctx, source); err != nil {
+ return err
+ }
+ approved, metadataAdmitted = true, true
+ return nil
+ }
+ observers.Submission = hooks
+ var result Result
+ if resume {
+ result, err = s.Resume(ctx, data, backend, digest, observers)
+ } else {
+ result, err = s.Submit(ctx, data, backend, observers)
+ }
+ return projectCapturedResult(result, approved, metadataAdmitted), err
+}
+
+func projectCapturedResult(result Result, approved, metadataAdmitted bool) Result {
+ if result.Source != nil && !approved {
+ result.Source = nil
+ result.SourceWithheld = true
+ }
+ result.Diagnostics = publicDiagnostics(result.Diagnostics)
+ if result.Submission != nil {
+ submission := *result.Submission
+ submission.Report = nil
+ submission.Diagnostics = publicDiagnostics(submission.Diagnostics)
+ if !approved {
+ submission.Terminal = nil
+ }
+ if !metadataAdmitted {
+ submission.Caller = nil
+ submission.Reservation = nil
+ submission.Decision = nil
+ submission.Completion = nil
+ }
+ result.Submission = &submission
+ }
+ return result
+}
+
+func publicDiagnostics(original []runnercontract.Diagnostic) []runnercontract.Diagnostic {
+ result := make([]runnercontract.Diagnostic, len(original))
+ for i, diagnostic := range original {
+ result[i] = runnercontract.Diagnostic{
+ Code: diagnostic.Code,
+ Message: "Checkpoint operation did not complete. Preserve source and partial records; inspect operator diagnostics. Unapproved source and invocation text are withheld.",
+ }
+ }
+ return result
+}
diff --git a/internal/testutil/manifestfixture/fixture.go b/internal/testutil/manifestfixture/fixture.go
new file mode 100644
index 000000000..9f45b3467
--- /dev/null
+++ b/internal/testutil/manifestfixture/fixture.go
@@ -0,0 +1,100 @@
+// Package manifestfixture creates public completed sources through the actual
+// mock runner, owner, capture and immutable store paths for integration tests.
+package manifestfixture
+
+import (
+ "encoding/json"
+ "os"
+ "testing"
+
+ "github.com/microsoft/waza/internal/artifactmanifest"
+ "github.com/microsoft/waza/internal/evidencestate"
+ "github.com/microsoft/waza/internal/executionledger"
+ "github.com/microsoft/waza/internal/models"
+ "github.com/microsoft/waza/internal/runnercontract"
+ "github.com/microsoft/waza/internal/runnerunit"
+)
+
+type Source struct {
+ Manifest string
+ Evidence string
+ Execution string
+}
+
+// Publish runs only the fixture's explicitly selected mock engine. Callers may
+// alter the original request/report to produce alternative and negative controls.
+func Publish(t *testing.T, backend *artifactmanifest.Local, requestPath string, changeRequest func(*runnercontract.Request), changeReport func(*runnerunit.Report)) Source {
+ t.Helper()
+ wire, err := os.ReadFile(requestPath)
+ if err != nil {
+ t.Fatal(err)
+ }
+ request, err := runnercontract.Parse(wire)
+ if err != nil {
+ t.Fatal(err)
+ }
+ if changeRequest != nil {
+ changeRequest(request)
+ wire, err = json.Marshal(request)
+ if err != nil {
+ t.Fatal(err)
+ }
+ }
+ report := runnerunit.NewService(nil).Run(t.Context(), wire)
+ if report.Mode != "mock" || report.Calls != 1 || report.Observation == nil || report.Status != "completed" {
+ t.Fatalf("fixture must execute exactly one actual mock call: %+v", report)
+ }
+ if changeReport != nil {
+ changeReport(&report)
+ }
+ prepared, err := evidencestate.CaptureReport(report, nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ _, binding := report.Terminal()
+ if binding == nil || binding.Identity == nil {
+ t.Fatal("fixture report has no original binding")
+ }
+ ledger, err := executionledger.New(backend)
+ if err != nil {
+ t.Fatal(err)
+ }
+ creator, err := ledger.Reserve(t.Context(), artifactmanifest.Binding{
+ Identity: *binding.Identity, Selection: binding.Selection,
+ ExecutionMode: binding.ExecutionMode, GraderIDs: binding.GraderIDs,
+ }, nil)
+ if err != nil {
+ t.Fatal(err)
+ }
+ published, err := evidencestate.PersistOwnedCapture(t.Context(), backend, prepared, ledger, creator)
+ if err != nil || published.Document == nil || published.Document.ExecutionSource == nil {
+ t.Fatalf("fixture must persist a verified completed public source: %+v %v", published, err)
+ }
+ return Source{Manifest: published.Document.OriginManifest, Evidence: published.EvidenceSHA256,
+ Execution: published.Document.ExecutionSource.ExecutionID}
+}
+
+// RecordedZero supplies explicit authored native usage, not observed billing.
+func RecordedZero(report *runnerunit.Report) {
+ zero := 0.0
+ report.Observation.Usage = &models.UsageStats{
+ AICredits: &zero, ModelMetrics: map[string]models.ModelUsage{"fixture-model": {AICredits: &zero}},
+ }
+}
+
+// RecordedNonzero supplies distinct authored counters, not observed SDK billing.
+func RecordedNonzero(factor int) func(*runnerunit.Report) {
+ return func(report *runnerunit.Report) {
+ credits := 1.25 * float64(factor)
+ report.Observation.Usage = &models.UsageStats{
+ Turns: factor, InputTokens: 11 * factor, OutputTokens: 7 * factor,
+ CacheReadTokens: 3 * factor, CacheWriteTokens: 2 * factor,
+ PremiumRequests: .5 * float64(factor), AICredits: &credits,
+ ModelMetrics: map[string]models.ModelUsage{"fixture-model": {
+ InputTokens: 11 * factor, OutputTokens: 7 * factor,
+ CacheReadTokens: 3 * factor, CacheWriteTokens: 2 * factor,
+ RequestCount: 2 * float64(factor), RequestCost: .25 * float64(factor), AICredits: &credits,
+ }},
+ }
+ }
+}
diff --git a/internal/webapi/aggregate_diagnostics_test.go b/internal/webapi/aggregate_diagnostics_test.go
new file mode 100644
index 000000000..56856a211
--- /dev/null
+++ b/internal/webapi/aggregate_diagnostics_test.go
@@ -0,0 +1,115 @@
+package webapi
+
+import (
+ "bytes"
+ "errors"
+ "io"
+ "log/slog"
+ "net/http"
+ "net/http/httptest"
+ "runtime"
+ "strings"
+ "sync"
+ "testing"
+ "time"
+
+ "github.com/microsoft/waza/internal/artifactmanifest"
+ "github.com/microsoft/waza/internal/manifestaggregate"
+ "github.com/stretchr/testify/require"
+)
+
+type aggregateResetFailure struct{ http.ResponseWriter }
+
+func (w aggregateResetFailure) Unwrap() http.ResponseWriter { return w.ResponseWriter }
+func (w aggregateResetFailure) SetReadDeadline(deadline time.Time) error {
+ if deadline.IsZero() {
+ return errors.New("private reset cause")
+ }
+ return http.NewResponseController(w.ResponseWriter).SetReadDeadline(deadline)
+}
+
+func TestAggregateHTTPBlockedDiagnosticsDoNotBlockDelivery(t *testing.T) {
+ require.Eventually(t, func() bool { return !runnerDiagnosticActive.Load() }, 2*time.Second, time.Millisecond)
+ sink := &blockedRunnerDiagnosticSink{entered: make(chan struct{}), release: make(chan struct{})}
+ var diagnostics bytes.Buffer
+ logger := slog.Default()
+ slog.SetDefault(slog.New(slog.NewTextHandler(io.MultiWriter(&diagnostics, sink), nil)))
+ var servers []*httptest.Server
+ var handlers sync.WaitGroup
+ t.Cleanup(func() {
+ close(sink.release)
+ for _, server := range servers {
+ server.Close()
+ }
+ handlers.Wait()
+ require.Eventually(t, func() bool { return !runnerDiagnosticActive.Load() }, 2*time.Second, time.Millisecond)
+ slog.SetDefault(logger)
+ require.Contains(t, diagnostics.String(), "private store cause", "operator diagnostics retain the actual cause")
+ })
+ backend, err := artifactmanifest.OpenLocal(t.TempDir())
+ require.NoError(t, err)
+ t.Cleanup(func() { require.NoError(t, backend.Close()) })
+ for index, control := range []struct {
+ status int
+ code string
+ reset bool
+ store bool
+ invalid bool
+ }{
+ {503, "aggregate_store_unavailable", false, false, false},
+ {500, "aggregate_transport_unavailable", true, false, false},
+ {409, "", false, true, false},
+ {503, "aggregate_evaluation_unavailable", false, false, true},
+ {503, "aggregate_store_unavailable", false, false, false},
+ } {
+ mux := http.NewServeMux()
+ RegisterAggregateRoutesWithSources(mux, func() (manifestaggregate.Sources, func() error, error) {
+ if control.invalid {
+ return manifestaggregate.Sources{}, func() error { return nil }, nil
+ }
+ if control.store {
+ return manifestaggregate.Sources{Objects: backend}, func() error { return nil }, nil
+ }
+ return manifestaggregate.Sources{}, nil, errors.New("private store cause")
+ })
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ handlers.Add(1)
+ defer handlers.Done()
+ if control.reset {
+ w = aggregateResetFailure{w}
+ }
+ mux.ServeHTTP(w, r)
+ }))
+ servers = append(servers, server)
+ server.Client().Timeout = time.Second
+ response, err := server.Client().Post(server.URL+"/api/v1/aggregates", "application/json",
+ strings.NewReader(`[{"manifest_sha256":"`+strings.Repeat("a", 64)+`"}]`))
+ require.NoError(t, err, "diagnostic sink must not block HTTP delivery")
+ body, readErr := io.ReadAll(response.Body)
+ closeErr := response.Body.Close()
+ require.NoError(t, readErr)
+ require.NoError(t, closeErr)
+ require.Equal(t, control.status, response.StatusCode)
+ require.NotContains(t, string(body), "private")
+ if control.code != "" {
+ require.JSONEq(t, `{"error":"`+control.code+`"}`, string(body))
+ } else {
+ report, err := manifestaggregate.Parse(body)
+ require.NoError(t, err)
+ require.Equal(t, "incomplete", report.State)
+ }
+ if index == 0 {
+ select {
+ case <-sink.entered:
+ case <-time.After(2 * time.Second):
+ t.Fatal("operator diagnostic did not reach blocked sink")
+ }
+ }
+ }
+ require.EqualValues(t, 1, sink.writes.Load())
+ require.GreaterOrEqual(t, runnerDiagnosticsSuppressed.Load(), uint64(4))
+ stack := make([]byte, 1<<20)
+ length := runtime.Stack(stack, true)
+ require.Less(t, length, len(stack))
+ require.Equal(t, 1, strings.Count(string(stack[:length]), "webapi.logRunnerError.func1("))
+}
diff --git a/internal/webapi/aggregate_transport_test.go b/internal/webapi/aggregate_transport_test.go
new file mode 100644
index 000000000..7de564239
--- /dev/null
+++ b/internal/webapi/aggregate_transport_test.go
@@ -0,0 +1,262 @@
+package webapi
+
+import (
+ "bufio"
+ "context"
+ "errors"
+ "fmt"
+ "io"
+ "net"
+ "net/http"
+ "net/http/httptest"
+ "strings"
+ "sync/atomic"
+ "testing"
+ "time"
+
+ "github.com/microsoft/waza/internal/artifactmanifest"
+ "github.com/microsoft/waza/internal/manifestaggregate"
+ "github.com/stretchr/testify/require"
+)
+
+func TestAggregateHTTPStalledBodyIsBounded(t *testing.T) {
+ var opened atomic.Int32
+ mux := http.NewServeMux()
+ RegisterAggregateRoutesWithSources(mux, func() (manifestaggregate.Sources, func() error, error) {
+ opened.Add(1)
+ return manifestaggregate.Sources{}, nil, errors.New("private provider error")
+ })
+ server := httptest.NewServer(mux)
+ defer server.Close()
+ connection, err := net.DialTimeout("tcp", strings.TrimPrefix(server.URL, "http://"), time.Second)
+ require.NoError(t, err)
+ defer func() { require.NoError(t, connection.Close()) }()
+ require.NoError(t, connection.SetDeadline(time.Now().Add(8*time.Second)))
+ _, err = fmt.Fprint(connection, "POST /api/v1/aggregates HTTP/1.1\r\nHost: localhost\r\nContent-Length: 2\r\n\r\n[")
+ require.NoError(t, err)
+ response, err := http.ReadResponse(bufio.NewReader(connection), &http.Request{Method: http.MethodPost})
+ require.NoError(t, err, "stalled input must produce a bounded explicit HTTP failure")
+ defer func() { require.NoError(t, response.Body.Close()) }()
+ body, err := io.ReadAll(response.Body)
+ require.NoError(t, err)
+ require.Equal(t, http.StatusRequestTimeout, response.StatusCode)
+ require.JSONEq(t, `{"error":"aggregate_request_timeout"}`, string(body))
+ require.True(t, response.Close, "partial input must not trigger an unbounded drain")
+ require.Zero(t, opened.Load())
+}
+
+func TestAggregateHTTPRejectedPartialInputDoesNotDrain(t *testing.T) {
+ for _, control := range []struct {
+ name string
+ cancel bool
+ }{
+ {name: "size limit with withheld final byte"},
+ {name: "request cancellation", cancel: true},
+ } {
+ t.Run(control.name, func(t *testing.T) {
+ var opened atomic.Int32
+ mux := http.NewServeMux()
+ RegisterAggregateRoutesWithSources(mux, func() (manifestaggregate.Sources, func() error, error) {
+ opened.Add(1)
+ return manifestaggregate.Sources{}, nil, errors.New("private provider error")
+ })
+ cancels := make(chan context.CancelFunc, 1)
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ ctx, cancel := context.WithCancel(r.Context())
+ defer cancel()
+ cancels <- cancel
+ mux.ServeHTTP(w, r.WithContext(ctx))
+ }))
+ defer server.Close()
+ connection, err := net.DialTimeout("tcp", strings.TrimPrefix(server.URL, "http://"), time.Second)
+ require.NoError(t, err)
+ defer func() { require.NoError(t, connection.Close()) }()
+ require.NoError(t, connection.SetDeadline(time.Now().Add(8*time.Second)))
+ payload := strings.Repeat(" ", artifactmanifest.MaxManifestBytes+1)
+ status, code := http.StatusBadRequest, "invalid_aggregate_request"
+ if control.cancel {
+ payload, status, code = "[", http.StatusRequestTimeout, "aggregate_request_timeout"
+ }
+ _, err = fmt.Fprintf(connection, "POST /api/v1/aggregates HTTP/1.1\r\nHost: localhost\r\nContent-Length: %d\r\n\r\n%s", len(payload)+1, payload)
+ require.NoError(t, err)
+ select {
+ case cancel := <-cancels:
+ if control.cancel {
+ cancel()
+ }
+ case <-time.After(8 * time.Second):
+ t.Fatal("request did not reach the registered handler")
+ }
+ require.NoError(t, connection.SetReadDeadline(time.Now().Add(2*time.Second)))
+ response, err := http.ReadResponse(bufio.NewReader(connection), &http.Request{Method: http.MethodPost})
+ require.NoError(t, err, "rejected partial input must not wait for the missing byte or five-second read deadline")
+ defer func() { require.NoError(t, response.Body.Close()) }()
+ body, err := io.ReadAll(response.Body)
+ require.NoError(t, err)
+ require.Equal(t, status, response.StatusCode)
+ require.JSONEq(t, `{"error":"`+code+`"}`, string(body))
+ require.True(t, response.Close)
+ require.Zero(t, opened.Load())
+ })
+ }
+}
+
+type aggregateDeadlineFault struct {
+ *httptest.ResponseRecorder
+ readDeadlines []time.Time
+ writeDeadlines []time.Time
+ readFailure int
+ writeFailure int
+ writeAlways bool
+ shortWrite bool
+ writeError bool
+ flushError bool
+ headers int
+}
+
+func (w *aggregateDeadlineFault) SetReadDeadline(deadline time.Time) error {
+ w.readDeadlines = append(w.readDeadlines, deadline)
+ if len(w.readDeadlines) == w.readFailure {
+ return errors.New("private read deadline failure")
+ }
+ return nil
+}
+
+func (w *aggregateDeadlineFault) SetWriteDeadline(deadline time.Time) error {
+ w.writeDeadlines = append(w.writeDeadlines, deadline)
+ if w.writeAlways || len(w.writeDeadlines) == w.writeFailure {
+ return http.ErrNotSupported
+ }
+ return nil
+}
+
+func (w *aggregateDeadlineFault) WriteHeader(status int) {
+ w.headers++
+ w.ResponseRecorder.WriteHeader(status)
+}
+
+func (w *aggregateDeadlineFault) Write(data []byte) (int, error) {
+ if w.writeError {
+ return 0, errors.New("private write failure")
+ }
+ if w.shortWrite {
+ return w.ResponseRecorder.Write(data[:len(data)-1])
+ }
+ return w.ResponseRecorder.Write(data)
+}
+
+func (w *aggregateDeadlineFault) FlushError() error {
+ if w.flushError {
+ return errors.New("private flush failure")
+ }
+ w.Flush()
+ return nil
+}
+
+type aggregateReadCounter struct {
+ io.Reader
+ calls int
+}
+
+func (r *aggregateReadCounter) Read(data []byte) (int, error) {
+ r.calls++
+ return r.Reader.Read(data)
+}
+
+func TestAggregateHTTPDeadlineAdmissionFailures(t *testing.T) {
+ for _, control := range []struct {
+ name string
+ readFailure int
+ writeFailure int
+ readBody bool
+ }{
+ {name: "read setup", readFailure: 1},
+ {name: "write support probe", writeFailure: 1},
+ {name: "read reset", readFailure: 2, readBody: true},
+ } {
+ t.Run(control.name, func(t *testing.T) {
+ opened := 0
+ mux := http.NewServeMux()
+ RegisterAggregateRoutesWithSources(mux, func() (manifestaggregate.Sources, func() error, error) {
+ opened++
+ return manifestaggregate.Sources{}, nil, errors.New("private provider error")
+ })
+ body := &aggregateReadCounter{Reader: strings.NewReader(`[{"manifest_sha256":"` + strings.Repeat("a", 64) + `"}]`)}
+ writer := &aggregateDeadlineFault{ResponseRecorder: httptest.NewRecorder(),
+ readFailure: control.readFailure, writeFailure: control.writeFailure}
+ mux.ServeHTTP(writer, httptest.NewRequest(http.MethodPost, "/api/v1/aggregates", body))
+ require.Equal(t, http.StatusInternalServerError, writer.Code)
+ require.JSONEq(t, `{"error":"aggregate_transport_unavailable"}`, writer.Body.String())
+ require.Equal(t, "close", writer.Header().Get("Connection"))
+ require.NotContains(t, writer.Body.String(), "private")
+ require.Equal(t, control.readBody, body.calls > 0)
+ require.Zero(t, opened)
+ require.False(t, writer.readDeadlines[0].IsZero())
+ if control.readBody {
+ require.Len(t, writer.readDeadlines, 2)
+ require.True(t, writer.readDeadlines[1].IsZero(), "reset failure must not admit storage")
+ }
+ require.False(t, writer.writeDeadlines[len(writer.writeDeadlines)-1].IsZero())
+ })
+ }
+}
+
+func TestAggregateHTTPDeliveryFailuresAbort(t *testing.T) {
+ for _, control := range []struct {
+ name string
+ writeAlways bool
+ shortWrite bool
+ writeError bool
+ flushError bool
+ }{
+ {name: "unsupported output", writeAlways: true},
+ {name: "short write", shortWrite: true},
+ {name: "write error", writeError: true},
+ {name: "flush error", flushError: true},
+ } {
+ t.Run(control.name, func(t *testing.T) {
+ opened := 0
+ mux := http.NewServeMux()
+ RegisterAggregateRoutesWithSources(mux, func() (manifestaggregate.Sources, func() error, error) {
+ opened++
+ return manifestaggregate.Sources{}, nil, errors.New("private provider error")
+ })
+ writer := &aggregateDeadlineFault{ResponseRecorder: httptest.NewRecorder(),
+ writeAlways: control.writeAlways, shortWrite: control.shortWrite,
+ writeError: control.writeError, flushError: control.flushError}
+ require.PanicsWithValue(t, http.ErrAbortHandler, func() {
+ mux.ServeHTTP(writer, httptest.NewRequest(http.MethodPost, "/api/v1/aggregates", strings.NewReader(`[]`)))
+ })
+ require.Zero(t, opened)
+ if control.writeAlways {
+ require.Zero(t, writer.headers, "unsupported delivery must not commit an implicit 200")
+ require.Empty(t, writer.Body.Bytes())
+ } else {
+ require.Equal(t, http.StatusBadRequest, writer.Code)
+ }
+ })
+ }
+}
+
+type aggregateUnsupportedOutput struct{ http.ResponseWriter }
+
+func (w aggregateUnsupportedOutput) Unwrap() http.ResponseWriter { return w.ResponseWriter }
+func (aggregateUnsupportedOutput) SetWriteDeadline(time.Time) error { return http.ErrNotSupported }
+
+func TestAggregateHTTPUnsupportedDeliveryHasNoSuccessResponse(t *testing.T) {
+ var opened atomic.Int32
+ mux := http.NewServeMux()
+ RegisterAggregateRoutesWithSources(mux, func() (manifestaggregate.Sources, func() error, error) {
+ opened.Add(1)
+ return manifestaggregate.Sources{}, nil, errors.New("private provider error")
+ })
+ server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
+ mux.ServeHTTP(aggregateUnsupportedOutput{w}, r)
+ }))
+ defer server.Close()
+ response, err := server.Client().Post(server.URL+"/api/v1/aggregates", "application/json",
+ strings.NewReader(`[{"manifest_sha256":"`+strings.Repeat("a", 64)+`"}]`))
+ require.Error(t, err, "unsupported delivery must abort instead of returning an implicit 200")
+ require.Nil(t, response)
+ require.Zero(t, opened.Load())
+}
diff --git a/internal/webapi/aggregates.go b/internal/webapi/aggregates.go
new file mode 100644
index 000000000..ebca05916
--- /dev/null
+++ b/internal/webapi/aggregates.go
@@ -0,0 +1,141 @@
+package webapi
+
+import (
+ "context"
+ "encoding/json"
+ "errors"
+ "io"
+ "net/http"
+ "time"
+
+ "github.com/microsoft/waza/internal/artifactmanifest"
+ "github.com/microsoft/waza/internal/manifestaggregate"
+ "github.com/microsoft/waza/internal/projectconfig"
+)
+
+// AggregateSources opens explicitly configured read-only capabilities per
+// request. Providers must not construct runtimes or discover worker-local state.
+type AggregateSources func() (manifestaggregate.Sources, func() error, error)
+
+func RegisterAggregateRoutes(mux *http.ServeMux, resultsDir string, config *projectconfig.StorageConfig) {
+ provider := manifestInspectorFor(resultsDir, config)
+ RegisterAggregateRoutesWithSources(mux, func() (manifestaggregate.Sources, func() error, error) {
+ if config != nil && config.Enabled && config.Provider != "" && config.Provider != "local" {
+ return manifestaggregate.Sources{}, nil, errors.New("aggregation does not support the configured active provider")
+ }
+ backend, closeBackend, err := provider.openBackend()
+ if err != nil {
+ return manifestaggregate.Sources{}, nil, err
+ }
+ namespace, err := artifactmanifest.NewNamespace(backend, artifactmanifest.ExecutionLedgerNamespaceV1)
+ if err != nil {
+ return manifestaggregate.Sources{}, nil, errors.Join(err, closeBackend())
+ }
+ reader, err := namespace.Reader(func(read artifactmanifest.ReadBackend) artifactmanifest.ReadBackend { return read })
+ if err != nil {
+ return manifestaggregate.Sources{}, nil, errors.Join(err, closeBackend())
+ }
+ return manifestaggregate.Sources{Objects: backend, Namespace: reader}, closeBackend, nil
+ })
+}
+
+func RegisterAggregateRoutesWithSources(mux *http.ServeMux, open AggregateSources) {
+ mux.HandleFunc("POST /api/v1/aggregates", func(w http.ResponseWriter, r *http.Request) {
+ inputComplete := false
+ emit := func(status int, data []byte) {
+ writeAggregateResponse(w, status, data, inputComplete)
+ }
+ reject := func(status int, code string) {
+ data, err := json.Marshal(map[string]string{"error": code})
+ if err != nil {
+ logRunnerError("encoding aggregate transport error", "error", err)
+ panic(http.ErrAbortHandler)
+ }
+ emit(status, append(data, '\n'))
+ }
+ controller := http.NewResponseController(w)
+ deadline := time.Now().Add(5 * time.Second)
+ if err := controller.SetReadDeadline(deadline); err != nil {
+ logRunnerError("setting aggregate read deadline", "error", err)
+ reject(http.StatusInternalServerError, "aggregate_transport_unavailable")
+ return
+ }
+ if err := controller.SetWriteDeadline(time.Time{}); err != nil {
+ logRunnerError("probing aggregate write deadline", "error", err)
+ reject(http.StatusInternalServerError, "aggregate_transport_unavailable")
+ return
+ }
+ data, inputErr := readRunnerHTTPBody(w, r, artifactmanifest.MaxManifestBytes, deadline)
+ if inputErr != nil {
+ logRunnerError("reading aggregate input", "code", inputErr.Code, "error", inputErr.Err)
+ switch inputErr.Status {
+ case http.StatusRequestTimeout:
+ reject(http.StatusRequestTimeout, "aggregate_request_timeout")
+ case http.StatusInternalServerError:
+ reject(http.StatusInternalServerError, "aggregate_transport_unavailable")
+ default:
+ reject(http.StatusBadRequest, "invalid_aggregate_request")
+ }
+ return
+ }
+ inputComplete = true
+ inputs, err := manifestaggregate.ParseInputs(data)
+ if err != nil {
+ reject(http.StatusBadRequest, "invalid_aggregate_request")
+ return
+ }
+ sources, closeSources, err := open()
+ if err != nil {
+ logRunnerError("aggregate store unavailable", "error", err)
+ reject(http.StatusServiceUnavailable, "aggregate_store_unavailable")
+ return
+ }
+ ctx, cancel := context.WithTimeout(r.Context(), 30*time.Second)
+ defer cancel()
+ report, buildErr := manifestaggregate.Build(ctx, sources, inputs, manifestaggregate.DefaultLimits())
+ closeErr := closeSources()
+ if report == nil || closeErr != nil {
+ logRunnerError("aggregate evaluation unavailable", "error", errors.Join(buildErr, closeErr))
+ reject(http.StatusServiceUnavailable, "aggregate_evaluation_unavailable")
+ return
+ }
+ wire, encodeErr := manifestaggregate.Encode(*report)
+ if encodeErr != nil {
+ logRunnerError("aggregate encoding failed", "error", encodeErr)
+ reject(http.StatusInternalServerError, "aggregate_encoding_failed")
+ return
+ }
+ status := http.StatusOK
+ if buildErr != nil {
+ status = http.StatusConflict
+ logRunnerError("aggregate admission incomplete", "error", buildErr)
+ }
+ emit(status, wire)
+ })
+}
+
+func writeAggregateResponse(w http.ResponseWriter, status int, data []byte, inputComplete bool) {
+ if !inputComplete {
+ w.Header().Set("Connection", "close")
+ }
+ controller := http.NewResponseController(w)
+ if err := controller.SetWriteDeadline(time.Now().Add(5 * time.Second)); err != nil {
+ logRunnerError("setting aggregate write deadline; aborting transport without output", "error", err)
+ panic(http.ErrAbortHandler)
+ }
+ w.Header().Set("Content-Type", "application/json")
+ w.Header().Set("Cache-Control", "no-store")
+ w.WriteHeader(status)
+ written, err := w.Write(data)
+ if written != len(data) {
+ err = errors.Join(err, io.ErrShortWrite)
+ }
+ if err != nil {
+ logRunnerError("writing aggregate response", "error", err)
+ panic(http.ErrAbortHandler)
+ }
+ if err := controller.Flush(); err != nil {
+ logRunnerError("flushing aggregate response", "error", err)
+ panic(http.ErrAbortHandler)
+ }
+}
diff --git a/internal/webapi/aggregates_test.go b/internal/webapi/aggregates_test.go
new file mode 100644
index 000000000..516e4341c
--- /dev/null
+++ b/internal/webapi/aggregates_test.go
@@ -0,0 +1,56 @@
+package webapi
+
+import (
+ "errors"
+ "io"
+ "net/http"
+ "net/http/httptest"
+ "strings"
+ "sync/atomic"
+ "testing"
+
+ "github.com/microsoft/waza/internal/artifactmanifest"
+ "github.com/microsoft/waza/internal/manifestaggregate"
+ "github.com/stretchr/testify/require"
+)
+
+func TestAggregateAPIRejectsInvalidInputBeforeOpeningProvider(t *testing.T) {
+ mux := http.NewServeMux()
+ var calls atomic.Int32
+ RegisterAggregateRoutesWithSources(mux, func() (manifestaggregate.Sources, func() error, error) {
+ calls.Add(1)
+ return manifestaggregate.Sources{}, nil, errors.New("private provider error")
+ })
+ server := httptest.NewServer(mux)
+ defer server.Close()
+ call := func(input string) (int, string) {
+ t.Helper()
+ response, err := server.Client().Post(server.URL+"/api/v1/aggregates", "application/json", strings.NewReader(input))
+ require.NoError(t, err)
+ defer func() { require.NoError(t, response.Body.Close()) }()
+ body, err := io.ReadAll(response.Body)
+ require.NoError(t, err)
+ return response.StatusCode, string(body)
+ }
+ for _, input := range []string{
+ `null`, `[]`, `[{"manifest_sha256":"bad"}]`,
+ `[{"manifest_sha256":"` + strings.Repeat("a", 64) + `","evidence_sha256":null}]`,
+ strings.Repeat(" ", (1<<20)+1),
+ } {
+ status, body := call(input)
+ require.Equal(t, http.StatusBadRequest, status)
+ require.JSONEq(t, `{"error":"invalid_aggregate_request"}`, body)
+ }
+ require.Zero(t, calls.Load())
+ valid := `[{"manifest_sha256":"` + strings.Repeat("a", 64) + `"}]`
+ for _, input := range []string{
+ valid,
+ valid + strings.Repeat(" ", artifactmanifest.MaxManifestBytes-len(valid)),
+ } {
+ status, body := call(input)
+ require.Equal(t, http.StatusServiceUnavailable, status)
+ require.NotContains(t, body, "private")
+ require.JSONEq(t, `{"error":"aggregate_store_unavailable"}`, body)
+ }
+ require.EqualValues(t, 2, calls.Load(), "the exact existing byte limit remains admitted")
+}
diff --git a/internal/webapi/checkpoint_browser_fixture_test.go b/internal/webapi/checkpoint_browser_fixture_test.go
new file mode 100644
index 000000000..9dee8e163
--- /dev/null
+++ b/internal/webapi/checkpoint_browser_fixture_test.go
@@ -0,0 +1,263 @@
+package webapi
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "errors"
+ "io"
+ "net/http"
+ "net/http/httptest"
+ "os"
+ "path/filepath"
+ "strconv"
+ "strings"
+ "sync/atomic"
+ "testing"
+
+ "github.com/microsoft/waza/internal/artifactmanifest"
+ "github.com/microsoft/waza/internal/execution"
+ "github.com/microsoft/waza/internal/executioncheckpoint"
+ "github.com/microsoft/waza/internal/models"
+ "github.com/microsoft/waza/internal/runnercheckpoint"
+ "github.com/microsoft/waza/internal/runnercontract"
+ "github.com/microsoft/waza/internal/runnerunit"
+ "github.com/stretchr/testify/require"
+)
+
+type checkpointBrowserFixture struct {
+ Path string `json:"path"`
+ Request json.RawMessage `json:"request"`
+ Admission json.RawMessage `json:"admission"`
+ HTTPStatus int `json:"http_status"`
+ Response string `json:"response"`
+}
+
+func TestCheckpointBrowserFixturesHaveActualProtectedHTTPProducers(t *testing.T) {
+ directory := t.TempDir()
+ var revision atomic.Pointer[string]
+ var failFinal atomic.Bool
+ var failBefore atomic.Bool
+ var factories atomic.Int64
+ initialRevision, changedRevision := strings.Repeat("c", 40), strings.Repeat("d", 40)
+ revision.Store(&initialRevision)
+ mux := http.NewServeMux()
+ service := runnerunit.NewService(func(config models.Config) (execution.AgentEngine, error) {
+ factories.Add(1)
+ return execution.NewMockEngine(config.ModelID), nil
+ })
+ RegisterRunnerRoutesWithService(mux, service)
+ registerCheckpointOperationRoutes(mux, checkpointOperations{
+ open: func() (artifactmanifest.Backend, func() error, error) {
+ return artifactmanifest.OpenBackend(directory, "")
+ },
+ runner: service, revision: func() (string, error) { return *revision.Load(), nil },
+ observers: runnercheckpoint.Hooks{Published: func(_ context.Context, reference runnercheckpoint.Reference) error {
+ if failBefore.Load() && reference.Boundary == executioncheckpoint.BeforeUnit {
+ return errors.New("fixture stops the original worker before dispatch")
+ }
+ if failFinal.Load() && reference.Boundary == executioncheckpoint.FinalizedUnit {
+ return errors.New("fixture rejects finalized progress delivery after actual completed execution")
+ }
+ return nil
+ }},
+ })
+ server := httptest.NewServer(mux)
+ defer server.Close()
+ post := func(path string, wire []byte, expectedStatus int) []byte {
+ response, err := server.Client().Post(server.URL+path, "application/json", bytes.NewReader(wire))
+ require.NoError(t, err)
+ body, err := io.ReadAll(response.Body)
+ require.NoError(t, err)
+ require.NoError(t, response.Body.Close())
+ require.Equal(t, expectedStatus, response.StatusCode, "%s", body)
+ if strings.HasPrefix(path, "/api/v1/checkpoints") {
+ result, err := runnercheckpoint.ParseResult(body)
+ require.NoError(t, err)
+ require.Equal(t, strconv.Itoa(result.ExitCode), response.Header.Get("X-Waza-Exit-Code"))
+ }
+ return body
+ }
+ fixtures := map[string]checkpointBrowserFixture{}
+ record := func(name, path string, wire []byte, status int) runnercheckpoint.Result {
+ body := post(path, wire, status)
+ result, err := runnercheckpoint.ParseResult(body)
+ require.NoError(t, err)
+ require.Equal(t, 1, bytes.Count(body, []byte{'\n'}))
+ request, err := runnerunit.ParseSubmission(wire)
+ require.NoError(t, err)
+ admission := post("/api/v1/runner/validate", request.Request, http.StatusOK)
+ fixtures[name] = checkpointBrowserFixture{
+ Path: path, Request: wire, Admission: admission, HTTPStatus: status, Response: string(body),
+ }
+ return *result
+ }
+ wire := checkpointOperationWire(t)
+ success := record("created", "/api/v1/checkpoints", wire, http.StatusOK)
+ require.Equal(t, "completed", success.Status)
+ require.Equal(t, 1, success.Submission.EngineCalls)
+
+ unicodeSubmission, err := runnerunit.ParseSubmission(wire)
+ require.NoError(t, err)
+ unicodeRequest, err := runnercontract.Parse(unicodeSubmission.Request)
+ require.NoError(t, err)
+ unicodeRequest.Input.Eval += "graders:\n" +
+ " - name: \"<>&\\u2028\\u2029\"\n type: text\n config:\n contains: [hello]\n" +
+ " - name: \"\\uE000\"\n type: text\n config:\n contains: [hello]\n" +
+ " - name: \"\\U00010000\"\n type: text\n config:\n contains: [hello]\n"
+ unicodeSubmission.Request, err = json.Marshal(unicodeRequest)
+ require.NoError(t, err)
+ unicodeWire, err := json.Marshal(unicodeSubmission)
+ require.NoError(t, err)
+ require.NotEqual(t, wire, unicodeWire)
+ _, err = runnerunit.ParseSubmission(unicodeWire)
+ require.NoError(t, err)
+ unicodeResult := record("unicode", "/api/v1/checkpoints", unicodeWire, http.StatusOK)
+ for _, identifier := range []string{"<>&\u2028\u2029", "\uE000", "\U00010000"} {
+ require.Contains(t, unicodeResult.Submission.Reservation.Binding.GraderIDs, identifier)
+ }
+
+ failedWire := bytes.Replace(wire, []byte("- hello"), []byte("- missing-browser-control"), 1)
+ require.NotEqual(t, wire, failedWire)
+ failed := record("failed", "/api/v1/checkpoints", failedWire, http.StatusOK)
+ require.Equal(t, 1, failed.ExitCode)
+ destination := bytes.Replace(failedWire, []byte(`"attempt":2`), []byte(`"attempt":3`), 1)
+ require.NotEqual(t, failedWire, destination)
+ reused := record("reused", "/api/v1/checkpoints/"+failed.FinalizedDigest+"/resume", destination, http.StatusOK)
+ require.Equal(t, "reused", reused.Status)
+ require.Zero(t, reused.Submission.EngineCalls)
+ require.Equal(t, 1, reused.ExitCode)
+ require.Equal(t, failed.Submission.Terminal, reused.Submission.Terminal)
+
+ // Fail the admitted concrete mock's actual workspace creation, not runtime
+ // admission or a synthetic terminal. Restore every platform's temp setting.
+ var restoreTemp []func()
+ for _, name := range []string{"TMPDIR", "TMP", "TEMP"} {
+ value, present := os.LookupEnv(name)
+ t.Setenv(name, filepath.Join(directory, "absent-runtime-temp"))
+ restoreTemp = append(restoreTemp, func() {
+ if present {
+ require.NoError(t, os.Setenv(name, value))
+ } else {
+ require.NoError(t, os.Unsetenv(name))
+ }
+ })
+ }
+ executionFailure := record("execution_error", "/api/v1/checkpoints", wire, http.StatusOK)
+ for _, restore := range restoreTemp {
+ restore()
+ }
+ require.Equal(t, "completed", executionFailure.Status)
+ require.Equal(t, 2, executionFailure.ExitCode)
+ require.Equal(t, "error", executionFailure.Submission.Terminal.Observation.Status)
+ require.Empty(t, executionFailure.Submission.Terminal.Observation.Validations)
+ calls := factories.Load()
+ errorDestination := bytes.Replace(wire, []byte(`"attempt":2`), []byte(`"attempt":3`), 1)
+ require.NotEqual(t, wire, errorDestination)
+ reusedError := record("reused_error", "/api/v1/checkpoints/"+executionFailure.FinalizedDigest+"/resume", errorDestination, http.StatusOK)
+ require.Equal(t, "reused", reusedError.Status)
+ require.Equal(t, 2, reusedError.ExitCode)
+ require.Zero(t, reusedError.Submission.EngineCalls)
+ require.Equal(t, calls, factories.Load())
+ require.Equal(t, executionFailure.Submission.Terminal, reusedError.Submission.Terminal)
+
+ failBefore.Store(true)
+ paused, err := runnercheckpoint.ParseResult(post("/api/v1/checkpoints", wire, http.StatusInternalServerError))
+ require.NoError(t, err)
+ require.NotEmpty(t, paused.BeforeDigest)
+ require.Zero(t, paused.Submission.EngineCalls)
+ require.Nil(t, paused.Submission.Completion)
+ failBefore.Store(false)
+ retryWire := bytes.Replace(wire, []byte(`"attempt":2`), []byte(`"attempt":3`), 1)
+ require.NotEqual(t, wire, retryWire)
+ retried := record("retried", "/api/v1/checkpoints/"+paused.BeforeDigest+"/resume", retryWire, http.StatusOK)
+ require.Equal(t, "completed", retried.Status)
+ require.Equal(t, executioncheckpoint.BeforeUnit, retried.Source.Boundary)
+ require.NotEqual(t, paused.Submission.Reservation.ExecutionID, retried.Submission.Reservation.ExecutionID)
+ require.Equal(t, paused.Submission.Reservation.ExecutionID, retried.Submission.Reservation.Parent.ExecutionID)
+ require.Equal(t, "retry", retried.Submission.Reservation.Parent.Relationship)
+ require.Equal(t, 1, retried.Submission.EngineCalls)
+ require.Equal(t, 3, retried.Submission.Terminal.Selection.Attempt)
+
+ revision.Store(&changedRevision)
+ incompatible := record("incompatible", "/api/v1/checkpoints/"+failed.FinalizedDigest+"/resume", destination, http.StatusConflict)
+ require.True(t, incompatible.SourceWithheld)
+ require.Nil(t, incompatible.Source)
+ require.Nil(t, incompatible.RestoredUnits)
+ require.Nil(t, incompatible.RemainingWork)
+ revision.Store(&initialRevision)
+ stale := record("stale", "/api/v1/checkpoints/"+success.BeforeDigest+"/resume", wire, http.StatusConflict)
+ require.True(t, stale.SourceWithheld)
+ require.Nil(t, stale.Submission)
+ failFinal.Store(true)
+ publicationFailure := record("checkpoint_error", "/api/v1/checkpoints", wire, http.StatusInternalServerError)
+ require.Equal(t, "checkpoint_error", publicationFailure.Status)
+ require.Equal(t, 2, publicationFailure.ExitCode)
+ require.Equal(t, "completed", publicationFailure.Submission.Status)
+ require.Zero(t, publicationFailure.Submission.ExitCode)
+ require.NotNil(t, publicationFailure.Submission.Completion)
+
+ path := filepath.Join("..", "..", "web", "e2e", "fixtures", "checkpoint-operations.json")
+ if os.Getenv("UPDATE_CHECKPOINT_BROWSER_FIXTURES") == "1" {
+ data, err := json.MarshalIndent(fixtures, "", " ")
+ require.NoError(t, err)
+ require.NoError(t, os.WriteFile(path, append(data, '\n'), 0o600))
+ }
+ data, err := os.ReadFile(path)
+ require.NoError(t, err, "generate actual HTTP fixtures with UPDATE_CHECKPOINT_BROWSER_FIXTURES=1")
+ var saved map[string]checkpointBrowserFixture
+ require.NoError(t, json.Unmarshal(data, &saved))
+ require.Len(t, saved, len(fixtures))
+ for name, current := range fixtures {
+ t.Run(name, func(t *testing.T) {
+ original, exists := saved[name]
+ require.True(t, exists)
+ require.Equal(t, strings.HasSuffix(current.Path, "/resume"), strings.HasSuffix(original.Path, "/resume"))
+ require.JSONEq(t, string(current.Request), string(original.Request))
+ require.JSONEq(t, string(current.Admission), string(original.Admission))
+ require.Equal(t, current.HTTPStatus, original.HTTPStatus)
+ actual, err := runnercheckpoint.ParseResult([]byte(current.Response))
+ require.NoError(t, err)
+ expected, err := runnercheckpoint.ParseResult([]byte(original.Response))
+ require.NoError(t, err)
+ require.Equal(t, checkpointBrowserValues(*expected), checkpointBrowserValues(*actual))
+ })
+ }
+}
+
+// IDs, content-addressed references and creation times vary with fresh owned
+// reservations. Strict codecs admit each unmodified response before comparison;
+// this projection compares rendered presence, original terminal and diagnostics,
+// not normalized objects presented as verified execution evidence.
+func checkpointBrowserValues(result runnercheckpoint.Result) map[string]any {
+ values := map[string]any{
+ "status": result.Status, "exit": result.ExitCode,
+ "withheld": result.SourceWithheld, "source": result.Source != nil,
+ "compatibility": result.CompatibilityResult, "differences": result.CompatibilityDifferences,
+ "restored": result.RestoredUnits, "remaining": result.RemainingWork, "diagnostics": result.Diagnostics,
+ "before": result.BeforeDigest != "", "finalized": result.FinalizedDigest != "",
+ "submission": result.Submission != nil,
+ }
+ if result.Source != nil {
+ values["boundary"] = result.Source.Boundary
+ }
+ if submission := result.Submission; submission != nil {
+ values["calls"] = submission.EngineCalls
+ values["terminal"] = submission.Terminal
+ values["report"] = len(submission.Report)
+ values["submission_diagnostics"] = submission.Diagnostics
+ values["caller"] = submission.Caller
+ values["reservation"] = submission.Reservation != nil
+ values["completion"] = submission.Completion != nil
+ values["decision"] = submission.Decision != nil
+ if submission.Reservation != nil {
+ values["selection"] = submission.Reservation.Binding.Selection
+ values["parent"] = submission.Reservation.Parent != nil
+ }
+ if submission.Decision != nil {
+ values["policy"] = submission.Decision.Policy
+ values["decision_outcome"] = submission.Decision.Outcome
+ }
+ }
+ return values
+}
diff --git a/internal/webapi/checkpoint_operations.go b/internal/webapi/checkpoint_operations.go
new file mode 100644
index 000000000..c0e4ab085
--- /dev/null
+++ b/internal/webapi/checkpoint_operations.go
@@ -0,0 +1,115 @@
+package webapi
+
+import (
+ "errors"
+ "fmt"
+ "net/http"
+
+ "github.com/microsoft/waza/internal/artifactmanifest"
+ "github.com/microsoft/waza/internal/executioncheckpoint"
+ "github.com/microsoft/waza/internal/projectconfig"
+ "github.com/microsoft/waza/internal/runnercheckpoint"
+ "github.com/microsoft/waza/internal/runnerunit"
+)
+
+type checkpointOperations struct {
+ open func() (artifactmanifest.Backend, func() error, error)
+ runner *runnerunit.Service
+ revision func() (string, error)
+ observers runnercheckpoint.Hooks
+}
+
+func RegisterCheckpointOperationRoutes(mux *http.ServeMux, resultsDir string, config *projectconfig.StorageConfig) {
+ registerCheckpointOperationRoutes(mux, checkpointOperations{
+ open: checkpointBackendFor(resultsDir, config),
+ runner: runnerunit.NewService(nil), revision: executioncheckpoint.CurrentRunnerRevision,
+ })
+}
+
+func checkpointBackendFor(resultsDir string, config *projectconfig.StorageConfig) func() (artifactmanifest.Backend, func() error, error) {
+ provider := manifestInspectorFor(resultsDir, config)
+ return func() (artifactmanifest.Backend, func() error, error) {
+ if config != nil && config.Enabled && config.Provider != "" &&
+ config.Provider != "local" && config.Provider != "azure-blob" {
+ return nil, nil, fmt.Errorf("checkpoint operations do not support the configured active provider")
+ }
+ return provider.openBackend()
+ }
+}
+
+func registerCheckpointOperationRoutes(mux *http.ServeMux, operations checkpointOperations) {
+ for _, action := range []string{"create", "resume"} {
+ path := "POST /api/v1/checkpoints"
+ if action == "resume" {
+ path = "POST /api/v1/checkpoints/{digest}/resume"
+ }
+ mux.HandleFunc(path, func(w http.ResponseWriter, r *http.Request) {
+ inputComplete := false
+ emit := func(status int, result runnercheckpoint.Result) {
+ data, err := runnercheckpoint.EncodeResult(result)
+ if err != nil {
+ logRunnerError("validating checkpoint operation output; aborting without disclosure", "error", err)
+ panic(http.ErrAbortHandler)
+ }
+ if err := WriteRunnerOperationJSON(w, status, result.ExitCode, data, inputComplete); err != nil {
+ logRunnerError("writing checkpoint operation output; no retry", "error", err)
+ }
+ }
+ data, inputErr := ReadRunnerOperationInput(w, r, runnerunit.MaxSubmissionBytes)
+ if inputErr != nil {
+ logRunnerError("checkpoint input admission failed", "error", inputErr.Err)
+ emit(inputErr.Status, runnercheckpoint.RejectPublic(inputErr.Code))
+ return
+ }
+ inputComplete = true
+ if action == "resume" {
+ if err := executioncheckpoint.ValidateDigest(r.PathValue("digest")); err != nil {
+ emit(http.StatusBadRequest, runnercheckpoint.RejectPublic("checkpoint_digest_invalid"))
+ return
+ }
+ }
+ if _, err := runnerunit.ParseSubmission(data); err != nil {
+ emit(http.StatusBadRequest, runnercheckpoint.RejectPublic("checkpoint_input_invalid"))
+ return
+ }
+ revision, err := operations.revision()
+ if err != nil {
+ logRunnerError("checkpoint build admission failed", "error", err)
+ emit(http.StatusServiceUnavailable, runnercheckpoint.RejectPublic("checkpoint_build_incompatible"))
+ return
+ }
+ checkpoints, err := runnercheckpoint.NewService(operations.runner, revision)
+ if err != nil {
+ logRunnerError("checkpoint runner unavailable", "error", err)
+ emit(http.StatusServiceUnavailable, runnercheckpoint.RejectPublic("checkpoint_runner_unavailable"))
+ return
+ }
+ backend, closeBackend, err := operations.open()
+ if err != nil {
+ logRunnerError("checkpoint immutable storage unavailable", "error", err)
+ emit(http.StatusServiceUnavailable, runnercheckpoint.RejectPublic("checkpoint_storage_unavailable"))
+ return
+ }
+ var result runnercheckpoint.Result
+ if action == "resume" {
+ result, err = checkpoints.ResumeCaptured(r.Context(), data, backend, r.PathValue("digest"), operations.observers)
+ } else {
+ result, err = checkpoints.SubmitCaptured(r.Context(), data, backend, operations.observers)
+ }
+ if closeErr := closeBackend(); closeErr != nil {
+ logRunnerError("checkpoint immutable store close failed; result not delivered", "error", errors.Join(err, closeErr))
+ emit(http.StatusServiceUnavailable, runnercheckpoint.RejectPublic("checkpoint_storage_close_failed"))
+ return
+ }
+ status := http.StatusOK
+ if err != nil {
+ logRunnerError("checkpoint operation rejected or failed", "error", err)
+ status = http.StatusConflict
+ if result.Status == "submission_error" || result.Status == "checkpoint_error" {
+ status = http.StatusInternalServerError
+ }
+ }
+ emit(status, result)
+ })
+ }
+}
diff --git a/internal/webapi/checkpoint_operations_test.go b/internal/webapi/checkpoint_operations_test.go
new file mode 100644
index 000000000..4816e0583
--- /dev/null
+++ b/internal/webapi/checkpoint_operations_test.go
@@ -0,0 +1,321 @@
+package webapi
+
+import (
+ "bytes"
+ "encoding/json"
+ "errors"
+ "io"
+ "net/http"
+ "net/http/httptest"
+ "os"
+ "path/filepath"
+ "strconv"
+ "strings"
+ "sync/atomic"
+ "testing"
+
+ "github.com/microsoft/waza/internal/artifactmanifest"
+ "github.com/microsoft/waza/internal/execution"
+ "github.com/microsoft/waza/internal/executionattempt"
+ "github.com/microsoft/waza/internal/executioncheckpoint"
+ "github.com/microsoft/waza/internal/executionledger"
+ "github.com/microsoft/waza/internal/models"
+ "github.com/microsoft/waza/internal/projectconfig"
+ "github.com/microsoft/waza/internal/runnercheckpoint"
+ "github.com/microsoft/waza/internal/runnerunit"
+ "github.com/stretchr/testify/require"
+)
+
+func TestCheckpointConfiguredBackendSharesManifestStore(t *testing.T) {
+ for _, control := range []struct {
+ name string
+ config *projectconfig.StorageConfig
+ deny bool
+ }{
+ {"default-local", nil, false},
+ {"enabled-local", &projectconfig.StorageConfig{Enabled: true, Provider: "local"}, false},
+ {"disabled-foreign", &projectconfig.StorageConfig{Provider: "unsupported"}, false},
+ {"enabled-foreign", &projectconfig.StorageConfig{Enabled: true, Provider: "unsupported"}, true},
+ } {
+ t.Run(control.name, func(t *testing.T) {
+ directory := t.TempDir()
+ original := t.TempDir()
+ require.NoError(t, os.Mkdir(filepath.Join(directory, ".waza-artifacts"), 0o700))
+ t.Chdir(original)
+ backend, closeBackend, err := checkpointBackendFor(directory, control.config)()
+ if control.deny {
+ require.Error(t, err)
+ require.Nil(t, backend)
+ require.Nil(t, closeBackend)
+ return
+ }
+ require.NoError(t, err)
+ data := []byte("configured-checkpoint-store-control")
+ digest := artifactmanifest.Hash(data)
+ created, err := backend.Create(t.Context(), "blob-"+digest, data)
+ require.NoError(t, err)
+ require.True(t, created)
+ require.NoError(t, closeBackend())
+ physical, err := os.ReadFile(filepath.Join(directory, ".waza-artifacts", "blob-"+digest))
+ require.NoError(t, err)
+ require.Equal(t, data, physical)
+ require.NoDirExists(t, filepath.Join(original, ".waza-artifacts"))
+ })
+ }
+}
+
+func checkpointOperationWire(t *testing.T) []byte {
+ t.Helper()
+ raw, err := os.ReadFile("../../examples/runner/unit.json")
+ require.NoError(t, err)
+ wire, err := json.Marshal(runnerunit.SubmissionRequest{
+ Kind: "waza.execution-submission", SchemaVersion: "1.0", Request: raw,
+ })
+ require.NoError(t, err)
+ return wire
+}
+
+func postCheckpointOperation(t *testing.T, server *httptest.Server, path string, wire []byte, status int) runnercheckpoint.Result {
+ t.Helper()
+ response, err := server.Client().Post(server.URL+path, "application/json", bytes.NewReader(wire))
+ require.NoError(t, err)
+ body, err := io.ReadAll(response.Body)
+ require.NoError(t, err)
+ require.NoError(t, response.Body.Close())
+ require.Equal(t, status, response.StatusCode, "%s", body)
+ require.Equal(t, 1, bytes.Count(body, []byte{'\n'}))
+ result, err := runnercheckpoint.ParseResult(body)
+ require.NoError(t, err, "%s", body)
+ require.Equal(t, strconv.Itoa(result.ExitCode), response.Header.Get("X-Waza-Exit-Code"))
+ return *result
+}
+
+func TestCheckpointOperationActualHTTPProducerAndFailedRestore(t *testing.T) {
+ directory := t.TempDir()
+ var factories atomic.Int32
+ var revision atomic.Pointer[string]
+ initialRevision, changedRevision := strings.Repeat("c", 40), strings.Repeat("d", 40)
+ revision.Store(&initialRevision)
+ service := runnerunit.NewService(func(config models.Config) (execution.AgentEngine, error) {
+ factories.Add(1)
+ return execution.NewMockEngine(config.ModelID), nil
+ })
+ mux := http.NewServeMux()
+ registerCheckpointOperationRoutes(mux, checkpointOperations{
+ open: func() (artifactmanifest.Backend, func() error, error) {
+ return artifactmanifest.OpenBackend(directory, "")
+ },
+ runner: service, revision: func() (string, error) { return *revision.Load(), nil },
+ })
+ server := httptest.NewServer(mux)
+ defer server.Close()
+ wire := checkpointOperationWire(t)
+ metadata := "http-retained-checkpoint-metadata-control"
+ changed := bytes.Replace(wire, []byte("id: selected"), []byte("id: "+metadata), 1)
+ require.NotEqual(t, wire, changed)
+ wire = changed
+ changed = bytes.Replace(wire, []byte(`"task":"selected"`), []byte(`"task":"`+metadata+`"`), 1)
+ require.NotEqual(t, wire, changed)
+ wire = changed
+ created := postCheckpointOperation(t, server, "/api/v1/checkpoints", wire, http.StatusOK)
+ require.NotNil(t, created.Submission.Completion)
+ require.Equal(t, int32(1), factories.Load())
+ require.Empty(t, created.Submission.Report)
+ original, err := json.Marshal(created.Submission.Terminal)
+ require.NoError(t, err)
+ destination := bytes.Replace(wire, []byte(`"attempt":2`), []byte(`"attempt":3`), 1)
+ require.NotEqual(t, wire, destination)
+ restored := postCheckpointOperation(t, server, "/api/v1/checkpoints/"+created.FinalizedDigest+"/resume", destination, http.StatusOK)
+ require.Equal(t, "reused", restored.Status)
+ require.Zero(t, restored.Submission.EngineCalls)
+ require.Equal(t, int32(1), factories.Load())
+ require.Equal(t, 2, restored.Submission.Terminal.Selection.Attempt)
+ terminal, err := json.Marshal(restored.Submission.Terminal)
+ require.NoError(t, err)
+ require.Equal(t, original, terminal)
+ stale := postCheckpointOperation(t, server, "/api/v1/checkpoints/"+created.BeforeDigest+"/resume", wire, http.StatusConflict)
+ require.True(t, stale.SourceWithheld)
+ require.Nil(t, stale.Source)
+ require.Nil(t, stale.RestoredUnits)
+ revision.Store(&changedRevision)
+ incompatible := postCheckpointOperation(t, server, "/api/v1/checkpoints/"+created.FinalizedDigest+"/resume", wire, http.StatusConflict)
+ require.Equal(t, "incompatible", incompatible.CompatibilityResult)
+ require.Equal(t, []string{"runner_revision"}, incompatible.CompatibilityDifferences)
+ require.True(t, incompatible.SourceWithheld)
+ require.Nil(t, incompatible.Source)
+ encoded, err := json.Marshal(incompatible)
+ require.NoError(t, err)
+ require.NotContains(t, string(encoded), metadata)
+ require.Equal(t, int32(1), factories.Load())
+}
+
+func TestCheckpointOperationHTTPDeniedReuseWithholdsSourceAndPreservesOriginal(t *testing.T) {
+ directory := t.TempDir()
+ var factories atomic.Int32
+ service := runnerunit.NewService(func(config models.Config) (execution.AgentEngine, error) {
+ factories.Add(1)
+ return execution.NewMockEngine(config.ModelID), nil
+ })
+ revision := strings.Repeat("c", 40)
+ backend, closeBackend, err := artifactmanifest.OpenBackend(directory, "")
+ require.NoError(t, err)
+ legacy, err := runnercheckpoint.NewService(service, revision)
+ require.NoError(t, err)
+ const private = "original-http-private-terminal-control"
+ input := checkpointOperationWire(t)
+ wire := bytes.Replace(input, []byte("- hello"), []byte("- "+private), 1)
+ require.NotEqual(t, input, wire)
+ for _, replacement := range [][2]string{
+ {"id: selected", "id: " + private},
+ {`"task":"selected"`, `"task":"` + private + `"`},
+ } {
+ changed := bytes.Replace(wire, []byte(replacement[0]), []byte(replacement[1]), 1)
+ require.NotEqual(t, wire, changed)
+ wire = changed
+ }
+ original, err := legacy.Submit(t.Context(), wire, backend, runnercheckpoint.Hooks{})
+ require.NoError(t, err)
+ require.NotEmpty(t, original.FinalizedDigest)
+ require.NotEmpty(t, original.Submission.Terminal.Diagnostics)
+ require.NotEqual(t, "[REDACTED]", original.Submission.Terminal.Diagnostics[0].Message)
+ owner, err := executionledger.New(backend)
+ require.NoError(t, err)
+ source := executionattempt.Source{
+ ExecutionID: original.Submission.Reservation.ExecutionID, ManifestDigest: original.Submission.Completion.ManifestDigest,
+ }
+ before, err := owner.ReadCompleted(t.Context(), source)
+ require.NoError(t, err)
+ var names []string
+ for _, artifact := range before.Publication.Manifest.Artifacts {
+ names = append(names, artifact.Name)
+ }
+ require.NotEmpty(t, names)
+ before, err = owner.ReadCompleted(t.Context(), source, names...)
+ require.NoError(t, err)
+ require.Len(t, before.Publication.ArtifactBytes, len(names))
+ require.Contains(t, string(before.Publication.ArtifactBytes["terminal.json"]), private)
+ checkpointBefore, err := backend.Get(t.Context(), "blob-"+original.FinalizedDigest, executioncheckpoint.MaxBytes)
+ require.NoError(t, err)
+ require.NoError(t, closeBackend())
+
+ mux := http.NewServeMux()
+ registerCheckpointOperationRoutes(mux, checkpointOperations{
+ open: func() (artifactmanifest.Backend, func() error, error) {
+ return artifactmanifest.OpenBackend(directory, "")
+ },
+ runner: service, revision: func() (string, error) { return revision, nil },
+ })
+ server := httptest.NewServer(mux)
+ defer server.Close()
+ denied := postCheckpointOperation(t, server, "/api/v1/checkpoints/"+original.FinalizedDigest+"/resume", wire, http.StatusInternalServerError)
+ require.Equal(t, "submission_error", denied.Status)
+ require.Equal(t, 2, denied.ExitCode)
+ require.True(t, denied.SourceWithheld)
+ require.Nil(t, denied.Source)
+ require.Nil(t, denied.RestoredUnits)
+ require.Nil(t, denied.RemainingWork)
+ require.NotNil(t, denied.Submission)
+ require.Zero(t, denied.Submission.EngineCalls)
+ require.Empty(t, denied.Submission.Report)
+ require.Nil(t, denied.Submission.Terminal)
+ require.Nil(t, denied.Submission.Completion)
+ require.Nil(t, denied.Submission.Reservation)
+ public, err := json.Marshal(denied)
+ require.NoError(t, err)
+ require.NotContains(t, string(public), private)
+ require.Equal(t, int32(1), factories.Load())
+
+ backend, closeBackend, err = artifactmanifest.OpenBackend(directory, "")
+ require.NoError(t, err)
+ defer func() { require.NoError(t, closeBackend()) }()
+ owner, err = executionledger.New(backend)
+ require.NoError(t, err)
+ after, err := owner.ReadCompleted(t.Context(), source, names...)
+ require.NoError(t, err)
+ require.Equal(t, before.Publication.ArtifactBytes, after.Publication.ArtifactBytes)
+ require.Equal(t, before.Publication.ManifestBytes, after.Publication.ManifestBytes)
+ require.Equal(t, before.Reservation.Bytes, after.Reservation.Bytes)
+ require.Equal(t, before.CompletionBytes, after.CompletionBytes)
+ checkpointAfter, err := backend.Get(t.Context(), "blob-"+original.FinalizedDigest, executioncheckpoint.MaxBytes)
+ require.NoError(t, err)
+ require.Equal(t, checkpointBefore, checkpointAfter)
+}
+
+func TestCheckpointOperationHTTPAdmissionPrecedesBuildStorageAndExecution(t *testing.T) {
+ var opens, revisions, factories atomic.Int32
+ mux := http.NewServeMux()
+ registerCheckpointOperationRoutes(mux, checkpointOperations{
+ open: func() (artifactmanifest.Backend, func() error, error) {
+ opens.Add(1)
+ return nil, nil, errors.New("private-provider-control")
+ },
+ runner: runnerunit.NewService(func(models.Config) (execution.AgentEngine, error) {
+ factories.Add(1)
+ return nil, errors.New("must not construct")
+ }),
+ revision: func() (string, error) {
+ revisions.Add(1)
+ return "", errors.New("private-build-control")
+ },
+ })
+ server := httptest.NewServer(mux)
+ defer server.Close()
+ postCheckpointOperation(t, server, "/api/v1/checkpoints", []byte("{}"), http.StatusBadRequest)
+ postCheckpointOperation(t, server, "/api/v1/checkpoints/"+strings.Repeat("A", 64)+"/resume",
+ checkpointOperationWire(t), http.StatusBadRequest)
+ postCheckpointOperation(t, server, "/api/v1/checkpoints",
+ bytes.Repeat([]byte(" "), runnerunit.MaxSubmissionBytes+1), http.StatusRequestEntityTooLarge)
+ require.Zero(t, revisions.Load())
+ result := postCheckpointOperation(t, server, "/api/v1/checkpoints", checkpointOperationWire(t), http.StatusServiceUnavailable)
+ require.Equal(t, "checkpoint_build_incompatible", result.Diagnostics[0].Code)
+ require.Equal(t, int32(1), revisions.Load())
+ require.Zero(t, opens.Load())
+ require.Zero(t, factories.Load())
+}
+
+func TestCheckpointOperationHTTPTrustDenialPrecedesInputAndStorage(t *testing.T) {
+ var opens, revisions atomic.Int32
+ mux := http.NewServeMux()
+ registerCheckpointOperationRoutes(mux, checkpointOperations{
+ open: func() (artifactmanifest.Backend, func() error, error) {
+ opens.Add(1)
+ return nil, nil, errors.New("must not open")
+ },
+ runner: runnerunit.NewService(nil),
+ revision: func() (string, error) {
+ revisions.Add(1)
+ return strings.Repeat("c", 40), nil
+ },
+ })
+ server := httptest.NewServer(mux)
+ defer server.Close()
+ for _, control := range []string{"foreign-host", "foreign-origin", "cross-site"} {
+ t.Run(control, func(t *testing.T) {
+ request, err := http.NewRequestWithContext(t.Context(), http.MethodPost, server.URL+"/api/v1/checkpoints",
+ bytes.NewReader(checkpointOperationWire(t)))
+ require.NoError(t, err)
+ request.Header.Set("Content-Type", "application/json")
+ switch control {
+ case "foreign-host":
+ request.Host = "example.invalid"
+ case "foreign-origin":
+ request.Header.Set("Origin", "https://example.invalid")
+ case "cross-site":
+ request.Header.Set("Sec-Fetch-Site", "cross-site")
+ }
+ response, err := server.Client().Do(request)
+ require.NoError(t, err)
+ body, err := io.ReadAll(response.Body)
+ require.NoError(t, err)
+ require.NoError(t, response.Body.Close())
+ require.Equal(t, http.StatusForbidden, response.StatusCode, "%s", body)
+ result, err := runnercheckpoint.ParseResult(body)
+ require.NoError(t, err)
+ require.Equal(t, "operation_http_denied", result.Diagnostics[0].Code)
+ require.Nil(t, result.Submission)
+ require.Equal(t, "6", response.Header.Get("X-Waza-Exit-Code"))
+ })
+ }
+ require.Zero(t, opens.Load())
+ require.Zero(t, revisions.Load())
+}
diff --git a/internal/webapi/manifests.go b/internal/webapi/manifests.go
index a350f5d2d..ae6c4d2c6 100644
--- a/internal/webapi/manifests.go
+++ b/internal/webapi/manifests.go
@@ -63,17 +63,21 @@ type configuredManifestInspector struct {
}
func (i configuredManifestInspector) open() (*artifactmanifest.Store, func() error, error) {
- var azureURL string
- if i.config != nil && i.config.Enabled && i.config.Provider == "azure-blob" {
- azureURL = "https://" + i.config.AccountName + ".blob.core.windows.net/" + i.config.ContainerName
- }
- backend, closeBackend, err := artifactmanifest.OpenBackend(i.directory, azureURL)
+ backend, closeBackend, err := i.openBackend()
if err != nil {
return nil, nil, err
}
return artifactmanifest.NewStore(backend), closeBackend, nil
}
+func (i configuredManifestInspector) openBackend() (artifactmanifest.Backend, func() error, error) {
+ var azureURL string
+ if i.config != nil && i.config.Enabled && i.config.Provider == "azure-blob" {
+ azureURL = "https://" + i.config.AccountName + ".blob.core.windows.net/" + i.config.ContainerName
+ }
+ return artifactmanifest.OpenBackend(i.directory, azureURL)
+}
+
func (i configuredManifestInspector) Catalog(ctx context.Context) (result artifactmanifest.Catalog, resultErr error) {
store, closeStore, err := i.open()
if err != nil {
@@ -128,7 +132,9 @@ func (i configuredManifestInspector) Inspect(ctx context.Context, digest string)
// The dashboard's configured results directory selects a separate opt-in store.
// Query input can select only a digest, never a host path or provider URL.
func RegisterManifestRoutes(mux *http.ServeMux, resultsDir string, config *projectconfig.StorageConfig) {
- RegisterManifestRoutesWithInspector(mux, configuredManifestInspector{
- directory: filepath.Join(resultsDir, ".waza-artifacts"), config: config,
- })
+ RegisterManifestRoutesWithInspector(mux, manifestInspectorFor(resultsDir, config))
+}
+
+func manifestInspectorFor(resultsDir string, config *projectconfig.StorageConfig) configuredManifestInspector {
+ return configuredManifestInspector{directory: filepath.Join(resultsDir, ".waza-artifacts"), config: config}
}
diff --git a/internal/webapi/runner_operation_transport.go b/internal/webapi/runner_operation_transport.go
new file mode 100644
index 000000000..aa855c089
--- /dev/null
+++ b/internal/webapi/runner_operation_transport.go
@@ -0,0 +1,116 @@
+package webapi
+
+import (
+ "bytes"
+ "context"
+ "encoding/json"
+ "errors"
+ "fmt"
+ "io"
+ "mime"
+ "net"
+ "net/http"
+ "strconv"
+ "sync/atomic"
+ "time"
+)
+
+type RunnerOperationError struct {
+ Status int
+ Code string
+ Err error
+}
+
+func (e *RunnerOperationError) Error() string { return e.Err.Error() }
+func (e *RunnerOperationError) Unwrap() error { return e.Err }
+
+// ReadRunnerOperationInput admits transport only; callers still apply the
+// strict request codec and capture policy before accessing immutable storage.
+func ReadRunnerOperationInput(w http.ResponseWriter, r *http.Request, maxBytes int64) ([]byte, *RunnerOperationError) {
+ if maxBytes <= 0 {
+ return nil, &RunnerOperationError{http.StatusInternalServerError, "input_limit", fmt.Errorf("runner operation input limit must be positive")}
+ }
+ if err := authorizeNativeRunnerHTTP(r); err != nil {
+ return nil, &RunnerOperationError{http.StatusForbidden, "operation_http_denied", err}
+ }
+ controller := http.NewResponseController(w)
+ deadline := time.Now().Add(5 * time.Second)
+ if err := controller.SetReadDeadline(deadline); err != nil {
+ return nil, &RunnerOperationError{http.StatusInternalServerError, "input_deadline", fmt.Errorf("setting runner read deadline: %w", err)}
+ }
+ if err := controller.SetWriteDeadline(time.Time{}); err != nil {
+ return nil, &RunnerOperationError{http.StatusInternalServerError, "output_deadline", fmt.Errorf("probing runner write deadline: %w", err)}
+ }
+ mediaType, _, err := mime.ParseMediaType(r.Header.Get("Content-Type"))
+ if err != nil || mediaType != "application/json" {
+ return nil, &RunnerOperationError{http.StatusUnsupportedMediaType, "invalid_media_type", fmt.Errorf("runner operation requires Content-Type: application/json")}
+ }
+ return readRunnerHTTPBody(w, r, maxBytes, deadline)
+}
+
+func readRunnerHTTPBody(w http.ResponseWriter, r *http.Request, maxBytes int64, deadline time.Time) ([]byte, *RunnerOperationError) {
+ controller := http.NewResponseController(w)
+ interrupted := make(chan struct{})
+ var canceledRead atomic.Bool
+ stop := context.AfterFunc(r.Context(), func() {
+ defer close(interrupted)
+ canceledRead.Store(time.Now().Before(deadline))
+ if err := controller.SetReadDeadline(time.Now()); err != nil {
+ logRunnerError("interrupting runner operation body admission", "error", err)
+ }
+ })
+ data, readErr := io.ReadAll(http.MaxBytesReader(w, r.Body, maxBytes))
+ if !stop() {
+ <-interrupted
+ }
+ if canceledRead.Load() {
+ return nil, &RunnerOperationError{http.StatusRequestTimeout, "input_canceled", fmt.Errorf("runner input canceled: %w", r.Context().Err())}
+ }
+ if timeout, ok := errors.AsType[net.Error](readErr); ok && timeout.Timeout() {
+ return nil, &RunnerOperationError{http.StatusRequestTimeout, "input_timeout", errors.Join(context.DeadlineExceeded, readErr)}
+ }
+ if err := r.Context().Err(); err != nil {
+ return nil, &RunnerOperationError{http.StatusRequestTimeout, "input_canceled", errors.Join(err, readErr)}
+ }
+ if readErr != nil {
+ if _, ok := errors.AsType[*http.MaxBytesError](readErr); ok {
+ return nil, &RunnerOperationError{http.StatusRequestEntityTooLarge, "request_too_large", readErr}
+ }
+ return nil, &RunnerOperationError{http.StatusBadRequest, "input_read", fmt.Errorf("reading runner request: %w", readErr)}
+ }
+ if err := controller.SetReadDeadline(time.Time{}); err != nil {
+ return nil, &RunnerOperationError{http.StatusInternalServerError, "input_deadline_reset", fmt.Errorf("resetting runner read deadline: %w", err)}
+ }
+ return data, nil
+}
+
+// WriteRunnerOperationJSON writes one capture-approved, caller-validated result
+// in the bounded delivery phase. It never retries or proves consumer receipt.
+func WriteRunnerOperationJSON(w http.ResponseWriter, status, exitCode int, data []byte, inputComplete bool) error {
+ if !json.Valid(data) {
+ return fmt.Errorf("runner operation output is not valid JSON")
+ }
+ if !inputComplete {
+ w.Header().Set("Connection", "close")
+ }
+ controller := http.NewResponseController(w)
+ if err := controller.SetWriteDeadline(time.Now().Add(5 * time.Second)); err != nil {
+ logRunnerError("setting runner operation write deadline; aborting transport without output", "error", err)
+ panic(http.ErrAbortHandler)
+ }
+ w.Header().Set("Content-Type", "application/json")
+ w.Header().Set("X-Waza-Exit-Code", strconv.Itoa(exitCode))
+ w.WriteHeader(status)
+ payload := append(bytes.Clone(bytes.TrimRight(data, " \t\r\n")), '\n')
+ written, err := w.Write(payload)
+ if err != nil {
+ return fmt.Errorf("writing runner operation output: %w", err)
+ }
+ if written != len(payload) {
+ return fmt.Errorf("writing runner operation output: %w", io.ErrShortWrite)
+ }
+ if err := controller.Flush(); err != nil {
+ return fmt.Errorf("flushing runner operation output: %w", err)
+ }
+ return nil
+}
diff --git a/internal/webserver/aggregate_test.go b/internal/webserver/aggregate_test.go
new file mode 100644
index 000000000..47832d3ca
--- /dev/null
+++ b/internal/webserver/aggregate_test.go
@@ -0,0 +1,142 @@
+package webserver
+
+import (
+ "bytes"
+ "encoding/json"
+ "flag"
+ "io"
+ "net/http"
+ "net/http/httptest"
+ "os"
+ "path/filepath"
+ "strings"
+ "testing"
+
+ "github.com/microsoft/waza/internal/artifactmanifest"
+ "github.com/microsoft/waza/internal/manifestaggregate"
+ "github.com/microsoft/waza/internal/projectconfig"
+ "github.com/microsoft/waza/internal/testutil/manifestfixture"
+ "github.com/microsoft/waza/internal/webapi"
+ "github.com/stretchr/testify/require"
+)
+
+var aggregateProductionFixture = flag.String("aggregate-production-fixture-output", "", "Exclusively create public sources and canonical API output for the embedded Lab browser control")
+
+func TestAggregateProductionConfiguredSourceAndBrowserFixture(t *testing.T) {
+ directory := t.TempDir()
+ if *aggregateProductionFixture != "" {
+ directory = *aggregateProductionFixture
+ require.True(t, filepath.IsAbs(directory))
+ require.NoError(t, os.Mkdir(directory, 0o700))
+ }
+ results := filepath.Join(directory, "results")
+ store := filepath.Join(results, ".waza-artifacts")
+ require.NoError(t, os.MkdirAll(store, 0o700))
+ backend, err := artifactmanifest.OpenLocal(store)
+ require.NoError(t, err)
+ source := manifestfixture.Publish(t, backend, "../../examples/runner/unit.json", nil, manifestfixture.RecordedNonzero(1))
+ require.NoError(t, backend.Close())
+ inputs := []manifestaggregate.Input{{Manifest: source.Manifest, Evidence: source.Evidence}}
+ wire, err := json.Marshal(inputs)
+ require.NoError(t, err)
+ // A different working directory must not alter either public capability.
+ t.Chdir(t.TempDir())
+ server, err := New(Config{ResultsDir: results, NoBrowser: true, Lab: true})
+ require.NoError(t, err)
+ call := func(body []byte) (int, []byte) {
+ t.Helper()
+ return callAggregateHTTP(t, server.Handler(), body)
+ }
+ status, body := call(wire)
+ require.Equal(t, http.StatusOK, status, string(body))
+ report, err := manifestaggregate.Parse(body)
+ require.NoError(t, err)
+ require.Equal(t, "complete", report.State)
+ require.Equal(t, 1, report.Counts.VerifiedExecutions, "configured namespace must verify the original execution")
+ require.Equal(t, source.Execution, report.Members[0].Execution)
+ require.Equal(t, "complete", report.Usage.State)
+ require.Equal(t, 1.25, *report.Usage.Recorded.AICredits)
+ canonical := bytes.Clone(body)
+ for _, control := range []struct {
+ name string
+ config *projectconfig.StorageConfig
+ }{
+ {"active-local", &projectconfig.StorageConfig{Enabled: true, Provider: "local"}},
+ {"disabled-remote", &projectconfig.StorageConfig{Enabled: false, Provider: "azure-blob"}},
+ } {
+ t.Run(control.name, func(t *testing.T) {
+ configured, err := New(Config{ResultsDir: results, NoBrowser: true, StorageConfig: control.config})
+ require.NoError(t, err)
+ status, body := callAggregateHTTP(t, configured.Handler(), wire)
+ require.Equal(t, http.StatusOK, status, string(body))
+ require.Equal(t, canonical, body)
+ })
+ }
+ t.Run("unsupported-active-provider-cannot-use-valid-local-neighbor", func(t *testing.T) {
+ configured, err := New(Config{ResultsDir: results, NoBrowser: true, StorageConfig: &projectconfig.StorageConfig{
+ Enabled: true, Provider: "unsupported",
+ }})
+ require.NoError(t, err)
+ status, body := callAggregateHTTP(t, configured.Handler(), wire)
+ require.Equal(t, http.StatusServiceUnavailable, status)
+ require.JSONEq(t, `{"error":"aggregate_store_unavailable"}`, string(body))
+ })
+ inputs = append(inputs, manifestaggregate.Input{Manifest: strings.Repeat("a", 64)})
+ incomplete, err := json.Marshal(inputs)
+ require.NoError(t, err)
+ status, body = call(incomplete)
+ require.Equal(t, http.StatusConflict, status, string(body))
+ report, err = manifestaggregate.Parse(body)
+ require.NoError(t, err)
+ require.Equal(t, "incomplete", report.State)
+ require.Equal(t, 2, report.Counts.Members)
+ require.Equal(t, 1, report.Counts.VerifiedExecutions)
+ require.Nil(t, report.Usage.Recorded)
+ status, _ = call([]byte(`[]`))
+ require.Equal(t, http.StatusBadRequest, status)
+ if *aggregateProductionFixture != "" {
+ require.NoError(t, os.WriteFile(filepath.Join(directory, "inputs.json"), wire, 0o600))
+ require.NoError(t, os.WriteFile(filepath.Join(directory, "canonical.json"), canonical, 0o600))
+ }
+}
+
+func TestAggregateProductionMissingStoreDoesNotFallBackToWorkingDirectory(t *testing.T) {
+ working := t.TempDir()
+ require.NoError(t, os.Mkdir(filepath.Join(working, ".waza-artifacts"), 0o700))
+ t.Chdir(working)
+ selected := t.TempDir()
+ server, err := New(Config{ResultsDir: selected, NoBrowser: true})
+ require.NoError(t, err)
+ status, body := callAggregateHTTP(t, server.Handler(), []byte(`[{"manifest_sha256":"`+strings.Repeat("a", 64)+`"}]`))
+ require.Equal(t, http.StatusServiceUnavailable, status)
+ require.JSONEq(t, `{"error":"aggregate_store_unavailable"}`, string(body))
+ require.NoDirExists(t, filepath.Join(selected, ".waza-artifacts"))
+}
+
+func TestAggregateConfiguredProviderDenialBeforeOpeningOrCredentials(t *testing.T) {
+ for _, provider := range []string{"azure-blob", "unsupported"} {
+ t.Run(provider, func(t *testing.T) {
+ directory := t.TempDir()
+ mux := http.NewServeMux()
+ webapi.RegisterAggregateRoutes(mux, directory, &projectconfig.StorageConfig{
+ Enabled: true, Provider: provider,
+ })
+ status, body := callAggregateHTTP(t, mux, []byte(`[{"manifest_sha256":"`+strings.Repeat("a", 64)+`"}]`))
+ require.Equal(t, http.StatusServiceUnavailable, status)
+ require.JSONEq(t, `{"error":"aggregate_store_unavailable"}`, string(body))
+ require.NoDirExists(t, filepath.Join(directory, ".waza-artifacts"))
+ })
+ }
+}
+
+func callAggregateHTTP(t *testing.T, handler http.Handler, input []byte) (int, []byte) {
+ t.Helper()
+ server := httptest.NewServer(handler)
+ defer server.Close()
+ response, err := server.Client().Post(server.URL+"/api/v1/aggregates", "application/json", bytes.NewReader(input))
+ require.NoError(t, err)
+ defer func() { require.NoError(t, response.Body.Close()) }()
+ body, err := io.ReadAll(response.Body)
+ require.NoError(t, err)
+ return response.StatusCode, body
+}
diff --git a/internal/webserver/checkpoint_operations_test.go b/internal/webserver/checkpoint_operations_test.go
new file mode 100644
index 000000000..a1a9da971
--- /dev/null
+++ b/internal/webserver/checkpoint_operations_test.go
@@ -0,0 +1,53 @@
+package webserver
+
+import (
+ "bytes"
+ "encoding/json"
+ "io"
+ "net/http"
+ "net/http/httptest"
+ "os"
+ "path/filepath"
+ "testing"
+
+ "github.com/microsoft/waza/internal/runnercheckpoint"
+ "github.com/microsoft/waza/internal/runnerunit"
+ "github.com/stretchr/testify/require"
+)
+
+func TestCheckpointProductionRouteAdmissionPrecedesStore(t *testing.T) {
+ directory := t.TempDir()
+ server, err := New(Config{ResultsDir: directory, NoBrowser: true})
+ require.NoError(t, err)
+ httpServer := httptest.NewServer(server.srv.Handler)
+ defer httpServer.Close()
+ raw, err := os.ReadFile("../../examples/runner/unit.json")
+ require.NoError(t, err)
+ wire, err := json.Marshal(runnerunit.SubmissionRequest{
+ Kind: "waza.execution-submission", SchemaVersion: "1.0", Request: raw,
+ })
+ require.NoError(t, err)
+ for _, control := range []struct {
+ name string
+ wire []byte
+ code string
+ http int
+ }{
+ {"invalid-wrapper", []byte("{}"), "checkpoint_input_invalid", http.StatusBadRequest},
+ {"not-owning-cli-build", wire, "checkpoint_build_incompatible", http.StatusServiceUnavailable},
+ } {
+ t.Run(control.name, func(t *testing.T) {
+ response, err := httpServer.Client().Post(httpServer.URL+"/api/v1/checkpoints", "application/json", bytes.NewReader(control.wire))
+ require.NoError(t, err)
+ body, err := io.ReadAll(response.Body)
+ require.NoError(t, err)
+ require.NoError(t, response.Body.Close())
+ require.Equal(t, control.http, response.StatusCode, "%s", body)
+ result, err := runnercheckpoint.ParseResult(body)
+ require.NoError(t, err)
+ require.Nil(t, result.Submission)
+ require.Equal(t, control.code, result.Diagnostics[0].Code)
+ require.NoDirExists(t, filepath.Join(directory, ".waza-artifacts"))
+ })
+ }
+}
diff --git a/internal/webserver/routes.go b/internal/webserver/routes.go
index 48d044d25..92a3a1218 100644
--- a/internal/webserver/routes.go
+++ b/internal/webserver/routes.go
@@ -61,7 +61,9 @@ func registerRoutes(mux *http.ServeMux, cfg Config) error {
webapi.RegisterRoutesWithStorage(mux, runStore, storageCfg)
webapi.RegisterReleaseRoutes(mux, cfg.ResultsDir)
webapi.RegisterRunnerRoutes(mux)
+ webapi.RegisterCheckpointOperationRoutes(mux, cfg.ResultsDir, cfg.StorageConfig)
webapi.RegisterManifestRoutes(mux, cfg.ResultsDir, cfg.StorageConfig)
+ webapi.RegisterAggregateRoutes(mux, cfg.ResultsDir, cfg.StorageConfig)
// SPA static files with HTML5 history API fallback
handler, err := spaHandler()
diff --git a/schemas/manifest-aggregate-1.0.schema.json b/schemas/manifest-aggregate-1.0.schema.json
new file mode 100644
index 000000000..810e13b4d
--- /dev/null
+++ b/schemas/manifest-aggregate-1.0.schema.json
@@ -0,0 +1,118 @@
+{
+ "$schema": "https://json-schema.org/draft/2020-12/schema",
+ "$id": "https://microsoft.github.io/waza/schemas/manifest-aggregate-1.0.schema.json",
+ "title": "Public completed manifest aggregate",
+ "type": "object",
+ "additionalProperties": false,
+ "required": ["kind", "schema_version", "state", "verification", "members", "groups", "counts", "usage"],
+ "properties": {
+ "kind": {"const": "waza.manifest-aggregate"},
+ "schema_version": {"const": "1.0"},
+ "state": {"enum": ["complete", "incomplete", "conflicting"]},
+ "verification": {"const": "current-content-and-original-record-context-only"},
+ "members": {"type": "array", "minItems": 1, "maxItems": 128, "items": {"$ref": "#/$defs/member"}},
+ "groups": {"type": "array", "maxItems": 128, "items": {
+ "type": "object", "additionalProperties": false, "required": ["profile_sha256", "members"],
+ "properties": {"profile_sha256": {"$ref": "#/$defs/digest"},
+ "members": {"type": "array", "minItems": 1, "maxItems": 128, "uniqueItems": true, "items": {"$ref": "#/$defs/input"}}}
+ }},
+ "counts": {
+ "type": "object", "additionalProperties": false,
+ "required": ["input_members", "unique_verified_manifests", "unique_verified_execution_records", "member_states", "member_outcomes"],
+ "properties": {
+ "input_members": {"type": "integer", "minimum": 1, "maximum": 128},
+ "unique_verified_manifests": {"$ref": "#/$defs/count"},
+ "unique_verified_execution_records": {"$ref": "#/$defs/count"},
+ "member_states": {"type": "object", "additionalProperties": false, "properties": {
+ "admitted": {"$ref": "#/$defs/count"}, "omitted": {"$ref": "#/$defs/count"}, "conflicting": {"$ref": "#/$defs/count"}}},
+ "member_outcomes": {"type": "object", "additionalProperties": false, "properties": {
+ "success": {"$ref": "#/$defs/count"}, "evaluation_failure": {"$ref": "#/$defs/count"},
+ "infrastructure_failure": {"$ref": "#/$defs/count"}, "cancelled": {"$ref": "#/$defs/count"},
+ "timeout": {"$ref": "#/$defs/count"}, "unsupported_capability": {"$ref": "#/$defs/count"},
+ "invalid_input": {"$ref": "#/$defs/count"}}}
+ }
+ },
+ "usage": {
+ "type": "object", "additionalProperties": false,
+ "required": ["state", "population", "executions", "recorded_historical_usage", "diagnostics"],
+ "properties": {
+ "state": {"enum": ["complete", "unavailable", "conflicting"]},
+ "population": {"const": "unique-verified-execution-records"},
+ "executions": {"$ref": "#/$defs/count"},
+ "recorded_historical_usage": {"oneOf": [{"type": "null"}, {"$ref": "#/$defs/usageStats"}]},
+ "diagnostics": {"$ref": "#/$defs/codes"}
+ },
+ "allOf": [
+ {"if": {"properties": {"state": {"const": "complete"}}}, "then": {"properties": {
+ "executions": {"minimum": 1}, "recorded_historical_usage": {"$ref": "#/$defs/usageStats"}, "diagnostics": {"maxItems": 0}
+ }}, "else": {"properties": {"recorded_historical_usage": {"type": "null"}, "diagnostics": {"minItems": 1}}}}
+ ]
+ }
+ },
+ "$defs": {
+ "digest": {"type": "string", "pattern": "^[0-9a-f]{64}$"},
+ "count": {"type": "integer", "minimum": 0, "maximum": 128},
+ "codes": {"type": "array", "maxItems": 16, "uniqueItems": true, "items": {
+ "enum": ["execution_and_usage_unavailable_without_sidecar", "execution_identity_unavailable",
+ "cached_historical_usage_excluded", "read_budget_exhausted", "execution_read_capability_unavailable",
+ "conflicting_source", "corrupt_source", "missing_source", "read_canceled", "unsupported_or_invalid_profile",
+ "source_read_error", "incompatible_recorded_usage", "complete_historical_usage_unavailable",
+ "recorded_usage_numeric_overflow", "recorded_usage_numeric_precision_unavailable",
+ "recorded_usage_report_size_unavailable"]
+ }},
+ "input": {
+ "type": "object", "additionalProperties": false, "required": ["manifest_sha256"],
+ "properties": {"manifest_sha256": {"$ref": "#/$defs/digest"}, "evidence_sha256": {"$ref": "#/$defs/digest"}}
+ },
+ "member": {
+ "type": "object", "additionalProperties": false,
+ "required": ["manifest_sha256", "state", "categories", "diagnostics"],
+ "properties": {
+ "manifest_sha256": {"$ref": "#/$defs/digest"}, "evidence_sha256": {"$ref": "#/$defs/digest"},
+ "state": {"enum": ["admitted", "omitted", "conflicting"]},
+ "binding": {"$ref": "artifact-manifest-1.0.schema.json#/properties/binding"},
+ "outcome": {"enum": ["success", "evaluation_failure", "infrastructure_failure", "cancelled", "timeout", "unsupported_capability", "invalid_input"]},
+ "execution_id": {"type": "string", "pattern": "^exec-[0-9a-f]{64}$"},
+ "categories": {"type": "array", "maxItems": 6, "items": {"$ref": "#/$defs/category"}},
+ "diagnostics": {"$ref": "#/$defs/codes"}
+ },
+ "allOf": [{"if": {"properties": {"state": {"const": "admitted"}}}, "then": {"required": ["binding", "outcome"]}}]
+ },
+ "category": {
+ "type": "object", "additionalProperties": false,
+ "required": ["category", "state", "source", "context", "verification"],
+ "properties": {
+ "category": {"enum": ["conversation", "tool", "grader", "reliability", "usage", "provenance"]},
+ "state": {"enum": ["present", "unavailable", "incomplete", "redacted", "conflicting"]},
+ "declared_state": {"enum": ["present", "unavailable", "incomplete", "redacted", "conflicting"]},
+ "source": {"enum": ["authored", "mock", "cached", "live", "unknown"]},
+ "context": {"enum": ["declared", "observed", "unknown"]},
+ "verification": {"enum": ["unverified", "declared-state-only", "source-content-and-recorded-category-shape-only",
+ "source-content-and-recorded-usage-shape-only", "recorded-degradation-only"]},
+ "reason": {"type": "string", "pattern": "^[a-z_]+$", "maxLength": 128},
+ "observed_count": {"type": "integer", "minimum": 0}
+ }
+ },
+ "number": {"type": "number", "minimum": 0, "maximum": 9007199254740991},
+ "tokens": {"type": "integer", "minimum": 0, "maximum": 9007199254740991},
+ "usageStats": {
+ "type": "object", "additionalProperties": false,
+ "required": ["turns", "input_tokens", "output_tokens", "cache_read_tokens", "cache_write_tokens", "premium_requests"],
+ "properties": {
+ "turns": {"$ref": "#/$defs/tokens"}, "input_tokens": {"$ref": "#/$defs/tokens"},
+ "output_tokens": {"$ref": "#/$defs/tokens"}, "cache_read_tokens": {"$ref": "#/$defs/tokens"},
+ "cache_write_tokens": {"$ref": "#/$defs/tokens"}, "premium_requests": {"$ref": "#/$defs/number"},
+ "ai_credits": {"$ref": "#/$defs/number"}, "provider": {"type": "string"}, "provider_host": {"type": "string"},
+ "model_metrics": {"type": "object", "additionalProperties": {
+ "type": "object", "additionalProperties": false,
+ "required": ["input_tokens", "output_tokens", "cache_read_tokens", "cache_write_tokens", "request_count", "request_cost"],
+ "properties": {
+ "input_tokens": {"$ref": "#/$defs/tokens"}, "output_tokens": {"$ref": "#/$defs/tokens"},
+ "cache_read_tokens": {"$ref": "#/$defs/tokens"}, "cache_write_tokens": {"$ref": "#/$defs/tokens"},
+ "request_count": {"$ref": "#/$defs/number"}, "request_cost": {"$ref": "#/$defs/number"}, "ai_credits": {"$ref": "#/$defs/number"}
+ }
+ }}
+ }
+ }
+ }
+}
diff --git a/schemas/manifest_aggregate.go b/schemas/manifest_aggregate.go
new file mode 100644
index 000000000..e63660562
--- /dev/null
+++ b/schemas/manifest_aggregate.go
@@ -0,0 +1,6 @@
+package schemas
+
+import _ "embed"
+
+//go:embed manifest-aggregate-1.0.schema.json
+var ManifestAggregateSchemaJSON string
diff --git a/site/astro.config.mjs b/site/astro.config.mjs
index fdeef5505..b3fcd2840 100644
--- a/site/astro.config.mjs
+++ b/site/astro.config.mjs
@@ -55,6 +55,7 @@ export default defineConfig({
{ label: 'Runner Logical Identities', slug: 'guides/runner-identities' },
{ label: 'Runner Terminal Outcomes', slug: 'guides/runner-terminal' },
{ label: 'Immutable Artifacts', slug: 'guides/immutable-artifacts' },
+ { label: 'Aggregate Completed Manifests', slug: 'guides/manifest-aggregation' },
{ label: 'Local Runner Compatibility', slug: 'guides/runner-compatibility' },
{ label: 'Local Runner Container', slug: 'guides/runner-container' },
{ label: 'Evaluating Custom Agents', slug: 'guides/custom-agents' },
diff --git a/site/e2e/manifest-aggregation.spec.ts b/site/e2e/manifest-aggregation.spec.ts
new file mode 100644
index 000000000..748b1cfaa
--- /dev/null
+++ b/site/e2e/manifest-aggregation.spec.ts
@@ -0,0 +1,20 @@
+import { expect, test } from '@playwright/test';
+
+test('aggregate guide navigation preserves the deployed base and population contract', async ({ page }) => {
+ await page.goto('/waza/guides/immutable-artifacts/');
+ const link = page.getByRole('link', { name: 'Aggregate Completed Manifests', exact: true });
+ await expect(link).toHaveAttribute('href', '/waza/guides/manifest-aggregation/');
+ await link.click();
+ await expect(page).toHaveURL(/\/waza\/guides\/manifest-aggregation\/$/);
+ await expect(page.getByRole('heading', { level: 1 })).toHaveText('Aggregate completed manifests');
+ const content = page.locator('.sl-markdown-content');
+ await expect(content).toContainText('input_members');
+ await expect(content).toContainText('unique_verified_manifests');
+ await expect(content).toContainText('unique_verified_execution_records');
+ await expect(content).toContainText('Compatible multiple views remain visible');
+ await expect(content).toContainText('Incomplete/conflicting reports are still emitted and saved');
+ await expect(content).toContainText('historical_usage');
+ await expect(content).toContainText('Unknown versions are explicit unsupported/invalid inputs');
+ await expect(content.getByRole('link', { name: 'immutable publication workflow' }))
+ .toHaveAttribute('href', '../immutable-artifacts/');
+});
diff --git a/site/e2e/runner-terminal.spec.ts b/site/e2e/runner-terminal.spec.ts
index 724e9d687..cbf719751 100644
--- a/site/e2e/runner-terminal.spec.ts
+++ b/site/e2e/runner-terminal.spec.ts
@@ -1,6 +1,32 @@
import { expect, test } from '@playwright/test';
import { readFileSync } from 'node:fs';
+test('checkpoint source commands link to bounded recovery and protected HTTP limits', async ({ page }) => {
+ await page.goto('/waza/reference/cli/');
+ await expect(page.getByRole('heading', { name: 'Checkpoint commands (unreleased source preview)', exact: true })).toBeVisible();
+ await page.getByRole('link', { name: 'complete workflow and recovery limits', exact: true }).click();
+ await expect(page).toHaveURL(/\/waza\/guides\/immutable-artifacts\/#checkpoint-public-source-preview$/);
+ const content = page.locator('main');
+ await expect(content).toContainText('Build the owning CLI from a clean Git tree with VCS stamping enabled.');
+ await expect(content).toContainText('zero new submission engine calls');
+ await expect(content).toContainText('Unavailable progress is null, never an invented zero.');
+ await expect(content).toContainText('POST /api/v1/checkpoints');
+ await expect(content).toContainText('POST /api/v1/checkpoints/{digest}/resume');
+ await expect(content).toContainText('Unsupported active providers fail closed');
+ await expect(content).toContainText('no raw checkpoint GET endpoint is available');
+ await expect(content).toContainText('A previously dispatched source is refused, not replayed.');
+ await expect(content).toContainText('available in the source Lab');
+ await expect(content).toContainText('Full native-platform and release acceptance remain pending');
+ await expect(content.locator('pre').filter({ hasText: 'waza runner checkpoint create' })).toContainText('schema_version:"1.0"');
+ await expect(content.locator('pre').filter({ hasText: 'waza runner checkpoint resume' })).toContainText('restored-result.json');
+ await page.goto('/waza/guides/dashboard-lab/');
+ const image = page.getByRole('img', { name: 'Source Lab showing a recorded protected mock API response: failed original terminal reused with zero new calls and exact JSON download.' });
+ await expect(image).toBeVisible();
+ expect(new URL((await image.getAttribute('src'))!, page.url()).pathname).toBe('/waza/images/dashboard-lab-checkpoint.png');
+ expect(await image.evaluate((img) => img instanceof HTMLImageElement && img.complete && img.naturalWidth > 0)).toBe(true);
+ await expect(page.locator('main')).toContainText('not the synthetic demo or live inference');
+});
+
test('attempt preview links serve exact independent sidecar schemas below the deployed base', async ({ page, request }) => {
await page.goto('/waza/guides/immutable-artifacts/');
await expect(page.getByRole('heading', { name: 'Logical attempts: embedding preview', exact: true })).toBeVisible();
diff --git a/site/public/images/dashboard-lab-checkpoint.png b/site/public/images/dashboard-lab-checkpoint.png
new file mode 100644
index 000000000..834cf5ffd
Binary files /dev/null and b/site/public/images/dashboard-lab-checkpoint.png differ
diff --git a/site/public/images/manifest-aggregate.png b/site/public/images/manifest-aggregate.png
new file mode 100644
index 000000000..4daa39ed3
Binary files /dev/null and b/site/public/images/manifest-aggregate.png differ
diff --git a/site/src/content/docs/guides/dashboard-lab.mdx b/site/src/content/docs/guides/dashboard-lab.mdx
index 83a4517be..d3f1c4804 100644
--- a/site/src/content/docs/guides/dashboard-lab.mdx
+++ b/site/src/content/docs/guides/dashboard-lab.mdx
@@ -12,7 +12,8 @@ or build current source as described in
`waza serve --lab` opens the **experimental real-results dashboard** at `/#/lab`.
The classic dashboard remains available at `/#/` on the same server. The Go
-evaluation engine is unchanged; the lab is a viewer, not an execution service.
+evaluation engine is unchanged. Saved-results workspaces are viewers; explicit
+runner actions use the separate exact-unit runner service.
@@ -38,6 +39,32 @@ This is separate from saved run evidence and is
unavailable in the synthetic demo. See [Runner contract and API](../runner-contract/)
and [logical identity normalization and migration](../runner-identities/).
+The unreleased source preview also offers **Create or resume an exact-unit
+checkpoint** in **Runner contract**. It uses protected POST operations, not raw
+checkpoint inspection. It shows compatibility, withheld source metadata, exact-unit
+progress, original failure/ordinal, retry parent or reuse source, and separate
+new/original engine calls. Unknown progress stays unavailable; recorded zero
+stays zero. **Download checkpoint result JSON** saves the exact protected response.
+Edits, new operations and reload errors clear prior results. Synthetic demo mode
+cannot dispatch checkpoint operations.
+Resume source declarations are checked against the requested checkpoint digest
+before display or download; this is declaration linkage, not storage authority.
+Completed resumes require a source; source-withheld failures remain available.
+This also applies when completed work has an outer publication error.
+Create responses cannot contain resume-only source or reuse state.
+
+Creation and before-unit resume execute work and may incur charges when the
+captured Copilot request explicitly consents. Finalized restore reuses the
+original terminal without new engine calls. Stopping a browser request is not
+proof that its server worker stopped. This is not a full ordered attempt history
+or whole-evaluation resume workflow. See [checkpoint boundaries and recovery
+limits](../immutable-artifacts/#checkpoint-public-source-preview).
+
+
+
+This checkpoint screenshot uses the producer-backed protected mock API fixture,
+not the synthetic demo or live inference.
+
For a dashboard built from this source tree:
```bash
diff --git a/site/src/content/docs/guides/immutable-artifacts.mdx b/site/src/content/docs/guides/immutable-artifacts.mdx
index 2c403c77a..6289cee08 100644
--- a/site/src/content/docs/guides/immutable-artifacts.mdx
+++ b/site/src/content/docs/guides/immutable-artifacts.mdx
@@ -53,7 +53,7 @@ See the independently versioned
[submission schema](../../schemas/execution-submission-1.0.schema.json), and
[submission-result schema](../../schemas/execution-submission-result-1.0.schema.json).
-### Checkpoint embedding preview
+### Checkpoint public source preview
The source-only checkpoint service supports two whole-unit safe boundaries:
`before-unit` (fresh retry child, after the original worker has stopped) and
@@ -77,12 +77,80 @@ does **not** mean it can be resumed: before-unit recovery still rejects truncate
discovery. Mutex waits and blocked providers are not made cancellable by the gate.
No in-flight process, SDK session, partial grader or workspace is restored.
-Protected public commands, Lab controls and clean-production-CLI kill/resume
-evidence remain pending. The
+The source CLI exposes capture-approved checkpoint creation and resume; it is
+not in installed v0.40.1 artifacts. Protected HTTP create/resume operations are
+registered and available in the source Lab. Full native-platform and release
+acceptance remain pending. The
[checkpoint schema](../../schemas/execution-checkpoint-1.0.schema.json) and
[checkpoint-result schema](../../schemas/runner-checkpoint-result-1.0.schema.json)
are independently versioned `1.0`; ordinary runner and legacy results are unchanged.
+Build the owning CLI from a clean Git tree with VCS stamping enabled. A dirty,
+unstamped, foreign, or unavailable build revision is rejected before store access;
+there is no user-supplied revision override. From the repository root, wrap the
+existing exact-unit example without changing its request:
+
+```bash
+mkdir -m700 .waza-artifacts
+jq -n --slurpfile request examples/runner/unit.json \
+ '{kind:"waza.execution-submission",schema_version:"1.0",request:$request[0]}' \
+ > submission.json
+waza runner checkpoint create --store .waza-artifacts \
+ < submission.json > checkpoint-result.json
+jq '{status,exit_code,before_checkpoint_sha256,finalized_checkpoint_sha256}' \
+ checkpoint-result.json
+```
+
+Stdout is one protected result; boundary/digest progress goes to stderr. Preserve
+the result even on a nonzero exit. To restore the exact finalized source:
+
+```bash
+DIGEST="$(jq -er '.finalized_checkpoint_sha256' checkpoint-result.json)"
+waza runner checkpoint resume "$DIGEST" --store .waza-artifacts \
+ < submission.json > restored-result.json
+jq '{status,exit_code,restored_units,remaining_work,submission}' restored-result.json
+```
+
+A valid finalized restore reports `reused`, `restored_units: 1`,
+`remaining_work: 0`, and zero new submission engine calls. The original terminal
+ordinal and outcome remain unchanged, including failed checks or cancellation.
+Unavailable progress is `null`, never an invented zero.
+
+The dashboard server accepts the same submission at
+`POST /api/v1/checkpoints` and
+`POST /api/v1/checkpoints/{digest}/resume`. Responses use the same protected
+result with `X-Waza-Exit-Code`. Operations require a loopback peer and literal
+loopback Host, same-origin browser metadata, and `application/json`.
+The existing results directory selects `.waza-artifacts`, unless the server has
+an active supported Azure storage configuration. Request JSON cannot choose
+host paths or credentials. Unsupported active providers fail closed; no raw
+checkpoint GET endpoint is available.
+
+In the source Lab, open **Runner contract** and **Create or resume an exact-unit
+checkpoint**. Supply the same captured submission and, for resume, the original
+checkpoint digest. The panel separates new calls from original terminal calls,
+preserves failed original outcomes, and shows retry parents or reuse sources
+without claiming an evaluation-wide history. **Download checkpoint result JSON**
+saves the exact protected response, including explicit zero or unavailable
+progress. Edits, new requests, and reload failures clear previous results;
+synthetic demo mode cannot dispatch operations. Stopping a browser request does
+not establish that its server worker stopped.
+
+For an interrupted **before-unit** source, first stop and join the original
+worker, then pass its original `before_checkpoint_sha256` to the same resume
+command. Resume creates a distinct retry child only when complete discovery
+establishes that no original dispatch or completion exists. A previously
+dispatched source is refused, not replayed. Do not select the before reference
+from an ordinarily completed execution: it is already stale.
+
+| Result | Operator action |
+|---|---|
+| Incompatible build or configuration | Preserve the checkpoint and use its compatible original binary and captured configuration; do not override the revision. |
+| Corrupt, missing, or unverified source | Preserve records and inspect the configured store; do not repair bytes in place. |
+| Stale before-unit source or incomplete discovery | Investigate the original execution. Missing completion does not authorize replay. |
+| `source_withheld: true` | Original-source capture admission did not succeed; no source metadata or terminal was approved for disclosure. |
+| Publication, output, or close failure | Preserve original and partial references. Do not infer completion or receipt from a digest. |
+
## Prepare a declaration
Keep the original dedicated [runner terminal envelope](../runner-terminal/) and
diff --git a/site/src/content/docs/guides/manifest-aggregation.mdx b/site/src/content/docs/guides/manifest-aggregation.mdx
new file mode 100644
index 000000000..63712847d
--- /dev/null
+++ b/site/src/content/docs/guides/manifest-aggregation.mdx
@@ -0,0 +1,140 @@
+---
+title: Aggregate completed manifests
+description: Build deterministic public-manifest aggregates without agents, worker ownership or invented usage.
+---
+
+**Unreleased v0.41 source preview.** Standalone aggregation reads an explicit set
+of completed public manifests from an existing local immutable store. It does
+not start agents, credential helpers or the original worker. Mutable result
+files and Azure credential-backed aggregation are not inputs to this profile.
+
+## Worked example: retained public sources
+
+Start with a completed manifest from the
+[immutable publication workflow](../immutable-artifacts/). Set
+`MANIFEST_SHA256` to its actual finalized digest. If available, set
+`EVIDENCE_SHA256` to the actual published public evidence-sidecar digest.
+These are references, not verification claims.
+
+```bash
+# Without a sidecar, admission can be complete but execution and usage are unknown.
+waza aggregate --store .waza-artifacts "$MANIFEST_SHA256" -o metadata-aggregate.json
+```
+
+Create a qualified input set without guessing digest strings or overwriting a file:
+
+```bash
+node --input-type=module <<'JS'
+import { writeFileSync } from 'node:fs';
+const manifest = process.env.MANIFEST_SHA256;
+const evidence = process.env.EVIDENCE_SHA256;
+if (!/^[0-9a-f]{64}$/.test(manifest ?? '') ||
+ !/^[0-9a-f]{64}$/.test(evidence ?? '')) {
+ throw new Error('Set actual MANIFEST_SHA256 and EVIDENCE_SHA256 public digests.');
+}
+writeFileSync('aggregate-inputs.json', JSON.stringify([
+ { manifest_sha256: manifest, evidence_sha256: evidence }
+]), { flag: 'wx', mode: 0o600 });
+JS
+
+waza aggregate --store .waza-artifacts --inputs aggregate-inputs.json -o aggregate.json
+```
+
+The evaluated report is emitted to stdout and saved to the new output file.
+Incomplete/conflicting reports are still emitted and saved, but exit nonzero.
+Do not treat a nonzero exit as permission to discard diagnostics or retry unchanged
+inputs until they appear complete. Existing output files are not replaced.
+
+## Understand the three populations
+
+`input_members` counts exact unique reference pairs; per-member state/outcome
+counts use this denominator. `unique_verified_manifests` counts current verified
+publications. `unique_verified_execution_records` counts original admitted source
+records, not legacy attempt ordinals. Compatible multiple views remain visible
+but count a record once. Distinct repeated executions are valid, not conflicts.
+Different identity/selection/mode/grader profiles remain separate display groups;
+there is no latest winner or pooled quality score.
+
+Complete admission is **not a passing task or execution attestation**. Failed and
+cancelled source outcomes remain unchanged. Supported legacy metadata-only sources,
+optional omitted sidecars and privacy-redacted categories can be complete inputs.
+Supplied invalid sidecars, missing sources and conflicting claims remain explicit.
+
+`usage.recorded_historical_usage` is either a complete historical total over unique
+admitted, non-cached execution records or null. No partial total, token-price
+estimate or new SDK billing is invented. Explicit recorded zero remains zero;
+missing credits/model attribution remain unavailable. Per-view source labels
+distinguish mock/authored/cached/unknown records from observed live evidence.
+Readers reject a complete usage claim if any supplied view lacks execution
+identity or a verified, present single usage observation, or declares cached
+categories/lineage exclusion. An eligible view cannot override another's
+exclusion. Complete metadata-only admission still retains unavailable/null
+usage. This consistency check is not source reverification or live-only
+attestation. Optional aggregate credits/model attribution may remain absent;
+missing values are unavailable, never recorded zero.
+
+If the full serialized report would exceed 32 MiB because of recorded usage,
+`recorded_usage_report_size_unavailable` withholds the entire historical total.
+Membership, diagnostics and the unique execution count remain available; the
+smaller records do not become an unlabeled subset total. Nothing is truncated.
+
+## Lab and JSON downloads
+
+Open `waza serve --lab --results-dir RESULTS_DIR` with the configured results directory containing its
+existing `.waza-artifacts` store, then select **Runner**. In **Aggregate completed public manifests**, enter one manifest
+digest and optionally its evidence digest per line. The server reads only this
+configured local store; browser input cannot select filesystem paths or providers.
+
+Uploads are limited to 1 MiB and five seconds before opening storage. Stalled
+uploads return HTTP 408 (`aggregate_request_timeout`); cancellation stops
+admission (408 when delivery is still possible). Deadline
+setup/reset failures return 500 (`aggregate_transport_unavailable`), and
+unsupported bounded delivery aborts the connection instead of returning success.
+After admission, evaluation has its separate 30-second budget. Response delivery
+is bounded to five seconds without changing canonical report bytes.
+The aggregate-only client deadline is 50 seconds, covering those phases plus ten
+seconds of headroom while preserving caller cancellation. Source open/close and
+synchronous filesystem I/O are separate constraints, not a universal lifetime
+guarantee. Blocked diagnostic sinks use the existing bounded reporter rather
+than blocking report delivery or accumulating a logger per request.
+
+The panel displays composition, profile counts, member state/outcome counts,
+conflicts, omissions, category states, historical usage availability and source
+manifest inspection links. Evaluated incomplete/conflicting HTTP 409 reports stay
+visible and downloadable. The JSON download preserves the actual canonical API
+response; transport/malformed responses clear earlier success data instead of
+showing it as current.
+
+
+
+This producer-backed example contains author-recorded mock usage, not observed
+SDK billing or execution attestation.
+
+The aggregate response limit is 32 MiB of UTF-8 bytes, including escaped
+identifiers. The client checks member-driven admission state (conflicts take
+priority), named populations, display groups and historical-usage completeness
+before showing or downloading a report. A schema-valid but internally inconsistent
+response is not accepted as complete.
+
+## Compatibility, migration and troubleshooting
+
+Aggregate schema 1.0 is separate from manifest, evidence and result versions.
+Completed manifest/terminal 1.0 sources are supported with or without a sidecar.
+Evidence/capture/attempt/completion 1.0 can qualify original record context.
+Unknown versions are explicit unsupported/invalid inputs; historical mutable
+results are not automatically converted and ownership/usage is never synthesized.
+Custom attempt artifact names are supported only when the selected descriptor
+matches the original reservation bytes; an unchanged execution ID alone is not
+sufficient.
+Ordering is deterministic for the same requested population and observed source
+state, including summation order. Diagnostics contain stable codes, not payloads
+or private provider errors.
+
+For `missing_source`, restore the original public source rather than dropping it.
+For corruption/conflicts, preserve evidence and inspect the source records.
+For `read_budget_exhausted`, the original population remains incomplete; a subset
+must not be relabeled as its complete aggregate. Missing namespace capability or
+usage stays unavailable, not zero. Reordering cannot repair a missing source.
+
+This is an opt-in source preview, not a published release or full acceptance claim.
+Existing result readers and ordinary evaluation behavior remain unchanged.
diff --git a/site/src/content/docs/reference/cli.mdx b/site/src/content/docs/reference/cli.mdx
index 991c55682..015ba55a4 100644
--- a/site/src/content/docs/reference/cli.mdx
+++ b/site/src/content/docs/reference/cli.mdx
@@ -1615,6 +1615,25 @@ Render a session timeline from an NDJSON event log.
waza session view
```
+## waza aggregate
+
+**Unreleased v0.41 source preview.** Aggregate completed public manifests without
+agents or original worker ownership. Explicit existing local stores only.
+
+```bash
+waza aggregate --store .waza-artifacts "$MANIFEST_SHA256" -o aggregate.json
+waza aggregate --store .waza-artifacts --inputs inputs.json -o qualified.json
+```
+
+`--inputs` selects a nonempty JSON reference array (at most 128 manifest/evidence
+pairs), mutually exclusive with positional digests. `--store` defaults to
+`.waza-artifacts`; no ambient provider discovery occurs. `-o, --output`
+exclusively creates a new canonical JSON file. Evaluated incomplete/conflicting
+reports are saved and emitted before a nonzero exit. Existing files are never
+replaced. Complete admission is not terminal success or execution attestation;
+historical usage is unavailable instead of an invented partial total.
+See [worked examples, conflict rules and migration](../../guides/manifest-aggregation/).
+
## waza artifact
**Unreleased v0.41 source preview.** Publish and verify immutable artifacts without
@@ -1685,6 +1704,29 @@ Save stdout with `> validation.json`; identity is not proof of execution or
duplicate suppression. See [logical identity fields and migration](../../guides/runner-identities/).
See [Runner contract, schemas, API and migration](../../guides/runner-contract/).
+### Checkpoint commands (unreleased source preview)
+
+```bash
+waza runner checkpoint create --store .waza-artifacts < submission.json
+waza runner checkpoint resume CHECKPOINT_SHA256 --store .waza-artifacts < submission.json
+```
+
+Both require one `waza.execution-submission` wrapper on stdin, not a bare runner
+request, and a clean VCS-stamped Waza CLI. `create` accepts no positional
+arguments; `resume` requires one full lowercase SHA-256 checkpoint digest.
+`--store` defaults to `.waza-artifacts`, an existing private local directory.
+Alternatively use `--azure-container-url` with existing authorized credentials;
+it is mutually exclusive with `--store` and does not initiate interactive login.
+
+Stdout is one capture-approved `waza.runner-checkpoint-result` JSON record.
+Stderr reports `checkpoint BOUNDARY DIGEST` after each published checkpoint.
+Finalized restore makes zero new engine calls and preserves original terminal
+exits, including failed checks 1, cancellation 3, and timeout 4. Invalid input
+or rejected restore exits 6; checkpoint publication, storage, and output failures
+exit 2. A reference can survive a failed publication and is not a success claim.
+See the [complete workflow and recovery limits](../../guides/immutable-artifacts/#checkpoint-public-source-preview).
+Installed v0.40.1 artifacts do not contain these commands.
+
## waza serve
Standalone v0.40.0 also exposes negotiation and exact-unit validation/run through HTTP.
diff --git a/site/src/content/docs/reference/schema-changes.md b/site/src/content/docs/reference/schema-changes.md
index e998ff735..461527411 100644
--- a/site/src/content/docs/reference/schema-changes.md
+++ b/site/src/content/docs/reference/schema-changes.md
@@ -16,6 +16,15 @@ scenario semantics; `2.1` and future majors are not supported scenario versions.
## Versioned artifacts
+The unreleased v0.41 standalone aggregate profile independently uses
+`kind: waza.manifest-aggregate`, exact `schema_version: "1.0"`. Canonical
+reference/profile/diagnostic ordering and three explicitly named populations
+preserve repeated executions and compatible views without synthesizing legacy
+ownership, missing evidence or billing. Failed/cancelled terminal outcomes are
+not aggregate admission failures. Historical mutable results are not converted;
+unsupported supplied source profiles remain explicit.
+See [aggregate compatibility and worked examples](../../guides/manifest-aggregation/).
+
This table describes standalone v0.40.0. Published standalone
`v0.39.1` writes results `1.7`; see [release availability](/waza/reference/releases/).
diff --git a/web/dist/index.html b/web/dist/index.html
index 27001416c..fe1125f56 100644
--- a/web/dist/index.html
+++ b/web/dist/index.html
@@ -4,7 +4,7 @@
waza — eval dashboard
-
+
diff --git a/web/e2e/aggregate-client.spec.ts b/web/e2e/aggregate-client.spec.ts
new file mode 100644
index 000000000..a4df54d4f
--- /dev/null
+++ b/web/e2e/aggregate-client.spec.ts
@@ -0,0 +1,390 @@
+import { expect, test } from "@playwright/test";
+import { createServer, type ViteDevServer } from "vite";
+import { execFileSync } from "node:child_process";
+import { createHash } from "node:crypto";
+import { mkdtemp, readFile, rm } from "node:fs/promises";
+import { tmpdir } from "node:os";
+import { join, resolve } from "node:path";
+
+const digest = "a".repeat(64);
+
+function omittedReport(state = "incomplete", memberState = "omitted") {
+ return {
+ kind: "waza.manifest-aggregate", schema_version: "1.0", state,
+ verification: "current-content-and-original-record-context-only",
+ members: [{
+ manifest_sha256: digest, state: memberState, categories: [],
+ diagnostics: [memberState === "conflicting" ? "conflicting_source" : "read_budget_exhausted"],
+ }],
+ groups: [],
+ counts: {
+ input_members: 1, unique_verified_manifests: 0, unique_verified_execution_records: 0,
+ member_states: { [memberState]: 1 }, member_outcomes: {},
+ },
+ usage: {
+ state: "unavailable", population: "unique-verified-execution-records", executions: 0,
+ recorded_historical_usage: null, diagnostics: ["complete_historical_usage_unavailable"],
+ },
+ };
+}
+
+test.describe("Actual aggregate fetch, component and download callers", () => {
+ let server: ViteDevServer;
+ let base: string;
+ let directory: string;
+ let mixed: {
+ inputs: { manifest_sha256: string; evidence_sha256?: string }[];
+ canonical: string;
+ forged: string;
+ eligibility: {
+ name: string;
+ inputs: { manifest_sha256: string; evidence_sha256?: string }[];
+ canonical: string;
+ forged: string;
+ }[];
+ };
+ let delayed: { wire: string; delay: number; started?: () => void; closed?: (completed: boolean) => void } | undefined;
+
+ test.beforeAll(async () => {
+ directory = await mkdtemp(join(tmpdir(), "waza-mixed-aggregate-"));
+ const output = join(directory, "mixed.json");
+ execFileSync("go", ["test", "-count=1", "-run", "^TestMixedIdentityPublicReportFixture$",
+ "./internal/manifestaggregate", "-args", "-aggregate-mixed-identity-fixture-output", output], {
+ cwd: resolve(".."), timeout: 120_000,
+ });
+ mixed = JSON.parse(await readFile(output, "utf8"));
+ server = await createServer({
+ server: { host: "127.0.0.1", port: 0 },
+ plugins: [{
+ name: "aggregate-delayed-http-controls",
+ configureServer(vite) {
+ vite.middlewares.use((request, response, next) => {
+ const control = delayed;
+ if (request.url !== "/api/v1/aggregates" || !control) return next();
+ request.resume();
+ const timer = setTimeout(() => {
+ response.setHeader("Content-Type", "application/json");
+ response.end(control.wire);
+ }, control.delay);
+ response.on("close", () => {
+ clearTimeout(timer);
+ control.closed?.(response.writableFinished);
+ });
+ control.started?.();
+ });
+ },
+ }],
+ });
+ await server.listen();
+ const address = server.httpServer?.address();
+ if (!address || typeof address === "string") throw new Error("Aggregate harness listener unavailable.");
+ base = `http://127.0.0.1:${address.port}`;
+ });
+ test.afterAll(async () => {
+ await server?.close();
+ if (directory) await rm(directory, { recursive: true, force: true });
+ });
+
+ test("rejects a producer-backed mixed identity subtotal and clears earlier valid admission", async ({ page }) => {
+ const original = JSON.parse(mixed.canonical);
+ const forged = JSON.parse(mixed.forged);
+ expect(original.members).toHaveLength(2);
+ expect(original.members.filter((member: { execution_id?: string }) => member.execution_id)).toHaveLength(1);
+ expect(forged).toEqual({ ...original, usage: forged.usage });
+ expect(original.usage.recorded_historical_usage).toBeNull();
+ expect(forged.usage.recorded_historical_usage.ai_credits).toBe(1.25);
+ let calls = 0;
+ await page.route("**/api/v1/aggregates", (route) => route.fulfill({
+ contentType: "application/json", body: calls++ === 0 ? mixed.canonical : mixed.forged,
+ }));
+ await page.goto(`${base}/e2e/fixtures/aggregate-harness.html`);
+ await page.getByRole("textbox").fill(mixed.inputs.map((input) =>
+ `${input.manifest_sha256}${input.evidence_sha256 ? ` ${input.evidence_sha256}` : ""}`).join("\n"));
+ await page.getByRole("button", { name: "Build aggregate", exact: true }).click();
+ await expect(page.getByRole("status", { name: "Evaluated manifest aggregate" })).toContainText("Aggregate admission: complete");
+ const download = page.waitForEvent("download");
+ await page.getByRole("button", { name: "Download aggregate JSON" }).click();
+ expect(await readFile((await (await download).path())!)).toEqual(Buffer.from(mixed.canonical));
+ await page.getByRole("button", { name: "Build aggregate", exact: true }).click();
+ await expect(page.getByRole("alert")).toContainText("inconsistent");
+ await expect(page.getByRole("status", { name: "Evaluated manifest aggregate" })).toHaveCount(0);
+ await expect(page.getByRole("button", { name: "Download aggregate JSON" })).toHaveCount(0);
+ expect(calls).toBe(2);
+ });
+
+ test("rejects producer-backed declared usage exclusions without excluding optional aggregate credits", async ({ page }) => {
+ let response = "";
+ await page.route("**/api/v1/aggregates", (route) => route.fulfill({ contentType: "application/json", body: response }));
+ await page.goto(`${base}/e2e/fixtures/aggregate-harness.html`);
+ for (const fixture of mixed.eligibility) {
+ const original = JSON.parse(fixture.canonical);
+ expect(original.members.every((member: { execution_id?: string }) => member.execution_id)).toBe(true);
+ const references = fixture.inputs.map((input) =>
+ `${input.manifest_sha256}${input.evidence_sha256 ? ` ${input.evidence_sha256}` : ""}`).join("\n");
+ await page.getByRole("textbox").fill(references);
+ response = fixture.canonical;
+ await page.getByRole("button", { name: "Build aggregate", exact: true }).click();
+ const result = page.getByRole("status", { name: "Evaluated manifest aggregate" });
+ await expect(result).toContainText("Aggregate admission: complete");
+ if (fixture.name === "optional aggregate credits absent") {
+ expect(original.usage.recorded_historical_usage.ai_credits).toBeUndefined();
+ await expect(result.locator("dd").nth(5)).toHaveText("Unavailable");
+ } else if (fixture.name === "optional aggregate attribution absent") {
+ expect(original.usage.recorded_historical_usage.model_metrics).toBeUndefined();
+ await expect(result.locator("dd").nth(5)).toHaveText("1.25");
+ } else if (fixture.name === "recorded zero credits") {
+ expect(original.usage.recorded_historical_usage.ai_credits).toBe(0);
+ await expect(result.locator("dd").nth(5)).toHaveText("0");
+ } else {
+ expect(original.usage.recorded_historical_usage).toBeNull();
+ }
+ const download = page.waitForEvent("download");
+ await page.getByRole("button", { name: "Download aggregate JSON" }).click();
+ expect(await readFile((await (await download).path())!)).toEqual(Buffer.from(fixture.canonical));
+ if (fixture.forged) {
+ const forged = JSON.parse(fixture.forged);
+ expect(forged).toEqual({ ...original, usage: forged.usage });
+ response = fixture.forged;
+ await page.getByRole("button", { name: "Build aggregate", exact: true }).click();
+ await expect(page.getByRole("alert")).toContainText("inconsistent");
+ await expect(result).toHaveCount(0);
+ }
+ }
+ expect(mixed.eligibility).toHaveLength(6);
+ });
+
+ test("requires a verified single usage observation and honors any cached category", async ({ page }) => {
+ const fixture = mixed.eligibility.find((control) => control.name === "optional aggregate credits absent")!;
+ let response = fixture.canonical;
+ await page.route("**/api/v1/aggregates", (route) => route.fulfill({ contentType: "application/json", body: response }));
+ await page.goto(`${base}/e2e/fixtures/aggregate-harness.html`);
+ await page.getByRole("textbox").fill(fixture.inputs.map((input) =>
+ `${input.manifest_sha256} ${input.evidence_sha256}`).join("\n"));
+ for (const change of ["missing category", "cached category", "unverified", "empty count", "withheld declaration"]) {
+ const report = JSON.parse(fixture.canonical);
+ const member = report.members[0];
+ const usage = member.categories.find((category: { category: string }) => category.category === "usage");
+ expect(usage.observed_count).toBe(1);
+ if (change === "missing category") member.categories = [];
+ else if (change === "cached category") member.categories[0].source = "cached";
+ else if (change === "unverified") usage.verification = "declared-state-only";
+ else if (change === "empty count") usage.observed_count = 0;
+ else usage.declared_state = "redacted";
+ response = JSON.stringify(report);
+ await page.getByRole("button", { name: "Build aggregate", exact: true }).click();
+ await expect(page.getByRole("alert")).toContainText("inconsistent");
+ await expect(page.getByRole("status", { name: "Evaluated manifest aggregate" })).toHaveCount(0);
+ }
+ });
+
+ test("delayed valid HTTP response outlives the old client deadline and downloads exact bytes", async ({ page }) => {
+ test.setTimeout(60_000);
+ delayed = { wire: mixed.canonical, delay: 36_000 };
+ try {
+ await page.goto(`${base}/e2e/fixtures/aggregate-harness.html`);
+ await page.getByRole("textbox").fill(mixed.inputs.map((input) =>
+ `${input.manifest_sha256}${input.evidence_sha256 ? ` ${input.evidence_sha256}` : ""}`).join("\n"));
+ await page.getByRole("button", { name: "Build aggregate", exact: true }).click();
+ await expect(page.getByRole("status", { name: "Evaluated manifest aggregate" }))
+ .toContainText("Aggregate admission: complete", { timeout: 40_000 });
+ const download = page.waitForEvent("download");
+ await page.getByRole("button", { name: "Download aggregate JSON" }).click();
+ expect(await readFile((await (await download).path())!)).toEqual(Buffer.from(mixed.canonical));
+ } finally {
+ delayed = undefined;
+ }
+ });
+
+ test("caller edits abort actual HTTP and never restore stale aggregate data", async ({ page }) => {
+ let started: (() => void) | undefined;
+ let closed: ((completed: boolean) => void) | undefined;
+ const requestStarted = new Promise((done) => { started = done; });
+ const requestClosed = new Promise((done) => { closed = done; });
+ delayed = { wire: mixed.canonical, delay: 60_000, started: () => started?.(), closed: (complete) => closed?.(complete) };
+ try {
+ await page.goto(`${base}/e2e/fixtures/aggregate-harness.html`);
+ const references = mixed.inputs.map((input) =>
+ `${input.manifest_sha256}${input.evidence_sha256 ? ` ${input.evidence_sha256}` : ""}`).join("\n");
+ await page.getByRole("textbox").fill(references);
+ await page.getByRole("button", { name: "Build aggregate", exact: true }).click();
+ await requestStarted;
+ await page.getByRole("textbox").fill(`${references}\n`);
+ expect(await requestClosed).toBe(false);
+ await expect(page.getByRole("status", { name: "Evaluated manifest aggregate" })).toHaveCount(0);
+ await expect(page.getByRole("alert")).toHaveCount(0);
+ delayed.delay = 0;
+ await page.getByRole("button", { name: "Build aggregate", exact: true }).click();
+ await expect(page.getByRole("status", { name: "Evaluated manifest aggregate" })).toContainText("Aggregate admission: complete");
+ } finally {
+ delayed = undefined;
+ }
+ });
+
+ test("finite client timeout clears previous report and closes actual HTTP", async ({ page }) => {
+ test.setTimeout(75_000);
+ let closed: ((completed: boolean) => void) | undefined;
+ const requestClosed = new Promise((done) => { closed = done; });
+ delayed = { wire: mixed.canonical, delay: 0 };
+ try {
+ await page.goto(`${base}/e2e/fixtures/aggregate-harness.html`);
+ await page.getByRole("textbox").fill(mixed.inputs.map((input) =>
+ `${input.manifest_sha256}${input.evidence_sha256 ? ` ${input.evidence_sha256}` : ""}`).join("\n"));
+ await page.getByRole("button", { name: "Build aggregate", exact: true }).click();
+ await expect(page.getByRole("status", { name: "Evaluated manifest aggregate" })).toContainText("Aggregate admission: complete");
+ delayed = { wire: mixed.canonical, delay: 60_000, closed: (complete) => closed?.(complete) };
+ await page.getByRole("button", { name: "Build aggregate", exact: true }).click();
+ await expect(page.getByRole("status", { name: "Evaluated manifest aggregate" })).toHaveCount(0);
+ await expect(page.getByRole("alert")).toContainText("could not be loaded", { timeout: 55_000 });
+ expect(await requestClosed).toBe(false);
+ await expect(page.getByRole("button", { name: "Download aggregate JSON" })).toHaveCount(0);
+ await expect(page.getByRole("button", { name: "Build aggregate", exact: true })).toBeEnabled();
+ } finally {
+ delayed = undefined;
+ }
+ });
+
+ test("rejects a matching-count omitted member disguised as complete", async ({ page }) => {
+ await page.route("**/api/v1/aggregates", (route) => route.fulfill({ json: omittedReport("complete") }));
+ await page.goto(`${base}/e2e/fixtures/aggregate-harness.html`);
+ await page.getByRole("textbox").fill(digest);
+ await page.getByRole("button", { name: "Build aggregate", exact: true }).click();
+ await expect(page.getByRole("alert")).toContainText("inconsistent");
+ await expect(page.getByRole("status", { name: "Evaluated manifest aggregate" })).toHaveCount(0);
+ });
+
+ test("rejects conflicting members disguised as merely incomplete", async ({ page }) => {
+ const report = omittedReport("incomplete", "conflicting");
+ await page.route("**/api/v1/aggregates", (route) => route.fulfill({ status: 409, json: report }));
+ await page.goto(`${base}/e2e/fixtures/aggregate-harness.html`);
+ await page.getByRole("textbox").fill(digest);
+ await page.getByRole("button", { name: "Build aggregate", exact: true }).click();
+ await expect(page.getByRole("alert")).toContainText("inconsistent");
+ });
+
+ test("rejects a usage execution count differing from the unique record population", async ({ page }) => {
+ const report = omittedReport();
+ report.usage.executions = 1;
+ await page.route("**/api/v1/aggregates", (route) => route.fulfill({ status: 409, json: report }));
+ await page.goto(`${base}/e2e/fixtures/aggregate-harness.html`);
+ await page.getByRole("textbox").fill(digest);
+ await page.getByRole("button", { name: "Build aggregate", exact: true }).click();
+ await expect(page.getByRole("alert")).toContainText("inconsistent");
+ });
+
+ test("rejects complete usage attached to incomplete aggregate admission", async ({ page }) => {
+ const report = {
+ ...omittedReport(),
+ members: [{ ...omittedReport().members[0]!, execution_id: `exec-${digest}` }],
+ counts: { ...omittedReport().counts, unique_verified_execution_records: 1 },
+ usage: {
+ ...omittedReport().usage, state: "complete", executions: 1, diagnostics: [],
+ recorded_historical_usage: {
+ turns: 0, input_tokens: 0, output_tokens: 0, cache_read_tokens: 0, cache_write_tokens: 0, premium_requests: 0,
+ },
+ },
+ };
+ await page.route("**/api/v1/aggregates", (route) => route.fulfill({ status: 409, json: report }));
+ await page.goto(`${base}/e2e/fixtures/aggregate-harness.html`);
+ await page.getByRole("textbox").fill(digest);
+ await page.getByRole("button", { name: "Build aggregate", exact: true }).click();
+ await expect(page.getByRole("alert")).toContainText("inconsistent");
+ });
+
+ for (const [state, memberState] of [["incomplete", "omitted"], ["conflicting", "conflicting"]]) {
+ test(`retains and downloads a valid ${state} evaluated response`, async ({ page }) => {
+ const canonical = JSON.stringify(omittedReport(state, memberState));
+ await page.route("**/api/v1/aggregates", (route) => route.fulfill({ status: 409, contentType: "application/json", body: canonical }));
+ await page.goto(`${base}/e2e/fixtures/aggregate-harness.html`);
+ await page.getByRole("textbox").fill(digest);
+ await page.getByRole("button", { name: "Build aggregate", exact: true }).click();
+ await expect(page.getByRole("status", { name: "Evaluated manifest aggregate" })).toContainText(`Aggregate admission: ${state}`);
+ const download = page.waitForEvent("download");
+ await page.getByRole("button", { name: "Download aggregate JSON" }).click();
+ const stream = await (await download).createReadStream();
+ const chunks: Buffer[] = [];
+ for await (const chunk of stream) chunks.push(Buffer.from(chunk));
+ expect(Buffer.concat(chunks).toString("utf8")).toBe(canonical);
+ });
+ }
+
+ test("preserves conflicting priority when a later member is omitted", async ({ page }) => {
+ const report = omittedReport("conflicting", "conflicting");
+ const missing = { ...omittedReport().members[0]!, manifest_sha256: "b".repeat(64) };
+ report.members.push(missing);
+ report.counts.input_members = 2;
+ report.counts.member_states.omitted = 1;
+ await page.route("**/api/v1/aggregates", (route) => route.fulfill({ status: 409, json: report }));
+ await page.goto(`${base}/e2e/fixtures/aggregate-harness.html`);
+ await page.getByRole("textbox").fill(`${digest}\n${missing.manifest_sha256}`);
+ await page.getByRole("button", { name: "Build aggregate", exact: true }).click();
+ await expect(page.getByRole("status", { name: "Evaluated manifest aggregate" })).toContainText("Aggregate admission: conflicting");
+ });
+
+ for (const extra of [0, 1]) {
+ test(`enforces the exact 32 MiB aggregate response boundary (${extra} excess bytes)`, async ({ page }) => {
+ const report = JSON.stringify(omittedReport());
+ const wire = " ".repeat(32 * 1024 * 1024 - report.length + extra) + report;
+ expect(Buffer.byteLength(wire)).toBe(32 * 1024 * 1024 + extra);
+ await page.route("**/api/v1/aggregates", (route) => route.fulfill({ status: 409, contentType: "application/json", body: wire }));
+ await page.goto(`${base}/e2e/fixtures/aggregate-harness.html`);
+ await page.getByRole("textbox").fill(digest);
+ await page.getByRole("button", { name: "Build aggregate", exact: true }).click();
+ if (extra) await expect(page.getByRole("alert")).toContainText("malformed JSON");
+ else await expect(page.getByRole("status", { name: "Evaluated manifest aggregate" })).toContainText("Aggregate admission: incomplete");
+ });
+ }
+
+ test("uses UTF-8 bytes for the aggregate limit, retaining the unrelated default limit", async ({ page }) => {
+ await page.route("**/api/v1/aggregates", (route) => route.fulfill({
+ contentType: "application/json", body: `"${"é".repeat(16 * 1024 * 1024)}"`,
+ }));
+ await page.goto(`${base}/e2e/fixtures/aggregate-harness.html`);
+ await page.getByRole("textbox").fill(digest);
+ await page.getByRole("button", { name: "Build aggregate", exact: true }).click();
+ await expect(page.getByRole("alert")).toContainText("malformed JSON");
+ const defaultBoundary = await page.evaluate(async () => {
+ const moduleURL = "/src/lib/strictJson.ts";
+ const { parseStrictJSON } = await import(moduleURL);
+ const accepted = parseStrictJSON(" ".repeat(16 * 1024 * 1024 - 1) + "0") === 0;
+ let rejected = false;
+ try { parseStrictJSON(" ".repeat(16 * 1024 * 1024) + "0"); }
+ catch { rejected = true; }
+ return { accepted, rejected };
+ });
+ expect(defaultBoundary).toEqual({ accepted: true, rejected: true });
+ });
+
+ test("consumes and downloads actual canonical public output between 16 and 32 MiB", async ({ page }) => {
+ test.setTimeout(240_000);
+ const directory = await mkdtemp(join(tmpdir(), "waza-aggregate-caller-"));
+ try {
+ const output = join(directory, "canonical.json");
+ execFileSync("go", ["test", "-count=1", "-run", "^TestLargeCanonicalPublicReportFitsAggregateProfile$",
+ "./internal/manifestaggregate", "-args", "-aggregate-browser-fixture-output", output], {
+ cwd: resolve(".."), timeout: 180_000, env: { ...process.env, NO_COLOR: "1" },
+ });
+ const canonical = await readFile(output);
+ expect(canonical.length).toBeGreaterThan(16 * 1024 * 1024);
+ expect(canonical.length).toBeLessThanOrEqual(32 * 1024 * 1024);
+ const report = JSON.parse(canonical.toString("utf8")) as { members: { manifest_sha256: string }[] };
+ expect(report.members).toHaveLength(128);
+ await page.route("**/api/v1/aggregates", (route) => route.fulfill({ contentType: "application/json", body: canonical }));
+ await page.goto(`${base}/e2e/fixtures/aggregate-harness.html`);
+ await page.getByRole("textbox").fill(report.members.map((member) => member.manifest_sha256).join("\n"));
+ await page.getByRole("button", { name: "Build aggregate", exact: true }).click();
+ const result = page.getByRole("status", { name: "Evaluated manifest aggregate" });
+ await expect(result).toContainText("Aggregate admission: complete", { timeout: 30_000 });
+ await expect(result.getByRole("link", { name: "Inspect source manifest" })).toHaveCount(128);
+ const download = page.waitForEvent("download");
+ await page.getByRole("button", { name: "Download aggregate JSON" }).click();
+ const stream = await (await download).createReadStream();
+ const chunks: Buffer[] = [];
+ for await (const chunk of stream) chunks.push(Buffer.from(chunk));
+ const downloaded = Buffer.concat(chunks);
+ expect(downloaded.length).toBe(canonical.length);
+ expect(createHash("sha256").update(downloaded).digest("hex")).toBe(createHash("sha256").update(canonical).digest("hex"));
+ } finally {
+ await rm(directory, { recursive: true, force: true });
+ }
+ });
+});
diff --git a/web/e2e/aggregate-production.spec.ts b/web/e2e/aggregate-production.spec.ts
new file mode 100644
index 000000000..deadd46d0
--- /dev/null
+++ b/web/e2e/aggregate-production.spec.ts
@@ -0,0 +1,159 @@
+import { expect, test } from "@playwright/test";
+import { execFileSync, spawn, spawnSync, type ChildProcess } from "node:child_process";
+import { mkdtemp, readFile, rename, rm } from "node:fs/promises";
+import { createServer } from "node:net";
+import { tmpdir } from "node:os";
+import { join, resolve } from "node:path";
+
+test.describe("Configured production aggregate CLI, server and embedded Lab", () => {
+ test.describe.configure({ mode: "serial" });
+ let directory: string;
+ let fixture: string;
+ let binary: string;
+ let base: string;
+ let server: ChildProcess;
+ let canonical: Buffer;
+ let inputs: { manifest_sha256: string; evidence_sha256: string }[];
+ let output = "";
+ const env = {
+ ...process.env, COPILOT_CLI_PATH: "/usr/bin/false", WAZA_NO_UPDATE_CHECK: "1",
+ ENABLE_COPILOT_TESTS: "false", ENABLE_COPILOT_SDK_TESTS: "false", ENABLE_LIVE_COPILOT_TESTS: "false",
+ };
+
+ test.beforeAll(async ({ request }) => {
+ test.setTimeout(180_000);
+ directory = await mkdtemp(join(tmpdir(), "waza-aggregate-production-"));
+ fixture = join(directory, "fixture");
+ binary = join(directory, "waza");
+ execFileSync("go", ["test", "-count=1", "-run", "^TestAggregateProductionConfiguredSourceAndBrowserFixture$",
+ "./internal/webserver", "-args", `-aggregate-production-fixture-output=${fixture}`], {
+ cwd: resolve(".."), env, timeout: 120_000,
+ });
+ execFileSync("go", ["build", "-o", binary, "./cmd/waza"], { cwd: resolve(".."), env, timeout: 120_000 });
+ inputs = JSON.parse(await readFile(join(fixture, "inputs.json"), "utf8"));
+ canonical = await readFile(join(fixture, "canonical.json"));
+ expect(inputs).toHaveLength(1);
+ const saved = join(directory, "saved-cli.json");
+ const cli = spawnSync(binary, ["aggregate", "--store", join(fixture, "results", ".waza-artifacts"),
+ "--inputs", join(fixture, "inputs.json"), "-o", saved], {
+ cwd: directory, env: { ...env, WAZA_NO_UPDATE_CHECK: "" }, timeout: 15_000,
+ });
+ expect(cli.error).toBeUndefined();
+ expect(cli.status, cli.stderr.toString()).toBe(0);
+ expect(await readFile(saved)).toEqual(canonical);
+ expect(cli.stdout).toEqual(Buffer.concat([canonical, Buffer.from("\n")]));
+ const listener = createServer();
+ await new Promise((done) => listener.listen(0, "127.0.0.1", done));
+ const address = listener.address();
+ if (!address || typeof address === "string") throw new Error("Missing aggregate test listener");
+ await new Promise((done, reject) => listener.close((error) => error ? reject(error) : done()));
+ base = `http://127.0.0.1:${address.port}`;
+ server = spawn(binary, ["serve", "--no-browser", "--lab", "--port", String(address.port),
+ "--results-dir", join(fixture, "results")], { cwd: directory, env, stdio: ["ignore", "pipe", "pipe"] });
+ server.stdout?.on("data", (chunk: Buffer) => { output += chunk.toString(); });
+ server.stderr?.on("data", (chunk: Buffer) => { output += chunk.toString(); });
+ await expect.poll(async () => {
+ if (server.exitCode !== null) throw new Error(`Aggregate server exited: ${output}`);
+ try { return (await request.post(`${base}/api/v1/aggregates`, { data: inputs })).status(); }
+ catch { return 0; }
+ }, { timeout: 15_000 }).toBe(200);
+ });
+
+ test.afterAll(async () => {
+ if (server && server.exitCode === null) {
+ const exited = new Promise((done) => server.once("exit", done));
+ server.kill("SIGTERM");
+ await exited;
+ }
+ if (directory) await rm(directory, { recursive: true, force: true });
+ });
+
+ test("saved CLI bytes reach real Lab, source inspection and exact download", async ({ page }) => {
+ const index = await page.request.get(base);
+ expect(await index.text()).toBe(await readFile(resolve("dist/index.html"), "utf8"));
+ await page.goto(`${base}/#/lab/runner`);
+ const panel = page.getByRole("region", { name: "Standalone manifest aggregation" });
+ await panel.getByRole("textbox").fill(`${inputs[0]!.manifest_sha256} ${inputs[0]!.evidence_sha256}`);
+ const response = page.waitForResponse(`${base}/api/v1/aggregates`);
+ await panel.getByRole("button", { name: "Build aggregate", exact: true }).click();
+ expect(await (await response).body()).toEqual(canonical);
+ const receipt = panel.getByRole("status", { name: "Evaluated manifest aggregate" });
+ await expect(receipt.getByRole("heading", { name: "Aggregate admission: complete", exact: true })).toBeVisible();
+ await expect(receipt.locator("dd").nth(2)).toHaveText("1");
+ await expect(receipt.locator("dd").nth(5)).toHaveText("1.25");
+ const link = receipt.getByRole("link", { name: "Inspect source manifest" });
+ const inspected = await page.request.get(`${base}${await link.getAttribute("href")}`);
+ expect(inspected.status()).toBe(200);
+ expect(await inspected.json()).toMatchObject({ manifest_sha256: inputs[0]!.manifest_sha256, integrity: "verified" });
+ const download = page.waitForEvent("download");
+ await receipt.getByRole("button", { name: "Download aggregate JSON" }).click();
+ const downloaded = await download;
+ const path = await downloaded.path();
+ expect(path).not.toBeNull();
+ expect(await readFile(path!)).toEqual(canonical);
+ for (const path of ["../docs/images/manifest-aggregate.png", "../site/public/images/manifest-aggregate.png"]) {
+ await panel.screenshot({ path, animations: "disabled" });
+ }
+ });
+
+ test("actual missing neighbor retains evaluated 409 and downloadable membership", async ({ page }) => {
+ await page.goto(`${base}/#/lab/runner`);
+ const panel = page.getByRole("region", { name: "Standalone manifest aggregation" });
+ const missing = "a".repeat(64);
+ await panel.getByRole("textbox").fill(`${inputs[0]!.manifest_sha256} ${inputs[0]!.evidence_sha256}\n${missing}`);
+ const response = page.waitForResponse(`${base}/api/v1/aggregates`);
+ await panel.getByRole("button", { name: "Build aggregate", exact: true }).click();
+ const evaluated = await response;
+ expect(evaluated.status()).toBe(409);
+ const wire = await evaluated.body();
+ expect(JSON.parse(wire.toString())).toMatchObject({
+ state: "incomplete", counts: { input_members: 2, unique_verified_execution_records: 1 },
+ usage: { state: "unavailable", recorded_historical_usage: null },
+ });
+ const receipt = panel.getByRole("status", { name: "Evaluated manifest aggregate" });
+ await expect(receipt.getByRole("heading", { name: "Aggregate admission: incomplete", exact: true })).toBeVisible();
+ await expect(receipt.getByRole("heading", { name: `omitted: ${missing}`, exact: true })).toBeVisible();
+ const download = page.waitForEvent("download");
+ await receipt.getByRole("button", { name: "Download aggregate JSON" }).click();
+ const path = await (await download).path();
+ expect(await readFile(path!)).toEqual(wire);
+ });
+
+ test("real store failure and transport loss clear previous success", async ({ page }) => {
+ await page.goto(`${base}/#/lab/runner`);
+ const panel = page.getByRole("region", { name: "Standalone manifest aggregation" });
+ await panel.getByRole("textbox").fill(`${inputs[0]!.manifest_sha256} ${inputs[0]!.evidence_sha256}`);
+ const button = panel.getByRole("button", { name: "Build aggregate", exact: true });
+ const receipt = panel.getByRole("status", { name: "Evaluated manifest aggregate" });
+ await button.click();
+ await expect(receipt).toBeVisible();
+ const store = join(fixture, "results", ".waza-artifacts");
+ const unavailable = join(fixture, "results", ".temporarily-unavailable");
+ await rename(store, unavailable);
+ try {
+ const response = page.waitForResponse(`${base}/api/v1/aggregates`);
+ await button.click();
+ expect((await response).status()).toBe(503);
+ await expect(panel.getByRole("alert")).toBeVisible();
+ await expect(receipt).toHaveCount(0);
+ } finally { await rename(unavailable, store); }
+ await button.click();
+ await expect(receipt).toBeVisible();
+ await page.route("**/api/v1/aggregates", (route) => route.abort("connectionfailed"));
+ await button.click();
+ await expect(panel.getByRole("alert")).toBeVisible();
+ await expect(receipt).toHaveCount(0);
+ await expect(panel.getByRole("button", { name: "Download aggregate JSON" })).toHaveCount(0);
+ });
+
+ test("synthetic demo never offers aggregation or requests its API", async ({ page }) => {
+ const aggregateRequests: string[] = [];
+ page.on("request", (request) => { if (request.url().endsWith("/api/v1/aggregates")) aggregateRequests.push(request.url()); });
+ await page.goto(`${base}/#/lab/demo/runner`);
+ const panel = page.getByRole("region", { name: "Standalone manifest aggregation" });
+ await expect(panel).toContainText("Aggregation is unavailable in the synthetic demo.");
+ await expect(panel.getByRole("textbox")).toHaveCount(0);
+ await expect(panel.getByRole("button", { name: "Build aggregate" })).toHaveCount(0);
+ expect(aggregateRequests).toEqual([]);
+ });
+});
diff --git a/web/e2e/checkpoint-operations.spec.ts b/web/e2e/checkpoint-operations.spec.ts
new file mode 100644
index 000000000..0892c0d62
--- /dev/null
+++ b/web/e2e/checkpoint-operations.spec.ts
@@ -0,0 +1,320 @@
+import { expect, test, type Page } from "@playwright/test";
+import { copyFile, readFile } from "node:fs/promises";
+import fixtures from "./fixtures/checkpoint-operations.json" with { type: "json" };
+import type { CheckpointOperationResult } from "../src/api/checkpointResults";
+
+interface Fixture {
+ path: string;
+ request: { request: unknown };
+ admission: unknown;
+ http_status: number;
+ response: string;
+}
+
+function panel(page: Page) {
+ return page.getByRole("region", { name: "Checkpoint creation and recovery" });
+}
+
+function result(page: Page) {
+ return panel(page).getByLabel("Checkpoint operation result", { exact: true });
+}
+
+function parsed(fixture: Fixture): CheckpointOperationResult {
+ return JSON.parse(fixture.response);
+}
+
+async function prepare(page: Page, fixture: Fixture, response = fixture.response, status = fixture.http_status, exit?: string) {
+ const methods: string[] = [];
+ await page.route("**/api/v1/runner/validate", (route) => {
+ expect(route.request().postDataJSON()).toEqual(fixture.request.request);
+ return route.fulfill({ json: fixture.admission });
+ });
+ await page.route("**/api/v1/checkpoints**", (route) => {
+ methods.push(route.request().method());
+ expect(route.request().method()).toBe("POST");
+ expect(new URL(route.request().url()).pathname).toBe(fixture.path);
+ expect(route.request().postDataJSON()).toEqual(fixture.request);
+ return route.fulfill({
+ status, body: response, contentType: "application/json",
+ headers: { "X-Waza-Exit-Code": exit ?? String(parsed(fixture).exit_code) },
+ });
+ });
+ await panel(page).getByRole("textbox", { name: "Captured checkpoint submission (JSON)" }).fill(JSON.stringify(fixture.request, null, 2));
+ if (fixture.path.endsWith("/resume")) {
+ await panel(page).getByRole("textbox", { name: "Checkpoint digest to resume" }).fill(fixture.path.split("/")[4]!);
+ }
+ return methods;
+}
+
+async function run(page: Page, fixture: Fixture) {
+ await panel(page).getByRole("button", {
+ name: fixture.path.endsWith("/resume") ? "Resume checkpoint" : "Create checkpoints and run unit", exact: true,
+ }).click();
+}
+
+test.describe("Protected checkpoint operations in Lab", () => {
+ test.beforeEach(async ({ page }) => {
+ await page.route("**/api/v1/lab/runs", (route) => route.fulfill({ json: [] }));
+ await page.goto("/#/lab/runner");
+ });
+
+ for (const name of ["created", "unicode", "failed", "reused", "retried", "execution_error", "reused_error", "incompatible", "stale", "checkpoint_error"] as const) {
+ test(`renders and downloads actual protected producer ${name} values`, async ({ page }) => {
+ const fixture = fixtures[name];
+ const methods = await prepare(page, fixture);
+ await run(page, fixture);
+ const value = parsed(fixture);
+ const submission = value.submission;
+ const terminal = submission?.terminal;
+ await expect(result(page).locator("dd")).toHaveText([
+ value.status, String(value.exit_code), value.source?.boundary ?? "Unavailable",
+ value.source_withheld ? "Withheld - capture admission did not succeed"
+ : value.source ? "Admitted original declaration" : "Unavailable",
+ value.source?.created_at ?? "Unavailable", value.compatibility_result,
+ value.compatibility_differences.join(", ") || "None reported",
+ String(value.restored_units ?? "Unavailable"), String(value.remaining_work ?? "Unavailable"),
+ String(submission?.engine_calls ?? "Unavailable"), terminal?.outcome ?? "Unavailable",
+ String(terminal?.engine_calls ?? "Unavailable"), String(terminal?.selection?.attempt ?? "Unavailable"),
+ submission?.reservation?.execution_id ?? "Unavailable", submission?.reservation?.parent?.execution_id ?? "Unavailable",
+ submission?.reservation?.parent?.relationship ?? "Unavailable", submission?.decision?.policy ?? "Unavailable",
+ submission?.decision?.source?.execution_id ?? "Unavailable",
+ value.before_checkpoint_sha256 ?? "Unavailable", value.finalized_checkpoint_sha256 ?? "Unavailable",
+ ]);
+ const download = page.waitForEvent("download");
+ await panel(page).getByRole("button", { name: "Download checkpoint result JSON" }).click();
+ const file = await download;
+ expect(file.suggestedFilename()).toBe("waza-checkpoint-result.json");
+ const path = await file.path();
+ expect(path).not.toBeNull();
+ expect(await readFile(path!, "utf8")).toBe(fixture.response);
+ expect(methods).toEqual(["POST"]);
+ if (name === "reused") {
+ expect(value.exit_code).toBe(1);
+ expect(value.submission?.engine_calls).toBe(0);
+ expect(value.submission?.terminal?.selection?.attempt).toBe(2);
+ }
+ if (name === "retried") {
+ expect(value.source?.boundary).toBe("before-unit");
+ expect(value.submission?.reservation?.parent?.execution_id).toBe(value.source?.execution_id);
+ expect(value.submission?.reservation?.execution_id).not.toBe(value.source?.execution_id);
+ expect(value.submission?.engine_calls).toBe(1);
+ expect(value.submission?.terminal?.selection?.attempt).toBe(3);
+ }
+ if (name === "incompatible" || name === "stale") {
+ expect(value.source).toBeUndefined();
+ expect(value.restored_units).toBeNull();
+ expect(value.remaining_work).toBeNull();
+ }
+ });
+ }
+
+ for (const control of [
+ { name: "raw report", fixture: "created", mutate: (value: CheckpointOperationResult) => Reflect.set(value.submission!, "report", fixtures.created.admission) },
+ { name: "retagged original terminal", fixture: "reused", mutate: (value: CheckpointOperationResult) => { value.submission!.terminal!.selection!.attempt = 3; } },
+ { name: "foreign source execution", fixture: "reused", mutate: (value: CheckpointOperationResult) => { value.source!.execution_id = parsed(fixtures.created).submission!.reservation!.execution_id; } },
+ { name: "foreign source completion digest", fixture: "reused", mutate: (value: CheckpointOperationResult) => { value.source!.completion_digest = "0".repeat(64); } },
+ { name: "foreign distinct execution decision", fixture: "created", mutate: (value: CheckpointOperationResult) => {
+ value.submission!.decision!.execution_id = parsed(fixtures.failed).submission!.reservation!.execution_id;
+ } },
+ { name: "foreign decision caller", fixture: "created", mutate: (value: CheckpointOperationResult) => {
+ value.submission!.decision!.binding.identity.canonical_digest = "0".repeat(64);
+ } },
+ { name: "paired fresh original attempt retagging", fixture: "created", mutate: (value: CheckpointOperationResult) => {
+ const attempt = "attempt-" + "0".repeat(64);
+ value.submission!.reservation!.binding.identity.attempt = attempt;
+ value.submission!.terminal!.identity!.attempt = attempt;
+ } },
+ { name: "foreign reuse execution mode", fixture: "reused", mutate: (value: CheckpointOperationResult) => {
+ value.submission!.decision!.binding.execution_mode = "copilot";
+ } },
+ { name: "contradictory observed grading status", fixture: "created", mutate: (value: CheckpointOperationResult) => {
+ value.submission!.terminal!.observation!.validations[0]!.passed = false;
+ } },
+ { name: "invented restored zero", fixture: "incompatible", mutate: (value: CheckpointOperationResult) => { value.restored_units = 0; } },
+ { name: "noncanonical original time", fixture: "reused", mutate: (value: CheckpointOperationResult) => { value.source!.created_at = "2026-11-07T16:00:00.000Z"; } },
+ { name: "same digest at different boundaries", fixture: "created", mutate: (value: CheckpointOperationResult) => { value.finalized_checkpoint_sha256 = value.before_checkpoint_sha256; } },
+ { name: "mismatched caller identity", fixture: "created", mutate: (value: CheckpointOperationResult) => { value.submission!.caller!.canonical_digest = "0".repeat(64); } },
+ { name: "contradictory original exit", fixture: "created", mutate: (value: CheckpointOperationResult) => { value.exit_code = 1; value.submission!.exit_code = 1; } },
+ { name: "contradictory new call count", fixture: "created", mutate: (value: CheckpointOperationResult) => { value.submission!.engine_calls = 0; } },
+ { name: "unknown error grader", fixture: "execution_error", mutate: (value: CheckpointOperationResult) => {
+ value.submission!.terminal!.observation!.validations = [{ identifier: "unadmitted", passed: false }];
+ } },
+ { name: "duplicate error grader", fixture: "execution_error", mutate: (value: CheckpointOperationResult) => {
+ const verdict = { identifier: value.submission!.reservation!.binding.grader_ids[0]!, passed: false };
+ value.submission!.terminal!.observation!.validations = [verdict, { ...verdict }];
+ } },
+ ...(["created", "checkpoint_error"] as const).flatMap((fixture) => [
+ { name: `${fixture} foreign completion execution`, fixture, mutate: (value: CheckpointOperationResult) => {
+ value.submission!.completion!.execution_id = parsed(fixtures.failed).submission!.reservation!.execution_id;
+ } },
+ { name: `${fixture} foreign completion reservation`, fixture, mutate: (value: CheckpointOperationResult) => {
+ value.submission!.completion!.reservation_digest = "0".repeat(64);
+ } },
+ ]),
+ ] as const) {
+ test(`rejects ${control.name} without rendering or exporting success`, async ({ page }) => {
+ const fixture = fixtures[control.fixture];
+ const value = parsed(fixture);
+ control.mutate(value);
+ await prepare(page, fixture, JSON.stringify(value), fixture.http_status, String(value.exit_code));
+ await run(page, fixture);
+ await expect(panel(page).getByRole("alert")).toBeVisible();
+ await expect(result(page)).toHaveCount(0);
+ await expect(panel(page).getByRole("button", { name: "Download checkpoint result JSON" })).toHaveCount(0);
+ });
+ }
+
+ for (const control of ["same-work checkpoint digest", "canonical source creation time"] as const) {
+ test(`rejects changed ${control} before rendering or downloading`, async ({ page }) => {
+ const fixture = { ...fixtures.reused };
+ const value = parsed(fixture);
+ let response = fixture.response;
+ if (control === "same-work checkpoint digest") {
+ const original = parsed(fixtures.failed);
+ expect(original.submission?.reservation?.execution_id).toBe(value.source?.execution_id);
+ expect(original.submission?.reservation?.logical.work).toBe(value.source?.compatibility.work);
+ expect(original.before_checkpoint_sha256).toMatch(/^[a-f0-9]{64}$/);
+ fixture.path = `/api/v1/checkpoints/${original.before_checkpoint_sha256}/resume`;
+ expect(fixture.path).not.toBe(fixtures.reused.path);
+ } else {
+ const changed = "2010-01-02T03:04:05.123456789Z";
+ expect(value.source!.created_at).not.toBe(changed);
+ value.source!.created_at = changed;
+ response = JSON.stringify(value);
+ }
+ const methods = await prepare(page, fixture, response);
+ await run(page, fixture);
+ await expect(panel(page).getByRole("alert")).toContainText("requested checkpoint digest");
+ await expect(result(page)).toHaveCount(0);
+ await expect(panel(page).getByRole("button", { name: "Download checkpoint result JSON" })).toHaveCount(0);
+ expect(methods).toEqual(["POST"]);
+ });
+ }
+
+ for (const name of ["created", "checkpoint_error"] as const) {
+ test(`rejects a source-less completed ${name} resume before rendering or downloading`, async ({ page }) => {
+ const digest = "0".repeat(64);
+ const fixture = { ...fixtures[name], path: `/api/v1/checkpoints/${digest}/resume` };
+ const value = parsed(fixture);
+ expect(value.submission?.status).toBe("completed");
+ expect(value.source).toBeUndefined();
+ expect(digest).not.toBe(value.before_checkpoint_sha256);
+ expect(digest).not.toBe(value.finalized_checkpoint_sha256);
+ const methods = await prepare(page, fixture);
+ await run(page, fixture);
+ await expect(panel(page).getByRole("alert")).toContainText("completed resume without its checkpoint source");
+ await expect(result(page)).toHaveCount(0);
+ await expect(panel(page).getByRole("button", { name: "Download checkpoint result JSON" })).toHaveCount(0);
+ expect(methods).toEqual(["POST"]);
+ });
+ }
+
+ test("rejects resume-only state replayed as a create response", async ({ page }) => {
+ const fixture = { ...fixtures.reused, path: "/api/v1/checkpoints" };
+ expect(parsed(fixture).status).toBe("reused");
+ const methods = await prepare(page, fixture);
+ await run(page, fixture);
+ await expect(panel(page).getByRole("alert")).toContainText("create response contains resume-only state");
+ await expect(result(page)).toHaveCount(0);
+ await expect(panel(page).getByRole("button", { name: "Download checkpoint result JSON" })).toHaveCount(0);
+ expect(methods).toEqual(["POST"]);
+ });
+
+ for (const name of ["reused", "retried"] as const) {
+ test(`admits ${name} source with independently reordered declaration keys`, async ({ page }) => {
+ const fixture = fixtures[name];
+ const response = JSON.stringify(parsed(fixture), (_key, value: unknown) =>
+ value !== null && typeof value === "object" && !Array.isArray(value)
+ ? Object.fromEntries(Object.entries(value).reverse()) : value);
+ expect(response).not.toBe(fixture.response);
+ const methods = await prepare(page, fixture, response);
+ await run(page, fixture);
+ await expect(result(page)).toBeVisible();
+ const download = page.waitForEvent("download");
+ await panel(page).getByRole("button", { name: "Download checkpoint result JSON" }).click();
+ const file = await download;
+ const path = await file.path();
+ expect(path).not.toBeNull();
+ expect(await readFile(path!, "utf8")).toBe(response);
+ expect(methods).toEqual(["POST"]);
+ });
+ }
+
+ test("invalid digest never sends admission or operation requests", async ({ page }) => {
+ let requests = 0;
+ await page.route("**/api/v1/checkpoints**", () => { requests++; });
+ await page.route("**/api/v1/runner/validate", () => { requests++; });
+ for (const digest of ["", "A".repeat(64), "a".repeat(63), "a".repeat(64) + " "]) {
+ await panel(page).getByRole("textbox", { name: "Checkpoint digest to resume" }).fill(digest);
+ await panel(page).getByRole("button", { name: "Resume checkpoint", exact: true }).click();
+ await expect(panel(page).getByRole("alert")).toContainText("full lowercase SHA-256");
+ }
+ expect(requests).toBe(0);
+ });
+
+ test("reload failure clears original success and withholds private malformed provider text", async ({ page }) => {
+ const fixture = fixtures.created;
+ await prepare(page, fixture);
+ await run(page, fixture);
+ await expect(result(page)).toBeVisible();
+ await page.route("**/api/v1/checkpoints**", (route) => route.fulfill({
+ status: 503, body: "https://private-provider.example/secret /private/operator/store",
+ }));
+ await run(page, fixture);
+ await expect(panel(page).getByRole("alert")).toBeVisible();
+ await expect(result(page)).toHaveCount(0);
+ await expect(panel(page)).not.toContainText("private-provider");
+ await expect(panel(page)).not.toContainText("/private/operator/store");
+ await expect(panel(page).getByRole("button", { name: "Download checkpoint result JSON" })).toHaveCount(0);
+ });
+
+ test("editing a pending request aborts old admission without dispatching or resurrecting a result", async ({ page }) => {
+ let release!: () => void;
+ const gate = new Promise((resolve) => { release = resolve; });
+ let admissions = 0;
+ let operations = 0;
+ await page.route("**/api/v1/runner/validate", async (route) => {
+ admissions++;
+ await gate;
+ await route.fulfill({ json: fixtures.created.admission });
+ });
+ await page.route("**/api/v1/checkpoints**", () => { operations++; });
+ await run(page, fixtures.created);
+ await expect.poll(() => admissions).toBe(1);
+ await panel(page).getByRole("textbox", { name: "Captured checkpoint submission (JSON)" }).fill(JSON.stringify(fixtures.failed.request));
+ release();
+ await expect(panel(page).getByRole("button", { name: "Create checkpoints and run unit", exact: true })).toBeEnabled();
+ await expect(result(page)).toHaveCount(0);
+ expect(operations).toBe(0);
+ });
+
+ test("header and HTTP contradictions cannot become completed results", async ({ page }) => {
+ for (const [status, exit] of [[200, "6"], [409, "0"]] as const) {
+ await prepare(page, fixtures.created, fixtures.created.response, status, exit);
+ await run(page, fixtures.created);
+ await expect(panel(page).getByRole("alert")).toContainText("transport status contradicts");
+ await expect(result(page)).toHaveCount(0);
+ }
+ });
+
+ test("synthetic demo has no operation inputs or mutable requests", async ({ page }) => {
+ const calls: string[] = [];
+ page.on("request", (request) => {
+ if (request.url().includes("/api/v1/checkpoints")) calls.push(request.method());
+ });
+ await page.goto("/#/lab/demo/runner");
+ await expect(panel(page)).toContainText("unavailable in the synthetic demo");
+ await expect(panel(page).getByRole("textbox")).toHaveCount(0);
+ await expect(panel(page).getByRole("button")).toHaveCount(0);
+ expect(calls).toEqual([]);
+ });
+
+ test("records the producer-backed failed-original reuse screenshot", async ({ page }) => {
+ await prepare(page, fixtures.reused);
+ await run(page, fixtures.reused);
+ await expect(result(page)).toContainText("evaluation_failure");
+ const path = "../docs/images/dashboard-lab-checkpoint.png";
+ await panel(page).screenshot({ path, animations: "disabled" });
+ await copyFile(path, "../site/public/images/dashboard-lab-checkpoint.png");
+ });
+});
diff --git a/web/e2e/fixtures/aggregate-harness.html b/web/e2e/fixtures/aggregate-harness.html
new file mode 100644
index 000000000..22d7afc79
--- /dev/null
+++ b/web/e2e/fixtures/aggregate-harness.html
@@ -0,0 +1,5 @@
+
+
+ Aggregate component caller control
+
+
diff --git a/web/e2e/fixtures/aggregate-harness.tsx b/web/e2e/fixtures/aggregate-harness.tsx
new file mode 100644
index 000000000..2c36baaba
--- /dev/null
+++ b/web/e2e/fixtures/aggregate-harness.tsx
@@ -0,0 +1,7 @@
+import { createRoot } from "react-dom/client";
+import ManifestAggregate from "../../src/lab/ManifestAggregate";
+import "../../src/lab/lab.css";
+
+const root = document.getElementById("root");
+if (!root) throw new Error("Aggregate caller harness root is missing.");
+createRoot(root).render();
diff --git a/web/e2e/fixtures/checkpoint-operations.json b/web/e2e/fixtures/checkpoint-operations.json
new file mode 100644
index 000000000..25cbe62a6
--- /dev/null
+++ b/web/e2e/fixtures/checkpoint-operations.json
@@ -0,0 +1,932 @@
+{
+ "checkpoint_error": {
+ "path": "/api/v1/checkpoints",
+ "request": {
+ "kind": "waza.execution-submission",
+ "schema_version": "1.0",
+ "request": {
+ "schema_version": "1.0",
+ "versions": [
+ "1.0"
+ ],
+ "required_capabilities": [
+ "exact-unit",
+ "inline.mock-text"
+ ],
+ "optional_capabilities": [],
+ "run_key": "single-unit-example",
+ "selection": {
+ "shard": "shard-a",
+ "task": "selected",
+ "trial": 2,
+ "attempt": 2
+ },
+ "input": {
+ "shard": "shard-a",
+ "eval": "schemaVersion: \"2.0\"\nscenario: runner-example\nname: Runner example\nconfig:\n executor: mock\n model: mock-model\n trials_per_task: 3\n max_attempts: 3\n timeout_seconds: 30\ntasks:\n - adjacent-must-not-be-opened/*.yaml\n",
+ "task": "id: selected\nname: Selected task\ninputs:\n prompt: hello\nexpected:\n output_contains:\n - hello\n"
+ },
+ "timeout_ms": 60000,
+ "allow_paid_execution": false
+ }
+ },
+ "admission": {
+ "kind": "waza.runner-unit",
+ "schema_version": "1.0",
+ "status": "ready",
+ "compatibility": {
+ "kind": "waza.runner-negotiation",
+ "schema_version": "1.0",
+ "status": "compatible",
+ "negotiation": {
+ "version": "1.0",
+ "requested_versions": [
+ "1.0"
+ ],
+ "required_capabilities": [
+ "exact-unit",
+ "inline.mock-text"
+ ],
+ "optional_capabilities": [],
+ "available_capabilities": [
+ "negotiation",
+ "exact-unit",
+ "inline.mock-text",
+ "terminal-result",
+ "inline.copilot-text"
+ ],
+ "selected_capabilities": [
+ "exact-unit",
+ "inline.mock-text"
+ ],
+ "unavailable_optional_capabilities": []
+ },
+ "diagnostics": []
+ },
+ "requested": {
+ "shard": "shard-a",
+ "task": "selected",
+ "trial": 2,
+ "attempt": 2
+ },
+ "resolved": {
+ "shard": "shard-a",
+ "task": "selected",
+ "trial": 2,
+ "attempt": 2
+ },
+ "identity": {
+ "version": "sha256-waza-c14n-v1",
+ "canonical_digest": "46c427b5aa8bfbb61ab297b8e7b780c31ea7ca831ddaa633099498ed749e8dc5",
+ "run": "run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f",
+ "shard": "shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81",
+ "task": "task-fe113f725e5c244fb694124dcff3f073cc78f789891d82e79789a58497699657",
+ "trial": "trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51",
+ "attempt": "attempt-03e5dd35b35d71a351cbbd8661d188279b4c35f3fe73d8789abf02f3e0186c70"
+ },
+ "validation": "valid",
+ "adjacent_work_selected": false,
+ "engine_calls": 0,
+ "diagnostics": []
+ },
+ "http_status": 500,
+ "response": "{\"kind\":\"waza.runner-checkpoint-result\",\"schema_version\":\"1.0\",\"status\":\"checkpoint_error\",\"exit_code\":2,\"submission\":{\"kind\":\"waza.execution-submission-result\",\"schema_version\":\"1.0\",\"status\":\"completed\",\"exit_code\":0,\"engine_calls\":1,\"caller\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"46c427b5aa8bfbb61ab297b8e7b780c31ea7ca831ddaa633099498ed749e8dc5\",\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"shard\":\"shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81\",\"task\":\"task-fe113f725e5c244fb694124dcff3f073cc78f789891d82e79789a58497699657\",\"trial\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\",\"attempt\":\"attempt-03e5dd35b35d71a351cbbd8661d188279b4c35f3fe73d8789abf02f3e0186c70\"},\"decision\":{\"kind\":\"waza.duplicate-decision\",\"schema_version\":\"1.0\",\"decision_id\":\"decision-e2b81b50b017474380411e52fa77e429065a537808c48dd1ad390f9d992f36ef\",\"logical\":{\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"work\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\"},\"binding\":{\"identity\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"46c427b5aa8bfbb61ab297b8e7b780c31ea7ca831ddaa633099498ed749e8dc5\",\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"shard\":\"shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81\",\"task\":\"task-fe113f725e5c244fb694124dcff3f073cc78f789891d82e79789a58497699657\",\"trial\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\",\"attempt\":\"attempt-03e5dd35b35d71a351cbbd8661d188279b4c35f3fe73d8789abf02f3e0186c70\"},\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":2},\"execution_mode\":\"mock\",\"grader_ids\":[\"_output_contains\"]},\"policy\":\"distinct\",\"outcome\":\"distinct\",\"execution_id\":\"exec-81c8642c3cca5f8edcf1c7a44959f7b84a6f12c7e7aba2b0a35b293487adde1a\"},\"reservation\":{\"kind\":\"waza.execution-attempt\",\"schema_version\":\"1.0\",\"logical\":{\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"work\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\"},\"execution_id\":\"exec-81c8642c3cca5f8edcf1c7a44959f7b84a6f12c7e7aba2b0a35b293487adde1a\",\"binding\":{\"identity\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"46c427b5aa8bfbb61ab297b8e7b780c31ea7ca831ddaa633099498ed749e8dc5\",\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"shard\":\"shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81\",\"task\":\"task-fe113f725e5c244fb694124dcff3f073cc78f789891d82e79789a58497699657\",\"trial\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\",\"attempt\":\"attempt-03e5dd35b35d71a351cbbd8661d188279b4c35f3fe73d8789abf02f3e0186c70\"},\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":2},\"execution_mode\":\"mock\",\"grader_ids\":[\"_output_contains\"]}},\"completion\":{\"kind\":\"waza.execution-completion\",\"schema_version\":\"1.0\",\"execution_id\":\"exec-81c8642c3cca5f8edcf1c7a44959f7b84a6f12c7e7aba2b0a35b293487adde1a\",\"reservation_digest\":\"7dbe9d9a80a54fe37e4103ee076fddf2544f3b58150700b98bb886b197896b77\",\"manifest_digest\":\"02e29877d8fb0308fd6c33b2dc69d69c1d3a86f2322f673957ffc0a6916cae12\"},\"terminal\":{\"kind\":\"waza.runner-terminal\",\"schema_version\":\"1.0\",\"outcome\":\"success\",\"exit_code\":0,\"failure_boundary\":\"none\",\"validation\":\"valid\",\"engine_calls\":1,\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":2},\"identity\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"46c427b5aa8bfbb61ab297b8e7b780c31ea7ca831ddaa633099498ed749e8dc5\",\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"shard\":\"shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81\",\"task\":\"task-fe113f725e5c244fb694124dcff3f073cc78f789891d82e79789a58497699657\",\"trial\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\",\"attempt\":\"attempt-03e5dd35b35d71a351cbbd8661d188279b4c35f3fe73d8789abf02f3e0186c70\"},\"observation\":{\"execution_mode\":\"mock\",\"engine_calls\":1,\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":2},\"run_number\":2,\"attempts\":2,\"status\":\"passed\",\"validations\":[{\"identifier\":\"_output_contains\",\"passed\":true}]},\"diagnostics\":[]},\"diagnostics\":[]},\"before_checkpoint_sha256\":\"651a346bf369f3dc11ca50ac436073bf04fd5cf4ebcf836e2562cf2d3e5db0a0\",\"finalized_checkpoint_sha256\":\"bd9c4b5239f321ce408919fd32cc7d7e915e1912547a8094dbbd5e9e0fe4c29a\",\"compatibility_result\":\"unavailable\",\"compatibility_differences\":[],\"restored_units\":null,\"remaining_work\":null,\"diagnostics\":[{\"code\":\"checkpoint_finalized_publication_failed\",\"message\":\"Checkpoint operation did not complete. Preserve source and partial records; inspect operator diagnostics. Unapproved source and invocation text are withheld.\"}]}\n"
+ },
+ "created": {
+ "path": "/api/v1/checkpoints",
+ "request": {
+ "kind": "waza.execution-submission",
+ "schema_version": "1.0",
+ "request": {
+ "schema_version": "1.0",
+ "versions": [
+ "1.0"
+ ],
+ "required_capabilities": [
+ "exact-unit",
+ "inline.mock-text"
+ ],
+ "optional_capabilities": [],
+ "run_key": "single-unit-example",
+ "selection": {
+ "shard": "shard-a",
+ "task": "selected",
+ "trial": 2,
+ "attempt": 2
+ },
+ "input": {
+ "shard": "shard-a",
+ "eval": "schemaVersion: \"2.0\"\nscenario: runner-example\nname: Runner example\nconfig:\n executor: mock\n model: mock-model\n trials_per_task: 3\n max_attempts: 3\n timeout_seconds: 30\ntasks:\n - adjacent-must-not-be-opened/*.yaml\n",
+ "task": "id: selected\nname: Selected task\ninputs:\n prompt: hello\nexpected:\n output_contains:\n - hello\n"
+ },
+ "timeout_ms": 60000,
+ "allow_paid_execution": false
+ }
+ },
+ "admission": {
+ "kind": "waza.runner-unit",
+ "schema_version": "1.0",
+ "status": "ready",
+ "compatibility": {
+ "kind": "waza.runner-negotiation",
+ "schema_version": "1.0",
+ "status": "compatible",
+ "negotiation": {
+ "version": "1.0",
+ "requested_versions": [
+ "1.0"
+ ],
+ "required_capabilities": [
+ "exact-unit",
+ "inline.mock-text"
+ ],
+ "optional_capabilities": [],
+ "available_capabilities": [
+ "negotiation",
+ "exact-unit",
+ "inline.mock-text",
+ "terminal-result",
+ "inline.copilot-text"
+ ],
+ "selected_capabilities": [
+ "exact-unit",
+ "inline.mock-text"
+ ],
+ "unavailable_optional_capabilities": []
+ },
+ "diagnostics": []
+ },
+ "requested": {
+ "shard": "shard-a",
+ "task": "selected",
+ "trial": 2,
+ "attempt": 2
+ },
+ "resolved": {
+ "shard": "shard-a",
+ "task": "selected",
+ "trial": 2,
+ "attempt": 2
+ },
+ "identity": {
+ "version": "sha256-waza-c14n-v1",
+ "canonical_digest": "46c427b5aa8bfbb61ab297b8e7b780c31ea7ca831ddaa633099498ed749e8dc5",
+ "run": "run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f",
+ "shard": "shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81",
+ "task": "task-fe113f725e5c244fb694124dcff3f073cc78f789891d82e79789a58497699657",
+ "trial": "trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51",
+ "attempt": "attempt-03e5dd35b35d71a351cbbd8661d188279b4c35f3fe73d8789abf02f3e0186c70"
+ },
+ "validation": "valid",
+ "adjacent_work_selected": false,
+ "engine_calls": 0,
+ "diagnostics": []
+ },
+ "http_status": 200,
+ "response": "{\"kind\":\"waza.runner-checkpoint-result\",\"schema_version\":\"1.0\",\"status\":\"completed\",\"exit_code\":0,\"submission\":{\"kind\":\"waza.execution-submission-result\",\"schema_version\":\"1.0\",\"status\":\"completed\",\"exit_code\":0,\"engine_calls\":1,\"caller\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"46c427b5aa8bfbb61ab297b8e7b780c31ea7ca831ddaa633099498ed749e8dc5\",\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"shard\":\"shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81\",\"task\":\"task-fe113f725e5c244fb694124dcff3f073cc78f789891d82e79789a58497699657\",\"trial\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\",\"attempt\":\"attempt-03e5dd35b35d71a351cbbd8661d188279b4c35f3fe73d8789abf02f3e0186c70\"},\"decision\":{\"kind\":\"waza.duplicate-decision\",\"schema_version\":\"1.0\",\"decision_id\":\"decision-7027eefcf89d3c4f8fc8daee9756ef339bde479fec71769608b6a48dda19c1dc\",\"logical\":{\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"work\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\"},\"binding\":{\"identity\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"46c427b5aa8bfbb61ab297b8e7b780c31ea7ca831ddaa633099498ed749e8dc5\",\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"shard\":\"shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81\",\"task\":\"task-fe113f725e5c244fb694124dcff3f073cc78f789891d82e79789a58497699657\",\"trial\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\",\"attempt\":\"attempt-03e5dd35b35d71a351cbbd8661d188279b4c35f3fe73d8789abf02f3e0186c70\"},\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":2},\"execution_mode\":\"mock\",\"grader_ids\":[\"_output_contains\"]},\"policy\":\"distinct\",\"outcome\":\"distinct\",\"execution_id\":\"exec-584dd4bac4c385883fd79453271bc57f49deb85c78359ce60b8e5df5f047b936\"},\"reservation\":{\"kind\":\"waza.execution-attempt\",\"schema_version\":\"1.0\",\"logical\":{\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"work\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\"},\"execution_id\":\"exec-584dd4bac4c385883fd79453271bc57f49deb85c78359ce60b8e5df5f047b936\",\"binding\":{\"identity\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"46c427b5aa8bfbb61ab297b8e7b780c31ea7ca831ddaa633099498ed749e8dc5\",\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"shard\":\"shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81\",\"task\":\"task-fe113f725e5c244fb694124dcff3f073cc78f789891d82e79789a58497699657\",\"trial\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\",\"attempt\":\"attempt-03e5dd35b35d71a351cbbd8661d188279b4c35f3fe73d8789abf02f3e0186c70\"},\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":2},\"execution_mode\":\"mock\",\"grader_ids\":[\"_output_contains\"]}},\"completion\":{\"kind\":\"waza.execution-completion\",\"schema_version\":\"1.0\",\"execution_id\":\"exec-584dd4bac4c385883fd79453271bc57f49deb85c78359ce60b8e5df5f047b936\",\"reservation_digest\":\"cf645a837791fdee8912e85a5d025b3d26399fd7b87710cf783e47a0f54c130e\",\"manifest_digest\":\"a3854628ff63d5cacf314cf3a33a63a49347821c2e3aba9020a0924697729b56\"},\"terminal\":{\"kind\":\"waza.runner-terminal\",\"schema_version\":\"1.0\",\"outcome\":\"success\",\"exit_code\":0,\"failure_boundary\":\"none\",\"validation\":\"valid\",\"engine_calls\":1,\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":2},\"identity\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"46c427b5aa8bfbb61ab297b8e7b780c31ea7ca831ddaa633099498ed749e8dc5\",\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"shard\":\"shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81\",\"task\":\"task-fe113f725e5c244fb694124dcff3f073cc78f789891d82e79789a58497699657\",\"trial\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\",\"attempt\":\"attempt-03e5dd35b35d71a351cbbd8661d188279b4c35f3fe73d8789abf02f3e0186c70\"},\"observation\":{\"execution_mode\":\"mock\",\"engine_calls\":1,\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":2},\"run_number\":2,\"attempts\":2,\"status\":\"passed\",\"validations\":[{\"identifier\":\"_output_contains\",\"passed\":true}]},\"diagnostics\":[]},\"diagnostics\":[]},\"before_checkpoint_sha256\":\"a35f2b159a5a6cced45f3667960848c03ac769088133d3e3ac547c9becee5be5\",\"finalized_checkpoint_sha256\":\"1c81d6ab451b48f2f1621f46bb097e149311209619282bd94d261f123574306b\",\"compatibility_result\":\"compatible\",\"compatibility_differences\":[],\"restored_units\":0,\"remaining_work\":0,\"diagnostics\":[]}\n"
+ },
+ "execution_error": {
+ "path": "/api/v1/checkpoints",
+ "request": {
+ "kind": "waza.execution-submission",
+ "schema_version": "1.0",
+ "request": {
+ "schema_version": "1.0",
+ "versions": [
+ "1.0"
+ ],
+ "required_capabilities": [
+ "exact-unit",
+ "inline.mock-text"
+ ],
+ "optional_capabilities": [],
+ "run_key": "single-unit-example",
+ "selection": {
+ "shard": "shard-a",
+ "task": "selected",
+ "trial": 2,
+ "attempt": 2
+ },
+ "input": {
+ "shard": "shard-a",
+ "eval": "schemaVersion: \"2.0\"\nscenario: runner-example\nname: Runner example\nconfig:\n executor: mock\n model: mock-model\n trials_per_task: 3\n max_attempts: 3\n timeout_seconds: 30\ntasks:\n - adjacent-must-not-be-opened/*.yaml\n",
+ "task": "id: selected\nname: Selected task\ninputs:\n prompt: hello\nexpected:\n output_contains:\n - hello\n"
+ },
+ "timeout_ms": 60000,
+ "allow_paid_execution": false
+ }
+ },
+ "admission": {
+ "kind": "waza.runner-unit",
+ "schema_version": "1.0",
+ "status": "ready",
+ "compatibility": {
+ "kind": "waza.runner-negotiation",
+ "schema_version": "1.0",
+ "status": "compatible",
+ "negotiation": {
+ "version": "1.0",
+ "requested_versions": [
+ "1.0"
+ ],
+ "required_capabilities": [
+ "exact-unit",
+ "inline.mock-text"
+ ],
+ "optional_capabilities": [],
+ "available_capabilities": [
+ "negotiation",
+ "exact-unit",
+ "inline.mock-text",
+ "terminal-result",
+ "inline.copilot-text"
+ ],
+ "selected_capabilities": [
+ "exact-unit",
+ "inline.mock-text"
+ ],
+ "unavailable_optional_capabilities": []
+ },
+ "diagnostics": []
+ },
+ "requested": {
+ "shard": "shard-a",
+ "task": "selected",
+ "trial": 2,
+ "attempt": 2
+ },
+ "resolved": {
+ "shard": "shard-a",
+ "task": "selected",
+ "trial": 2,
+ "attempt": 2
+ },
+ "identity": {
+ "version": "sha256-waza-c14n-v1",
+ "canonical_digest": "46c427b5aa8bfbb61ab297b8e7b780c31ea7ca831ddaa633099498ed749e8dc5",
+ "run": "run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f",
+ "shard": "shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81",
+ "task": "task-fe113f725e5c244fb694124dcff3f073cc78f789891d82e79789a58497699657",
+ "trial": "trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51",
+ "attempt": "attempt-03e5dd35b35d71a351cbbd8661d188279b4c35f3fe73d8789abf02f3e0186c70"
+ },
+ "validation": "valid",
+ "adjacent_work_selected": false,
+ "engine_calls": 0,
+ "diagnostics": []
+ },
+ "http_status": 200,
+ "response": "{\"kind\":\"waza.runner-checkpoint-result\",\"schema_version\":\"1.0\",\"status\":\"completed\",\"exit_code\":2,\"submission\":{\"kind\":\"waza.execution-submission-result\",\"schema_version\":\"1.0\",\"status\":\"completed\",\"exit_code\":2,\"engine_calls\":1,\"caller\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"46c427b5aa8bfbb61ab297b8e7b780c31ea7ca831ddaa633099498ed749e8dc5\",\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"shard\":\"shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81\",\"task\":\"task-fe113f725e5c244fb694124dcff3f073cc78f789891d82e79789a58497699657\",\"trial\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\",\"attempt\":\"attempt-03e5dd35b35d71a351cbbd8661d188279b4c35f3fe73d8789abf02f3e0186c70\"},\"decision\":{\"kind\":\"waza.duplicate-decision\",\"schema_version\":\"1.0\",\"decision_id\":\"decision-c4e5ddab4839d68e34b588649fe3cd8c2d782150d97ccc65ffca136653b41e16\",\"logical\":{\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"work\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\"},\"binding\":{\"identity\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"46c427b5aa8bfbb61ab297b8e7b780c31ea7ca831ddaa633099498ed749e8dc5\",\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"shard\":\"shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81\",\"task\":\"task-fe113f725e5c244fb694124dcff3f073cc78f789891d82e79789a58497699657\",\"trial\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\",\"attempt\":\"attempt-03e5dd35b35d71a351cbbd8661d188279b4c35f3fe73d8789abf02f3e0186c70\"},\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":2},\"execution_mode\":\"mock\",\"grader_ids\":[\"_output_contains\"]},\"policy\":\"distinct\",\"outcome\":\"distinct\",\"execution_id\":\"exec-9cb99e52d9c306cc6d5039b351d59e2915b2f14cafb2f485c0667b3e1c89b3a6\"},\"reservation\":{\"kind\":\"waza.execution-attempt\",\"schema_version\":\"1.0\",\"logical\":{\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"work\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\"},\"execution_id\":\"exec-9cb99e52d9c306cc6d5039b351d59e2915b2f14cafb2f485c0667b3e1c89b3a6\",\"binding\":{\"identity\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"46c427b5aa8bfbb61ab297b8e7b780c31ea7ca831ddaa633099498ed749e8dc5\",\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"shard\":\"shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81\",\"task\":\"task-fe113f725e5c244fb694124dcff3f073cc78f789891d82e79789a58497699657\",\"trial\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\",\"attempt\":\"attempt-03e5dd35b35d71a351cbbd8661d188279b4c35f3fe73d8789abf02f3e0186c70\"},\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":2},\"execution_mode\":\"mock\",\"grader_ids\":[\"_output_contains\"]}},\"completion\":{\"kind\":\"waza.execution-completion\",\"schema_version\":\"1.0\",\"execution_id\":\"exec-9cb99e52d9c306cc6d5039b351d59e2915b2f14cafb2f485c0667b3e1c89b3a6\",\"reservation_digest\":\"1560682ab8ba7d9b70b4f538ef4e90441ee6ddcb052d87a0e4c54bc798ec8667\",\"manifest_digest\":\"b72402a5d0a131e3c40a7cb32666bba285d4ef7bbfbd4871477069daee9fd245\"},\"terminal\":{\"kind\":\"waza.runner-terminal\",\"schema_version\":\"1.0\",\"outcome\":\"infrastructure_failure\",\"exit_code\":2,\"failure_boundary\":\"execution\",\"validation\":\"valid\",\"engine_calls\":1,\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":2},\"identity\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"46c427b5aa8bfbb61ab297b8e7b780c31ea7ca831ddaa633099498ed749e8dc5\",\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"shard\":\"shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81\",\"task\":\"task-fe113f725e5c244fb694124dcff3f073cc78f789891d82e79789a58497699657\",\"trial\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\",\"attempt\":\"attempt-03e5dd35b35d71a351cbbd8661d188279b4c35f3fe73d8789abf02f3e0186c70\"},\"observation\":{\"execution_mode\":\"mock\",\"engine_calls\":1,\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":2},\"run_number\":2,\"attempts\":2,\"status\":\"error\",\"validations\":[],\"error_msg\":\"[REDACTED]\"},\"diagnostics\":[{\"code\":\"infrastructure_failure\",\"message\":\"[REDACTED]\"}]},\"diagnostics\":[]},\"before_checkpoint_sha256\":\"e3ee5c75713b32b8ab370eaed41446cfb11339d89f1cc1b6efa675b12017cf2e\",\"finalized_checkpoint_sha256\":\"8052eaabbc39713fbe3f485835de8ddbcbf8a0224c7e3a10a3a7e6f3ec9faf96\",\"compatibility_result\":\"compatible\",\"compatibility_differences\":[],\"restored_units\":0,\"remaining_work\":0,\"diagnostics\":[]}\n"
+ },
+ "failed": {
+ "path": "/api/v1/checkpoints",
+ "request": {
+ "kind": "waza.execution-submission",
+ "schema_version": "1.0",
+ "request": {
+ "schema_version": "1.0",
+ "versions": [
+ "1.0"
+ ],
+ "required_capabilities": [
+ "exact-unit",
+ "inline.mock-text"
+ ],
+ "optional_capabilities": [],
+ "run_key": "single-unit-example",
+ "selection": {
+ "shard": "shard-a",
+ "task": "selected",
+ "trial": 2,
+ "attempt": 2
+ },
+ "input": {
+ "shard": "shard-a",
+ "eval": "schemaVersion: \"2.0\"\nscenario: runner-example\nname: Runner example\nconfig:\n executor: mock\n model: mock-model\n trials_per_task: 3\n max_attempts: 3\n timeout_seconds: 30\ntasks:\n - adjacent-must-not-be-opened/*.yaml\n",
+ "task": "id: selected\nname: Selected task\ninputs:\n prompt: hello\nexpected:\n output_contains:\n - missing-browser-control\n"
+ },
+ "timeout_ms": 60000,
+ "allow_paid_execution": false
+ }
+ },
+ "admission": {
+ "kind": "waza.runner-unit",
+ "schema_version": "1.0",
+ "status": "ready",
+ "compatibility": {
+ "kind": "waza.runner-negotiation",
+ "schema_version": "1.0",
+ "status": "compatible",
+ "negotiation": {
+ "version": "1.0",
+ "requested_versions": [
+ "1.0"
+ ],
+ "required_capabilities": [
+ "exact-unit",
+ "inline.mock-text"
+ ],
+ "optional_capabilities": [],
+ "available_capabilities": [
+ "negotiation",
+ "exact-unit",
+ "inline.mock-text",
+ "terminal-result",
+ "inline.copilot-text"
+ ],
+ "selected_capabilities": [
+ "exact-unit",
+ "inline.mock-text"
+ ],
+ "unavailable_optional_capabilities": []
+ },
+ "diagnostics": []
+ },
+ "requested": {
+ "shard": "shard-a",
+ "task": "selected",
+ "trial": 2,
+ "attempt": 2
+ },
+ "resolved": {
+ "shard": "shard-a",
+ "task": "selected",
+ "trial": 2,
+ "attempt": 2
+ },
+ "identity": {
+ "version": "sha256-waza-c14n-v1",
+ "canonical_digest": "ef33ecf81cdb3979423271e2e28a0a6dfa65d688324be8f105624b5045e5d975",
+ "run": "run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f",
+ "shard": "shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81",
+ "task": "task-26c14ed6ea71f44b902eb09c2c96e157a5f3e6383f88802906090674d0cb7b1f",
+ "trial": "trial-3dc7c42454b11017a3b08056ad1f92fc1f0ed94dfb5a6306608c0aad78634974",
+ "attempt": "attempt-953b4e1ad40d66c77fb41691a9c583595d603eb725514d26f16bad2485f23385"
+ },
+ "validation": "valid",
+ "adjacent_work_selected": false,
+ "engine_calls": 0,
+ "diagnostics": []
+ },
+ "http_status": 200,
+ "response": "{\"kind\":\"waza.runner-checkpoint-result\",\"schema_version\":\"1.0\",\"status\":\"completed\",\"exit_code\":1,\"submission\":{\"kind\":\"waza.execution-submission-result\",\"schema_version\":\"1.0\",\"status\":\"completed\",\"exit_code\":1,\"engine_calls\":1,\"caller\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"ef33ecf81cdb3979423271e2e28a0a6dfa65d688324be8f105624b5045e5d975\",\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"shard\":\"shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81\",\"task\":\"task-26c14ed6ea71f44b902eb09c2c96e157a5f3e6383f88802906090674d0cb7b1f\",\"trial\":\"trial-3dc7c42454b11017a3b08056ad1f92fc1f0ed94dfb5a6306608c0aad78634974\",\"attempt\":\"attempt-953b4e1ad40d66c77fb41691a9c583595d603eb725514d26f16bad2485f23385\"},\"decision\":{\"kind\":\"waza.duplicate-decision\",\"schema_version\":\"1.0\",\"decision_id\":\"decision-6191b29fe02f2393b257c3f2b9d624284ddc492aa28a44c2971388429a868c0f\",\"logical\":{\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"work\":\"trial-3dc7c42454b11017a3b08056ad1f92fc1f0ed94dfb5a6306608c0aad78634974\"},\"binding\":{\"identity\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"ef33ecf81cdb3979423271e2e28a0a6dfa65d688324be8f105624b5045e5d975\",\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"shard\":\"shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81\",\"task\":\"task-26c14ed6ea71f44b902eb09c2c96e157a5f3e6383f88802906090674d0cb7b1f\",\"trial\":\"trial-3dc7c42454b11017a3b08056ad1f92fc1f0ed94dfb5a6306608c0aad78634974\",\"attempt\":\"attempt-953b4e1ad40d66c77fb41691a9c583595d603eb725514d26f16bad2485f23385\"},\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":2},\"execution_mode\":\"mock\",\"grader_ids\":[\"_output_contains\"]},\"policy\":\"distinct\",\"outcome\":\"distinct\",\"execution_id\":\"exec-0f70ffac1812236a4f3acb06c8dbc977e2b325c525338c17817cae873edb5f9a\"},\"reservation\":{\"kind\":\"waza.execution-attempt\",\"schema_version\":\"1.0\",\"logical\":{\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"work\":\"trial-3dc7c42454b11017a3b08056ad1f92fc1f0ed94dfb5a6306608c0aad78634974\"},\"execution_id\":\"exec-0f70ffac1812236a4f3acb06c8dbc977e2b325c525338c17817cae873edb5f9a\",\"binding\":{\"identity\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"ef33ecf81cdb3979423271e2e28a0a6dfa65d688324be8f105624b5045e5d975\",\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"shard\":\"shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81\",\"task\":\"task-26c14ed6ea71f44b902eb09c2c96e157a5f3e6383f88802906090674d0cb7b1f\",\"trial\":\"trial-3dc7c42454b11017a3b08056ad1f92fc1f0ed94dfb5a6306608c0aad78634974\",\"attempt\":\"attempt-953b4e1ad40d66c77fb41691a9c583595d603eb725514d26f16bad2485f23385\"},\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":2},\"execution_mode\":\"mock\",\"grader_ids\":[\"_output_contains\"]}},\"completion\":{\"kind\":\"waza.execution-completion\",\"schema_version\":\"1.0\",\"execution_id\":\"exec-0f70ffac1812236a4f3acb06c8dbc977e2b325c525338c17817cae873edb5f9a\",\"reservation_digest\":\"f9662ae0bf654e5f5995f4689df8c3a770e089388aba7d7373f631cf584a7f5e\",\"manifest_digest\":\"8c8aa300a8043594c7838343b5a330ba890cf05fcba9eda518e586e431e53f55\"},\"terminal\":{\"kind\":\"waza.runner-terminal\",\"schema_version\":\"1.0\",\"outcome\":\"evaluation_failure\",\"exit_code\":1,\"failure_boundary\":\"none\",\"validation\":\"valid\",\"engine_calls\":1,\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":2},\"identity\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"ef33ecf81cdb3979423271e2e28a0a6dfa65d688324be8f105624b5045e5d975\",\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"shard\":\"shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81\",\"task\":\"task-26c14ed6ea71f44b902eb09c2c96e157a5f3e6383f88802906090674d0cb7b1f\",\"trial\":\"trial-3dc7c42454b11017a3b08056ad1f92fc1f0ed94dfb5a6306608c0aad78634974\",\"attempt\":\"attempt-953b4e1ad40d66c77fb41691a9c583595d603eb725514d26f16bad2485f23385\"},\"observation\":{\"execution_mode\":\"mock\",\"engine_calls\":1,\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":2},\"run_number\":2,\"attempts\":2,\"status\":\"failed\",\"validations\":[{\"identifier\":\"_output_contains\",\"passed\":false}]},\"diagnostics\":[{\"code\":\"failed_checks\",\"message\":\"[REDACTED]\"}]},\"diagnostics\":[]},\"before_checkpoint_sha256\":\"c8a3724f264bb5c73f543fb5fe6e1d9f09d8e262c9ed42d37937c4749034614b\",\"finalized_checkpoint_sha256\":\"8d6e79951f76cd7f39d4497b672406029ac5a98f8d0656637e21025d43d49493\",\"compatibility_result\":\"compatible\",\"compatibility_differences\":[],\"restored_units\":0,\"remaining_work\":0,\"diagnostics\":[]}\n"
+ },
+ "incompatible": {
+ "path": "/api/v1/checkpoints/8d6e79951f76cd7f39d4497b672406029ac5a98f8d0656637e21025d43d49493/resume",
+ "request": {
+ "kind": "waza.execution-submission",
+ "schema_version": "1.0",
+ "request": {
+ "schema_version": "1.0",
+ "versions": [
+ "1.0"
+ ],
+ "required_capabilities": [
+ "exact-unit",
+ "inline.mock-text"
+ ],
+ "optional_capabilities": [],
+ "run_key": "single-unit-example",
+ "selection": {
+ "shard": "shard-a",
+ "task": "selected",
+ "trial": 2,
+ "attempt": 3
+ },
+ "input": {
+ "shard": "shard-a",
+ "eval": "schemaVersion: \"2.0\"\nscenario: runner-example\nname: Runner example\nconfig:\n executor: mock\n model: mock-model\n trials_per_task: 3\n max_attempts: 3\n timeout_seconds: 30\ntasks:\n - adjacent-must-not-be-opened/*.yaml\n",
+ "task": "id: selected\nname: Selected task\ninputs:\n prompt: hello\nexpected:\n output_contains:\n - missing-browser-control\n"
+ },
+ "timeout_ms": 60000,
+ "allow_paid_execution": false
+ }
+ },
+ "admission": {
+ "kind": "waza.runner-unit",
+ "schema_version": "1.0",
+ "status": "ready",
+ "compatibility": {
+ "kind": "waza.runner-negotiation",
+ "schema_version": "1.0",
+ "status": "compatible",
+ "negotiation": {
+ "version": "1.0",
+ "requested_versions": [
+ "1.0"
+ ],
+ "required_capabilities": [
+ "exact-unit",
+ "inline.mock-text"
+ ],
+ "optional_capabilities": [],
+ "available_capabilities": [
+ "negotiation",
+ "exact-unit",
+ "inline.mock-text",
+ "terminal-result",
+ "inline.copilot-text"
+ ],
+ "selected_capabilities": [
+ "exact-unit",
+ "inline.mock-text"
+ ],
+ "unavailable_optional_capabilities": []
+ },
+ "diagnostics": []
+ },
+ "requested": {
+ "shard": "shard-a",
+ "task": "selected",
+ "trial": 2,
+ "attempt": 3
+ },
+ "resolved": {
+ "shard": "shard-a",
+ "task": "selected",
+ "trial": 2,
+ "attempt": 3
+ },
+ "identity": {
+ "version": "sha256-waza-c14n-v1",
+ "canonical_digest": "5c739eeef17ae3074af226de23020625236dbfd08615454aa9c38ea523906d50",
+ "run": "run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f",
+ "shard": "shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81",
+ "task": "task-26c14ed6ea71f44b902eb09c2c96e157a5f3e6383f88802906090674d0cb7b1f",
+ "trial": "trial-3dc7c42454b11017a3b08056ad1f92fc1f0ed94dfb5a6306608c0aad78634974",
+ "attempt": "attempt-e08bcbd5fb54eb1088d8f66b087505380c772e6970e2f20ddd842bcb415010a5"
+ },
+ "validation": "valid",
+ "adjacent_work_selected": false,
+ "engine_calls": 0,
+ "diagnostics": []
+ },
+ "http_status": 409,
+ "response": "{\"kind\":\"waza.runner-checkpoint-result\",\"schema_version\":\"1.0\",\"status\":\"rejected\",\"exit_code\":6,\"source_withheld\":true,\"compatibility_result\":\"incompatible\",\"compatibility_differences\":[\"runner_revision\"],\"restored_units\":null,\"remaining_work\":null,\"diagnostics\":[{\"code\":\"checkpoint_incompatible\",\"message\":\"Checkpoint operation did not complete. Preserve source and partial records; inspect operator diagnostics. Unapproved source and invocation text are withheld.\"}]}\n"
+ },
+ "retried": {
+ "path": "/api/v1/checkpoints/ba99385052fa1ecfec66893cf5cf37c172554238e266020f2e3affde2004c958/resume",
+ "request": {
+ "kind": "waza.execution-submission",
+ "schema_version": "1.0",
+ "request": {
+ "schema_version": "1.0",
+ "versions": [
+ "1.0"
+ ],
+ "required_capabilities": [
+ "exact-unit",
+ "inline.mock-text"
+ ],
+ "optional_capabilities": [],
+ "run_key": "single-unit-example",
+ "selection": {
+ "shard": "shard-a",
+ "task": "selected",
+ "trial": 2,
+ "attempt": 3
+ },
+ "input": {
+ "shard": "shard-a",
+ "eval": "schemaVersion: \"2.0\"\nscenario: runner-example\nname: Runner example\nconfig:\n executor: mock\n model: mock-model\n trials_per_task: 3\n max_attempts: 3\n timeout_seconds: 30\ntasks:\n - adjacent-must-not-be-opened/*.yaml\n",
+ "task": "id: selected\nname: Selected task\ninputs:\n prompt: hello\nexpected:\n output_contains:\n - hello\n"
+ },
+ "timeout_ms": 60000,
+ "allow_paid_execution": false
+ }
+ },
+ "admission": {
+ "kind": "waza.runner-unit",
+ "schema_version": "1.0",
+ "status": "ready",
+ "compatibility": {
+ "kind": "waza.runner-negotiation",
+ "schema_version": "1.0",
+ "status": "compatible",
+ "negotiation": {
+ "version": "1.0",
+ "requested_versions": [
+ "1.0"
+ ],
+ "required_capabilities": [
+ "exact-unit",
+ "inline.mock-text"
+ ],
+ "optional_capabilities": [],
+ "available_capabilities": [
+ "negotiation",
+ "exact-unit",
+ "inline.mock-text",
+ "terminal-result",
+ "inline.copilot-text"
+ ],
+ "selected_capabilities": [
+ "exact-unit",
+ "inline.mock-text"
+ ],
+ "unavailable_optional_capabilities": []
+ },
+ "diagnostics": []
+ },
+ "requested": {
+ "shard": "shard-a",
+ "task": "selected",
+ "trial": 2,
+ "attempt": 3
+ },
+ "resolved": {
+ "shard": "shard-a",
+ "task": "selected",
+ "trial": 2,
+ "attempt": 3
+ },
+ "identity": {
+ "version": "sha256-waza-c14n-v1",
+ "canonical_digest": "563ed279d1aa5ae9f46a65dcc1983096fc641f5a9860b6c9b7bf80ae8d362065",
+ "run": "run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f",
+ "shard": "shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81",
+ "task": "task-fe113f725e5c244fb694124dcff3f073cc78f789891d82e79789a58497699657",
+ "trial": "trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51",
+ "attempt": "attempt-f10272493f0c171f1723cc682cfd16649c9d6f1aecbcc816f513356772895077"
+ },
+ "validation": "valid",
+ "adjacent_work_selected": false,
+ "engine_calls": 0,
+ "diagnostics": []
+ },
+ "http_status": 200,
+ "response": "{\"kind\":\"waza.runner-checkpoint-result\",\"schema_version\":\"1.0\",\"status\":\"completed\",\"exit_code\":0,\"submission\":{\"kind\":\"waza.execution-submission-result\",\"schema_version\":\"1.0\",\"status\":\"completed\",\"exit_code\":0,\"engine_calls\":1,\"caller\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"563ed279d1aa5ae9f46a65dcc1983096fc641f5a9860b6c9b7bf80ae8d362065\",\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"shard\":\"shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81\",\"task\":\"task-fe113f725e5c244fb694124dcff3f073cc78f789891d82e79789a58497699657\",\"trial\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\",\"attempt\":\"attempt-f10272493f0c171f1723cc682cfd16649c9d6f1aecbcc816f513356772895077\"},\"decision\":{\"kind\":\"waza.duplicate-decision\",\"schema_version\":\"1.0\",\"decision_id\":\"decision-38fb758842fbf3536676bfc6dd94ece4944d9315b3aefc31f97418a8c0b709ce\",\"logical\":{\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"work\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\"},\"binding\":{\"identity\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"563ed279d1aa5ae9f46a65dcc1983096fc641f5a9860b6c9b7bf80ae8d362065\",\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"shard\":\"shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81\",\"task\":\"task-fe113f725e5c244fb694124dcff3f073cc78f789891d82e79789a58497699657\",\"trial\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\",\"attempt\":\"attempt-f10272493f0c171f1723cc682cfd16649c9d6f1aecbcc816f513356772895077\"},\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":3},\"execution_mode\":\"mock\",\"grader_ids\":[\"_output_contains\"]},\"policy\":\"distinct\",\"outcome\":\"distinct\",\"execution_id\":\"exec-d110cbea011399a1c7ae8004a3acf0098454dc215b928ed57e5978397524b926\"},\"reservation\":{\"kind\":\"waza.execution-attempt\",\"schema_version\":\"1.0\",\"logical\":{\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"work\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\"},\"execution_id\":\"exec-d110cbea011399a1c7ae8004a3acf0098454dc215b928ed57e5978397524b926\",\"binding\":{\"identity\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"563ed279d1aa5ae9f46a65dcc1983096fc641f5a9860b6c9b7bf80ae8d362065\",\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"shard\":\"shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81\",\"task\":\"task-fe113f725e5c244fb694124dcff3f073cc78f789891d82e79789a58497699657\",\"trial\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\",\"attempt\":\"attempt-f10272493f0c171f1723cc682cfd16649c9d6f1aecbcc816f513356772895077\"},\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":3},\"execution_mode\":\"mock\",\"grader_ids\":[\"_output_contains\"]},\"parent\":{\"execution_id\":\"exec-bc0f057677a4c8bf6107138b5882c1121c778f6d94e79d57affb41ff33152baa\",\"relationship\":\"retry\"}},\"completion\":{\"kind\":\"waza.execution-completion\",\"schema_version\":\"1.0\",\"execution_id\":\"exec-d110cbea011399a1c7ae8004a3acf0098454dc215b928ed57e5978397524b926\",\"reservation_digest\":\"4168af6a72bb802d24d7ef863f616a87e8d98644c68caa42d9b2bd9e61efdbae\",\"manifest_digest\":\"ee88e425735eb11d161e85455fc4d11cee30cbc23b1d4cb70e9a923853b830ef\"},\"terminal\":{\"kind\":\"waza.runner-terminal\",\"schema_version\":\"1.0\",\"outcome\":\"success\",\"exit_code\":0,\"failure_boundary\":\"none\",\"validation\":\"valid\",\"engine_calls\":1,\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":3},\"identity\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"563ed279d1aa5ae9f46a65dcc1983096fc641f5a9860b6c9b7bf80ae8d362065\",\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"shard\":\"shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81\",\"task\":\"task-fe113f725e5c244fb694124dcff3f073cc78f789891d82e79789a58497699657\",\"trial\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\",\"attempt\":\"attempt-f10272493f0c171f1723cc682cfd16649c9d6f1aecbcc816f513356772895077\"},\"observation\":{\"execution_mode\":\"mock\",\"engine_calls\":1,\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":3},\"run_number\":2,\"attempts\":3,\"status\":\"passed\",\"validations\":[{\"identifier\":\"_output_contains\",\"passed\":true}]},\"diagnostics\":[]},\"diagnostics\":[]},\"source\":{\"kind\":\"waza.execution-checkpoint\",\"schema_version\":\"1.0\",\"boundary\":\"before-unit\",\"created_at\":\"2026-11-10T21:42:09.628503866Z\",\"execution_id\":\"exec-bc0f057677a4c8bf6107138b5882c1121c778f6d94e79d57affb41ff33152baa\",\"reservation_digest\":\"8f07636ae74b37147d9f874c09a24289a0696574359473e5ddcb5afe5af0f3f3\",\"compatibility\":{\"runner_revision\":\"cccccccccccccccccccccccccccccccccccccccc\",\"request_version\":\"1.0\",\"identity_version\":\"sha256-waza-c14n-v1\",\"terminal_version\":\"1.0\",\"manifest_version\":\"1.0\",\"attempt_version\":\"1.0\",\"completion_version\":\"1.0\",\"work\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\",\"request_policy_digest\":\"502111e81ae54b56b6d3d38b9ada4e1047c038706aa4e8dcc0282d7635b77674\"}},\"before_checkpoint_sha256\":\"fe134c5afb43449ad4178dd9d06399e4fda4e28d59e41e5bcdab062b1a9b7793\",\"finalized_checkpoint_sha256\":\"624f74473c4d8a599f3617084c9a0892abfd47a2f317a3e99a5c19e73453b357\",\"compatibility_result\":\"compatible\",\"compatibility_differences\":[],\"restored_units\":0,\"remaining_work\":0,\"diagnostics\":[]}\n"
+ },
+ "reused": {
+ "path": "/api/v1/checkpoints/8d6e79951f76cd7f39d4497b672406029ac5a98f8d0656637e21025d43d49493/resume",
+ "request": {
+ "kind": "waza.execution-submission",
+ "schema_version": "1.0",
+ "request": {
+ "schema_version": "1.0",
+ "versions": [
+ "1.0"
+ ],
+ "required_capabilities": [
+ "exact-unit",
+ "inline.mock-text"
+ ],
+ "optional_capabilities": [],
+ "run_key": "single-unit-example",
+ "selection": {
+ "shard": "shard-a",
+ "task": "selected",
+ "trial": 2,
+ "attempt": 3
+ },
+ "input": {
+ "shard": "shard-a",
+ "eval": "schemaVersion: \"2.0\"\nscenario: runner-example\nname: Runner example\nconfig:\n executor: mock\n model: mock-model\n trials_per_task: 3\n max_attempts: 3\n timeout_seconds: 30\ntasks:\n - adjacent-must-not-be-opened/*.yaml\n",
+ "task": "id: selected\nname: Selected task\ninputs:\n prompt: hello\nexpected:\n output_contains:\n - missing-browser-control\n"
+ },
+ "timeout_ms": 60000,
+ "allow_paid_execution": false
+ }
+ },
+ "admission": {
+ "kind": "waza.runner-unit",
+ "schema_version": "1.0",
+ "status": "ready",
+ "compatibility": {
+ "kind": "waza.runner-negotiation",
+ "schema_version": "1.0",
+ "status": "compatible",
+ "negotiation": {
+ "version": "1.0",
+ "requested_versions": [
+ "1.0"
+ ],
+ "required_capabilities": [
+ "exact-unit",
+ "inline.mock-text"
+ ],
+ "optional_capabilities": [],
+ "available_capabilities": [
+ "negotiation",
+ "exact-unit",
+ "inline.mock-text",
+ "terminal-result",
+ "inline.copilot-text"
+ ],
+ "selected_capabilities": [
+ "exact-unit",
+ "inline.mock-text"
+ ],
+ "unavailable_optional_capabilities": []
+ },
+ "diagnostics": []
+ },
+ "requested": {
+ "shard": "shard-a",
+ "task": "selected",
+ "trial": 2,
+ "attempt": 3
+ },
+ "resolved": {
+ "shard": "shard-a",
+ "task": "selected",
+ "trial": 2,
+ "attempt": 3
+ },
+ "identity": {
+ "version": "sha256-waza-c14n-v1",
+ "canonical_digest": "5c739eeef17ae3074af226de23020625236dbfd08615454aa9c38ea523906d50",
+ "run": "run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f",
+ "shard": "shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81",
+ "task": "task-26c14ed6ea71f44b902eb09c2c96e157a5f3e6383f88802906090674d0cb7b1f",
+ "trial": "trial-3dc7c42454b11017a3b08056ad1f92fc1f0ed94dfb5a6306608c0aad78634974",
+ "attempt": "attempt-e08bcbd5fb54eb1088d8f66b087505380c772e6970e2f20ddd842bcb415010a5"
+ },
+ "validation": "valid",
+ "adjacent_work_selected": false,
+ "engine_calls": 0,
+ "diagnostics": []
+ },
+ "http_status": 200,
+ "response": "{\"kind\":\"waza.runner-checkpoint-result\",\"schema_version\":\"1.0\",\"status\":\"reused\",\"exit_code\":1,\"submission\":{\"kind\":\"waza.execution-submission-result\",\"schema_version\":\"1.0\",\"status\":\"reused\",\"exit_code\":1,\"engine_calls\":0,\"caller\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"5c739eeef17ae3074af226de23020625236dbfd08615454aa9c38ea523906d50\",\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"shard\":\"shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81\",\"task\":\"task-26c14ed6ea71f44b902eb09c2c96e157a5f3e6383f88802906090674d0cb7b1f\",\"trial\":\"trial-3dc7c42454b11017a3b08056ad1f92fc1f0ed94dfb5a6306608c0aad78634974\",\"attempt\":\"attempt-e08bcbd5fb54eb1088d8f66b087505380c772e6970e2f20ddd842bcb415010a5\"},\"decision\":{\"kind\":\"waza.duplicate-decision\",\"schema_version\":\"1.0\",\"decision_id\":\"decision-c95bcc14e2bf88acc750bb7671e1c2d813f23c2007877f614810f942084b3a5b\",\"logical\":{\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"work\":\"trial-3dc7c42454b11017a3b08056ad1f92fc1f0ed94dfb5a6306608c0aad78634974\"},\"binding\":{\"identity\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"5c739eeef17ae3074af226de23020625236dbfd08615454aa9c38ea523906d50\",\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"shard\":\"shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81\",\"task\":\"task-26c14ed6ea71f44b902eb09c2c96e157a5f3e6383f88802906090674d0cb7b1f\",\"trial\":\"trial-3dc7c42454b11017a3b08056ad1f92fc1f0ed94dfb5a6306608c0aad78634974\",\"attempt\":\"attempt-e08bcbd5fb54eb1088d8f66b087505380c772e6970e2f20ddd842bcb415010a5\"},\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":3},\"execution_mode\":\"mock\",\"grader_ids\":[\"_output_contains\"]},\"policy\":\"reuse\",\"outcome\":\"reused\",\"source\":{\"execution_id\":\"exec-0f70ffac1812236a4f3acb06c8dbc977e2b325c525338c17817cae873edb5f9a\",\"manifest_digest\":\"8c8aa300a8043594c7838343b5a330ba890cf05fcba9eda518e586e431e53f55\"}},\"reservation\":{\"kind\":\"waza.execution-attempt\",\"schema_version\":\"1.0\",\"logical\":{\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"work\":\"trial-3dc7c42454b11017a3b08056ad1f92fc1f0ed94dfb5a6306608c0aad78634974\"},\"execution_id\":\"exec-0f70ffac1812236a4f3acb06c8dbc977e2b325c525338c17817cae873edb5f9a\",\"binding\":{\"identity\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"ef33ecf81cdb3979423271e2e28a0a6dfa65d688324be8f105624b5045e5d975\",\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"shard\":\"shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81\",\"task\":\"task-26c14ed6ea71f44b902eb09c2c96e157a5f3e6383f88802906090674d0cb7b1f\",\"trial\":\"trial-3dc7c42454b11017a3b08056ad1f92fc1f0ed94dfb5a6306608c0aad78634974\",\"attempt\":\"attempt-953b4e1ad40d66c77fb41691a9c583595d603eb725514d26f16bad2485f23385\"},\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":2},\"execution_mode\":\"mock\",\"grader_ids\":[\"_output_contains\"]}},\"completion\":{\"kind\":\"waza.execution-completion\",\"schema_version\":\"1.0\",\"execution_id\":\"exec-0f70ffac1812236a4f3acb06c8dbc977e2b325c525338c17817cae873edb5f9a\",\"reservation_digest\":\"f9662ae0bf654e5f5995f4689df8c3a770e089388aba7d7373f631cf584a7f5e\",\"manifest_digest\":\"8c8aa300a8043594c7838343b5a330ba890cf05fcba9eda518e586e431e53f55\"},\"terminal\":{\"kind\":\"waza.runner-terminal\",\"schema_version\":\"1.0\",\"outcome\":\"evaluation_failure\",\"exit_code\":1,\"failure_boundary\":\"none\",\"validation\":\"valid\",\"engine_calls\":1,\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":2},\"identity\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"ef33ecf81cdb3979423271e2e28a0a6dfa65d688324be8f105624b5045e5d975\",\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"shard\":\"shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81\",\"task\":\"task-26c14ed6ea71f44b902eb09c2c96e157a5f3e6383f88802906090674d0cb7b1f\",\"trial\":\"trial-3dc7c42454b11017a3b08056ad1f92fc1f0ed94dfb5a6306608c0aad78634974\",\"attempt\":\"attempt-953b4e1ad40d66c77fb41691a9c583595d603eb725514d26f16bad2485f23385\"},\"observation\":{\"execution_mode\":\"mock\",\"engine_calls\":1,\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":2},\"run_number\":2,\"attempts\":2,\"status\":\"failed\",\"validations\":[{\"identifier\":\"_output_contains\",\"passed\":false}]},\"diagnostics\":[{\"code\":\"failed_checks\",\"message\":\"[REDACTED]\"}]},\"diagnostics\":[]},\"source\":{\"kind\":\"waza.execution-checkpoint\",\"schema_version\":\"1.0\",\"boundary\":\"finalized-unit\",\"created_at\":\"2026-11-10T21:42:09.565954205Z\",\"execution_id\":\"exec-0f70ffac1812236a4f3acb06c8dbc977e2b325c525338c17817cae873edb5f9a\",\"reservation_digest\":\"f9662ae0bf654e5f5995f4689df8c3a770e089388aba7d7373f631cf584a7f5e\",\"compatibility\":{\"runner_revision\":\"cccccccccccccccccccccccccccccccccccccccc\",\"request_version\":\"1.0\",\"identity_version\":\"sha256-waza-c14n-v1\",\"terminal_version\":\"1.0\",\"manifest_version\":\"1.0\",\"attempt_version\":\"1.0\",\"completion_version\":\"1.0\",\"work\":\"trial-3dc7c42454b11017a3b08056ad1f92fc1f0ed94dfb5a6306608c0aad78634974\",\"request_policy_digest\":\"502111e81ae54b56b6d3d38b9ada4e1047c038706aa4e8dcc0282d7635b77674\"},\"completion_digest\":\"5915ef07b671894e441fa9052d3ae2fa560f14e93da4ce8b8a9b89ebe8d40bec\",\"manifest_digest\":\"8c8aa300a8043594c7838343b5a330ba890cf05fcba9eda518e586e431e53f55\"},\"compatibility_result\":\"compatible\",\"compatibility_differences\":[],\"restored_units\":1,\"remaining_work\":0,\"diagnostics\":[]}\n"
+ },
+ "reused_error": {
+ "path": "/api/v1/checkpoints/8052eaabbc39713fbe3f485835de8ddbcbf8a0224c7e3a10a3a7e6f3ec9faf96/resume",
+ "request": {
+ "kind": "waza.execution-submission",
+ "schema_version": "1.0",
+ "request": {
+ "schema_version": "1.0",
+ "versions": [
+ "1.0"
+ ],
+ "required_capabilities": [
+ "exact-unit",
+ "inline.mock-text"
+ ],
+ "optional_capabilities": [],
+ "run_key": "single-unit-example",
+ "selection": {
+ "shard": "shard-a",
+ "task": "selected",
+ "trial": 2,
+ "attempt": 3
+ },
+ "input": {
+ "shard": "shard-a",
+ "eval": "schemaVersion: \"2.0\"\nscenario: runner-example\nname: Runner example\nconfig:\n executor: mock\n model: mock-model\n trials_per_task: 3\n max_attempts: 3\n timeout_seconds: 30\ntasks:\n - adjacent-must-not-be-opened/*.yaml\n",
+ "task": "id: selected\nname: Selected task\ninputs:\n prompt: hello\nexpected:\n output_contains:\n - hello\n"
+ },
+ "timeout_ms": 60000,
+ "allow_paid_execution": false
+ }
+ },
+ "admission": {
+ "kind": "waza.runner-unit",
+ "schema_version": "1.0",
+ "status": "ready",
+ "compatibility": {
+ "kind": "waza.runner-negotiation",
+ "schema_version": "1.0",
+ "status": "compatible",
+ "negotiation": {
+ "version": "1.0",
+ "requested_versions": [
+ "1.0"
+ ],
+ "required_capabilities": [
+ "exact-unit",
+ "inline.mock-text"
+ ],
+ "optional_capabilities": [],
+ "available_capabilities": [
+ "negotiation",
+ "exact-unit",
+ "inline.mock-text",
+ "terminal-result",
+ "inline.copilot-text"
+ ],
+ "selected_capabilities": [
+ "exact-unit",
+ "inline.mock-text"
+ ],
+ "unavailable_optional_capabilities": []
+ },
+ "diagnostics": []
+ },
+ "requested": {
+ "shard": "shard-a",
+ "task": "selected",
+ "trial": 2,
+ "attempt": 3
+ },
+ "resolved": {
+ "shard": "shard-a",
+ "task": "selected",
+ "trial": 2,
+ "attempt": 3
+ },
+ "identity": {
+ "version": "sha256-waza-c14n-v1",
+ "canonical_digest": "563ed279d1aa5ae9f46a65dcc1983096fc641f5a9860b6c9b7bf80ae8d362065",
+ "run": "run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f",
+ "shard": "shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81",
+ "task": "task-fe113f725e5c244fb694124dcff3f073cc78f789891d82e79789a58497699657",
+ "trial": "trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51",
+ "attempt": "attempt-f10272493f0c171f1723cc682cfd16649c9d6f1aecbcc816f513356772895077"
+ },
+ "validation": "valid",
+ "adjacent_work_selected": false,
+ "engine_calls": 0,
+ "diagnostics": []
+ },
+ "http_status": 200,
+ "response": "{\"kind\":\"waza.runner-checkpoint-result\",\"schema_version\":\"1.0\",\"status\":\"reused\",\"exit_code\":2,\"submission\":{\"kind\":\"waza.execution-submission-result\",\"schema_version\":\"1.0\",\"status\":\"reused\",\"exit_code\":2,\"engine_calls\":0,\"caller\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"563ed279d1aa5ae9f46a65dcc1983096fc641f5a9860b6c9b7bf80ae8d362065\",\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"shard\":\"shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81\",\"task\":\"task-fe113f725e5c244fb694124dcff3f073cc78f789891d82e79789a58497699657\",\"trial\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\",\"attempt\":\"attempt-f10272493f0c171f1723cc682cfd16649c9d6f1aecbcc816f513356772895077\"},\"decision\":{\"kind\":\"waza.duplicate-decision\",\"schema_version\":\"1.0\",\"decision_id\":\"decision-cc59e2d5dd9abb95952d2db4fc2f3caa579da03fada28d15f61f092cb9d4f072\",\"logical\":{\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"work\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\"},\"binding\":{\"identity\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"563ed279d1aa5ae9f46a65dcc1983096fc641f5a9860b6c9b7bf80ae8d362065\",\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"shard\":\"shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81\",\"task\":\"task-fe113f725e5c244fb694124dcff3f073cc78f789891d82e79789a58497699657\",\"trial\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\",\"attempt\":\"attempt-f10272493f0c171f1723cc682cfd16649c9d6f1aecbcc816f513356772895077\"},\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":3},\"execution_mode\":\"mock\",\"grader_ids\":[\"_output_contains\"]},\"policy\":\"reuse\",\"outcome\":\"reused\",\"source\":{\"execution_id\":\"exec-9cb99e52d9c306cc6d5039b351d59e2915b2f14cafb2f485c0667b3e1c89b3a6\",\"manifest_digest\":\"b72402a5d0a131e3c40a7cb32666bba285d4ef7bbfbd4871477069daee9fd245\"}},\"reservation\":{\"kind\":\"waza.execution-attempt\",\"schema_version\":\"1.0\",\"logical\":{\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"work\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\"},\"execution_id\":\"exec-9cb99e52d9c306cc6d5039b351d59e2915b2f14cafb2f485c0667b3e1c89b3a6\",\"binding\":{\"identity\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"46c427b5aa8bfbb61ab297b8e7b780c31ea7ca831ddaa633099498ed749e8dc5\",\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"shard\":\"shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81\",\"task\":\"task-fe113f725e5c244fb694124dcff3f073cc78f789891d82e79789a58497699657\",\"trial\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\",\"attempt\":\"attempt-03e5dd35b35d71a351cbbd8661d188279b4c35f3fe73d8789abf02f3e0186c70\"},\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":2},\"execution_mode\":\"mock\",\"grader_ids\":[\"_output_contains\"]}},\"completion\":{\"kind\":\"waza.execution-completion\",\"schema_version\":\"1.0\",\"execution_id\":\"exec-9cb99e52d9c306cc6d5039b351d59e2915b2f14cafb2f485c0667b3e1c89b3a6\",\"reservation_digest\":\"1560682ab8ba7d9b70b4f538ef4e90441ee6ddcb052d87a0e4c54bc798ec8667\",\"manifest_digest\":\"b72402a5d0a131e3c40a7cb32666bba285d4ef7bbfbd4871477069daee9fd245\"},\"terminal\":{\"kind\":\"waza.runner-terminal\",\"schema_version\":\"1.0\",\"outcome\":\"infrastructure_failure\",\"exit_code\":2,\"failure_boundary\":\"execution\",\"validation\":\"valid\",\"engine_calls\":1,\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":2},\"identity\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"46c427b5aa8bfbb61ab297b8e7b780c31ea7ca831ddaa633099498ed749e8dc5\",\"run\":\"run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f\",\"shard\":\"shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81\",\"task\":\"task-fe113f725e5c244fb694124dcff3f073cc78f789891d82e79789a58497699657\",\"trial\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\",\"attempt\":\"attempt-03e5dd35b35d71a351cbbd8661d188279b4c35f3fe73d8789abf02f3e0186c70\"},\"observation\":{\"execution_mode\":\"mock\",\"engine_calls\":1,\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":2},\"run_number\":2,\"attempts\":2,\"status\":\"error\",\"validations\":[],\"error_msg\":\"[REDACTED]\"},\"diagnostics\":[{\"code\":\"infrastructure_failure\",\"message\":\"[REDACTED]\"}]},\"diagnostics\":[]},\"source\":{\"kind\":\"waza.execution-checkpoint\",\"schema_version\":\"1.0\",\"boundary\":\"finalized-unit\",\"created_at\":\"2026-11-10T21:42:09.602782968Z\",\"execution_id\":\"exec-9cb99e52d9c306cc6d5039b351d59e2915b2f14cafb2f485c0667b3e1c89b3a6\",\"reservation_digest\":\"1560682ab8ba7d9b70b4f538ef4e90441ee6ddcb052d87a0e4c54bc798ec8667\",\"compatibility\":{\"runner_revision\":\"cccccccccccccccccccccccccccccccccccccccc\",\"request_version\":\"1.0\",\"identity_version\":\"sha256-waza-c14n-v1\",\"terminal_version\":\"1.0\",\"manifest_version\":\"1.0\",\"attempt_version\":\"1.0\",\"completion_version\":\"1.0\",\"work\":\"trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51\",\"request_policy_digest\":\"502111e81ae54b56b6d3d38b9ada4e1047c038706aa4e8dcc0282d7635b77674\"},\"completion_digest\":\"8ecfae1d76f0e5a9d7430fce932d7bad2251e2610b61f30f4d2b00f94c3a18d8\",\"manifest_digest\":\"b72402a5d0a131e3c40a7cb32666bba285d4ef7bbfbd4871477069daee9fd245\"},\"compatibility_result\":\"compatible\",\"compatibility_differences\":[],\"restored_units\":1,\"remaining_work\":0,\"diagnostics\":[]}\n"
+ },
+ "stale": {
+ "path": "/api/v1/checkpoints/a35f2b159a5a6cced45f3667960848c03ac769088133d3e3ac547c9becee5be5/resume",
+ "request": {
+ "kind": "waza.execution-submission",
+ "schema_version": "1.0",
+ "request": {
+ "schema_version": "1.0",
+ "versions": [
+ "1.0"
+ ],
+ "required_capabilities": [
+ "exact-unit",
+ "inline.mock-text"
+ ],
+ "optional_capabilities": [],
+ "run_key": "single-unit-example",
+ "selection": {
+ "shard": "shard-a",
+ "task": "selected",
+ "trial": 2,
+ "attempt": 2
+ },
+ "input": {
+ "shard": "shard-a",
+ "eval": "schemaVersion: \"2.0\"\nscenario: runner-example\nname: Runner example\nconfig:\n executor: mock\n model: mock-model\n trials_per_task: 3\n max_attempts: 3\n timeout_seconds: 30\ntasks:\n - adjacent-must-not-be-opened/*.yaml\n",
+ "task": "id: selected\nname: Selected task\ninputs:\n prompt: hello\nexpected:\n output_contains:\n - hello\n"
+ },
+ "timeout_ms": 60000,
+ "allow_paid_execution": false
+ }
+ },
+ "admission": {
+ "kind": "waza.runner-unit",
+ "schema_version": "1.0",
+ "status": "ready",
+ "compatibility": {
+ "kind": "waza.runner-negotiation",
+ "schema_version": "1.0",
+ "status": "compatible",
+ "negotiation": {
+ "version": "1.0",
+ "requested_versions": [
+ "1.0"
+ ],
+ "required_capabilities": [
+ "exact-unit",
+ "inline.mock-text"
+ ],
+ "optional_capabilities": [],
+ "available_capabilities": [
+ "negotiation",
+ "exact-unit",
+ "inline.mock-text",
+ "terminal-result",
+ "inline.copilot-text"
+ ],
+ "selected_capabilities": [
+ "exact-unit",
+ "inline.mock-text"
+ ],
+ "unavailable_optional_capabilities": []
+ },
+ "diagnostics": []
+ },
+ "requested": {
+ "shard": "shard-a",
+ "task": "selected",
+ "trial": 2,
+ "attempt": 2
+ },
+ "resolved": {
+ "shard": "shard-a",
+ "task": "selected",
+ "trial": 2,
+ "attempt": 2
+ },
+ "identity": {
+ "version": "sha256-waza-c14n-v1",
+ "canonical_digest": "46c427b5aa8bfbb61ab297b8e7b780c31ea7ca831ddaa633099498ed749e8dc5",
+ "run": "run-a4fe0d7265abd37befd01c024046df1174de8b649ad8bc53a8cbc317e7fcca9f",
+ "shard": "shard-780d0dd8d03603a682b6228ec2916020839fbb21324dee1ce83bc8f81e81ab81",
+ "task": "task-fe113f725e5c244fb694124dcff3f073cc78f789891d82e79789a58497699657",
+ "trial": "trial-ccb6f415db4210a7e071f65c1bcf3dec32860c4fc53e23492070da597a7e8b51",
+ "attempt": "attempt-03e5dd35b35d71a351cbbd8661d188279b4c35f3fe73d8789abf02f3e0186c70"
+ },
+ "validation": "valid",
+ "adjacent_work_selected": false,
+ "engine_calls": 0,
+ "diagnostics": []
+ },
+ "http_status": 409,
+ "response": "{\"kind\":\"waza.runner-checkpoint-result\",\"schema_version\":\"1.0\",\"status\":\"rejected\",\"exit_code\":6,\"source_withheld\":true,\"compatibility_result\":\"unavailable\",\"compatibility_differences\":[],\"restored_units\":null,\"remaining_work\":null,\"diagnostics\":[{\"code\":\"checkpoint_before_source_rejected\",\"message\":\"Checkpoint operation did not complete. Preserve source and partial records; inspect operator diagnostics. Unapproved source and invocation text are withheld.\"}]}\n"
+ },
+ "unicode": {
+ "path": "/api/v1/checkpoints",
+ "request": {
+ "kind": "waza.execution-submission",
+ "schema_version": "1.0",
+ "request": {
+ "schema_version": "1.0",
+ "versions": [
+ "1.0"
+ ],
+ "required_capabilities": [
+ "exact-unit",
+ "inline.mock-text"
+ ],
+ "optional_capabilities": [],
+ "run_key": "single-unit-example",
+ "selection": {
+ "shard": "shard-a",
+ "task": "selected",
+ "trial": 2,
+ "attempt": 2
+ },
+ "input": {
+ "shard": "shard-a",
+ "eval": "schemaVersion: \"2.0\"\nscenario: runner-example\nname: Runner example\nconfig:\n executor: mock\n model: mock-model\n trials_per_task: 3\n max_attempts: 3\n timeout_seconds: 30\ntasks:\n - adjacent-must-not-be-opened/*.yaml\ngraders:\n - name: \"\u003c\u003e\u0026\\u2028\\u2029\"\n type: text\n config:\n contains: [hello]\n - name: \"\\uE000\"\n type: text\n config:\n contains: [hello]\n - name: \"\\U00010000\"\n type: text\n config:\n contains: [hello]\n",
+ "task": "id: selected\nname: Selected task\ninputs:\n prompt: hello\nexpected:\n output_contains:\n - hello\n"
+ },
+ "timeout_ms": 60000,
+ "allow_paid_execution": false
+ }
+ },
+ "admission": {
+ "kind": "waza.runner-unit",
+ "schema_version": "1.0",
+ "status": "ready",
+ "compatibility": {
+ "kind": "waza.runner-negotiation",
+ "schema_version": "1.0",
+ "status": "compatible",
+ "negotiation": {
+ "version": "1.0",
+ "requested_versions": [
+ "1.0"
+ ],
+ "required_capabilities": [
+ "exact-unit",
+ "inline.mock-text"
+ ],
+ "optional_capabilities": [],
+ "available_capabilities": [
+ "negotiation",
+ "exact-unit",
+ "inline.mock-text",
+ "terminal-result",
+ "inline.copilot-text"
+ ],
+ "selected_capabilities": [
+ "exact-unit",
+ "inline.mock-text"
+ ],
+ "unavailable_optional_capabilities": []
+ },
+ "diagnostics": []
+ },
+ "requested": {
+ "shard": "shard-a",
+ "task": "selected",
+ "trial": 2,
+ "attempt": 2
+ },
+ "resolved": {
+ "shard": "shard-a",
+ "task": "selected",
+ "trial": 2,
+ "attempt": 2
+ },
+ "identity": {
+ "version": "sha256-waza-c14n-v1",
+ "canonical_digest": "8b87af77b7ea3a5653e8cece8c49c8c93542eb99cb475dadc3a322b8ca132177",
+ "run": "run-0ffdc3d4f349aa8f16917dc83223b0decdb5ad31636f397b7b306b960e255fb6",
+ "shard": "shard-f8a02c113aefee2dd99fb3fc5cde85ac6611ac6d412210cb7c0d02f13965ec34",
+ "task": "task-2beb920b903b355aca3c33d410d66e7bf058cb0254c8e2c2f5c8761cabc65f81",
+ "trial": "trial-b00f30c616aede2e22634831c434ba9112a3a214930d7fa3e30e4e3f17c0bb4d",
+ "attempt": "attempt-951276d40f08a614a364192405813a7f05c45ff5cf2ffd3bed6d9cd3538443fd"
+ },
+ "validation": "valid",
+ "adjacent_work_selected": false,
+ "engine_calls": 0,
+ "diagnostics": []
+ },
+ "http_status": 200,
+ "response": "{\"kind\":\"waza.runner-checkpoint-result\",\"schema_version\":\"1.0\",\"status\":\"completed\",\"exit_code\":0,\"submission\":{\"kind\":\"waza.execution-submission-result\",\"schema_version\":\"1.0\",\"status\":\"completed\",\"exit_code\":0,\"engine_calls\":1,\"caller\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"8b87af77b7ea3a5653e8cece8c49c8c93542eb99cb475dadc3a322b8ca132177\",\"run\":\"run-0ffdc3d4f349aa8f16917dc83223b0decdb5ad31636f397b7b306b960e255fb6\",\"shard\":\"shard-f8a02c113aefee2dd99fb3fc5cde85ac6611ac6d412210cb7c0d02f13965ec34\",\"task\":\"task-2beb920b903b355aca3c33d410d66e7bf058cb0254c8e2c2f5c8761cabc65f81\",\"trial\":\"trial-b00f30c616aede2e22634831c434ba9112a3a214930d7fa3e30e4e3f17c0bb4d\",\"attempt\":\"attempt-951276d40f08a614a364192405813a7f05c45ff5cf2ffd3bed6d9cd3538443fd\"},\"decision\":{\"kind\":\"waza.duplicate-decision\",\"schema_version\":\"1.0\",\"decision_id\":\"decision-24128d12a72146c481bf1e37c7884c2e7e21b6ff8ca759f8e4e83dcde412d551\",\"logical\":{\"run\":\"run-0ffdc3d4f349aa8f16917dc83223b0decdb5ad31636f397b7b306b960e255fb6\",\"work\":\"trial-b00f30c616aede2e22634831c434ba9112a3a214930d7fa3e30e4e3f17c0bb4d\"},\"binding\":{\"identity\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"8b87af77b7ea3a5653e8cece8c49c8c93542eb99cb475dadc3a322b8ca132177\",\"run\":\"run-0ffdc3d4f349aa8f16917dc83223b0decdb5ad31636f397b7b306b960e255fb6\",\"shard\":\"shard-f8a02c113aefee2dd99fb3fc5cde85ac6611ac6d412210cb7c0d02f13965ec34\",\"task\":\"task-2beb920b903b355aca3c33d410d66e7bf058cb0254c8e2c2f5c8761cabc65f81\",\"trial\":\"trial-b00f30c616aede2e22634831c434ba9112a3a214930d7fa3e30e4e3f17c0bb4d\",\"attempt\":\"attempt-951276d40f08a614a364192405813a7f05c45ff5cf2ffd3bed6d9cd3538443fd\"},\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":2},\"execution_mode\":\"mock\",\"grader_ids\":[\"\\u003c\\u003e\\u0026\\u2028\\u2029\",\"_output_contains\",\"\",\"𐀀\"]},\"policy\":\"distinct\",\"outcome\":\"distinct\",\"execution_id\":\"exec-2a4d0149d96ffa6ad3761323b1060b1fbfc1891a4b85cb6e11b0635b5960f6da\"},\"reservation\":{\"kind\":\"waza.execution-attempt\",\"schema_version\":\"1.0\",\"logical\":{\"run\":\"run-0ffdc3d4f349aa8f16917dc83223b0decdb5ad31636f397b7b306b960e255fb6\",\"work\":\"trial-b00f30c616aede2e22634831c434ba9112a3a214930d7fa3e30e4e3f17c0bb4d\"},\"execution_id\":\"exec-2a4d0149d96ffa6ad3761323b1060b1fbfc1891a4b85cb6e11b0635b5960f6da\",\"binding\":{\"identity\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"8b87af77b7ea3a5653e8cece8c49c8c93542eb99cb475dadc3a322b8ca132177\",\"run\":\"run-0ffdc3d4f349aa8f16917dc83223b0decdb5ad31636f397b7b306b960e255fb6\",\"shard\":\"shard-f8a02c113aefee2dd99fb3fc5cde85ac6611ac6d412210cb7c0d02f13965ec34\",\"task\":\"task-2beb920b903b355aca3c33d410d66e7bf058cb0254c8e2c2f5c8761cabc65f81\",\"trial\":\"trial-b00f30c616aede2e22634831c434ba9112a3a214930d7fa3e30e4e3f17c0bb4d\",\"attempt\":\"attempt-951276d40f08a614a364192405813a7f05c45ff5cf2ffd3bed6d9cd3538443fd\"},\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":2},\"execution_mode\":\"mock\",\"grader_ids\":[\"\\u003c\\u003e\\u0026\\u2028\\u2029\",\"_output_contains\",\"\",\"𐀀\"]}},\"completion\":{\"kind\":\"waza.execution-completion\",\"schema_version\":\"1.0\",\"execution_id\":\"exec-2a4d0149d96ffa6ad3761323b1060b1fbfc1891a4b85cb6e11b0635b5960f6da\",\"reservation_digest\":\"4160f0533542f9b792120f53dcc67a16742c752dc8e1b2dddf04093eeb97e19f\",\"manifest_digest\":\"9fc7d44b480c2410ff9dc5c51aa0ae56f0c89c4e5545ff79484d2a75abe4cf1c\"},\"terminal\":{\"kind\":\"waza.runner-terminal\",\"schema_version\":\"1.0\",\"outcome\":\"success\",\"exit_code\":0,\"failure_boundary\":\"none\",\"validation\":\"valid\",\"engine_calls\":1,\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":2},\"identity\":{\"version\":\"sha256-waza-c14n-v1\",\"canonical_digest\":\"8b87af77b7ea3a5653e8cece8c49c8c93542eb99cb475dadc3a322b8ca132177\",\"run\":\"run-0ffdc3d4f349aa8f16917dc83223b0decdb5ad31636f397b7b306b960e255fb6\",\"shard\":\"shard-f8a02c113aefee2dd99fb3fc5cde85ac6611ac6d412210cb7c0d02f13965ec34\",\"task\":\"task-2beb920b903b355aca3c33d410d66e7bf058cb0254c8e2c2f5c8761cabc65f81\",\"trial\":\"trial-b00f30c616aede2e22634831c434ba9112a3a214930d7fa3e30e4e3f17c0bb4d\",\"attempt\":\"attempt-951276d40f08a614a364192405813a7f05c45ff5cf2ffd3bed6d9cd3538443fd\"},\"observation\":{\"execution_mode\":\"mock\",\"engine_calls\":1,\"selection\":{\"shard\":\"shard-a\",\"task\":\"selected\",\"trial\":2,\"attempt\":2},\"run_number\":2,\"attempts\":2,\"status\":\"passed\",\"validations\":[{\"identifier\":\"\\u003c\\u003e\\u0026\\u2028\\u2029\",\"passed\":true},{\"identifier\":\"_output_contains\",\"passed\":true},{\"identifier\":\"\",\"passed\":true},{\"identifier\":\"𐀀\",\"passed\":true}]},\"diagnostics\":[]},\"diagnostics\":[]},\"before_checkpoint_sha256\":\"19ce6c9f03521ac712b61a1e9fbe908266e2afd33191cfb0318e73ef5d3fbcf5\",\"finalized_checkpoint_sha256\":\"a0d9bc243164eb737c06bbc3e99a82c986d13747d6a1dc23122b7f4d0f8c1649\",\"compatibility_result\":\"compatible\",\"compatibility_differences\":[],\"restored_units\":0,\"remaining_work\":0,\"diagnostics\":[]}\n"
+ }
+}
diff --git a/web/src/api/aggregates.ts b/web/src/api/aggregates.ts
new file mode 100644
index 000000000..9e5671f39
--- /dev/null
+++ b/web/src/api/aggregates.ts
@@ -0,0 +1,209 @@
+import Ajv2020 from "ajv/dist/2020";
+import aggregateSchema from "../../../schemas/manifest-aggregate-1.0.schema.json";
+import manifestSchema from "../../../schemas/artifact-manifest-1.0.schema.json";
+import terminalSchema from "../../../schemas/runner-terminal-1.0.schema.json";
+import { parseStrictJSON } from "../lib/strictJson";
+
+export interface AggregateInput {
+ manifest_sha256: string;
+ evidence_sha256?: string;
+}
+
+export interface AggregateReport {
+ kind: "waza.manifest-aggregate";
+ schema_version: "1.0";
+ state: "complete" | "incomplete" | "conflicting";
+ verification: "current-content-and-original-record-context-only";
+ members: (AggregateInput & {
+ state: "admitted" | "omitted" | "conflicting";
+ outcome?: string;
+ execution_id?: string;
+ binding?: {
+ identity: {
+ version: string; canonical_digest: string; run: string; shard: string;
+ task: string; trial: string; attempt: string;
+ };
+ selection: { shard: string; task: string; trial: number; attempt: number };
+ execution_mode: string;
+ grader_ids: string[];
+ };
+ categories: {
+ category: string;
+ state: string;
+ declared_state?: string;
+ source: string;
+ context: string;
+ verification: string;
+ reason?: string;
+ observed_count?: number;
+ }[];
+ diagnostics: string[];
+ })[];
+ groups: { profile_sha256: string; members: AggregateInput[] }[];
+ counts: {
+ input_members: number;
+ unique_verified_manifests: number;
+ unique_verified_execution_records: number;
+ member_states: Record;
+ member_outcomes: Record;
+ };
+ usage: {
+ state: "complete" | "unavailable" | "conflicting";
+ population: "unique-verified-execution-records";
+ executions: number;
+ recorded_historical_usage: {
+ turns: number;
+ input_tokens: number;
+ output_tokens: number;
+ cache_read_tokens: number;
+ cache_write_tokens: number;
+ premium_requests: number;
+ ai_credits?: number;
+ } | null;
+ diagnostics: string[];
+ };
+}
+
+export interface AggregateResult {
+ report: AggregateReport;
+ canonical: string;
+ status: number;
+}
+
+export class AggregateError extends Error {}
+
+const validateReport = new Ajv2020({ strict: false })
+ .addSchema(terminalSchema)
+ .addSchema(manifestSchema)
+ .compile(aggregateSchema);
+
+function referenceKey(input: AggregateInput): string {
+ return `${input.manifest_sha256}:${input.evidence_sha256 ?? ""}`;
+}
+
+function memberUsageEligible(member: AggregateReport["members"][number]): boolean {
+ return !!member.execution_id && !member.diagnostics.includes("cached_historical_usage_excluded") &&
+ !member.categories.some((category) => category.source === "cached") &&
+ member.categories.some((category) =>
+ category.category === "usage" && category.state === "present" && category.declared_state === "present" &&
+ category.verification === "source-content-and-recorded-usage-shape-only" && category.observed_count === 1);
+}
+
+function sorted(values: string[], compare: (a: string, b: string) => number = (a, b) => a < b ? -1 : a > b ? 1 : 0): boolean {
+ return values.every((value, index) => index === 0 || compare(values[index - 1]!, value) <= 0);
+}
+
+function compareUTF8(a: string, b: string): number {
+ const left = new TextEncoder().encode(a);
+ const right = new TextEncoder().encode(b);
+ for (let index = 0; index < Math.min(left.length, right.length); index++) {
+ if (left[index] !== right[index]) return left[index]! - right[index]!;
+ }
+ return left.length - right.length;
+}
+
+async function bindingProfile(binding: NonNullable): Promise {
+ const { version, canonical_digest, run, shard, task, trial, attempt } = binding.identity;
+ const selection = binding.selection;
+ // Match the Go typed binding order and encoding/json string escaping.
+ const wire = JSON.stringify({
+ identity: { version, canonical_digest, run, shard, task, trial, attempt },
+ selection: { shard: selection.shard, task: selection.task, trial: selection.trial, attempt: selection.attempt },
+ execution_mode: binding.execution_mode, grader_ids: binding.grader_ids,
+ }).replace(/[<>&\u2028\u2029]/g, (character) => `\\u${character.charCodeAt(0).toString(16).padStart(4, "0")}`);
+ const hash = await crypto.subtle.digest("SHA-256", new TextEncoder().encode(wire));
+ return Array.from(new Uint8Array(hash), (byte) => byte.toString(16).padStart(2, "0")).join("");
+}
+
+export function parseAggregateReferences(text: string): AggregateInput[] {
+ const lines = text.trim().split(/\r?\n/);
+ if (!text.trim() || lines.length > 128) {
+ throw new AggregateError("Enter 1 to 128 explicit manifest references.");
+ }
+ const unique = new Map();
+ for (const line of lines) {
+ const fields = line.trim().split(/\s+/);
+ if (fields.length < 1 || fields.length > 2 || fields.some((field) => !/^[0-9a-f]{64}$/.test(field))) {
+ throw new AggregateError("Each line must contain a full lowercase manifest digest and optionally an evidence digest.");
+ }
+ const manifest = fields[0];
+ if (!manifest) throw new AggregateError("A manifest digest is required.");
+ const input: AggregateInput = { manifest_sha256: manifest };
+ if (fields[1]) input.evidence_sha256 = fields[1];
+ unique.set(referenceKey(input), input);
+ }
+ return [...unique.entries()].sort(([a], [b]) => a < b ? -1 : a > b ? 1 : 0).map(([, input]) => input);
+}
+
+export async function buildAggregate(inputs: AggregateInput[], signal: AbortSignal): Promise {
+ let response: Response;
+ let text: string;
+ try {
+ response = await fetch("/api/v1/aggregates", {
+ method: "POST",
+ headers: { "Content-Type": "application/json" },
+ body: JSON.stringify(inputs),
+ // 5s admission + 30s evaluation + 5s delivery + 10s client headroom.
+ signal: AbortSignal.any([signal, AbortSignal.timeout(50_000)]),
+ });
+ text = await response.text();
+ } catch {
+ throw new AggregateError("Aggregate could not be loaded. Check the server connection and retry.");
+ }
+ if (response.status !== 200 && response.status !== 409) {
+ throw new AggregateError(`Aggregate API returned HTTP ${response.status}; no evaluated report is available.`);
+ }
+ let value: unknown;
+ try {
+ const maximumBytes = 32 * 1024 * 1024;
+ if (text.length > maximumBytes || new TextEncoder().encode(text).byteLength > maximumBytes) {
+ throw new AggregateError("Aggregate report exceeds its byte limit.");
+ }
+ value = parseStrictJSON(text, maximumBytes);
+ } catch {
+ throw new AggregateError("Aggregate API returned malformed JSON.");
+ }
+ if (!validateReport(value)) throw new AggregateError("Aggregate API returned an invalid report.");
+ const expected = [...new Set(inputs.map(referenceKey))].sort();
+ const received = value.members.map(referenceKey);
+ const memberCounts: Record = {};
+ const outcomes: Record = {};
+ const profiles = new Map();
+ let state = "complete";
+ let ordered = sorted(value.usage.diagnostics);
+ for (const member of value.members) {
+ memberCounts[member.state] = (memberCounts[member.state] ?? 0) + 1;
+ if (member.outcome) outcomes[member.outcome] = (outcomes[member.outcome] ?? 0) + 1;
+ if (member.state === "conflicting") state = "conflicting";
+ else if (member.state !== "admitted" && state !== "conflicting") state = "incomplete";
+ ordered &&= sorted(member.diagnostics) &&
+ member.categories.every((category, index) => index === 0 || member.categories[index - 1]!.category < category.category);
+ if (member.binding) {
+ ordered &&= sorted(member.binding.grader_ids, compareUTF8);
+ const profile = await bindingProfile(member.binding);
+ const members = profiles.get(profile) ?? [];
+ members.push(referenceKey(member));
+ profiles.set(profile, members);
+ }
+ }
+ const expectedGroups = [...profiles.entries()].sort(([a], [b]) => a < b ? -1 : a > b ? 1 : 0);
+ const actualGroups = value.groups.map((group) => [group.profile_sha256, group.members.map(referenceKey)]);
+ const executions = new Set(value.members.flatMap((member) => member.execution_id ? [member.execution_id] : [])).size;
+ const equalCounts = (a: Record, b: Record) =>
+ JSON.stringify(Object.entries(a).sort()) === JSON.stringify(Object.entries(b).sort());
+ if (!ordered || value.state !== state ||
+ JSON.stringify(expectedGroups) !== JSON.stringify(actualGroups) ||
+ JSON.stringify(expected) !== JSON.stringify(received) ||
+ value.counts.input_members !== received.length ||
+ !equalCounts(memberCounts, value.counts.member_states) ||
+ !equalCounts(outcomes, value.counts.member_outcomes) ||
+ value.counts.unique_verified_execution_records !== executions ||
+ value.usage.executions !== executions ||
+ (value.usage.state === "complete" && (executions === 0 || state !== "complete" ||
+ !value.members.every(memberUsageEligible))) ||
+ value.counts.unique_verified_manifests !== new Set(value.members.filter((m) => m.binding).map((m) => m.manifest_sha256)).size ||
+ (response.status === 200) !== (value.state === "complete")) {
+ throw new AggregateError("Aggregate API returned inconsistent membership or population counts.");
+ }
+ return { report: value, canonical: text, status: response.status };
+}
diff --git a/web/src/api/checkpointResults.ts b/web/src/api/checkpointResults.ts
new file mode 100644
index 000000000..9b97b67b5
--- /dev/null
+++ b/web/src/api/checkpointResults.ts
@@ -0,0 +1,358 @@
+import Ajv2020 from "ajv/dist/2020";
+import resultSchema from "../../../schemas/runner-checkpoint-result-1.0.schema.json";
+import submissionSchema from "../../../schemas/execution-submission-result-1.0.schema.json";
+import checkpointSchema from "../../../schemas/execution-checkpoint-1.0.schema.json";
+import attemptSchema from "../../../schemas/execution-attempt-1.0.schema.json";
+import completionSchema from "../../../schemas/execution-completion-1.0.schema.json";
+import manifestSchema from "../../../schemas/artifact-manifest-1.0.schema.json";
+import terminalSchema from "../../../schemas/runner-terminal-1.0.schema.json";
+import unitSchema from "../../../schemas/runner-unit-1.0.schema.json";
+import negotiationSchema from "../../../schemas/runner-negotiation-response-1.0.schema.json";
+import {
+ sameRunnerIdentity, sameSelection, validateRunnerSelection,
+ type RunnerIdentity, type RunnerTerminal, type WorkSelection,
+} from "./runner";
+import { parseStrictJSON } from "../lib/strictJson";
+
+interface SourceReference {
+ execution_id: string;
+ manifest_digest: string;
+}
+
+interface SubmissionResult {
+ kind: "waza.execution-submission-result";
+ schema_version: "1.0";
+ status: "rejected" | "submission_error" | "completed" | "reused";
+ exit_code: number;
+ engine_calls: number;
+ caller?: RunnerIdentity;
+ reservation?: {
+ kind: "waza.execution-attempt";
+ schema_version: "1.0";
+ execution_id: string;
+ logical: { run: string; work: string };
+ parent?: { execution_id: string; relationship: "retry" | "branch" };
+ binding: {
+ identity: RunnerIdentity;
+ selection: WorkSelection;
+ execution_mode: "mock" | "copilot";
+ grader_ids: string[];
+ };
+ };
+ completion?: {
+ kind: "waza.execution-completion";
+ schema_version: "1.0";
+ execution_id: string;
+ manifest_digest: string;
+ reservation_digest: string;
+ };
+ decision?: {
+ policy: "distinct" | "reuse";
+ outcome: string;
+ execution_id?: string;
+ logical: { run: string; work: string };
+ binding: NonNullable["binding"];
+ source?: SourceReference;
+ };
+ terminal?: RunnerTerminal;
+ diagnostics: { code: string; message: string }[];
+}
+
+interface CheckpointDeclaration {
+ kind: "waza.execution-checkpoint";
+ schema_version: "1.0";
+ boundary: "before-unit" | "finalized-unit";
+ created_at: string;
+ execution_id: string;
+ reservation_digest: string;
+ compatibility: {
+ runner_revision: string;
+ request_version: string;
+ identity_version: string;
+ terminal_version: string;
+ manifest_version: string;
+ attempt_version: string;
+ completion_version: string;
+ work: string;
+ request_policy_digest: string;
+ };
+ completion_digest?: string;
+ manifest_digest?: string;
+}
+
+export interface CheckpointOperationResult {
+ kind: "waza.runner-checkpoint-result";
+ schema_version: "1.0";
+ status: "rejected" | "submission_error" | "checkpoint_error" | "completed" | "reused";
+ exit_code: number;
+ submission?: SubmissionResult;
+ source?: CheckpointDeclaration;
+ source_withheld?: true;
+ before_checkpoint_sha256?: string;
+ finalized_checkpoint_sha256?: string;
+ compatibility_result: "unavailable" | "incompatible" | "compatible";
+ compatibility_differences: string[];
+ restored_units: 0 | 1 | null;
+ remaining_work: 0 | 1 | null;
+ diagnostics: { code: string; message: string }[];
+}
+
+const ajv = new Ajv2020({ strict: false });
+for (const schema of [
+ negotiationSchema, unitSchema, terminalSchema, manifestSchema,
+ attemptSchema, completionSchema, submissionSchema, checkpointSchema,
+]) ajv.addSchema(schema);
+const validate = ajv.compile(resultSchema);
+
+async function goJSONDigest(value: object): Promise {
+ const canonical = JSON.stringify(value).replace(
+ /[<>&\u2028\u2029]/g, (character) => `\\u${character.charCodeAt(0).toString(16).padStart(4, "0")}`,
+ );
+ if (!globalThis.crypto?.subtle) throw new Error("Checkpoint declaration hashing requires browser Web Crypto.");
+ const digest = await crypto.subtle.digest("SHA-256", new TextEncoder().encode(canonical));
+ return Array.from(new Uint8Array(digest), (byte) => byte.toString(16).padStart(2, "0")).join("");
+}
+
+function sortedGraders(ids: string[]): string[] {
+ const encoder = new TextEncoder();
+ return ids.map((id) => ({ id, bytes: encoder.encode(id) })).sort((a, b) => {
+ for (let index = 0; index < Math.min(a.bytes.length, b.bytes.length); index++) {
+ const difference = a.bytes[index]! - b.bytes[index]!;
+ if (difference) return difference;
+ }
+ return a.bytes.length - b.bytes.length;
+ }).map(({ id }) => id);
+}
+
+function sameGraders(left: string[], right: string[]): boolean {
+ return left.length === right.length && left.every((id, index) => id === right[index]);
+}
+
+async function reservationDigest(reservation: NonNullable): Promise {
+ const { identity, selection, execution_mode, grader_ids } = reservation.binding;
+ // Match executionattempt.EncodeAttempt: typed field order, UTF-8 grader
+ // ordering and compact Go JSON escaping, independent of response key order.
+ return goJSONDigest({
+ kind: reservation.kind, schema_version: reservation.schema_version,
+ logical: { run: reservation.logical.run, work: reservation.logical.work },
+ execution_id: reservation.execution_id,
+ binding: {
+ identity: {
+ version: identity.version, canonical_digest: identity.canonical_digest,
+ run: identity.run, shard: identity.shard, task: identity.task,
+ trial: identity.trial, attempt: identity.attempt,
+ },
+ selection: { shard: selection.shard, task: selection.task, trial: selection.trial, attempt: selection.attempt },
+ execution_mode, grader_ids: sortedGraders(grader_ids),
+ },
+ ...(reservation.parent ? { parent: {
+ execution_id: reservation.parent.execution_id, relationship: reservation.parent.relationship,
+ } } : {}),
+ });
+}
+
+async function checkpointDigest(source: CheckpointDeclaration): Promise {
+ const compatibility = source.compatibility;
+ // Match executioncheckpoint.Encode's typed field order and optional fields.
+ return goJSONDigest({
+ kind: source.kind, schema_version: source.schema_version, boundary: source.boundary,
+ created_at: source.created_at, execution_id: source.execution_id,
+ reservation_digest: source.reservation_digest,
+ compatibility: {
+ runner_revision: compatibility.runner_revision, request_version: compatibility.request_version,
+ identity_version: compatibility.identity_version, terminal_version: compatibility.terminal_version,
+ manifest_version: compatibility.manifest_version, attempt_version: compatibility.attempt_version,
+ completion_version: compatibility.completion_version, work: compatibility.work,
+ request_policy_digest: compatibility.request_policy_digest,
+ },
+ ...(source.completion_digest ? { completion_digest: source.completion_digest } : {}),
+ ...(source.manifest_digest ? { manifest_digest: source.manifest_digest } : {}),
+ });
+}
+
+function isCanonicalCheckpointTime(value: string): boolean {
+ const match = value.match(/^([0-9]{4}-[0-9]{2}-[0-9]{2}T[0-9]{2}:[0-9]{2}:[0-9]{2})(?:\.[0-9]{0,8}[1-9])?Z$/);
+ const seconds = match?.[1];
+ if (!seconds || seconds.startsWith("0000")) return false;
+ const time = Date.parse(`${seconds}Z`);
+ return Number.isFinite(time) && new Date(time).toISOString() === `${seconds}.000Z`;
+}
+
+// These checks consume protected responses; they do not authorize disclosure
+// or turn declaration-only hashes into runtime or recovery evidence.
+export async function parseCheckpointOperationResult(text: string): Promise {
+ const value = parseStrictJSON(text);
+ if (!validate(value)) throw new Error("Checkpoint API returned an invalid operation result.");
+ if (value.source && !isCanonicalCheckpointTime(value.source.created_at)) {
+ throw new Error("Checkpoint API returned a noncanonical source creation time.");
+ }
+ if (value.before_checkpoint_sha256 &&
+ value.before_checkpoint_sha256 === value.finalized_checkpoint_sha256) {
+ throw new Error("Checkpoint API returned one reference for different safe boundaries.");
+ }
+ const submission = value.submission;
+ if (submission && "report" in submission) {
+ throw new Error("Checkpoint API returned an unapproved raw report.");
+ }
+ if (submission &&
+ (value.status === "checkpoint_error"
+ ? submission.status !== "completed"
+ : value.status !== submission.status || value.exit_code !== submission.exit_code)) {
+ throw new Error("Checkpoint operation contradicts its actual submission.");
+ }
+ const binding = submission?.reservation?.binding;
+ const terminal = submission?.terminal;
+ const reservation = submission?.reservation;
+ const decision = submission?.decision;
+ if (reservation && binding &&
+ (reservation.logical.run !== binding.identity.run || reservation.logical.work !== binding.identity.trial ||
+ submission?.caller?.run !== reservation.logical.run || submission.caller.trial !== reservation.logical.work ||
+ (!decision && !sameRunnerIdentity(submission.caller, binding.identity)))) {
+ throw new Error("Checkpoint reservation contradicts its original logical work.");
+ }
+ if (decision && reservation && binding) {
+ const caller = decision.binding;
+ if (!sameRunnerIdentity(caller.identity, submission?.caller) ||
+ decision.logical.run !== reservation.logical.run || decision.logical.work !== reservation.logical.work) {
+ throw new Error("Checkpoint decision contradicts its caller or original logical work.");
+ }
+ if (decision.policy === "distinct"
+ ? decision.execution_id !== reservation.execution_id ||
+ !sameRunnerIdentity(caller.identity, binding.identity) || !sameSelection(caller.selection, binding.selection) ||
+ caller.execution_mode !== binding.execution_mode || !sameGraders(caller.grader_ids, binding.grader_ids)
+ : decision.outcome !== "reused" || decision.source?.execution_id !== reservation.execution_id ||
+ (["version", "run", "shard", "task", "trial"] as const).some((key) => caller.identity[key] !== binding.identity[key]) ||
+ caller.selection.shard !== binding.selection.shard || caller.selection.task !== binding.selection.task ||
+ caller.selection.trial !== binding.selection.trial || caller.execution_mode !== binding.execution_mode ||
+ !sameGraders(sortedGraders(caller.grader_ids), binding.grader_ids)) {
+ throw new Error("Checkpoint decision contradicts its original execution binding.");
+ }
+ }
+ if (submission?.completion &&
+ (!submission.reservation || submission.completion.execution_id !== submission.reservation.execution_id ||
+ submission.completion.reservation_digest !== await reservationDigest(submission.reservation))) {
+ throw new Error("Checkpoint completion contradicts its canonical original reservation.");
+ }
+ if (terminal && submission &&
+ ((submission.status !== "submission_error" && submission.exit_code !== terminal.exit_code) ||
+ (submission.status !== "reused" && submission.engine_calls !== terminal.engine_calls))) {
+ throw new Error("Checkpoint submission contradicts its original terminal outcome or call count.");
+ }
+ if (terminal && (!binding || !sameSelection(terminal.selection, binding.selection) ||
+ !sameRunnerIdentity(terminal.identity, binding.identity))) {
+ throw new Error("Checkpoint terminal contradicts its original reservation.");
+ }
+ if (terminal?.observation && binding) {
+ const observed = terminal.observation;
+ const ids = observed.validations.map((verdict) => verdict.identifier);
+ if (!sameSelection(observed.selection, terminal.selection) ||
+ observed.execution_mode !== binding.execution_mode ||
+ observed.run_number !== terminal.selection?.trial || observed.attempts !== terminal.selection?.attempt ||
+ observed.engine_calls !== terminal.engine_calls || new Set(ids).size !== ids.length ||
+ (observed.status !== "error" && ids.length !== binding.grader_ids.length) ||
+ ids.some((id) => !binding.grader_ids.includes(id)) ||
+ (observed.status !== "error" &&
+ (observed.status === "failed") !== observed.validations.some((verdict) => !verdict.passed))) {
+ throw new Error("Checkpoint observation contradicts its original binding.");
+ }
+ }
+ if (value.source && submission?.reservation &&
+ value.source.compatibility.work !== submission.reservation.logical.work) {
+ throw new Error("Checkpoint source contradicts the actual submission work.");
+ }
+ if (value.status === "completed" && value.source?.boundary === "finalized-unit") {
+ throw new Error("Finalized checkpoint work must be reused, not executed again.");
+ }
+ if (value.status === "reused") {
+ const source = value.source;
+ if (!source || submission?.engine_calls !== 0 ||
+ submission.reservation?.execution_id !== source.execution_id ||
+ submission.completion?.execution_id !== source.execution_id ||
+ submission.completion.manifest_digest !== source.manifest_digest ||
+ submission.completion.reservation_digest !== source.reservation_digest ||
+ submission.decision?.source?.execution_id !== source.execution_id ||
+ submission.decision.source.manifest_digest !== source.manifest_digest) {
+ throw new Error("Checkpoint reuse does not retain its exact original source.");
+ }
+ const completion = submission.completion;
+ if (source.completion_digest !== await goJSONDigest({
+ kind: completion.kind, schema_version: completion.schema_version, execution_id: completion.execution_id,
+ reservation_digest: completion.reservation_digest, manifest_digest: completion.manifest_digest,
+ })) {
+ throw new Error("Checkpoint reuse contradicts its canonical original completion.");
+ }
+ }
+ if (value.source?.boundary === "before-unit" && submission?.reservation) {
+ const parent = submission.reservation.parent;
+ if (!parent || parent.relationship !== "retry" ||
+ parent.execution_id !== value.source.execution_id ||
+ submission.reservation.execution_id === value.source.execution_id) {
+ throw new Error("Before-unit recovery does not retain a fresh original-source retry child.");
+ }
+ }
+ return value;
+}
+
+export interface CheckpointOperationResponse {
+ result: CheckpointOperationResult;
+ json: string;
+}
+
+export async function runCheckpointOperation(
+ action: "create" | "resume", request: string, digest: string, signal: AbortSignal,
+): Promise {
+ if (action === "resume" && !/^[a-f0-9]{64}$/.test(digest)) {
+ throw new Error("Provide a full lowercase SHA-256 checkpoint digest before resume.");
+ }
+ const wrapper = parseStrictJSON(request);
+ if (typeof wrapper !== "object" || wrapper === null || !("request" in wrapper) ||
+ typeof wrapper.request !== "object" || wrapper.request === null ||
+ !("timeout_ms" in wrapper.request) || typeof wrapper.request.timeout_ms !== "number" ||
+ !Number.isInteger(wrapper.request.timeout_ms) ||
+ wrapper.request.timeout_ms < 1 || wrapper.request.timeout_ms > 86_400_000) {
+ throw new Error("Provide one captured submission with an explicit native request timeout_ms.");
+ }
+ const admitted = await validateRunnerSelection(JSON.stringify(wrapper.request), signal);
+ if (admitted.status !== "ready") {
+ throw new Error("Captured runner selection was not admitted; no checkpoint operation was sent.");
+ }
+ const response = await fetch(action === "create"
+ ? "/api/v1/checkpoints" : `/api/v1/checkpoints/${digest}/resume`, {
+ method: "POST", headers: { "Content-Type": "application/json" }, body: request,
+ signal: AbortSignal.any([signal, AbortSignal.timeout(wrapper.request.timeout_ms + 30_000)]),
+ });
+ const json = await response.text();
+ const result = await parseCheckpointOperationResult(json);
+ if (action === "create" && (result.source || result.source_withheld || result.status === "reused")) {
+ throw new Error("Checkpoint create response contains resume-only state.");
+ }
+ if (action === "resume" && !result.source &&
+ (result.status === "completed" || (result.submission?.status === "completed" && !result.source_withheld))) {
+ throw new Error("Checkpoint API returned completed resume without its checkpoint source.");
+ }
+ if (action === "resume" && result.source && await checkpointDigest(result.source) !== digest) {
+ throw new Error("Checkpoint source contradicts the requested checkpoint digest.");
+ }
+ if (response.headers.get("X-Waza-Exit-Code") !== String(result.exit_code) ||
+ (!response.ok && (result.status === "completed" || result.status === "reused"))) {
+ throw new Error("Checkpoint transport status contradicts its operation result.");
+ }
+ const caller = result.submission?.caller;
+ if (caller && !sameRunnerIdentity(caller, admitted.identity)) {
+ throw new Error("Checkpoint caller contradicts the admitted captured request.");
+ }
+ const binding = result.submission?.reservation?.binding;
+ if (binding) {
+ const original = binding.selection;
+ const current = admitted.resolved;
+ if (!current || original.shard !== current.shard || original.task !== current.task || original.trial !== current.trial ||
+ (result.status !== "reused" && !sameSelection(original, current)) ||
+ !admitted.identity || (result.status === "reused"
+ ? (["version", "run", "shard", "task", "trial"] as const)
+ .some((key) => admitted.identity?.[key] !== binding.identity[key])
+ : !sameRunnerIdentity(binding.identity, admitted.identity))) {
+ throw new Error("Checkpoint original work contradicts the admitted captured request.");
+ }
+ }
+ return { result, json };
+}
diff --git a/web/src/api/runner.ts b/web/src/api/runner.ts
index 2768a2426..676fefcbd 100644
--- a/web/src/api/runner.ts
+++ b/web/src/api/runner.ts
@@ -78,10 +78,14 @@ export interface RunnerTerminal {
const validateTerminal = new Ajv2020({ strict: false }).compile(terminalSchema);
-function sameSelection(a: WorkSelection | undefined, b: WorkSelection | undefined): boolean {
+export function sameSelection(a: WorkSelection | undefined, b: WorkSelection | undefined): boolean {
return !!a && !!b && a.shard === b.shard && a.task === b.task && a.trial === b.trial && a.attempt === b.attempt;
}
+export function sameRunnerIdentity(a: RunnerIdentity | undefined, b: RunnerIdentity | undefined): boolean {
+ return !!a && !!b && (Object.keys(a) as (keyof RunnerIdentity)[]).every((key) => a[key] === b[key]);
+}
+
// Server admission is authoritative for grader bindings. The browser checks the
// exact original JSON, schema, selection and identity against offline admission;
// it neither derives identities nor treats them as proof of live execution.
@@ -113,8 +117,8 @@ export async function runRunnerUnit(request: string, signal: AbortSignal): Promi
throw new Error("Runner API terminal exit header contradicts its body.");
}
if (value.identity) {
- if (admitted.status !== "ready" || !sameSelection(value.selection, admitted.resolved) || !admitted.identity ||
- (Object.keys(admitted.identity) as (keyof RunnerIdentity)[]).some((key) => admitted.identity?.[key] !== value.identity?.[key])) {
+ if (admitted.status !== "ready" || !sameSelection(value.selection, admitted.resolved) ||
+ !sameRunnerIdentity(admitted.identity, value.identity)) {
throw new Error("Terminal identity or selection does not match admitted input.");
}
} else if (value.selection || value.observation || value.engine_calls !== 0) {
diff --git a/web/src/lab/CheckpointOperations.tsx b/web/src/lab/CheckpointOperations.tsx
new file mode 100644
index 000000000..97a623f1c
--- /dev/null
+++ b/web/src/lab/CheckpointOperations.tsx
@@ -0,0 +1,90 @@
+import { useEffect, useRef, useState } from "react";
+import unit from "../../../examples/runner/unit.json";
+import { runCheckpointOperation, type CheckpointOperationResponse } from "../api/checkpointResults";
+import { downloadText } from "../lib/export";
+import CheckpointResult from "./CheckpointResult";
+
+const example = JSON.stringify({
+ kind: "waza.execution-submission", schema_version: "1.0", request: unit,
+}, null, 2);
+
+function Operations() {
+ const [request, setRequest] = useState(example);
+ const [digest, setDigest] = useState("");
+ const [response, setResponse] = useState(null);
+ const [error, setError] = useState("");
+ const [pending, setPending] = useState(false);
+ const active = useRef(null);
+ useEffect(() => () => active.current?.abort(), []);
+
+ function clear() {
+ active.current?.abort();
+ setResponse(null);
+ setError("");
+ setPending(false);
+ }
+
+ async function run(action: "create" | "resume") {
+ active.current?.abort();
+ const controller = new AbortController();
+ active.current = controller;
+ setResponse(null);
+ setError("");
+ setPending(true);
+ try {
+ const value = await runCheckpointOperation(action, request, digest, controller.signal);
+ if (!controller.signal.aborted) setResponse(value);
+ } catch (error) {
+ if (!controller.signal.aborted) setError(
+ error instanceof Error ? error.message : "Checkpoint operation could not be loaded.",
+ );
+ } finally {
+ if (!controller.signal.aborted) setPending(false);
+ }
+ }
+
+ return (
+ <>
+
+