feat: Wave A — testing system (Phase-0 detection + design-time Test Strategy + autonomous Release Gate) [v5.7.0] - #82
Merged
Merged
Conversation
…feature Durable source-of-truth plan so multi-wave work survives compaction/sessions. Captures: 4 pillars, verified 2026 research synthesis (Playwright/Testcontainers stack + gotchas + 3 killed myths), locked decisions, two-wave execution plan (Wave A = P0+P2+P3 testing system; Wave B = P1 architecture review), open questions, and a status tracker. Note: lives in docs/roadmap/ because docs/superflow/ is gitignored (reserved for superflow-generated artifacts). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ter generated); A1 next Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…env.sh + test-env.json schema + phase0 wiring) - tools/detect-test-env.sh: read-only, idempotent probe that writes .superflow/test-env.json atomically. Detects Docker runtime (desktop/colima/rancher/podman), emits required exports for non-Desktop runtimes. Detects Node runners (vitest/jest/playwright/cypress) via package.json, Python runners (pytest) via pyproject.toml/requirements*.txt, Playwright browser availability via the LOCAL binary only (never bare npx). Classifies project_type (web/backend-only/library). Computes per-type readiness verdict (ready/partial/blocked) with missing[] and recommendations[] arrays. Every external probe timeout-wrapped via gtimeout/timeout/perl fallback. Shellcheck -S error clean. - templates/test-env.schema.json: JSON Schema 2020-12 documenting the .superflow/test-env.json shape; matches the event-schema.json style used by the rest of the skill. - references/phase0/stage1-detect.md: invoke bash tools/detect-test-env.sh in parallel with the existing preflight probes. - references/phase0/stage3-report.md: read .superflow/test-env.json and surface the Testing Infrastructure section (runtime, runners, browsers, verdict, recommendations) in the health report. - prompts/claude-md-writer.md: add <testing_section> guidance to emit an idempotent Testing section (marked with <!-- superflow:testing -->) into the target project's CLAUDE.md when test-env.json exists. Includes non-Desktop Docker export instructions and per-OS Playwright install commands (--with-deps on Debian/Ubuntu, binaries-only on macOS). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… timeout, real browser-binary check, unit-runner count, colima fallback, library blocked FIX 1 (HIGH): _classify_project — broadened frontend framework list (astro/nuxt/gatsby/ remix/vite/preact/solid-js/qwik/eleventy/vuepress/docusaurus/gridsome + @sveltejs/*, @remix-run/*, @builder.io/*, @11ty/*, @docusaurus/*); added src/pages and src/app dirs; added SSG output check (dist/site/_site/out/build *.html); ambiguous JS runnable app (scripts.start/dev/serve) → web; positive library signals required to emit library (main/module/exports/bin with no serve; [build-system]/setup.py/setup.cfg; bare repo). Invariant: superflow repo (no package.json) → still library. Updated schema description. FIX 2 (HIGH): _timeout — removed bare `"$@"` unbounded fallback; now returns 1 (fail closed) when gtimeout/timeout/perl are all absent. Probes treat return 1 as unavailable. Non-negotiable: a dead socket must never hang the autonomous run. FIX 3 (HIGH): Playwright browser detection replaced install --list (reports installable names, not installed binaries) with on-disk cache dir check: macOS ~/Library/Caches/ms-playwright, Linux ~/.cache/ms-playwright, honoring PLAYWRIGHT_BROWSERS_PATH. A browser counts only when chromium-*/firefox-*/webkit-* subdir exists. New helper _detect_playwright_browsers(). read-only, timeout-free (stat-only, no external proc). e2e_tooling=true only with ≥1 real binary dir. FIX 4 (MEDIUM): _compute_readiness — unit_node_count now filters .runners[] to only vitest and jest (playwright and cypress are E2E tools, excluded). unit_py_count unchanged (pytest only). Updated schema readiness.unit.description. FIX 5 (LOW): _detect_docker — added colima fallback in the `*)` case: if colima status reports running (timeout-wrapped) OR the default colima socket file exists, classify runtime=colima and populate exports. Covers misconfigured active context. FIX 6 (LOW): library verdict — no unit runner now emits blocked (was partial). Consistent with schema definition: "no unit runner = cannot run any tests at all". Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…GHT_BROWSERS_PATH=0, conservative colima detection, NestJS backend, schema doc FIX A: wrap find in _detect_playwright_browsers with _timeout 5; use head -1 + || true pattern to avoid pipefail SIGPIPE false-negative when find exits after head closes the pipe — a stuck FS can no longer hang. FIX B: PLAYWRIGHT_BROWSERS_PATH=0 is now treated as the playwright package-local install sentinel (node_modules/playwright-core/.local-browsers/ with pnpm/yarn fallback to node_modules/.cache/ms-playwright/), not as a literal directory path "0". FIX C: Remove colima heuristics that could mis-classify the active endpoint. The *) case now requires positive proof before promoting to colima: (1) DOCKER_HOST env already points at .colima/, (2) docker context inspect endpoint contains .colima/, or (3) /var/run/docker.sock is a symlink into $HOME/.colima/. If none of the three proofs fire, stays as "desktop". FIX D: Update templates/test-env.schema.json node.playwright.browsers.description to say "detected by checking real on-disk binary dirs … honoring PLAYWRIGHT_BROWSERS_PATH; special value '0' means in-package location". No reference to install --list remains in the schema. FIX E: Add @nestjs/, @adonisjs/, @hapi/ prefix-match to the recognized Node backend list in _classify_project. This check runs BEFORE the ambiguous start/dev/serve→web fallback, so NestJS apps with a "start" script now classify as "backend-only" rather than "web". Update project_type.description in schema to document the ordering invariant. Verification: - shellcheck -S error: PASS - superflow repo run: library + blocked - idempotency: byte-identical - jq empty on both files: PASS - classifier spot-checks: Astro→web, NestJS+start→backend-only, unknown+start→web, pure-lib→library, FastAPI→backend-only - browser probe: PLAYWRIGHT_BROWSERS_PATH=0 with chromium dir→detected, empty PLAYWRIGHT_BROWSERS_PATH dir→e2e_tooling=false - verify-phase2-dag.sh: ALL CHECKS PASSED (33/0) - jq empty tracked JSON: all 5 files OK Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ed colima socket, timeout-wrap SSG scan
FIX A2: replace `|| true` with status-aware `_br_rc` capture in
_detect_playwright_browsers. Pattern: `_br_out=$(...) || _br_rc=$?`
then `[ _br_rc -eq 0 ] && [ -n _br_out ]`. Any non-zero rc (including
124 timeout) → browser treated as absent, even when partial output was
captured before the kill. `-print -quit` eliminates the head-pipe SIGPIPE
that the previous || true was papering over.
FIX C2: the Proof 3 colima fallback (symlink-based) now exports
`unix://${sock_target}` (the actual readlink target) instead of the
hard-coded `unix://${HOME}/.colima/default/docker.sock`. A non-default
Colima profile (e.g. myprofile) now gets the correct socket path.
Proofs 1 and 2 were already using the resolved value; only Proof 3 had
the hard-coded default. Named colima/colima-* context cases derive the
profile from the context name and are unchanged.
FIX D2: wrap the SSG output-dir heuristic in _classify_project with
_timeout 5 and -print -quit (no head pipe). A huge/networked dist/build
dir can no longer hang the classifier. On timeout or error, the signal is
absent — web-bias applies only when the probe fires cleanly.
Verification:
- shellcheck -S error: PASS
- superflow repo: project_type=library, verdict=blocked
- idempotency: byte-identical
- jq empty + DAG verifier: 33 PASS / 0 FAIL
- FIX A2 unit test: case rc=124+partial-output → ABSENT (fail-closed);
old ||true pattern → PRESENT (illustrates the pre-fix bug)
- FIX C2 spot-check: symlink /var/run/docker.sock → .colima/myprofile/docker.sock
→ DOCKER_HOST exported as unix://...myprofile/docker.sock, not ...default/
- Classifier: Astro→web, NestJS+start→backend-only, unknown+start→web,
pure-lib→library, FastAPI→backend-only (all PASS)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… step + charter test_strategy block + journey-authoring guidance) - references/phase1-discovery.md: insert Step 13a inside Step 13 (Generate Autonomy Charter) — reads .superflow/test-env.json, derives active levels by project_type, builds test_strategy frontmatter block and body section; enforces owning_sprint constraint (unowned journey = Release Gate FAIL). Updates charter YAML template with test_strategy: field (levels/journeys/ coverage/runtime_matrix/per_sprint_acceptance); updates Body guidance to require a ## Test Strategy section. - prompts/test-strategy.md: new concise guidance prompt — level derivation table (web/backend-only/library), journey object spec (id, title, steps, expected_outcome, spec_path, spec_title, owning_sprint), library path (coverage threshold + runtime_matrix), per_sprint_acceptance string format, and two complete examples (web with 2 journeys; library with coverage gate). No new runtime deps. Baseline green: verify-phase2-dag.sh PASS (33/33), jq validation PASS (5 files), shellcheck unaffected, forbidden-token gate clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…cker A1 done/A2 in-progress Discovered during Wave A execution: named background reviewers can go idle without delivering their verdict fence, stalling the orchestrator. §8 records the mitigation (prefer synchronous reviewer dispatch / explicit verdict-delivery instruction / bounded-wait + cold re-dispatch on idle-without-verdict). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… self-check, spec_tag per-journey key, example alignment FIX 1 (level-convention): always emit all three level keys; type-INACTIVE levels use "N/A — <type>; <reason>", type-applicable-but-missing use "not configured — <install cmd>". Applied in phase1-discovery.md template comment and prompts/test-strategy.md Step 1 + both worked examples. Also removes docker.present parenthetical from derivation table (FIX 5), since readiness.integration already encodes docker availability. FIX 2 (library journeys wording): prompts/test-strategy.md Step 3 now reads "Set journeys: [] … and add coverage + runtime_matrix" instead of "Replace the journeys list". FIX 3 (owning_sprint placeholder + self-check): template placeholder changed from owning_sprint: 0 to owning_sprint: 2 with a "MUST NOT be 0" inline comment. Added binding MUST validation paragraph in Step 13a requiring all journey owning_sprints to be positive integers matching existing plan sprints before the charter file is written. FIX 4 (spec_tag field): added spec_tag to the journey schema in both files (phase1-discovery.md charter template + prompts/test-strategy.md Required journey fields + Complete web example). spec_title now includes the @<tag> annotation. per_sprint_acceptance wording updated to reference spec_tag explicitly, removing the ambiguous "appears in title or tag" phrasing. FIX 5 (example alignment): Step 4 web per_sprint_acceptance example now uses J1-login / J2-checkout, matching the Complete web example. Required journey fields example updated from J1-checkout to J1-login for consistency. Baseline green: verify-phase2-dag.sh 33/33 PASS, jq 5 files OK. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…t engine + phase-level release_gate step + enforcement rule + ordering hooks) - tools/release-gate.sh: pure-computation bash+jq verdict engine. Flags: --project-type (web|backend-only|library), --journeys (JSON), --results (JSON), --evidence-dir (optional). Writes .superflow/release-gate/verdict.json atomically (mktemp+mv). Exit 0=PASS/SKIPPED, exit 1=FAIL. shellcheck -S error clean. Verdict matrix: library→SKIPPED; web→per-journey spec_tag coverage check (no-vacuous-pass, browsers-absent FAIL, docker-absent LOUD note non-blocking); backend-only→integration-only gate (docker absent=conservative FAIL). - references/phase2/steps/release-gate.md: orchestrator stage instructions — assembly, test-env.json reading, image-version pinning (derive PW version from installed package), app-boot via Playwright webServer, Testcontainers integration (Ryuk per CI/ryuk_forced_disabled precedence + cleanup backstop), E2E headless workers=1, per-journey tag extraction, results.json assembly, release-gate.sh call, verdict interpretation. Artefact paths table. Conditional matrix table. - references/phase2/workflow.json: add phase_gates.release_gate top-level node (post-sprint-loop, post-holistic, pre-completion-report; mandatory_for web/backend-only; skipped_for library) + step_files["release_gate"]="release-gate.md". 7-stage sequence and 9-cell decision matrix unchanged; DAG verifier passes. - superflow-enforcement.md: Rule 14 — Release Gate mandatory before Phase 3 when project is runnable; library substitutes coverage threshold; persisted verdict.json required; no vacuous pass; per-journey spec_tag coverage. - references/phase2/overview.md + references/phase2-execution.md: explicit post-sprint-loop ordering hook: sprint loop → holistic → RELEASE GATE → Completion Report → Phase 3. Fixture evidence (8 cases): A1/PASS, A2/FAIL(J2-checkout forced-fail), A3/SKIPPED, A4/FAIL(docker-absent LOUD), A6/FAIL(no-vacuous-pass), journey-uncovered/FAIL, A7/FAIL(browsers-absent), backend-only/PASS. All shellcheck+DAG+jq gates green. NOT in this sprint: phase3-merge.md wiring (A4), verify-phase2-dag.sh extension for phase_gates (A4), real app-boot self-test (A4). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…nput-schema, integration fail-closed, pipeline exit), fix Playwright extraction, drop yq
FIX 1 [CRITICAL] — web + empty journeys → FAIL
release-gate.sh: added early check in _compute_verdict_web(): journey_count==0 → FAIL
("web project but zero journeys supplied"). Web must supply ≥1 journey; empty array
on a web project can never vacuously PASS.
FIX 2 [CRITICAL] — input schema validation + exact set-membership
release-gate.sh: added _validate_inputs() called before dispatch. Validates:
- journeys file is a JSON array
- every journey has a non-empty string spec_tag
- e2e_covered_tags is a JSON array of strings (not a string, number, etc.)
- e2e_failed_tags is a JSON array of strings
Malformed input → FAIL closed, never PASS.
Replaced index($t) with any($arr[]; . == $t) for strict element equality;
a comma-joined string field can no longer substring-match a journey tag.
FIX 3 [CRITICAL] — web integration fail-closed on non-{pass,skipped}
release-gate.sh: _compute_verdict_web() now uses case/whitelist on _INTEGRATION:
pass → ok; skipped → LOUD non-blocking note (web: E2E is primary gate);
anything else (fail, failed, error, fatal, unknown) → FAIL, fail-closed.
"skipped" is the only recognised non-pass; every other string → FAIL.
FIX 4 [CRITICAL] — pipeline exit code masking
release-gate.md Steps 5+6: replaced `cmd | tee log; EXIT=$?` (captures tee=0)
with `cmd > log 2>&1; EXIT=$? ; cat log` pattern (captures actual command exit).
FIX 5 [HIGH] — Playwright JSON schema
release-gate.md Step 7: replaced broken `.suites[].specs[]` + `.tests[].status=="passed"`
with: `.. | objects | select(has("specs")) | .specs[]` (recursive) + `select(.ok==true)`.
spec.ok is the canonical pass signal; test.status is expected/unexpected/flaky/skipped.
FIX 6 [MEDIUM] — native Playwright tags + permissive regex
release-gate.md Step 7: prefer spec.tags[] (ltrimstr("@")), fall back to
capture("@(?<tag>[A-Za-z][A-Za-z0-9_-]*)") — handles J2-checkoutV2, J1-sign_in.
Regex uses jq-native (?<name>) syntax (not Python-style (?P<name>) which jq rejects).
FIX 7 [LOW] — timeout kill → specs_ran=false
release-gate.md Step 7: E2E exit 124 (SIGTERM) / 137 (SIGKILL) → SPECS_RAN=false;
only non-timeout non-zero exits count as "specs ran but failed".
FIX 8 [CRITICAL-adjacent] — remove yq; orchestrator emits journeys.json directly
release-gate.md Step 8: removed yq mention; specified that the orchestrator (Phase-2
LLM) reads the short charter file (Rule 11 allowed), extracts journeys, and emits
journeys.json via jq -n inline — no YAML parser, no new dependency.
12 fixtures verified (8 original + 4 new): all correct. shellcheck+DAG+jq all pass.
Playwright jq sample: covered=["J1-login"], failed=["J2-checkout"] ✓
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…validation), correct Playwright JSON reporter invocation, free-form tag fallback
FIX A (CRITICAL): jq's // operator treats boolean false as absent, masking type
errors before validation — e.g. {"integration":false} → "skipped" (false PASS).
Removed all // defaults from the read section; replaced with explicit
if-has-not-null guards. Rewrote _validate_inputs() to check raw file types
before any defaulting: array fields (e2e_covered_tags, e2e_failed_tags), string
field (integration), boolean fields (specs_ran, browsers_present, docker_present).
A genuinely absent/null key still falls to the correct else-branch default.
FIX B (HIGH): Playwright has no --output-file flag; the old invocation silently
dropped the flag and left pw-results.json unwritten. Replaced with the correct
PLAYWRIGHT_JSON_OUTPUT_NAME env var.
FIX C (MEDIUM): Documented the title-fallback regex constraint: [A-Za-z][A-Za-z0-9_-]*
matches only alphanumeric/hyphen/underscore — spec_tags with . : / will NOT be
matched. Charter authors should use kebab-slug IDs.
Tests: 15/15 pass (12 regression + 3 new adversarial).
shellcheck, DAG verifier, jq validation, forbidden-token gate all green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…losed), keep absent-key defaults
The previous if-has-not-null guard treated {"integration":null} as absent and
silently defaulted it to "skipped", bypassing the type check. Contract fix:
absent key = OK, defaults apply
present key = must be exact type; null is NOT acceptable for any typed field
Validation change (all three field groups now use has($f) alone):
- Array fields (e2e_covered_tags, e2e_failed_tags): if present → type=="array"
and all elements are strings; null → type=="null" → FAIL.
- String field (integration): if present → type=="string"; null → FAIL.
- Boolean fields (specs_ran, browsers_present, docker_present): if present →
type=="boolean"; null → FAIL. A real false still passes (Docker absent is valid).
Read section simplified: after validation guarantees a present key is correct-type,
if-has reads are safe — no need for the != null guard.
Fixture spot-checks:
{"integration":null,...} (web, J1-login) → FAIL (present-null rejected)
{"e2e_failed_tags":null,...} (web, J1-login) → FAIL (present-null rejected)
integration key ABSENT + docker absent → PASS loud (absent-key default ok)
Full suite: 17/17 pass (12 original + 3 bool-false + 2 present-null).
shellcheck, DAG verifier, jq validation, forbidden-token gate all green.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…finalize + phase3 verdict precondition + Ryuk reconciliation + DAG verifier extension + self-test
1. Journeys→scenarios handoff: added explicit end-to-end chain in
references/phase1-discovery.md (Step 13a) and
references/phase2/steps/release-gate.md stating who authors what
(owning_sprint implementer) and how the gate joins them (spec_tag
matching). Integration-branch assembly for multi-PR modes flagged
NOT-YET-VALIDATED more prominently.
2. Enforcement finalized: Rule 14 already numbered correctly; added
rationalization-prevention line "The gate passed but I'll merge
anyway → NEVER; re-run gate if code changes post-verdict."
3. Phase-3 merge precondition: added pre-merge step 0a to
references/phase3-merge.md — refuses merge unless
.superflow/release-gate/verdict.json holds PASS or SKIPPED with
exact jq check:
jq -e '.verdict == "PASS" or .verdict == "SKIPPED"' verdict.json
Verdict also folded into .par-evidence.json as release_gate field;
par-evidence.md schema updated with new optional/final-required field
+ example showing mid-run (omit) vs final (required) PAR shape.
4. Ryuk reconciliation: updated 7 mirrors to the same two-case rule —
Ryuk ENABLED except (a) CI===true or (b) docker.ryuk_forced_disabled=
true (rootless Podman). Mandatory cleanup-testcontainers.sh backstop
in case (b). Files: superflow-enforcement.md Rule 6, codex/AGENTS.md
Rule 14, agents/{deep,standard,fast}-implementer.md, prompts/
implementer.md, references/phase2/overview.md.
5. DAG verifier extended: added Check 7 to tools/verify-phase2-dag.sh
validating phase_gates.release_gate node, step_files["release_gate"]
entry, and on-disk release-gate.md existence + mandatory_for/
skipped_for fields. Passes 39/0 (was 33/0 pre-A4).
6. Roadmap: docs/roadmap/2026-06-30-testing-release-gate.md §7 — A2
and A3 marked DONE with HEAD refs; A4 marked IN PROGRESS.
Self-test (tools run LIVE; browser/Docker inputs SIMULATED):
detect-test-env.sh web-app → project_type=web LIVE
detect-test-env.sh library → project_type=library LIVE
release-gate.sh web PASS (2/2 journeys green) SIMULATED inputs
release-gate.sh web FAIL (J2-checkout fails) SIMULATED inputs
release-gate.sh web zero-specs → FAIL (no vacuous) SIMULATED inputs
release-gate.sh library → SKIPPED SIMULATED inputs
All 4 verdict scenarios confirmed correct.
CI: shellcheck clean, DAG 39/0, JSON valid, no forbidden tokens.
CLAUDE.md Ryuk mirror deferred to doc stage (task constraint).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…ock, DAG Check 7 semantics, phase3 SKIPPED scope, roadmap Ryuk canon FIX 1: Release-Gate hard rule added to codex/AGENTS.md as Rule 15 (Rule 14 = Ryuk — no collision). Both superflow-enforcement.md Rule 14 and codex/AGENTS.md Rule 15 now explicitly state "Phase 3 merge BLOCKED until verdict=PASS or SKIPPED (library only); env-blocked runs emit FAIL, not SKIPPED, and also block the merge." Compaction-surviving on both runtimes. FIX 2: verify-phase2-dag.sh Check 7 extended with semantic assertions (not just field existence): mandatory_for must contain 'web' + 'backend- only'; skipped_for must contain 'library' and must NOT contain 'web' or 'backend-only'; when must contain 'post-sprint-loop' and 'pre-completion'. Negative checks: adding 'web' to skipped_for → FAIL "DANGER: gate would be skipped for web"; dropping 'web' from mandatory_for → FAIL "must include 'web'". DAG verifier now 44/0 (was 39/0). FIX 3: references/phase3-merge.md SKIPPED prose tightened: "SKIPPED = library projects ONLY; env-blocked runs (docker absent, browsers absent) emit FAIL, not SKIPPED — those block the merge; no env-degraded bypass." FIX 4: docs/roadmap §3 Ryuk research note updated: "Research finding; canon superseded in Sprint A4" + pointer to authoritative Rule 6/Rule 14. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
… Strategy + Release Gate) + reconcile CLAUDE.md Ryuk canon Register the Wave A testing files/capabilities across CLAUDE.md, llms.txt, and CHANGELOG (new [5.7.0] entry); bump SKILL.md/CLAUDE.md/llms.txt to v5.7.0; disambiguate the roadmap Ryuk cross-reference to enforcement Test & Process Discipline §6. Reconcile the last stale single-case Ryuk mirror in CLAUDE.md (and llms.txt) to the finalized two-case canon (CI OR rootless-Podman-forced, label-based cleanup backstop). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…r parity guard, explicit fail-closed INTEGRATION_RESULT FIX 1 (HIGH): release-gate.md — add Step 1b that re-runs detect-test-env.sh before reading project_type. detect runs only at Phase 0; a mid-run shape change (backend-only → web after sprints add a frontend) would produce a silent stale-type PASS. Re-detecting at gate start means a stale backend-only type + freshly detected web project_type + charter journeys:[] hits the zero-journey FAIL guard (loud failure) instead of a vacuous PASS or SKIPPED. FIX 2 (MEDIUM): release-gate.sh — add optional --expected-journey-count N flag. When provided, asserts (journeys.json | length) == N; on mismatch → FAIL with reason "journeys.json count X != charter journey count N — transcription mismatch". Validates N is a non-negative integer (exit 2 on bad value). release-gate.md Step 8 updated to instruct the orchestrator to count charter journeys and pass the count on every invocation. Backward-compatible: omitting the flag preserves all prior behavior. FIX 3 (LOW): release-gate.md Step 5 — add explicit fail-closed INTEGRATION_RESULT derivation (docker absent→skipped, exit 0→pass, else→fail). Step 7 results.json build removes the :-skipped fallback so an unset result cannot silently become non-blocking. A web project with INTEGRATION_EXIT=1 + green E2E now produces integration=fail → verdict=FAIL instead of the prior silent PASS. Tests: 19/19 pass (17 regression + 2 new FIX-2 parity cases). shellcheck exit 0, DAG verifier 44 PASS / 0 FAIL, jq 5/5, forbidden-token clean. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
…tags have no @ prefix in JSON; Scenario C fails via per-journey coverage FIX 1: spec.tags description and proof sample corrected. Real Playwright (verified 1.61.1) stores tags WITHOUT the @ prefix — e.g. ["J1-login"] not ["@J1-login"]. Updated: schema bullet, Tag extraction prose (ltrimstr reframed as defensive/no-op on real output), and the proof JSON sample (both tags entries). The jq is unchanged. FIX 2: "Determine specs_ran" section rewritten to describe both no-vacuous-pass paths. "No tests found" exits with code 1 (not 124/137), so SPECS_RAN=true — the gate still FAILs via the per-journey coverage check (journey spec_tag absent from e2e_covered_tags), not the specs_ran guard. Added explicit (a)/(b) description with the Playwright 1.61.1 verification note. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Wave A — Testing System (v5.7.0)
Hardens both ends of the Superflow pipeline: design-time test rigor (decide how it will be tested before code) and release-time verification (spin up the assembled system and actually run integration + browser E2E, autonomously, before merge). Scope = pillars P0 + P2 + P3 (P1 architecture review is Wave B). Pure-Markdown skill — no new orchestrator runtime dependency (bash + jq only).
What's in it (4 sprints)
tools/detect-test-env.sh(read-only/idempotent/atomic) →.superflow/test-env.json: docker + runtime (Docker Desktop/Colima/Rancher/Podman exports;ryuk_forced_disabledfor rootless Podman), node/python + runners, Playwright browsers (real on-disk cache check, never installs), a 3-wayproject_typeclassifier (web / backend-only / library; ambiguous → web), and a readiness verdict + recommendations. Schema:templates/test-env.schema.json.prompts/test-strategy.md: the chartertest_strategyblock — critical journeys each with a stablespec_tag+spec_path/spec_title/owning_sprint(the machine-checkable P2→P3 contract); library path = coverage threshold + version matrix.tools/release-gate.sh(pure bash+jq verdict engine, fail-closed) +references/phase2/steps/release-gate.md(assemble → boot app via PlaywrightwebServer→ Testcontainers integration → headless E2Eworkers=1tagged byspec_tag→ evidence →verdict.json). No vacuous pass: per-journey coverage byspec_tag— a web project with journeys but zero executed specs → FAIL.phase_gates.release_gateinworkflow.json; DAG verifier Check 7.phase3-merge.mdrefuses merge unlessverdict.json= PASS or library-SKIPPED (defends the documented post-compaction merge regression) +.par-evidence.jsonrelease_gatefold; Ryuk two-case precedence reconciled across all mirrors; docs (CLAUDE.md/llms.txt/CHANGELOG).Review rigor
Every sprint passed a dual-lens unified review (Claude product + codex/Claude technical). The merge-blocking Release Gate (A3) took 3 fix rounds that closed 7 distinct false-PASS paths (empty-journeys, substring vs exact
spec_tagmembership, integration not fail-closed,tee-masked exit codes, two jq-//boolean-masking traps, present-null). The final holistic review (whole feature as one system) caught two cross-sprint seams — a frozen Phase-0project_typethe gate never re-derived (a web UI added to a backend-only repo could merge un-gated) and a journeys.json transcription-parity gap — both fixed.Self-test
The feature gated its own merge:
detect-test-env.sh→project_type=library(pure-Markdown skill) →release-gate.sh→ verdict=SKIPPED (library; coverage substitutes) → Phase-3 precondition → merge-eligible.Known limitations (honestly flagged)
stacked_prs/parallel_wave_prs) is not-yet-validated — thissolo_single_prrun doesn't exercise it.Verification
shellcheck -S errorclean ·verify-phase2-dag.sh44/0 · all tracked JSON valid · forbidden-token gate clean.🤖 Generated with Claude Code