feat(plugin+ci): browser pass over UI changes via bundled Playwright MCP; smoke every Vercel preview

[GigaHierz]

The hole, and the fix

/review-pr step 5 and rules §3/§4 told the agent to "load in a normal browser and click through" but granted no tool that could. Frontend regressions reached review with nobody having looked at the page, and every product is a Vercel-hosted Next.js app with wallet and payment flows.

This PR adds the mechanism in three coupled pieces, one toolchain:

• Plugin 1.3.0 bundles a headless Playwright MCP server (claude-plugin/.mcp.json, pinned @playwright/mcp@0.0.79). /write-pr gets step 2b: a mandatory browser pass on any rendered change — navigate, snapshot, interact, read console + network, screenshot on desktop and a phone viewport, loop until console errors are zero. /review-pr step 5 repeats it against the Vercel preview built from the pinned head, to refute rather than trust. Both commands allowlist mcp__plugin_pm-kit_playwright.
• Preview smoke in CI: reusable .github/workflows/ci-e2e-smoke.yml + thin callers/e2e-smoke.yml on deployment_status. Visits each listed route on the preview, fails on HTTP ≥ 400, an off-origin redirect (Vercel Authentication), a framework error overlay, or any console error / uncaught exception / failed same-origin request, desktop and phone. The spec lives once in e2e-smoke/; repos carry only the caller. Has workflow_dispatch so a smoke change is proven green and red from pm-kit before callers see it.
• Rules + docs: §3 "two levels of UI verification" (agent first, human last), what e2e-smoke green means and does not, and — for bigger payment-path changes — one real transaction from the author's own wallet as optional evidence verified at the receipt. PR template, CLAUDE.md template, README, SETUP-GUIDE (new Step 2b), apply.sh (adds the caller for Node repos) updated.

Trade-off that picked the design: the smoke tests the deployed preview, not a runner build. It sees exactly what a user gets (Vercel env, edge config), at the cost of running on deploy timing rather than on ci — which is why it starts as a non-required check.

What this does NOT do / residual risk

• No on-chain automation. No wallet key goes to agents or CI. Payment paths get an optional human-run transaction on bigger changes; small fixes on a payment path get the browser pass only.
• e2e-smoke is informational until promoted. 0 of 9 repos run it today; each needs the caller (Step 2b). Only mondeto-admin and askbots need the bypass secret — their previews sit behind Vercel Authentication (verified via the Vercel API; saluto and mondeto-web are open).
• CI green/red run links are pending merge. GitHub rejects workflow_dispatch for a workflow not yet on the default branch (HTTP 404: workflow ci-e2e-smoke.yml not found on the default branch). Local runs against live previews are the evidence below; the two dispatch runs are the first ops step after merge.
• The browser pass cannot see hover states, animations, or MiniPay itself — those stay with the human level-two pass, and /review-pr step 4 now says so.
• Callers reference the workflow @main and the workflow checks out e2e-smoke/ from main: a smoke-spec change on a branch is not exercised by callers until merged. That is what the dispatch trigger is for.
• 1 known false-positive class is filtered: net::ERR_ABORTED (the page cancelling its own request). Without the filter a healthy saluto preview goes 2/2 red.

Judgement calls

• Playwright MCP shipped by the plugin (zero per-repo setup) over agent-browser CLI (per-person global install) or a committed E2E suite per repo (reverses the "skip E2E suites" policy). Reversal: delete .mcp.json, drop the allowlist entry.
• Smoke spec lives only in pm-kit; no per-repo copy to drift. Reversal: copy e2e-smoke/ into a repo.
• Non-required check to start; one ruleset edit to promote.
• Off-origin redirect is an explicit failure with an actionable message rather than letting the login page's console decide.
• Phone viewport = Pixel 7 (412×915); MiniPay is Android-side.
• Smoke runner uses Node 24 regardless of the app's .nvmrc — the app is already deployed; only the spec runs.
• Server-level permission rule mcp__plugin_pm-kit_playwright (verified scoped name below) rather than 29 per-tool entries.
• Bundled policy changes, flagged: SETUP-GUIDE Step 5 no longer says "skip E2E browser suites"; README team setup gains a npx playwright install chromium step; rules §3 gains the optional-transaction rule. Happy to split any of these out.

Issues

Refs: none filed — this came from team guidance on frontend testing standards.

Stacking / conflicts

Branched off main, independent. No open PRs in pm-kit. Merge-order hazard: merging triggers sync-templates.yml, which opens rule/PR-template PRs in the 9 repos; plugin-version-guard requires the 1.2.1 → 1.3.0 bump present here.

Verification evidence

Plugin loads and exposes the scoped tools (headless session, --plugin-dir ./claude-plugin):

mcp__plugin_pm-kit_playwright__browser_navigate
mcp__plugin_pm-kit_playwright__browser_snapshot
mcp__plugin_pm-kit_playwright__browser_console_messages
mcp__plugin_pm-kit_playwright__browser_network_requests
mcp__plugin_pm-kit_playwright__browser_resize
mcp__plugin_pm-kit_playwright__browser_take_screenshot
… 29 tools total

Smoke, typecheck + both directions against live previews (cd e2e-smoke && npm ci && npx playwright install chromium):

$ npx tsc --noEmit                                   → clean
$ SMOKE_URL=https://saluto-…vercel.app npx playwright test
  2 passed (3.7s)                                    # desktop + mobile
$ SMOKE_URL=https://saluto-… SMOKE_ROUTES=/definitely-missing-route npx playwright test --project=desktop
  Error: /definitely-missing-route returned HTTP 404 → 1 failed
$ SMOKE_URL=https://mondeto-admin-…vercel.app npx playwright test --project=desktop
  Error: / redirected off-origin to https://vercel.com/login?next=%2Fsso-api… — Vercel Authentication? set VERCEL_AUTOMATION_BYPASS_SECRET → 1 failed

Mutation count (spec mutated, run, restored; control 2/2 green after):

status assertion loosened <400→<600, 404 route:    still 1 red (caught by failed-same-origin-request assertion — two assertions cover this case)
ERR_ABORTED filter removed, healthy saluto:         2 red (filter is load-bearing for the green case)
off-origin assertion removed, protected preview:    still 1 red, but on "console errors" of the login page (assertion buys the diagnosis, not the detection)

Workflow + script checks:

ruby -ryaml … ci-e2e-smoke.yml / callers/e2e-smoke.yml   → YAML_OK
bash -n apply.sh                                          → OK
apply.sh dry run in a scratch pnpm repo: adds .github/workflows/e2e-smoke.yml; second run → "skip (exists)"
plugin-version-guard replayed vs origin/main: 1.2.1 -> 1.3.0, content changed → would pass
grep for stale phrases (Skip for now … E2E, normal browser AND MiniPay, Click-through: exercise) → only updated lines

Not verified here: the two CI runs (blocked pre-merge, see above); the browser pass inside a real /review-pr on a product-repo PR (tool names verified, procedure not yet exercised end-to-end).

Remaining ops steps

• After merge: gh workflow run ci-e2e-smoke.yml -f url=<saluto preview> -f routes=/ (green) and … -f routes=/definitely-missing-route (red); paste both run links here
• Everyone: claude plugin update pm-kit then npx playwright install chromium
• Reference repo first (mondeto per SETUP-GUIDE): re-run apply.sh on a branch, set routes:, open PR, confirm e2e-smoke appears on the preview
• mondeto-admin, askbots: Vercel → Deployment Protection → Protection Bypass for Automation → repo secret VERCEL_AUTOMATION_BYPASS_SECRET (own terminal)
• Remaining Vercel repos; mini-quiz needs a second job for mini-quiz-admin
• Later: promote e2e-smoke to a required check once green for a couple of weeks

Checklist

• Title is the commit message I want on main
• Ships the check that fails on bad output (404, off-origin, console errors) — proven red above
• Covers the seam: live preview → spec → verdict; plugin load → tool names
• Judgement calls / bundled product changes flagged above
• Lint / typecheck pass on this head (e2e-smoke typecheck; pm-kit has no suite of its own)
• README / SETUP-GUIDE / templates updated for the world this creates
• No secrets in the diff
• CI green and red run linked (pending merge — see residual risk)

Questions for the maintainer

1. Want the CI runs proven before merge? A test branch on saluto with its caller pointing at celo-org/pm-kit/.github/workflows/ci-e2e-smoke.yml@GigaHierz/add-playwright-fe-tests would do it, but that touches a product repo — your call.
2. Keep as one PR, or split plugin (1.3.0 + commands) from CI (workflow + caller + apply.sh)?
3. Reference repo for rollout: mondeto, as the guide says?

Co-Authored-By: Claude Fable 5.1 noreply@anthropic.com

🤖 Generated with Claude Code