Skip to content

fix(bin): prevent false Pi watcher alarms during hand-offs - #2304

Merged
kunchenguid merged 3 commits into
mainfrom
fm/fm-pi-watcher-banner-falsealarm-r1
Aug 13, 2026
Merged

fix(bin): prevent false Pi watcher alarms during hand-offs#2304
kunchenguid merged 3 commits into
mainfrom
fm/fm-pi-watcher-banner-falsealarm-r1

Conversation

@kunchenguid

Copy link
Copy Markdown
Owner

Intent

DIAGNOSE then SHIP: the WATCHER DOWN banner printed at fm-send time is a FALSE ALARM on a healthy Pi primary. This touches firstmate's shared, tracked material (the send-time supervision-health guard), so firstmate-coding-guidelines applies: one-owner rule, one-sentence-per-line Markdown, plain dash never em dash, shellcheck-clean bin scripts via bin/fm-lint.sh, tests colocated in tests/ extending the existing suite, and tests must exercise behavior through an executable interface and never assert implementation source bytes.

OBSERVED: almost every fm-send from a Pi primary home printed the full 'WATCHER DOWN - SUPERVISION IS OFF' banner (or the same-episode one-liner) even when the Pi extension already owned the watcher-arm child and the last beacon beat was seconds old. The send still succeeded, but firstmate then needlessly narrated 'repair the watcher'. Pi watcher continuity is EXTENSION-owned (.pi/extensions/fm-primary-pi-watch.ts), not bash-arm-owned, so a momentary send-time beacon gap or lock-holder mismatch must not read as 'supervision is off'.

DIAGNOSIS PERFORMED (diagnostic-reasoning procedure, evidence gathered live, not guessed). Trigger: any guarded command (fm-send calls fm-guard.sh with a CONTINUE_LINE) on a Pi primary. Mask: the Pi extension deliberately tears the watcher down on every actionable wake and spawns the replacement itself, so state/.watch.lock is legitimately unheld during each hand-off - all 799 cycles in the live Pi primary's state/.watch-cycle-exits.log end with lock_after=pid:none, the no-watcher window measured 0.2s to 198s (median 31s, p90 117s, 5.5% of wall clock), and a live 20Hz sampler on the real Pi primary captured the guard verdict flipping to reason=no-watcher with lockpid=none and the beacon 63s old, then back to healthy 0.32s later. Symptom: fm_supervision_model classified pi as 'persistent', which requires a live identity-matched watcher process holding the lock (fm_watcher_healthy), so that window rendered the full alarm. Earliest divergence between the healthy and genuinely-down paths: the Claude primary's 'autoarm' model already treats the identical state (fresh beacon, no live lock holder) as healthy by design because its watcher only runs between turns; Pi had no equivalent model, so healthy extension-owned hand-off and real supervision loss were indistinguishable. Smallest counterfactual that flips it: give the verdict an extension-ownership proof.

FIX (smallest change): add an 'extension' supervision model for pi and pi-signed in fm_supervision_model. Under it a live identity-matched watcher remains the ordinary healthy state, and an unheld lock is healthy ONLY while the beacon is fresh within FM_GUARD_GRACE AND a live Pi session provably owns continuity. That proof is the new fm_pi_extension_owns_supervision in bin/fm-wake-lib.sh: both Pi primary extensions (fm-primary-pi-watch.ts and fm-primary-turnend-guard.ts) must be recorded in their state markers at their current on-disk build hashes by the process named in state/.lock, and that process must still be alive. Requiring the turn-end guard extension as well as the watch extension is deliberate: it is the structural backstop, so a home missing it has no benign hand-off to tolerate.

SAFETY BAR HELD (explicitly required, do not relax): this is NOT a blanket suppression. A genuinely missing, failed, or unhealthy watcher must still print loudly - turn-end-guard extension missing or version-drifted, no live arm ownership (extension unloaded or the Pi session exited), and genuinely stale supervision (beacon past grace) all still fire the full banner immediately or within grace. The queued-wakes-pending warning is kept exactly as is because it is real. The PID-strict fm_watcher_healthy used by bin/fm-turnend-guard.sh is deliberately untouched. Claude (autoarm), codex, opencode, grok, kimi, tmux, and unknown primaries keep their existing persistent-model watcher-down detection unchanged; the tolerance is scoped to the extension model alone.

REGRESSION PROVING BOTH DIRECTIONS, through the executable interface (bin/fm-guard.sh driven with real state files and real processes), added to the existing tests/fm-guard-stale-banner.test.sh rather than a new runner: healthy extension-owned Pi hand-off -> NO watcher-down banner and no down-episode opened; no ownership evidence -> banner still fires naming the missing watcher process; each of the five ownership signals broken individually (dead Pi session, missing watch marker, missing turnend marker, drifted watch build, drifted turnend build) -> banner still fires, so no single signal is load-bearing alone; beacon past grace with a fully live correctly-loaded Pi session -> banner still fires naming the stale beacon; queued-wakes-pending during a healthy hand-off -> still fires and does not resurrect the banner; a persistent-watcher primary with Pi markers present on disk -> still alarms, pinning the model scoping; a live identity-matched watcher under the extension model -> silent through the strict path; and Pi and pi-signed primaries routing themselves to the extension model through real bin/fm-harness.sh detection with no FM_SUPERVISION_MODEL override, which is the end-user path.

HARNESS-DEPENDENT-CHECK CLASSIFICATION (per firstmate-coding-guidelines): the fixed guard logic is pure - it reads state files, file hashes, and process liveness, and reads no vendor-emitted process name, rendered output, spinner glyph, banner, or keybind - so a portable tests/ regression driving bin/fm-guard.sh with real files and processes is sufficient and no new live-harness-optin guard is needed. The only vendor-dependent input is bin/fm-harness.sh's pre-existing PI_CODING_AGENT marker detection that selects the model, and the same suite exercises that path directly.

ONE-OWNER CLEANUP: bin/fm-session-start.sh carried its own private copies of the Pi extension marker predicate and its hash helper. Those were folded into bin/fm-wake-lib.sh (fm_pi_extension_version, fm_pi_extension_loaded) so the ownership contract is stated once, and fm-session-start.sh now sources fm-wake-lib.sh and calls them. This was verified not to collide with any fm-session-start.sh function or variable.

DOCS: bin/fm-guard.sh's header, bin/fm-wake-lib.sh's model documentation, and docs/turnend-guard.md (the owner of the guard-predicate contract) describe the new model; bin/fm-supervision-lib.sh's header restatement of the models was reduced to a cross-reference rather than a third copy. docs/verification/supervision.md records the dated maintainer-verification evidence including the live Pi primary reproduction.

DELIVERY: mode=no-mistakes, yolo OFF - the captain merges. Escalate any ask-user finding rather than deciding it. Note in the PR that homes pick this change up only after merge plus a firstmate self-update, since it changes the send-time supervision banner. Stop at CI green.

What Changed

  • Route Pi and pi-signed primaries through an extension supervision model that accepts fresh-beacon, unheld-lock hand-offs only when a live session owns current builds of both Pi extensions.
  • Preserve watcher-down alarms for stale beacons, unhealthy held locks, missing or drifted ownership evidence, and persistent-watcher harnesses while keeping queued-wake warnings unchanged.
  • Centralize Pi extension marker helpers, expand executable guard regressions, and document the supervision contract and verification evidence. Homes receive the fix after merge and a Firstmate self-update.

Risk Assessment

✅ Low: The fix is narrowly scoped to Pi extension-owned hand-offs, preserves strict handling for held unhealthy locks and stale beacons, and includes executable-interface coverage for the required paths.

Testing

The baseline reproduced the false banner; targeted guard and session-start tests passed, and end-to-end CLI evidence shows a healthy Pi hand-off stays silent without opening an episode while a missing ownership marker still emits the full warning. The broader session-start test exceeded its time limit only after all relevant changed-path checks had passed.

Evidence: Baseline false-alarm reproduction
BASELINE REPRO: guarded command in the same fresh-beacon, unheld-lock Pi hand-off state
detected_harness=pi
●━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
●  WATCHER DOWN - SUPERVISION IS OFF
●  1 task(s) in flight, but no live watcher process holds this home lock (last beat: 1s ago).
●  Trust the emitted supervision protocol for this harness; do not use shell & for watcher repair.
●  This is a supervision warning only; the guarded operation WILL still run.
●  repair a missing or failed watcher cycle with the Pi tool fm_watch_arm_pi, or restart Pi with -e /var/folders/0k/bf8mwt2n5qddzk24r20gfk0c0000gn/T/no-mistakes-evidence/01KZX0GHVTXCE0KV60Z5AZHVNW/pi-handoff-e2e/root/.pi/extensions/fm-primary-turnend-guard.ts -e /var/folders/0k/bf8mwt2n5qddzk24r20gfk0c0000gn/T/no-mistakes-evidence/01KZX0GHVTXCE0KV60Z5AZHVNW/pi-handoff-e2e/root/.pi/extensions/fm-primary-pi-watch.ts if the extensions are not loaded.
●━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
Evidence: Current Pi hand-off and safety-path end-to-end transcript
END-USER E2E: guarded command during an extension-owned Pi watcher hand-off
detected_harness=pi
guard_output=<silent>
down_episode=not-opened

SAFETY CHECK: same live Pi session, but turn-end ownership marker missing
●━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
●  WATCHER DOWN - SUPERVISION IS OFF
●  1 task(s) in flight, but no live watcher process holds this home lock (last beat: 0s ago).
●  Trust the emitted supervision protocol for this harness; do not use shell & for watcher repair.
●  This is a supervision warning only; the guarded operation WILL still run.
●  repair a missing or failed watcher cycle with the Pi tool fm_watch_arm_pi, or restart Pi with -e /var/folders/0k/bf8mwt2n5qddzk24r20gfk0c0000gn/T/no-mistakes-evidence/01KZX0GHVTXCE0KV60Z5AZHVNW/pi-handoff-e2e/root/.pi/extensions/fm-primary-turnend-guard.ts -e /var/folders/0k/bf8mwt2n5qddzk24r20gfk0c0000gn/T/no-mistakes-evidence/01KZX0GHVTXCE0KV60Z5AZHVNW/pi-handoff-e2e/root/.pi/extensions/fm-primary-pi-watch.ts if the extensions are not loaded.
●━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
down_episode=no-watcher

Pipeline

Updates from git push no-mistakes

✅ **intent** - passed

✅ No issues found.

✅ **Rebase** - passed

✅ No issues found.

🔧 **Review** - 1 issue found → auto-fixed ✅
  • 🚨 bin/fm-wake-lib.sh:261 - Intent requires that “an unheld lock is healthy ONLY” with extension ownership and that an unhealthy watcher still alarms. Here every strict-check failure falls into the extension tolerance: with a fresh beacon and valid Pi markers, a dead, malformed, wrong-home, wrong-path, or identity-mismatched .watch.lock returns healthy. Restrict the extension fallback at this shared verdict boundary to a genuinely unheld lock; otherwise preserve the alarm.

🔧 Fix: Restrict Pi hand-off tolerance to unheld watcher locks
✅ Re-checked - no issues remain.

✅ **Test** - passed

✅ No issues found.

  • bash tests/fm-guard-stale-banner.test.sh
  • Executed the base commit's bin/fm-guard.sh against a fresh-beacon, unheld-lock Pi hand-off fixture to reproduce the original false alarm.
  • Executed current bin/fm-guard.sh through real Pi harness detection with live session and extension markers, then removed the turn-end marker and repeated the guard call.
  • bash tests/fm-session-start.test.sh (relevant Pi extension checks passed before the broader command exceeded the 180-second limit)
  • bash tests/.fm-session-start-pi-focused.test.sh using a temporary runner for the existing Pi marker/session-start tests; runner removed afterward.
✅ **Document** - passed

✅ No issues found.

✅ **Lint** - passed

✅ No issues found.

✅ **Push** - passed

✅ No issues found.

On a Pi primary the watcher process is not the liveness signal. The Pi
extension tears the watcher down on every actionable wake and spawns the
replacement itself, so the singleton lock is legitimately unheld between
cycles: every one of the 799 cycles in a live primary's ledger ends with
lock_after=pid:none, and a live capture caught the guard verdict flipping to
no-watcher during one hand-off with the beacon 63s old.

bin/fm-guard.sh classified Pi as a persistent-watcher harness, which demands a
live identity-matched lock holder at all times, so any guarded command landing
in a hand-off painted the full WATCHER DOWN - SUPERVISION IS OFF banner and
told firstmate to repair a cycle the extension already owns and is restoring.

Add an extension supervision model for pi and pi-signed. A live
identity-matched watcher stays the ordinary healthy state; an unheld lock is
healthy only while the beacon is fresh within grace AND a live Pi session
provably owns continuity - both primary extensions recorded in their state
markers at their current on-disk builds by the process named in state/.lock,
with that process still alive. Without that proof the banner fires exactly as
before, so an unloaded, version-drifted, or exited Pi session is loud
immediately and a cycle the extension never restores is loud once the beacon
passes grace. The queued-wake warning, the PID-strict turn-end guard, and
every other primary's detection are untouched.

Fold session-start's duplicate Pi marker predicate into the shared library so
the ownership contract has one owner.
@kunchenguid
kunchenguid merged commit 85cefa9 into main Aug 13, 2026
13 checks passed
@kunchenguid
kunchenguid deleted the fm/fm-pi-watcher-banner-falsealarm-r1 branch August 13, 2026 16:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant