P2: owner cutover sidecar aborts after activation when the healthz probe reuses a closed keep-alive connection (fetch failed)

Problem

The owner cutover sidecar (~/.devspace/worktrees/issue-413-wave1-sidecar.mjs, the only governed activation route after #414 removed the coordination_* MCP tools) probes http://127.0.0.1:<port>/healthz with Node's global fetch several times across the lifecycle. The probe issued right after bindCutoverActivation fails with fetch failed whenever the release copy + hashReleaseTree takes longer than the server's keep-alive idle timeout (~5 s): undici reuses the pooled connection the server already closed. The sidecar then aborts between activation_bound and restart_scheduled, leaving the generation drained with the pointer already rewritten.

Fresh evidence (Dev M5, 2026-10-09, cutover `4323d0b3`, `b6cd329` → `668f5df`)

Sidecar log: cutover_drained 00:08:59.070Z → activation_bound 00:09:18.969Z (19.9 s copy of a ~900 MB release) → {"ok":false,"error":"fetch failed","at":"00:09:18.974Z"} 5 ms later.
Server log has no http_request for that probe (connection-level failure, not an HTTP error). The same probe succeeded at 00:08:59 (predecessor_verified) on a fresh connection.
State after abort: cutover status phase drained, current-release.json already pointing at the new release, live still old generation in drain mode.
Recovery that worked: carrier rotate-credential → cutover restart-bound --package-root <build-ready root> → a finish-only sidecar calling finishCutover with the witness pair. cutover observe is refused as legacy (COORDINATION_REQUIRED), so without the finish-only script the generation stays drained until expiry.

Expected

The sidecar's healthz probe should not depend on connection reuse: open a fresh connection per probe (e.g. fetch(url, { headers: { connection: "close" } }) or a per-call undici.Agent({ keepAliveTimeout: 1 })), and retry a connection-level failure once before aborting.
The sidecar should resume from drained + activationBinding present instead of starting a new generation, so a probe failure after activation never requires hand-written recovery.
The repo should ship the finish-only path (or make cutover observe coordination-aware) so a coordination-bound generation whose restart already happened can be closed from the CLI with the carrier.

Acceptance

Simulated slow activation (>10 s) no longer aborts the sidecar.
A sidecar started against a drained generation with an activation binding proceeds to restart/finish without a new cutover_start.
A CLI or documented script closes a coordination-bound generation after the replacement is up, using the approved carrier.

Refs #399, #405, #414

🤖 Generated with Claude Code

Authority: RESEARCH_OBSERVATION_ONLY / NO_ENGINEERING_AUTHORITY

Hybrid replication capture receipt

{
  "body_gzip_base64": "H4sIAAAAAAACA4VWy5IbNRTd+ytueRbYxN1jzyuxK5WqySTAVJEHw7AJRU3L3dduMXKrkdR2TCpZ8gEs2LLhy/gCPoEjqdt2IBQ1i7Hbku65555z1EdH9NroueJVr3dbMulNxYbyxuk1/ltZcC4MDbIPx2nBa1uLnI832tw7w2yPpbUNJ2eT02Qj1jxJ2vXp6kebjcj58yq1paU/rOKCRO7kWjipKzK6cUxi4VDm6GxyRoZXWFaEXVmutSlkFZbefZ7Ri6vX5LRWdkg14LKlrHSunh0fT04epmP8TWaPa23ck+OShXLlzxltpCvppS74M0tLpedCUbZgl5cZWQYgfHdyhaNEbrS1obCSC863ueKUPBuhFoUuCzJyWboWcTaXVXEVWbrcNZXRQkhlY+VYKzzhAmhKrnzVUMawYmGZcl1v6QF6Eba8ic9uQWxGTtwDmNLVMuwQVdhm2eAEtHPPXCdCyTXAFYpDH+CTBh/OyQ5n1FSFzCXKNJZjYzXIQw+5rirOwwD2B5JQhkWxpVxpy0VsvRs9llUk5qDW0pzdhvE128/xbq5RLCNRFZQZtk4Yd2fzkovGtz0iNLWW1TKUW4ICE8efFUbIKhDjyYoQZeUO0BjeGOkcV2mvd3REX+D0kngNXFXONHjGa3pxPqKT8clFMhkn4+lop9vs7PTktBjPTwFgfpEXpyfTOf35y690cfFocV4sLoa9XkLfti0qvZxBcnHv3Q7YeDwbP5qdT9Pxw/GbsPsTfftF09nkUTq9mL6hwWSaTsnGueoFCfowHY/pxdNu4sN4zru+vu/PFkJZHvXZGG36s/6hXvqjvnB4tj/+4dmb/vuMzmkFXQjwlPoO4vzQAEFCVOloizvDPzWYBfSog3xcq+TBfv6JghpVKNcYHmEvtF3RV7e3rykgGrY6EKvOB7bJc+bC+9jt2EE21AYPc7ZWmzvAkQsJ+oawPvpfhKntywbUTuysH5S1Z58gINfYjOrS+2OnkpFfYQxXLmmZTH+03nCdWIJ4vM4AzYup4k3HOTTojWKdVIq0Kg5lKDslZrRCUnh0N5x7JNvIm886LjxAYYwEQKM9+iT3PaOiUFkcaddA64Ek6IOSBIl5L5acGA2CH88bqYqk1TeePIm7wZOspC2TEJed9XKhlG8piz+2cXPgGHyoQDvVQpp0D0HPg68zBBfgLJABmBhUw0uRbzGvq1evbp5dv7y8vX718u7m+TffXd88f5YNR2R1ONsniT/+I0y5kbX7p4vR6tYeeLkBI4r4bS3NNrr2+dsag+fC++0gVpBhbU532kJVjMarsOCaQR2OP0irEGUz0jV/SlVUo+1W4pwu0zZ8B41RI3rnSxVs7Awf93tm1A9516f39H6YEZwi/DmJp52yGKHpJbp1g3chcS+9jm5j1M5o4neBNJ98hh0EI+i/7IVmEFlzhh05Sh5zTT+mpKMArTWw3MLo1QGzDw7D5ykuHxwAl2Ax8EHG1qFHHzlBfcEIwQP7YYXxipalDldrwv21HO8onx8SZyNVqiJpYxhPozM64IZr3aG2paz/JZpaQKgDELvCdfYJfR7e8onYCIN8DCA/+iE66UB1mxJj64y2S4BS1BCHv+MQY2gxXmaRRw/s6uvrvXNaN0eNXiLWaicwongrrBofsOhJ6c0hN4MnkzGuVx+z7c3c3otuP0XPzeV+pB6hd99S+BGhs/1EDxvysID6oNY8jtiPy2cuauiu4+NI8c6qcdC7+yusySIO3zPoL3QOSVWhp+jjQI79X6ajPOIbS63w7rcKarPU1CNqbHevg3kT3t32tN7wwtLR6RSX8tHZ+HwU3vF6vb9+/+M3+jIWwPrQ+PdXSjQFBoQA/mHg7y+L97o8PExzvWo/Jjl+H/b+Bp4u5vOxCgAA",
  "capture_sha256": "be2989ab0ba908475f09d75a37a96bacc90ffc729d8da9cd2ec1ca5a671bc3aa",
  "captured_at": "2026-10-09T00:23:34Z",
  "contract_sha256": "c108a30cb0640eaa84a0d36a49e135eedd046b1c768d981c946709cde05b3f26",
  "created_at": "2026-10-09T00:23:26Z",
  "default_branch": "main",
  "issue_number": 462,
  "issue_updated_at": "2026-10-09T00:23:26Z",
  "pre_implementation_revision": "668f5df6a84bed09f0c8d8e4f715676cf182a64d",
  "repository": "James3014/devspace",
  "schema": "nexus.hybrid_replication.capture.v1",
  "source_event_id": "actions:37864527993:1",
  "title": "P2: owner cutover sidecar aborts after activation when the healthz probe reuses a closed keep-alive connection (fetch failed)"
}

Status 2026-10-09: (c) CLI closure path is in PR #464 (devspace cutover finish-bound). (a) The health() probe fix (connection: close + one retry on connection-level failure) is applied on Dev M5 to both ~/.devspace/worktrees/issue-413-wave1-sidecar.mjs and the 668f5df copy (backups *.bak-health-20261009, node --check clean); the snippet is reproduced below for the next host. (b) resume-from-drained in the sidecar remains open.

async function health(config) {
  const url = `http://${config.host}:${config.port}/healthz`;
  const probe = async () => {
    // connection: close makes undici open a fresh socket, never a pooled one the server already closed.
    const response = await fetch(url, { redirect: "error", headers: { connection: "close" } });
    if (!response.ok) throw new Error(`healthz HTTP ${response.status}`);
    return await response.json();
  };
  const connectionLevel = (e) => e instanceof TypeError || ["ECONNRESET", "ECONNREFUSED"].includes(e?.code ?? e?.cause?.code);
  try {
    return await probe();
  } catch (error) {
    if (!connectionLevel(error)) throw error;
    await new Promise((r) => setTimeout(r, 500));
    return await probe();
  }
}

Authority: RESEARCH_OBSERVATION_ONLY / NO_ENGINEERING_AUTHORITY

{
  "body_gzip_base64": "H4sIAAAAAAACA4VWy3LbNhTd6ytu5UWlRqSpt6XJeMax1dYzjZ0qTqaTTkcEyUsRNUWwAChFzSTLfkAX3XbTL+sX9BN6AZCS0qbT8cISBeCee+45B3z6mefB3eK7Vy9X1/fLxWrx+vZmcXe9WL26u329WL5czEFlbDCezIPJdHgxHvWH6aA/jlI27o8xmCZTHATRYBoF01EwCuL+bDwKcHTBMJ0xDIJBnAz7I8R4Bp532WqdncELKaIcN63WQ4YgdgVKiCsttvRf8QRjJqETfjj3E9yqksV4vhPyUUtEdc6VqtAjDN6ObbHv1ev9zY8q7IE25xX5HtbmsAITYLHmW6a5KECKSiOwVFOZs1F/BBI3tCyxu8JYCJnwwi5dfRHC8+sXoIXIVRdKgosKwkzrcn5+3h9M/YD++vOnpZD68jxDluvs5xB2XGdwJxL8XME6FxHLIUxRx1kICgkQfdd8Q0exWAqlbOGcpxjv4xx9MGzYWmC7TEDydaZrxGHEi+TasXR1aCqElPFcucquln2CCaHJsDBVbRmJOTKFEItyD0+oF6aypXv2QMSGoNkjActFsbY7WGG3KZR0ArXziFh6LOdbApfkaPsgPqHzYQyqO4eqSHjMqUyl0DVWEnnUQyyKAmM7gOOBwHKJLNlDnAuFiWu9GT0tK4BFRK2CCPUO6Wt4nOMqElQsBFYkEEpUmkm9UnGGSWXa7gE1teXF2pZbEwXSjT9MJOOFJcaQ5SDyQp+gkbiTXGssfCvTL+n0DHBLuIoYoXODW3g+7sEgGEy8fuAFs95Bt+FoOBgmQTQkANGEJD+YRfDnL7/CZHKRjpN00m21PHhZt5iL9Zwk5/auDsCCYB5czMczP5gGb+zuT/RtFs3m/Qt/Npm9gU5/5s9AubmKFBh8mAUBPH/WTLzrznnXFo/tecpyhb02Silke94+1Uu712aanh2Pn47etN+HMIYN6YIRT77pwM2PGgCSEBTC2WIl8aeKZkF6FFY+ulZy5zh/Lyc15rZcJbFHe0nbBXz98PACLKJurQO2aXygqjhGTIyP9YEdyoZS0sMYlRJyRXB4yom+Llmf+k/t1I5lLWrNDta3yjqyDyQgXakQysz446CSnlkhJRbaq5n0f1TGcI1YrHiMzgiaEVOBu4Zz0qAxitI8z0HkyakMeaPEEDaUFAbdEmODZO94M1mHiQHIpOQEUAqD3otNz1SR5aEbadNA7QHP6oMylhLzka3Rk4IIfhpVPE+8Wt/05NLtJp54wVXm2bhsrBezPDcthe7HOm5OHEMfCqIdSsalf4QgIuvrkIKL4KSUATQxUg2uWbyneV3f3y9vbu+uHm7v71bLxbevbpeLm7DbAyXs2SZJzPEfYYolL/U/XUyt7tWJlytiJAd8W3K5d65dvC1p8JgYv53ECmVYndONtqgqjcaoMMESiTo6/iStbJTNQZT4KVVBSW3XEkd/7dfh26lk3oN3plSCUs3p43HPHNo279rwHt53QyCnMHOOZ2iH0EWof0Xd6s47m7hXRkcPLmrn0De7iDSTfBI1CYbBf9mLmqHIipDsiE7yNFf/Y0oaCqi1iiyXSrE5YfbJafg8o8uHDiCX0GLCRzJWmno0kWPVZ41gPXAclh0vq1lqcNUmPF7L7o4y+cHpbEqVIvHqGKanzhkNcImlaFCrjJf/Ek3JSKgdInZD19kn9Hl6y3tsxyTlowX50Q/OSSeq22U0tsZohwTIWEniMHccxRi16C4zx6MBdv3N7dE5tZudRq8o1krNaETuVthUJmCpp1zsTrnpXPYDul5NzNY3c30v6uMUDTdXx5EahMZ9a2ZGRJ0dJ3rakIFFqE9qRW7EZlwmc6mGaDo+dxQfrOoGfbi/7JrQ4TA9E/2JiElShe3J+diSo/6XaScP98ZS5vTut7FqU1CVPahUc68T89K+ux1pXWKq4Gw4o0v5bBSMe/Ydr9X66/c/foOvXAFabxv//jpnVUIDogD+oWPuL0XvdbF96MdiU3/0Yvq9+zeBuIzBIAsAAA==",
  "delta_sha256": "e0676c680a6d12d5de77734bd3fec6f3728d391b748ac4c341648e22985649d1",
  "edited_at": "2026-10-09T01:06:37Z",
  "issue_updated_at": "2026-10-09T01:05:49Z",
  "original_capture_sha256": "be2989ab0ba908475f09d75a37a96bacc90ffc729d8da9cd2ec1ca5a671bc3aa",
  "schema": "nexus.hybrid_replication.contract_delta.v1",
  "source_event_id": "actions:37868081990:1",
  "task_key": "James3014/devspace#462",
  "title": "P2: owner cutover sidecar aborts after activation when the healthz probe reuses a closed keep-alive connection (fetch failed)"
}

feat(cutover): add carrier-bound finish-bound CLI for drained generat…

Closing: all three acceptance items are addressed. (c) devspace cutover finish-bound merged in PR #464 (c402b4c). (a) the sidecar healthz probe fix (connection: close + one retry) is applied on Dev M5 to both sidecar copies. (b) resume-from-drained is implemented in the host-side orchestrator installed on Dev M5 (~/.local/bin/m5-cutover.sh + ~/.local/bin/m5-cutover-sidecar.mjs): it detects a drained generation with an activation binding and resumes at restart or finish instead of re-issuing cutover_start. The host-side sidecar remains outside the repo by design (owner authority stays on the host).

Navigation Menu

P2: owner cutover sidecar aborts after activation when the healthz probe reuses a closed keep-alive connection (fetch failed) #462

Problem

Fresh evidence (Dev M5, 2026-10-09, cutover `4323d0b3`, `b6cd329` → `668f5df`)

Expected

Acceptance

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

P2: owner cutover sidecar aborts after activation when the healthz probe reuses a closed keep-alive connection (fetch failed) #462

Description

Problem

Fresh evidence (Dev M5, 2026-10-09, cutover 4323d0b3, b6cd329 → 668f5df)

Expected

Acceptance

Activity

github-actions commented on Oct 9, 2026

James3014 commented on Oct 9, 2026

github-actions commented on Oct 9, 2026

James3014 commented on Oct 9, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions

Fresh evidence (Dev M5, 2026-10-09, cutover `4323d0b3`, `b6cd329` → `668f5df`)