Skip to content

P1: bound cutover finish leaves the state-root effect lease held; next prepare fails with overlapping-scope #472

Description

@James3014

Problem

A coordination-bound cutover that is closed through DurableOperations.finishCutover (MCP finish, cutover finish-bound from #464, or the owner sidecar) records the terminal receipt and closes the cutover generation, but the state-root effect lease that cutover_start acquired is never released. finishCutover calls consumer.finish(...), which only runs ControlPlaneOwnership.finishOperation (sets operation_state='finished', clears the operation handle). terminal_state stays null, so assertNoOverlap treats the state root as still leased until the lease expires.

Observed on Dev M5 on 2026-10-09 after cutover 8eaa5969 (668f5df → 5c008b0) was finished: the next cutover_start prepare failed with overlapping resource scope is already leased. The only way out was the manual cutover release-terminal-lease --cutover-id … --lease-id … --lease-version <current> --carrier <owner> --carrier-version <current> --terminal-record-hash <cutoverTerminalRecordHash(store.get())> --confirm <cutover-id> (CarrierBindingStore.releaseClosedCutoverLeaseLocal), which the M5 orchestrator now runs as a workaround step after every finish.

Expected

  • On the non-recovery path, finishCutover releases the lease in the same atomic section that records the terminal receipt (lifecycleTerminal: true, terminalRecordHash), using the same terminal-record-hash correlation that releaseClosedCutoverLeaseLocal enforces. A terminal replay of an already-released lease must be a no-op, not a CAS error.
  • On the expired/recovery path the lease is already reconciled through reconcileCutoverFinish; confirm it ends terminal, or release it the same way.
  • cutover finish-bound output reports the lease terminal state so an operator can see it was released.
  • cutover release-terminal-lease stays as the recovery entrypoint for historical records closed before this fix.

Acceptance

  • Unit test: bound cutover start → drain → finish; the state-root lease is terminal (released) and a second cutover_start on the same state root succeeds without release-terminal-lease.
  • Unit test: replaying finish after release returns the closed record and leaves the lease unchanged.
  • Unit test: recovery-path finish leaves no held lease.
  • npm run typecheck, npm run build, affected tsx src/*.test.ts suites green.

Scope note: #471 keeps the bound cutover path as historical recovery; this fix is confined to that path and must not block or be blocked by #471.

Refs #462, #464, #471

Activity

  1. James3014 commented on Oct 9, 2026

    @James3014
    OwnerAuthor

    Related uncommitted work found on Dev M5 (~/.devspace/worktrees/issue240-terminal-lease-hotfix, 2026-09-26, base = main@#286), saved before the worktree was removed. It extends releaseClosedCutoverLeaseLocal with an externallyReloadedCompletion case (closed generation whose reconciliationReceipt.closedByServerInstanceId differs from the old server identity, with drain evidence and no restart request). PR #477 makes the normal finish release the lease itself, but does not cover a generation closed by an externally reloaded instance; this patch is a candidate follow-up, unreviewed.

    diff --git a/src/carrier-binding.test.ts b/src/carrier-binding.test.ts
    index f386cf7..9436f8e 100644
    --- a/src/carrier-binding.test.ts
    +++ b/src/carrier-binding.test.ts
    @@ -277,6 +277,62 @@ test("terminal hygiene releases the exact active prepared-replacement cutover le
       }
     });
     
    +test("terminal hygiene releases an externally reloaded drained replacement after exact reconciliation",async()=>{
    +  const f=fixture();
    +  let manager:DurableOperationManager|undefined;
    +  try {
    +    const context={clientId:"shared-oauth",sessionId:"externally-reloaded-terminal-hygiene"};
    +    const pairing=f.store.requestPairing(context);
    +    const cutover={
    +      stateRoot:f.root,
    +      attemptKey:"externally-reloaded-terminal-hygiene",
    +      currentIdentity:{serverInstanceId:"original",sourceCommit:f.contract.baseRevision,buildId:"old",capabilityManifestSha256:"c".repeat(64)},
    +      expectedIdentity:{sourceCommit:"b".repeat(40),buildId:"new",capabilityManifestSha256:"d".repeat(64)},
    +      expiresAt:new Date(f.clock()+60000).toISOString(),
    +      restart:{buildReady:{verifiedBy:"independent",verifiedAt:new Date(f.clock()).toISOString(),evidence:"exact package digest"},actuator:"launchd-self" as const,serviceLabel:"test.service",launchdTarget:"gui/501/test.service"},
    +      finish:{workspaceId:"ws_external_cleanup",agentId:"agt_external_cleanup"},
    +    };
    +    const contract:CarrierContract={...f.contract,scope:[f.root],operations:["cutover_start"],cutover};
    +    const approved=f.store.approveLocal(pairing.pendingId,contract);
    +    f.store.redeem(context,pairing.credential);
    +    const plan=planCutoverStart(f.root,cutover);
    +    const acquired=f.store.prepareEffect(context,plan.subject);
    +    const config=loadConfig({DEVSPACE_CONFIG_DIR:join(f.root,"config"),DEVSPACE_ALLOWED_ROOTS:f.workspace,DEVSPACE_WORKTREE_ROOT:join(f.root,"worktrees"),DEVSPACE_STATE_DIR:f.root,DEVSPACE_OAUTH_OWNER_TOKEN:"test-owner-token-long-enough",PORT:"1"});
    +    manager=new DurableOperationManager(config,undefined,undefined,undefined,f.store.readers);
    +    const start=manager.startCutover(cutover,context);
    +    const cutoverId=start.receipt!.cutoverId as string;
    +    f.store.readers.approveCutoverLifecycle=()=>true;
    +    manager.drainCutover(cutoverId,cutover.currentIdentity,()=>({activeSessions:0,oldestAgeMs:0}),context);
    +    const replacement={serverInstanceId:"replacement",...cutover.expectedIdentity};
    +    const witness={workspaceQueryable:true,agentQueryable:true,agentReconciled:true,witnessWorkspaceId:cutover.finish.workspaceId,witnessAgentId:cutover.finish.agentId};
    +    await manager.finishCutover(cutoverId,replacement,cutover.finish,async()=>witness,context);
    +
    +    const closed=new CutoverStateStore(f.root).get()!;
    +    assert.ok(closed.drainEvidence);
    +    assert.equal(closed.restartRequest,undefined);
    +    const terminalHash=cutoverTerminalRecordHash(closed);
    +    const terminalLease=f.store.ownership.get(acquired.leaseId)!;
    +    const revoked=f.store.revokeLocal(approved.id,1);
    +    assert.equal(revoked.version,2);
    +    f.store.forgetSession(context.sessionId);
    +
    +    const released=f.store.releaseClosedCutoverLeaseLocal({
    +      cutoverId,
    +      leaseId:acquired.leaseId,
    +      expectedLeaseVersion:terminalLease.version,
    +      carrierId:approved.id,
    +      expectedCarrierVersion:2,
    +      expectedTerminalRecordHash:terminalHash,
    +      confirmCutoverId:cutoverId,
    +    });
    +    assert.equal(released.lease.terminalState,"released");
    +    assert.equal(released.lease.version,terminalLease.version+1);
    +  } finally {
    +    manager?.close();
    +    f.close();
    +  }
    +});
    +
     test("expired coordination-bound prepared cutover can terminally reconcile an exact observed replacement",async()=>{
       const f=fixture();
       try {
    diff --git a/src/carrier-binding.ts b/src/carrier-binding.ts
    index bf079a6..470e7eb 100644
    --- a/src/carrier-binding.ts
    +++ b/src/carrier-binding.ts
    @@ -943,10 +943,16 @@ export class CarrierBindingStore {
         );
         const normalCompletion=Boolean(closed.drainEvidence && closed.restartRequest?.restartScheduledAt);
         const preparedReplacementCompletion=closed.drainEvidence===undefined && closed.restartRequest===undefined;
    +    const externallyReloadedCompletion=Boolean(
    +      closed.drainEvidence &&
    +      closed.restartRequest===undefined &&
    +      closed.reconciliationReceipt?.closedByServerInstanceId &&
    +      closed.reconciliationReceipt.closedByServerInstanceId!==closed.oldServerIdentity.serverInstanceId
    +    );
         if(closed.coordinationBinding.leaseId!==input.leaseId ||
            closed.expiredPreparedNoEffect || closed.capabilityExpectationMismatch || closed.observedReplacement ||
            closed.supersession || closed.bindingRepair ||
    -       (!normalCompletion && !preparedReplacementCompletion) || !positiveWitness) {
    +       (!normalCompletion && !preparedReplacementCompletion && !externallyReloadedCompletion) || !positiveWitness) {
           deny("Terminal lease release requires one supported terminal cutover generation");
         }
     
    @@ -989,7 +995,7 @@ export class CarrierBindingStore {
              operation.receipt?.lifecycleTerminal!==true || operation.receipt?.terminalRecordHash!==terminalHash) {
             deny("Terminal lease release requires the exact successful terminal cutover operation");
           }
    -      if(preparedReplacementCompletion) {
    +      if(preparedReplacementCompletion || externallyReloadedCompletion) {
             const action=operation.receipt.lifecycleAction as {
               action?:unknown;
               cutoverId?:unknown;
    @@ -1005,7 +1011,7 @@ export class CarrierBindingStore {
                action.preferredPair?.workspaceId!==approved.finish.workspaceId ||
                action.preferredPair?.agentId!==approved.finish.agentId ||
                closed.reconciliationReceipt?.closedByServerInstanceId!==identity.serverInstanceId) {
    -          deny("Terminal lease release prepared replacement evidence is not exact");
    +          deny("Terminal lease release replacement evidence is not exact");
             }
           }
           const {coordinationBinding,...request}=operation.request;
  2. James3014 commented on Oct 10, 2026

    @James3014
    OwnerAuthor

    Wave 3 merge-gate reconciliation — #472 / PR #477 (2026-10-10)

    Classification: KEEP_OPEN / TRANSPORT_OR_TOOLING / MERGE_GATE_BLOCKER, not product-verifier FAIL.

    No source, PR, workflow, ruleset, deployment or runtime mutation occurred in this reconciliation.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions