Reconcile late durable workflow outcomes

[witbrock]

What changed

• reconcile a timed-out ordinary-turn effect when its canonical durable workflow later reaches completed, failed, or cancelled
• preserve the immutable turn-time partial receipt while adding an actor-scoped late_terminal journal phase
• persist tool_invocations on failed turns and recover represented workflow, instance, effect, and transport identities in Jira-facing diagnostics
• hydrate zero-count persisted diagnostics from bounded invocation evidence

Root cause

The workflow handler could durably submit an instance and then exhaust the MCP database/handler deadline. The turn correctly returned a partial receipt, but PyMongo deadline unwinding meant the handler-level late callback never observed the eventual durable terminal state. The failed-turn projection also read the obsolete invocations key, while the adaptive path emits tool_invocations, leaving the durable record with zero calls.

Impact

Post-turn diagnostics now distinguish the immutable turn result from the later canonical workflow outcome. A completed durable instance resolves the effect journal without retrying the effect, and exact workflow/instance/execution identifiers remain available for recovery and support.

Validation

• Ruff on all changed Python and test files
• 7 exact receipt, reconciliation, routing, diagnostics, and failed-turn projection tests
• 58 durable-instance-manager and adaptive-turn late/effect tests
• live canonical read-back: request live-kb-prompt-641f144c-4820-42ac-8622-51c2fc695ea0 now shows 3 tool calls, exact instance aa8ad9a5-b78e-4542-9431-f0a546196d4d, and late_terminal=succeeded

A broader touched-file run passed 126 tests; 11 pre-existing benchmark/dashboard tests still fail at their operator-authority guard before reaching the changed code.

[Copilot]

Pull request overview

This PR improves post-turn diagnosability and recovery by reconciling ordinary-turn “timed out after durable submission” effects with the eventual terminal state of the canonical durable workflow instance, while preserving the immutable turn-time receipt.

Changes:

• Add late-terminal reconciliation that records an actor-scoped late_terminal phase when a durable workflow later reaches completed / failed / cancelled.
• Persist and prefer identity-bearing tool_invocations for failed turns and MCP-facing diagnostics, with bounded hydration when persisted diagnostics are empty.
• Extend tests to cover routing reconstruction, terminal reconciliation idempotency, and diagnostics hydration/preference rules.

Reviewed changes

Copilot reviewed 8 out of 8 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/backend/test_turn_execution_record_service_execution_summary.py	Adds coverage for routing reconstruction and durable terminal reconciliation journaling/idempotency.
tests/backend/test_subworkflow_budget_diagnosability.py	Ensures failed-turn persistence includes tool_invocations and workflow identity fields.
tests/backend/test_rag_turn_execution_records_mcp_read_tools.py	Verifies MCP diagnostics prefer top-level identity-bearing invocations and hydrate empty tool history.
tests/backend/test_durable_workflow_system.py	Ensures durable instance completion triggers reconciliation for conversation-turn sourced instances.
src/backend/workflows/durable/instance_manager.py	Triggers best-effort reconciliation on durable terminal transitions for conversation turns.
src/backend/services/turn_execution_record_service.py	Adds reconciliation function and improves debug-driven routing/tool-invocation extraction.
src/backend/services/turn_execution_diagnostics_service.py	Adds bounded tool-invocation summarisation and hydration fallbacks for diagnostics.
src/backend/integrations/internal_mcp/catalogue.py	Prefers top-level invocations and enriches summaries with workflow/instance/transport identities.

Comments suppressed due to low confidence (1)

src/backend/services/turn_execution_record_service.py:12837

• Routing synthesis reads only llm_debug["tool_invocations"]. Many legacy debug payloads still store the tool list under invocations, which means this function will ignore tool activity entirely and default to plain_response. Consider falling back to invocations when tool_invocations isn't present (while still respecting an explicitly empty tool_invocations list).

        for raw_invocation in tool_invocations:
            if not isinstance(raw_invocation, Mapping):
                continue