Improve Thinking workflow labels and model telemetry - #351
Merged
Merged
Conversation
There was a problem hiding this comment.
Pull request overview
This PR improves how “represented workflow” capabilities are labelled and displayed in Thinking cards by carrying a trusted human-readable display name alongside stable capability/workflow identities, and it expands workflow LLM telemetry to preserve requested/selected/provider-observed model identities.
Changes:
- Add
capability_display_nameto represented-workflow invocation summaries/diagnostics and propagate it through adaptive turn execution and MCP catalogue summaries. - Update Thinking semantic projection + frontend rendering to prefer human workflow labels in Default/Expert modes (with canonical IDs still inspectable, and raw invocation IDs reserved for Debug), including legacy-card name recovery without DB lookups.
- Extend durable LLM-call recording to capture
requested_model,selected_model,effective_model, andmodel_identity_source, with updated backend test coverage.
Reviewed changes
Copilot reviewed 15 out of 15 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| tests/backend/test_turn_execution_record_service_execution_summary.py | Asserts capability_display_name is preserved in execution summaries for represented workflows. |
| tests/backend/test_tool_result_hint_actions.py | Verifies durable LLM-call outputs include requested/selected model identities across fallbacks. |
| tests/backend/test_thinking_semantic_projection_service.py | Adds coverage for human workflow labels while retaining stable identities and legacy normalisation behaviour. |
| tests/backend/test_rag_turn_execution_records_mcp_read_tools.py | Ensures diagnostics hydration and turn-execution read paths include capability_display_name. |
| tests/backend/test_llm_step_executor.py | Expands assertions to confirm gateway LLM-call metadata is preserved end-to-end. |
| tests/backend/test_adaptive_turn_service.py | Validates represented workflow invocations and progress events carry the display name and semantic operation label. |
| src/frontend/web/von_interface/static/js/test/chatTab.test.js | Adds Thinking UI tests for default/expert/debug display rules and legacy workflow-name recovery. |
| src/frontend/web/von_interface/static/js/chatTab.js | Implements capability descriptor extraction + opaque-label replacement + mode-specific workflow display behaviour. |
| src/backend/workflows/llm_step_executor.py | Records requested/selected/effective model identities (and source) into workflow LLM-call telemetry. |
| src/backend/workflows/durable/tool_result_hint_actions.py | Mirrors LLM-call metadata recording for durable nested workflow recorders. |
| src/backend/services/turn_execution_record_service.py | Includes capability_display_name in summarised tool invocation projections. |
| src/backend/services/turn_execution_diagnostics_service.py | Includes capability_display_name in summarised tool invocation history for diagnostics. |
| src/backend/services/thinking_semantic_projection_service.py | Introduces bounded capability label selection logic and uses it in semantic operation normalisation/projection. |
| src/backend/services/adaptive_turn_service.py | Threads workflow display names through contained capability execution and semantic projection emission. |
| src/backend/integrations/internal_mcp/catalogue.py | Adds capability_display_name to MCP catalogue turn-execution tool invocation summaries. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Why
Represented workflows were surfaced as
Represented Workflow <hash>, so ordinary users could not tell what Von was doing. Separately, nested workflow recorders accepted only the selected model name and dropped the requested/effective identity fields already produced by the gateway.User impact
Default Thinking now explains the named workflow in human terms. Expert retains semantic and canonical identity, while Debug adds the raw invocation identifier. Workflow telemetry can now distinguish the model the user requested, the model routing selected, and the model the provider reported.
Validation
git diff --checkpassed727deaec-f483-47ce-a752-f2746b9b38b7: Default displayed “Arxiv Paper Representation Workflow” without the generated or raw hash; Expert retained the human workflow name; Debug exposed the raw invocation ID alongside itThe historic represented-workflow invocation used for display validation had failed; the browser replay validates display and legacy recovery, not that historic workflow effect.
Jira: JVNAUTOSCI-2619