Add semantic task retrieval with scoped embeddings and RAG context

[witbrock]

Merge decision: ready — Von can retrieve related tasks by meaning through its existing task-search tool and API, with current visibility checks, citation context and explicit lexical fallback.

User outcome

Native task #V#task_agent_9a8deead9e6cb572a422aa411a7c2897 requests semantic and embedding indexing of tasks with RAG support. search_mode=semantic finds related task documents while retaining existing filters and lexical search as the default.

Material changes

• Define a versioned task document preserving identity, task fields, context, provenance and timestamps.
• Reuse the pinned RAG embedding runtime with a disposable actor/organisation-partitioned SQLite vector cache, document hashes, incremental refresh, pruning and full actor-scoped reindex/recovery.
• Read canonical task data before ranking and recheck visibility/existence after embedding; cached rows cannot supply task content or authority. Server actor context supplies scope; payload filters cannot impersonate an actor.
• Return cosine ranking, bounded citation-ready RAG excerpts and redacted diagnostics through the task API and MCP tool. Failures explicitly return lexical fallback.
• Add synthetic relevance fixtures, an opt-in configured-embedder evaluation harness, and operational documentation.

Evidence

116 targeted tests pass:

VON_USE_MOCK_DB=1 pdm run pytest tests/backend/test_task_semantic_index_service.py tests/backend/test_task_management_service.py tests/backend/test_task_routes_parity_fields.py
VON_USE_MOCK_DB=1 pdm run pytest tests/backend/test_internal_mcp_task_read_actor_scope.py tests/backend/test_internal_mcp_task_parity_tools.py

The semantic suite covers the real task service, canonical repository visibility filters, SQLite cache, Flask route and MCP handler with isolated mock storage and a fixture embedder: paraphrase ranking, filters, pagination, current text after edits, model-version changes, corruption, partial-batch recovery, actor/organisation/public boundaries, revocation, deletion and deletion during embedding. Syntax compilation and git diff --check pass.

A broader test, test_get_task_route_adds_local_hrefs_for_external_resource_actions, returns 404 instead of 200. It reproduces on an untouched worktree at starting revision d9ac13e09; it is outside semantic search and does not change the merge decision.

Ship boundary

• Minimum ship criteria met: useful semantic retrieval plumbing, current actor-visible context, existing filters and compatibility, observable fallback and recoverable indexing.
• Stop-ship: candidate-caused access leakage, stale cached task text returned as canonical, or a broken existing search route. None observed on the tested paths.
• Non-blocking limits: fixture embeddings prove ranking plumbing, not live model quality. The evaluation harness can measure recall@k, reciprocal rank, nDCG and latency against the substring baseline; no live provider was invoked. Cold indexing and exact ranking over large task sets remain unmeasured. Filter changes can evict reusable cache rows.
• Non-goals: task meaning/schema migration, attachment/comment ingestion, approximate vector search, new prompt/workflow policy, and deployment. This task does not request public DGX deployment.

No secrets or runtime data committed; the JSON fixture contains synthetic tasks only.