JVNAUTOSCI-2612: Recover visible-subject knowledge safely

[witbrock]

Outcome

• Recover denied canonical knowledge writes as provenance-bearing actor/organisation-scoped assertions when the concepts are visible but not canonically writable.
• Keep base publication and actor-effective views explicit across relation reads, Testing Theory comparison, and RAG.
• Add a compact semantic-coherence invariant and routed context-evolution guide so future changes preserve context, provenance, audience/authority, publication, lifecycle, and storage as distinct dimensions without selecting a universal context engine.

Authority and scale

• Bind mutations and reads to the trusted server actor; reject untrusted payload and raw-stdio namespace claims.
• Recheck live authority for base and scoped RAG candidates, keep authority-hidden diagnostics opaque, and bound reindex/stale-pruning memory and query work.
• This bounded slice deliberately does not add cross-namespace physical RAG fan-out or choose a general microtheory/context architecture.

Validation

• 253 affected tests passed.
• 17 manifest-parity and MCP process-guard tests passed.
• Ruff and git diff --check origin/main passed.
• Exact live Atlas replay verified canonical denial, successful scoped recovery, durable provenance-bearing read-back, base/effective view separation, idempotent retraction, and no probe residue.

Jira: JVNAUTOSCI-2612

[Copilot]

Pull request overview

This PR extends Von’s knowledge write/read surfaces to safely recover “denied canonical” writes as provenance-bearing actor/organisation-scoped assertions, while keeping base publication vs actor-effective views explicit across relation reads, Testing Theory comparison, and RAG retrieval.

Changes:

• Introduces explicit context views (base_publication vs actor_effective) for text-relation and relation reads, and threads them through MCP surfaces and Testing Theory checks.
• Adds scoped-assertion-aware RAG behaviour: live authority re-checking for base + scoped candidates and authority-opaque retrieval diagnostics.
• Adds bounded, non-inline derived-index maintenance (background queue) plus iterator-based reindexing that can also prune stale RAG docs.

Reviewed changes

Copilot reviewed 34 out of 34 changed files in this pull request and generated 1 comment.

Show a summary per file

File	Description
tests/backend/test_reindex_text_relations_cli.py	Updates CLI tests for iterator-based scanning, combined sources, batching, and stale-only deletion batching.
tests/backend/test_rag_text_relations_mcp_read_tools.py	Expands MCP read-tool tests to cover base vs scoped indexed rows and new row identity fields.
tests/backend/test_rag_text_relation_change_hook_service.py	Adds tests for queued (non-inline) RAG sync scheduling, coalescing, saturation handling, and rerun behaviour.
tests/backend/test_rag_service.py	Adds tests ensuring scoped candidates participate in concepts-mode queries and are live-filtered after revocation/visibility changes.
tests/backend/test_rag_retrieval_state.py	Updates retrieval-state expectations to be authority-opaque and adds tests for hidden candidates not appearing in diagnostics.
tests/backend/test_rag_provenance_contract.py	Adds tests enforcing trusted server actor binding and rejecting conflicting/untrusted namespaces before backend access.
tests/backend/test_mongo_collection_index_guards.py	Adds index-guard coverage for scoped assertions and new/updated text-relation index shapes.
tests/backend/test_internal_mcp_scoped_assertion_authority.py	Adds comprehensive gateway tests for trusted-actor binding, payload identity rejection/degradation, paging semantics, and receipts.
tests/backend/test_internal_mcp_concept_exists_gateway.py	Tightens access-denied behaviour to avoid leaking payload identity and ensure access diagnostics are used.
tests/backend/test_internal_mcp_catalogue_builds.py	Ensures scoped assertion tools are registered and have appropriate effect admission windows and trusted bindings.
tests/backend/test_description_metadata_preservation.py	Updates description metadata tests to include subject concept id and visibility filtering plumbing.
tests/backend/test_context_semantic_coherence.py	Adds end-to-end semantic coherence invariants around base vs effective views, summary ID separation, and authority enforcement.
tests/backend/test_adaptive_turn_service.py	Adds adaptive-turn tests verifying canonical denial recovery via upsert_scoped_assertion with correct affordances.
src/utilities/reindex_text_relations.py	Switches reindexing to an iterator-based source, adds stale pruning support, and improves progress/limit semantics.
src/backend/services/text_value_service.py	Adds explicit text context views, overlays scoped assertions for actor-effective reads, and extends summaries with typed IDs.
src/backend/services/text_relation_predicate_validation_service.py	Adds helper to normalise predicate concept IDs for storage/authority checks.
src/backend/services/testing_theory_service.py	Ensures canonical-text checks explicitly use base_publication view.
src/backend/services/scoped_rag_authority_service.py	Adds live authority re-checking for RAG candidates (base relations + scoped assertions) using current visibility + audience.
src/backend/services/rag_text_relation_change_hook_service.py	Implements bounded in-memory queue + worker for best-effort derived RAG refresh outside primary write paths.
src/backend/services/rag_backends/llamaindex_backend.py	Applies live authority filtering before visible diagnostics, supports scoped candidates in concepts mode, and degrades bounded empties.
src/backend/services/concept_relation_service.py	Adds explicit context view for relation lookup, integrates scoped assertions, and preserves lower-bound/truncation signalling.
src/backend/services/adaptive_turn_service.py	Adds bounded recovery affordances for subject-authority denials and correlates successful scoped recovery with prior failures.
src/backend/security/access_control.py	Adds force_access_control_enforcement() to enforce global-only visibility when no actor is bound (stdio/MCP).
src/backend/mcp_server/vontology_mcp.json	Updates tool schemas/descriptions to reflect predicate typing options and explicit base vs effective semantics.
src/backend/mcp_server/mcp_stdio_server.py	Forces access-control enforcement for stdio reads and routes RAG search through trusted actor context resolution.
src/backend/db/mongo_client.py	Adds scoped assertions collection + index ensuring; expands text-relations indexes to support new query patterns.
docs/engineering/contextual_knowledge_evolution.md	Adds routed design guide for preserving context/provenance/audience/publication/storage distinctions.
docs/design_index.md	Routes readers to the new contextual knowledge evolution guide and updates review date.
AGENTS.md	Adds invariant to preserve contextual degrees of freedom and routes to the new guide.

Comments suppressed due to low confidence (1)

tests/backend/test_rag_retrieval_state.py:290

• The third SimpleNamespace candidate has the same indentation issue as the previous block; as written, the metadata dict is not correctly nested under SimpleNamespace(, which will raise a syntax/indentation error when importing the test module.

            node=SimpleNamespace(
                    metadata={
                        "type": "text_relation",
                        "relation_id": "filtered",
                        "user_id": "#V#other_user",