Partition-owning types declared in mesh content: nested instances, bulk-create compensation, one resolution per create, i18n mirror sync

[rbuergi]

Follow-up to #4447 (every partition-owning NodeType can be created at the top level, not only Space). Review findings deliberately left out of that PR because none of them is made worse by it — each is pre-existing:

1. Nested instances of an in-mesh owning type are not refused. OwnsPartitionProvisioningValidator refuses a nested instance only for types registered in src/; for a type declared in mesh content (Crm/Client) a nested create was never refused and still is not. Closing it adds a definition read to every nested create of every in-mesh type, which is why it was not folded into a fix for the top-level path. Decide: read per create (non-System only), or record OwnsPartition where the create path already has the type.
2. Bulk create does not compensate a failed critical post-creation handler. CreateNodesRequest.BulkRefusal permits top-level owning roots; the bulk runner invokes FailsCreateOnError handlers but does not run CompensateFailedCreate when Handle faults, so a failed owner grant leaves the root behind. Same gap exists for Space.
3. The ownership declaration is resolved once per check (RLS, write guard, provisioning, post-creation) for a top-level in-mesh create. Every disagreement fails closed after fix: every partition-owning NodeType can be created at the top level, not only Space #4447, but one resolution per create operation would be cheaper and consistent.
4. i18n mirror sync: fix: every partition-owning NodeType can be created at the top level, not only Space #4447 adds access.partitionCreate.* keys to strings.{en,de}.json; run npm run sync:i18n -- --ref <merged core sha> in MeshWeaver.Plugins (clients/react/src/i18n/) after it merges.

🤖 Generated with Claude Code

[rbuergi]

{"body":"Item 4 is done (MeshWeaver.Plugins#1966). Item 2 is worse than the filing says, and the gap is asserted-but-false in a comment — details below so whoever takes it does not have to re-derive them.\n\n## Item 4 — the mirror sync, done\n\nSynced to core main's TIP rather than #4447's merge sha, because a pin landing on #4447 alone would be stale on arrival: 73 keys added, 14 removed, 0 value-changed per locale (9429593ad -> f0ea2ceb3). The 14 removals are all plugins.discovery.* / plugins.reconcile.deferred.*, core carries zero of them since #3603, and no React source references any (102 .ts/.tsx files scanned; 24 distinct dotted tokens; 13 are catalog keys and all 13 survive).\n\n## Item 2 — what is actually there\n\nMeshExtensions.cs, bulk phase 7:\n\ncsharp\n// ——— Phase 7: post-creation handlers per created node,\n// sequential — same semantics as the singular create\n// (FailsCreateOnError propagates; best-effort handlers\n// log-and-continue inside the runner). ———\nreturn list\n .Select(saved => RunPostCreationHandlersObs(hub, saved, capturedBy, logger))\n .Concat()\n .ToList()\n .Select(_ => (list, existingPaths));\n\n\n🚨 "same semantics as the singular create" is asserted and false. The singular path, on the same fault, calls CompensateFailedCreate and deletes the row it wrote — with the reason stated in place: "Reporting Fail while LEAVING the row was the whole defect: the caller cannot retry (create answers 'already exists') and cannot clean up (nobody has rights on the partition)." The bulk path has no compensation at all; the fault propagates to the outer subscriber and every row stays.\n\n## It is BROADER than "leaves the root behind"\n\nStorage writes all rows before phase 7 (the list.Count != stamped.Count check above it is the all-or-nothing storage gate). Handlers then run sequentially via .Concat(). So a critical fault at index k leaves two distinct populations of ghost rows, not one:\n\n| nodes | state | today |\n|---|---|---|\n| 0 … k-1 | created, handlers COMPLETED | fine — keep |\n| k | created, its critical handler FAILED | row kept (singular path would delete it) |\n| k+1 … n | created, handlers NEVER RAN | row kept, no owner grant ever attempted |\n\nThe third row is the one the filing does not mention and is the larger hazard: those nodes are indistinguishable from successfully-created ones by inspection, yet no post-creation handler ever touched them.\n\n## The decision it needs\n\nCompensating k … n (the failed one plus the never-reached ones) and keeping 0 … k-1 is exact parity with the singular path, per node. It is also a destructive change — it deletes rows a caller may currently be recovering by hand — so it wants a deliberate call rather than being folded in. Two things make it safer than it sounds: CompensateFailedCreate re-reads the row and compares CreatedDate before deleting, so it cannot remove a node this request did not create; and everything in list is by construction this request's own (pre-existing paths are tracked separately in existingPaths).\n\nSuggested order when someone does it: compensate in reverse (n … k), so a child is removed before its parent.\n\nNot claiming it — recording it because this issue had no comments and the comment in the code says the opposite of what the code does."}

[rbuergi]

Item 2 is done — #4503, merged as ae5cf0e195.

The fix follows the mechanism recorded above, including the part the original filing does not mention: a critical fault at index k leaves TWO populations of ghost rows, and both are compensated (k…n, in reverse), while 0…k-1 are kept. Both safety facts were re-verified in the code as it stands before anything was deleted.

Two things the implementation added that were not in the plan, and both were found by measurement rather than by reading:

1. The stop cannot be a Take(1). Catching per index into a failure element and taking the first compensates the right nodes and does not stop the chain — Observable.Concat(IEnumerable<IObservable<T>>) went on to subscribe every later node although the downstream had disposed, so k+1…n ran their handlers anyway and would have written side effects for rows about to be removed. The per-index Catch therefore re-throws tagged with the index (a fault terminates a Concat by construction).
2. It has to be deferred. The runner resolves handlers and asks each Matches on the way to returning its observable, so a handler throwing there bypassed the tag entirely and the batch reported a partial landing having compensated nothing (Copilot review; reproduced, then pinned by its own test).

Full write-up, with the measurements and the rollback's four reported states: Doc/Architecture/BulkCreateCompensation.

Residuals from review, filed rather than folded in: #4506 (the lineage check is read-then-delete, and CreatedDate is caller-supplied — both pre-existing and identical on the singular path since #638) and #4507 (CreateNodesResponse.Error is English-only, as is every sibling message on that surface).

Items 1 and 3 of this issue are untouched.

🤖 Generated with Claude Code

[rbuergi]

{"body":"## Items 1 and 3 read as one change. They are not — measured on today's main.\n\nI expected item 3 (resolve once per create) to make item 1 nearly free: if the declaration is already in hand, the nested check could consult it at no extra cost. That does not hold, and the reason is worth writing down before anyone else assumes it.\n\n### All four resolutions are on the TOP-LEVEL path\n\nPartitionOwningTypes.OwnsPartition(hub, nodeType) has exactly four call sites, and every one is gated to top-level creates:\n\n| site | nested behaviour |\n|---|---|\n| OwnsPartitionProvisioningValidator:97 | if (!string.IsNullOrEmpty(context.Node.Namespace)) return Valid() |\n| RlsNodeValidator.CheckPartitionOwnerCreate:284 | \\|\\| !string.IsNullOrEmpty(context.Node.Namespace) → returns null |\n| PartitionWriteGuardValidator:210 | inside the top-level branch only |\n| InMeshPartitionOwnerPostCreationHandler:67 | Matches requires PartitionDefinition.IsPartitionRoot, i.e. top-level |\n\nSo the two items act on disjoint populations: item 3 removes repeated work on top-level creates; item 1 would add work to nested ones, where today there is none. Sharing a resolution buys item 1 nothing.\n\n### What item 3 is actually worth\n\nFor an in-mesh type, one resolution is two round trips — an anchored namespace:… nodeType:… query to establish the definition exists, then GetMeshNodeStream(nodeType).Take(1) as System to read it. No cache. So a top-level create of an in-mesh owning type resolves the same fact up to four times = up to eight reads.\n\n🚨 But it is not purely a saving, and that is the part to decide deliberately. The four checks run at different moments (RLS → write guard → provisioning → post-creation). Today, a definition changing mid-create would be seen by a later check and fails closed on the disagreement (#4447). Resolving once removes the disagreement by removing the second look. That is almost certainly the better semantic — a create should be judged against ONE view of its type — but it is a change in behaviour on a security-adjacent path, not a pure optimisation, and it should be argued rather than slipped in as a perf fix.\n\n### Item 1's two options, sharpened\n\nThe filing offers "read per create (non-System only)" or "record OwnsPartition where the create path already has the type". The second is only cheap if some existing step on the NESTED path already materialises the definition (the type-registration check is the candidate). If it does not, option 2 collapses into option 1 and the honest cost is one extra read on every nested create of every in-mesh type, fleet-wide.\n\nThat is the measurement whoever takes item 1 should make first, and it is not the measurement I made — I only established that the four existing reads are not on that path.\n\nItem 2 is done (#4503), item 4 is done (MeshWeaver.Plugins#1966)."}

[rbuergi]

Item 3 is done — #4515. Item 1 is MEASURED and deliberately not paid for; the issue stays open on it.

Full write-up, committed rather than left in this thread: Doc/Architecture/PartitionOwnershipResolution.

Item 3 — one resolution per create

The three validators run back to back on ONE NodeValidationContext and now share a single resolution through PartitionOwningTypes.OwnsPartitionOnce, memoized on NodeValidationContext.PartitionOwnership. A top-level create of an in-mesh owning type went from three resolutions (six reads) to one (two).

The semantic change, stated rather than slipped in. Three independent resolutions could disagree if the declaration moved between them, and after #4447 the disagreement failed the create closed. Sharing removes that detection for those three — deliberately, on the argument you made: a create should be judged against ONE view of its type, and the window those three spanned is the microseconds between two validators of the same Concat.

The wide window is NOT collapsed, and that is the part worth keeping. InMeshPartitionOwnerPostCreationHandler sits on the far side of the storage write — the only window in a create long enough for a declaration to actually move — and keeps its own resolution. A declaration that reads "not owning", or does not answer, there still faults the create. So the change removes three microsecond-wide disagreement checks and keeps the one that spans the write.

Fail-closed survives by construction. The memo stores the resolver's tri-state including null ("could not be established"), so a starved resolution is remembered AS starved and every later check answers Undetermined exactly as it would have alone. A stored null is a recorded decision, never an absent entry.

The carrier. NodeValidationContext is built fresh per operation, so the memo's lifetime IS that operation's — it can leak across neither operations nor users, and it is not a cache (nothing in it is keyed by anything an earlier operation wrote). Keyed by node TYPE, so a context copied for a different node cannot read another type's answer. No static anywhere; ImmutableDictionary + ImmutableInterlocked, no gate.

Negative control, run: reverting all three validators to the unshared resolver fails the new end-to-end test with …Got 0 — the stated reason — while the create itself still succeeds, which is what makes the test a measurement of sharing rather than of behaviour.

Item 1 — the measurement you asked for, and a recommendation

You asked whether option 2 ("record OwnsPartition where the create path already has the type") is cheap, i.e. whether some step already on the NESTED path materialises the definition. It does not:

• NodeTypeResolution.Resolves — the type-registration check, the named candidate, run on every create — probes IStorageAdapter.Exists(nodeType) and yields a bool. It establishes that the definition exists; it never hands back its content.
• CreatableTypesCreationValidator is the one create-path validator that does materialise a NodeTypeDefinition on a nested create — the parent's type, for its CreatableTypes whitelist, not the created node's own.
• No other INodeValidator in the Create chain reads the created node's type definition at all.

So option 2 collapses into option 1, and the honest cost is one full resolution — two round trips — on every nested create of every in-mesh type, fleet-wide, for every non-platform writer.

🚨 And the cost is not only reads, which is the part that changed my recommendation. The second half of a resolution is GetMeshNodeStream(<type>), and for a per-node hub the read IS the activation. A cold NodeType activation compiles; the create path's own ControlPlaneActivationBudget remark puts a cold per-node activation at 5–45 s in CI, against PartitionOwningTypes.ProbeTimeout of 10 s — and a timeout answers null, which every caller turns into a fail-closed refusal. Paying it on every nested create would put an intermittent refusal, whose trigger is whether the type's hub happened to be warm, in front of routine content creation. (How often a hub is cold at that moment on a live portal is not measured — the coupling is, from the code.) The top-level path accepts that risk because a top-level create of an owning type IS a partition-creation act and is rare; a nested create of a package-declared type is the ordinary content path.

Against that, the benefit is narrow and is not a privilege escalation: the node lands inside a partition the caller already holds Create on, IsPartitionRoot is false so no post-creation handler makes it a partition, no schema and no grant are written, and PartitionDropPostDeletionHandler will not fire for it. It is a data-shape inconsistency — the type author said instances own their partition, and here is one that does not.

Recommendation: leave item 1 open, and close it properly rather than cheaply. What would make it cheap is a projection answering "does this type own its partition?" without activating the type's hub — the shape NodeTypeInstanceLocations already has for instanceLocations, fed by each definition's own hub. That projection is deliberately fail-open (a type whose hub is not live on this process is unknown, and the query fans out in full), which is right for narrowing a query and wrong for deciding a refusal: enforcement that varies with which hubs happen to be warm is worse than enforcement that is honestly absent. Giving that lane a fail-closed denominator is the real prerequisite, and it is a larger design than this issue frames.

One adjacent observation, recorded but not acted on: CreateLayoutArea forces the namespace to root for a partition-owning type only when FindStaticNode answers, so the create FORM is itself a route by which a nested instance of an in-mesh owning type gets authored. CreatableTypesProvider.BuildInfoFromMeshNode already materialises each offered type's definition, so carrying ownsPartition onto CreatableTypeInfo would close that route at zero extra read cost — but it is a UI convenience, not the boundary refusal item 1 asks for, and it should not be mistaken for one.

A gate defect the change surfaced, fixed at the root in the same PR

Worth knowing about because it is fleet-wide and it fired as a FALSE POSITIVE an author could not have fixed on the code side. Interface additions (implementers declared) failed #4515 on IOwnerEnforcedNodeValidator.PartitionOwnership — a member that is not on that interface, or on any interface. parse_members in scripts/check-type-forwards.py located a type's body by testing the declaration line for a TRAILING {, so public interface IOwnerEnforcedNodeValidator { } matched neither that nor the ; terminator, the scan walked past it, and it adopted the NEXT type's body: every member of NodeValidationContext was indexed under the marker, where members are public-by-default and oblige an implementer. The same mistake pointed the other way too — the record scanned as having no members of its own, so a REMOVAL from it would have been reported against the interface rather than as a member removal.

The body is now located by the line's brace BALANCE, with an empty inline body scanned correctly and a one-line body WITH members refused by name (measured: 4 empty inline bodies under src/, 0 with members). check-parser-delta.py priced the fix at −6 on publicMembersAtBase and implementerObligationsAtBase over an identical tree — exactly NodeValidationContext's six members at the merge base, the phantoms — and both are declared in the PR body. Written up in Doc/Architecture/CrossRepoPairGate.

Items 2 (#4503) and 4 (MeshWeaver.Plugins#1966) remain done.

🤖 Generated with Claude Code

[rbuergi]

Item 1 is implemented — #4589. Item 3 (#4515) is unblocked and landing first.

#4515 (item 3) is MERGED — 0de659120a — and the queue rule did not need to change

#4515 sat as a draft because the merge queue runs check-parser-delta.py strictly (no pull-request body on merge_group) and its detector fix measured −6 there. That shrink was irreducible over that merge base only: all six were phantoms caused by one source construct, public interface IOwnerEnforcedNodeValidator { } written inline above NodeValidationContext. #4572 (merged as 188af2f6d1) moved that empty body onto its own lines — formatting only, invisible to the control because the detector did not change — and over the new main the same queue-form check reads +0 on all four counters. #4515 then went through the queue with no rule change and no one-time exception, merging as 0de659120a; its two Copilot threads were already addressed in 25b3adb2, and are answered and resolved. The recipe is recorded in Doc/Architecture/CrossRepoPairGate.

Item 1 — the design, and how "unknown" is handled

A hub-fed projection (the NodeTypeInstanceLocations shape) cannot give a fail-closed denominator: its denominator is the hubs live on this process, and every answer for unknown is wrong for a refusal (allow → varies with warm hubs; refuse → intermittent refusal of ordinary content; activate → the 5–45 s compile this was avoiding).

The nested create instead reads the definition's durable row (IStorageAdapter.Read(<type>)) after the static registry — exactly the two sources NodeTypeResolution.Resolves consults. The set of types a create can land with is static ∪ stored, so the denominator is complete by construction and "not known on this process" cannot occur; no hub is activated.

• owns → refused, InvalidPath, keyed access.partitionCreate.nestedOwningType (caller's language; the static-type refusal now uses it too)
• does not own (declares otherwise / not a definition / row absent / no storage) → proceeds; an absent type is refused as unregistered next
• could not be established (store fault or timeout) → refused as Unavailable, retryable. The same store answers the existence probe a moment later, so this adds no availability dependency.

Measured while proving it: with the definition's row read faulting, the listing half of the top-level resolver (in-memory query provider) answered "absent" and a nested instance wired to it was created — a query-backed existence check fails open here; the durable read does not.

The create form carries CreatableTypeInfo.OwnsPartition and places an in-mesh owning type at root — a convenience beside the boundary refusal, not instead of it.

Proved on a real mesh (InMeshPartitionOwnerNestedCreateTest): refusal in English and German with the owning type's hub asserted not hosted before and after; top-level create of the same type still works; a nested non-owning in-mesh create lands; the unknown case refuses as unavailable while a static type and a readable non-owning type under the same parent land. Negative controls watched failing: old behaviour, the activating resolver (structural assertion fails: hub hosted), fault folded into "does not own", and the form ignoring the flag.

Write-up: Doc/Architecture/PartitionOwnershipResolution.

Left

• The four TOP-LEVEL checks still resolve through listing + GetMeshNodeStream, so a top-level create of an in-mesh owning type still carries the cold-activation exposure; the durable row would serve them too. Not folded in — it changes a separate, security-adjacent path including the post-creation handler's independent re-establishment.
• i18n mirror: Refuse a nested instance of a partition-owning type declared in mesh content, from the definition's durable row (#4449 item 1) #4589 adds access.partitionCreate.nestedOwningType and re-words access.partitionCreate.undetermined; MeshWeaver.Plugins needs npm run sync:i18n -- --ref <merge sha>.
• Create rollback: the lineage check is read-then-delete, and CreatedDate is caller-supplied #4506 and Create/bulk-create response Error text is English-only (the whole surface, not one message) #4507 are separate and untouched.

🤖 Generated with Claude Code