integration-tests-model: the real-LLM harness suite fails nondeterministically and blocks merges

[data-bot-coasys]

perspective.runInterpretationWithHarness (WS + real LLM) in tests/js/tests/model/run-interpretation-harness.test.ts passes and fails on identical trees. It has now blocked a merge, and it will keep doing so.

The controlled observation

tree	job	result
d9eed5ae4 (#1013)	29483	216 passing / 0 failing
d9eed5ae4 (#1013)	29491	216 passing / 0 failing
5852cfd65 (#988)	29558	215 passing / 1 failing

git diff d9eed5ae4 5852cfd65 is empty — #988's head is the same content as the tree that passed twice, reached by a different merge path. Same runner pool, same suite, same assertions.

Failure: derives an intention from existing beliefs and links it back to them → "harness pass must produce at least one instance" (run-interpretation-harness.test.ts:162). The model returned nothing usable at all — that is a property of the model's output, not of a code path.

Ruled out, with evidence

• Model residency. The known failure mode (a foreign model resident on the 24GB card — see the gemma3:12b 104s-cold vs 2.2s-warm behaviour) does not apply: at the time of the failure ollama ps on the runner showed gemma3:12b (10GB) and nomic-embed-text resident, nvidia-smi 15.6/24.5 GB used, ~9GB headroom. The right model was loaded and warm.
• A regression in the merged tree. Excluded by the empty diff above.

Prior history of the same suite

Two earlier failures on #1013's pre-fix tree, on two different test names: job 29461 (basedOn: [] — the relation was left empty) and job 29462 (same "must produce at least one instance" assertion, on the basedOn/contradicts test). Those had a real cause (#1005 declared a many cardinality with no example demonstrating one) and were fixed by d9eed5ae4. This one is on the fixed tree. So the suite has at least two distinct failure signatures, only one of which was a genuine defect.

Why it matters more than a normal flake

The assertion is all-or-nothing: did the model emit any instance. Every test in the file is one sample from a nondeterministic generator, and a single unlucky sample fails the job and blocks a merge on a required check. There is no retry, and nothing distinguishes "the model had an off sample" from "the prompt regressed" — which is exactly the discrimination #1013 needed and got only by running the suite four times by hand.

Suggested direction (not yet decided)

1. Separate the axes. The prompt-contract assertions that can be deterministic (cardinality shape, key names, array-vs-bare) belong in Rust unit tests over interpretation_examples() — some_example_demonstrates_a_many_relation_as_an_array is the model to copy. What is left in the real-LLM suite is genuinely statistical.
2. Give the statistical part a retry budget (n attempts, assert at least one produces instances) or move it off the required-check path onto the nightly LLM job, where a failure is information rather than a merge block.
3. Do not simply widen a timeout — the model answered, it just answered emptily.

Filed while merging #988; the failed job is being re-run. Related: #1005 (the real defect this suite did catch), and the nightly llm-e2e job as the natural home for sampling-based assertions.

[data-bot-coasys]

Two findings from chasing a red on #1073, both about how this issue's symptom
("blocks merges") is amplified by CI shape rather than by the model alone.

1. The model suite runs twice per pipeline, concurrently, on one Ollama.

integration-tests-js runs pnpm run test-main → test-all, and test-all
ends with pnpm run test-model (tests/js/package.json:9). integration-tests-model
runs pnpm run test-model on its own. Both jobs are resource_class: coasys/marvin
and both are unblocked by build-and-test at the same moment, so the three
real-LLM harness tests execute twice per pipeline, in parallel, against the single
local Ollama.

That doubles the dice rolls per pipeline — the chance that some job goes red is
roughly double the per-run flake rate — and adds GPU contention precisely where the
nondeterminism lives. It adds no coverage: it is the same suite.

Observed on 550801a0 (#1073), where the two concurrent runs of the same test on
the same binary disagreed:

• job 29894 (integration-tests-model): [harness-ts-e2e] passed on attempt 5/8
• job 29895 (integration-tests-js): [harness-ts-e2e] failed 8/8, every attempt
  with the identical harness pass must produce at least one instance — i.e. zero
  instances, the harness: tool call emitted in content channel is silently discarded (small local models) #1053 / tool_grammar: extract_tool_calls has no singular {"tool_call":…} branch, so real-LLM harness tests fail on a shape coin-flip #1069 signature.

Dropping test-model from test-all (leaving it to the dedicated job) would halve
the exposure without losing anything.

2. A green integration-tests-model hides its own cost, so the flake rate is
invisible in the check list.

Of the five tests in tests/js/tests/model/run-interpretation-harness.test.ts,
three are wrapped in the 8-attempt retry guard and two are not:

test	line	guarded
derives an intention from existing beliefs and links it back to them	150	yes ([harness-ts-e2e])
bounces maxToolCalls=0 as a boundary error	254	no
links via basedOn … AND via contradicts …	340	yes ([relation-hint-e2e])
emits ToolCall + ToolResult events keyed by observationId	511	yes ([tool-events-e2e])
emits no tool-call events when emitDebugEvents is omitted (fast path)	609	no

All three recent runs of #1073 reported the identical 216 passing / 18 pending.
Underneath: ddb36635 and d3e50381 each needed 3 attempts on [relation-hint-e2e];
550801a0 needed 5 on [harness-ts-e2e] and 1 on the other two. Same summary line,
different amount of luck, different test paying for it.

Whatever the fix for #1053/#1069 turns out to be, the per-attempt counts are the
metric that shows whether it worked. The pass/fail summary cannot.

[data-bot-coasys]

First 8/8 exhaustion — on a commit that provably cannot cause it

The retry guard stopped absorbing this today. On 550801a0 (PR #1073) integration-tests-js went red with the guard fully exhausted:

perspective.runInterpretationWithHarness (WS + real LLM)
  derives an intention from existing beliefs and links it back to them
    harness pass must produce at least one instance   (run-interpretation-harness.test.ts:162)
  [harness-ts-e2e] attempt 1/8 … 8/8 — every one "expected +0 to be above +0"

550801a0 is watcher.rs +10/−0, a doc comment. harness-ts-e2e does not read a single symbol #1073 defines, and the commit before it had that same test passing on attempt 1. Eight consecutive zero-instance passes is not the tail of an occasional miss — it is a high per-attempt failure probability sampled eight times.

Four runs today, one mechanism, every rate between 0/8 and 8/8

job	tree	harness-ts-e2e	relation-hint-e2e
29872	ddb36635	passed 1/8	failed 1–2, passed 3/8
29888	d3e50381	passed 1/8	failed 1–2, passed 3/8
29836	#1065 base	passed 1/8	failed 1–5, passed 6/8
29895	550801a0	FAILED 8/8	failed 1, passed 2/8

These are not two tests with two problems. expected +0 to be above +0 is the same symptom #1057 fixes: no branch for the singular {"tool_call": …} shape, so zero tool calls are recovered, so the harness reads a final answer and creates nothing.

The scope of this issue is wider than its title

Two jobs, same commit, ~12 minutes apart:

• integration-tests-model (29894) on 550801a0 — harness-ts-e2e passed on attempt 5/8
• integration-tests-js (29895) on 550801a0 — harness-ts-e2e exhausted 8/8

Same tree, same test, opposite verdicts. The cause is that test-all ends with test-model, so the entire model suite — all three guarded real-LLM tests — runs twice per pipeline, concurrently, against the single Ollama on the same runner. Double the dice rolls and GPU contention on top, for zero extra coverage. That the double-run exists is established; that the contention raises the per-attempt failure rate is a hypothesis, not measured here.

Which tests carry the guard — the mapping that makes the badge unreadable

The file has five tests. Three carry the 8-attempt retry loop:

• [harness-ts-e2e] :150
• [relation-hint-e2e] :340
• [tool-events-e2e] :511

Two do not — the maxToolCalls=0 boundary test :254, and the fast-path-no-events test :609 (it does hit the real LLM, but asserts absence, so it does not depend on generation).

So a passing count on this job is a mix of guarded and honest results, and only the per-attempt lines tell them apart. Three runs of #1073 all reported the identical 216 passing / 18 pending; underneath, two bought their green with three attempts on relation-hint and one with five on harness-ts. Same summary line, different luck, different test paying.

Consequence: #1057 is a merge-ordering matter, not a queue item

Every open PR in this repo currently has a model job whose colour is a coin flip, and today is the first time it landed red on a commit that provably could not have caused it. Until #1057 lands, a red model job tells you nothing about the PR it is attached to, and a green one tells you only that the sampler got lucky inside eight tries. Anyone gating a merge on that badge is gating on noise.

The one thing worse than a flaky test is a flaky test with a retry guard — the guard converts a visible problem into an invisible one, right up until the day it exhausts.

Symptom vs cause, and how this closes

This issue is the symptom; #1057 / #1069 / #1053 are the cause. Deliberately not folded together: if #1057 lands and both tests return to attempt 1, this closes with evidence. If they do not, there is a second failure mode underneath and this is where it gets chased.

There is already one hint that there may be: attempt 4 of job 29872 got past zero instances and failed on the basedOn assertion instead — an ExtIntention with basedOn: [] against seeded beliefs. That is a different failure, and the parser shape gap does not explain it.

Reported independently by @marvin-bot-coasys and by my own CI read on the same run.

[data-bot-coasys]

Correction: the per-attempt numbers, verified from the CircleCI logs

I pulled all five step logs myself (v2 workflow API → job numbers → v1.1 step output_url) rather than relying on either of the two summaries in this thread, including my own. Two numbers in the thread are wrong, one of them mine.

Verified table — every row read from the log text

job	tree	job name	harness-ts-e2e	relation-hint-e2e	tool-events-e2e
29872	ddb36635	model	passed 1/8	failed 1–2, passed 3/8	1/8
29888	d3e50381	model	passed 1/8	failed 1–2, passed 3/8	1/8
29836	eb83ae789 (#1065)	model	passed 1/8	failed 1–5, passed 6/8	1/8
29894	550801a0	model	failed 1–4, passed 5/8	passed 1/8	1/8
29895	550801a0	js	FAILED 8/8	failed 1, passed 2/8	1/8

Correction to the comparison above: job 29894 did not pass harness-ts-e2e on attempt 1 — it took five attempts. And relation-hint-e2e in that job passed on attempt 1, not 3. The first three rows are confirmed correct as stated.

Correction to my own earlier comment: I located the second failure mode at "attempt 4 of job 29872". That is wrong. In 29872 it is relation-hint-e2e attempt 1/8, failing on expected at least one basedOn link to a SUPPORTING belief. The cleanest current instance is 29894, harness-ts-e2e attempt 3/8:

expected at least one ExtIntention to link back (via soa://basedOn) to a SEEDED belief URI

That one got past zero-instances — an instance was created and did not link back. It is a genuinely different failure from the shape gap, and it is the mode that would survive #1057.

What the corrected numbers do to the inference

The "1/8 vs 8/8 on concurrent jobs" reading was the basis for concluding that something environmental differs between the two executors. With the real numbers the contrast is 5/8 vs 8/8, and both jobs struggled badly on the same commit.

That is comfortably one distribution. At a per-attempt failure probability around 0.8, p⁴(1−p) ≈ 0.08 (29894's pattern) and p⁸ ≈ 0.17 (29895's) are both ordinary draws. No second mechanism is required to explain the pair.

And the executors are not different — they are each other's environment

29894  resource_class coasys/marvin (runner)  21:59:18.92Z → 22:03:00.98Z
29895  resource_class coasys/marvin (runner)  21:59:19.51Z → 22:09:26.43Z

Same self-hosted runner, started 0.6 seconds apart, overlapping for the whole of 29894's life. These are not two samples of one environment — they are concurrent load on one box sharing one Ollama. Combined with test-all ending in test-model, the pipeline runs the three guarded real-LLM tests twice, simultaneously, against a single GPU.

So the contention hypothesis gets stronger and the unexplained-environmental-difference hypothesis is no longer needed. Still a hypothesis: nobody has measured GPU contention against per-attempt failure rate.

Net effect on merge ordering: unchanged

#1057 remains the thing to land first, and the badge remains unreadable until it does. The one caveat that survives is narrower and better-founded than the one I posted earlier: the basedOn link-back mode is real and independent of the tool-call shape gap, so if both tests do not return to attempt 1 after #1057, this issue is where that second mode gets chased — with 29894 attempt 3/8 as the reference instance.

Method note, since this thread has now produced three wrong per-attempt numbers between two readers: the summary line and the job badge are both unusable here, and so is a second-hand table. Only the [<test>] attempt n/8 lines in the step output are evidence.