Camera-ready: uncertainty, chunk-size ablation, repeated runs, and the multi-agent rerun - #2
Merged
Conversation
…result depends on Reviewers asked for intervals, repeated runs and a versioned release. The per-task verdicts already in docs/exports/ answer most of that without a rerun: Wilson intervals, exact McNemar over the paired tasks, and Holm across the family. The AST-versus-window question is answered on the index itself, by sweeping window size and measuring how whole and how pure each unit is, so chunk size is ruled out without running a model. Two defects surfaced while standing the stack up: /api/health reported retrieval ready while every S2 run was refused, because the database probe overwrote the model server's provisioning state; and both script bind mounts resolve to nothing under WSL with Docker Desktop, which is now documented. Agent CLIs and JDT.LS were unpinned, so two builds a week apart drove different agents.
…us anything A pass rate over one run cannot say whether an agent would answer the same way twice, which is what the reviewers asked. The runner launches a configuration k times over the same tasks and keeps every per-task verdict; the report scores them and, more usefully, names the individual tasks whose verdict changed between identical runs. Tasks are selected rather than sampled: one that always passes or always fails carries no information about run-to-run variation, so the interesting set is the tasks an earlier campaign found unstable or impossible. Batches are small and checkpointed because a long campaign is interrupted, and a repeat is not a batch -- they are pooled back before anything is scored.
The three analysis scripts opened by citing review feedback, which dates them to a process rather than explaining the tool. The technical reason stands on its own and is what a reader needs. Two docstrings also pointed at scripts that do not exist.
…run happened Chunk boundaries are decided by the parsers as much as by the cutting rules, but the chunker version was a hand-written string. Because that string is part of the index identity, upgrading tree-sitter changed how Java was cut while the identity still claimed the cached index was equivalent; the pins in the image were the only thing between a result and a silently different one. The version now carries the interpreter and parser versions it was produced with. A hosted model identifier is also not a version, since a provider can re-point a name at new weights without renaming it. The date is what pins which checkpoint answered, so the runs that carry one now say so, and the ones that do not are named as such rather than left to look precise.
Two datasets the analysis scripts read. The repeat probe covers six execution regimes at k=3 over tasks an earlier campaign found unstable, and answers whether repeating a run tells us anything: it does for retrieval, where pass@3 is roughly double pass@1, and does not for the baseline or for sub-agent delegation, which returned identical verdicts every time. The multi-agent run replaces a truncated one. Native sub-agents over the full hundred descriptive tasks score 66, against 86 for AST retrieval on the same tasks, and the platform records that in nineteen of those tasks the agent never delegated at all -- so the rate over tasks that actually exercised the regime, 70, is reported alongside it.
Two tags pointed at the same commit and one of them had to be explained in terms of the other, which is a sign there was only ever one thing to name.
The sub-agent numbers in the report still came from the run that stopped after twenty-six tasks, because the full-length rerun was not wired into the analysis -- it sat in the exports and was quoted by hand. Now it is loaded like every other configuration and passes through the same paired test and correction. The two exports name tasks differently: the campaign records repository and prompt mode alongside the task, the archive exports carry the bare id. They are the same benchmark tasks, so the qualifiers are stripped to pair them, and a comparison whose two sides share no tasks now stops with that reason instead of dividing by zero. Conformance is per run rather than per task in this export, so the rate over tasks that actually delegated stays quoted from the run records rather than recomputed here.
Adds a static academic project page under site/, deployed to GitHub Pages via a new Actions workflow, with the paper's abstract, authors, BibTeX and key RefactorBench/SWE-Refactor results. Updates the README with the EMNLP 2026 System Demonstrations acceptance notice and a Contributors section, and adds a back-link from docs/README.md.
Adding landing page and announce the paper's acceptance
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Camera-ready work for the EMNLP 2026 demo paper, on top of
refactor-demo. Fivecommits, no changes to existing behaviour beyond two bug fixes, each with a
regression test.
What reviewers asked for
Numbers without uncertainty.
scripts/uncertainty.pycomputes Wilsonintervals for every reported rate and tests the paired comparisons the paper
makes — McNemar's exact test on the per-task verdicts, a paired bootstrap on the
difference, and Holm correction across the family.
scripts/test_uncertainty.pychecks it against hand-computed values. The results are in
docs/exports/uncertainty.json.The headline comparisons survive correction: AST chunking over line windows is
worth +25 to +30 points at p < 1e-5 on all three prompt modes, and retrieval over
line windows scores below no retrieval at all, −16 points at p = 0.007.
Is the chunking result just chunk size? No, and it is now separable.
scripts/chunking_ablation.pysweeps the window from 20 to 240 lines over ninerepositories and measures two things per setting: how often a window contains a
whole definition, and how much unrelated code it drags in. Line windows never
reach AST chunking at any size — small windows split definitions, large ones
dilute them. It needs no model, so anyone can rerun it. The figure is in
docs/results.md.Does repeating a run change the conclusion? Depends entirely on the regime,
which is itself a finding.
scripts/repeat_campaign.pyruns a configuration ktimes, resumably;
scripts/repeat_stats.pyreports mean, a Student-t interval,pass@k and how many tasks flipped verdict between otherwise identical runs. Over
six regimes at k=3: AST retrieval goes from 27.8 % pass@1 to 50.0 % pass@3, while
the plain baseline and sub-agent delegation returned identical verdicts every
time and flipped nothing.
The S2/S3 mismatch
An earlier multi-agent run had been truncated. Rerun over the full 100
descriptive tasks with native sub-agents: 66/100 against 86/100 for AST
retrieval on the same tasks, a 20-point gap at p < 0.00001 with no task where
sub-agents won and retrieval lost.
The platform's own compliance record explains part of it: in 19 of those tasks the
agent never delegated at all. Over the 81 tasks that did exercise the regime the
rate is 70 %. Both numbers are reported — the second is the one that describes
sub-agent delegation, the first is what you get when you ask for it and don't
check.
Two bugs, both found by the above
server/app/retrieval/health.py— a model server still pulling its weights wasreported as healthy, because the database probe's result overwrote the
provisioning state. S2 runs were accepted and then failed.
server/app/retrieval/chunking.py— the chunker version was a hardcodedstring, so upgrading the tree-sitter grammar silently reused indexes built by
the old parser. It is now derived from the installed versions and forms part of
the index identity.
Both covered in
server/tests/test_retrieval_s2.py.Reproducibility
docker/backend.Dockerfilepins the agent CLIs and the JDT.LS snapshot, whichpreviously floated to whatever was newest at build time — a rebuild could not
reproduce a run.
CITATION.cffanddocs/release.mdrecord how a release istagged and what a result depends on.
Not merging this myself — flagging it for review.