Mirrors the GitHub Issues tab. Every item here has an issue and vice versa — except the foundational entries at the foot of Done, which predate the tracker. When one ships, check it off here so neither side drifts.
Bugs first — the cost-guard and answer-loss class this tool exists to prevent, then security, then features/optimisation, then tests. Each has a GitHub issue and vice versa.
Epic: encrypted SQLite is the authoritative store for lanes, secrets and (next) metrics — YAML retired (#105) — DONE 2026-10-06
- Split config validation from YAML parsing; route scripts through
rt.fetch_keys(#106) — shipped in PR #113 - Encrypted store:
db init|migrate|doctorandsecrets set|check|list|rotate|rm|rekey(#107) — shipped in PR #114 - Lane config in the store:
lanes,defaults,globals,lane_versions,db export|import,import-yaml(#108) — shipped in PR #123 - Run from the store:
load_configreads the DB,fetch_keysreads secrets, YAML retired (#109) — shipped in PR #124 -
db backup|restoreand the telesto restic job (#110) — shipped in PR #125 - Fleet rollout notes and the Python 3.11 floor (#111) — shipped in PR #126
Epic: run the roundtable from inside Orca — local web UI, no fork (#115) — DONE 2026-10-03
The UI lives in its own private repo, FlatlineRoundtableUI. AGENTS.md § Scope keeps long-lived processes out of this one.
-
--run-idties an--eachrun's transcripts together;--list --jsonfor tools (#116). Merged in PR #117. - UI: server skeleton and read-only run history browser (UI#1)
- UI: start a run, with live per-lane status and cancel (UI#2)
- UI:
--diff,--reviseand--discusson a finished run (UI#3) - UI: Orca launcher (quick command and
orca tab create) (UI#4) - Upstream Orca: let plugin panels reach loopback or their own worker (UI#5). Filed as stablyai/orca#25129.
-
--synthesize: the--diffreaders over a finished run's transcripts (#118). Merged in PR #120. - UI: a run cancelled before any lane answered survives a server restart (UI#6)
-
toolpanel.py: GLaDOS (grok cli) fails tool rounds with "Max turns reached" —cli_with_toolshas no grok branch, so the lane runs its tool-less args (--tools "" --max-turns 3) against a brief that tells it to use tools (#134) — fixed on fix/134-135-toolpanel-grok-stderr:--tools read_file,glob --deny '*__*' --permission-mode default --max-turns 30, verified unable to create a file -
toolpanel.py: TheDixieFlatline (agy) returns EMPTY on tool rounds; stderr is dropped on exit 0, and the lane record'smodeldisagrees with the model its args invoke (#135) — fixed on fix/134-135-toolpanel-grok-stderr: an EMPTY answer carries the stderr tail, the header names the model the args invoke; the EMPTY itself did not reproduce (a 5 KB brief answered with and without--sandbox) - Transcripts clobber each other under
--each -j N, losing paid answers — one-second filename stamp + shared.json.tmpcollide; #58's failure mode via the filename instead of write order (#70) — shipped in PR #88 -
--eachsilently defeats--max-spend— the pre-dispatch budget gate runs after the fan-out returns, so the parent never checks the whole-panel estimate (#71) — fixed on fix/open-issues-batch - Unknown model id estimates to
$0, so--max-spendcannot bind — an http lane with no resolvable price fails open on the guard meant to fail closed (#72) — fixed on fix/open-issues-batch - Transcripts are world-readable (
0755dir /0644file) — full briefs and answers exposed to any local user or tool-bearingclilane (#73) — fixed on fix/open-issues-batch -
--diffis silently dropped under--each— the flag isn't forwarded to children, so synthesis no-ops with exit 0 on the recommended invocation (#74) — fixed on fix/open-issues-batch -
--revise latest:Ncounts files, not lanes — wrong for a--panelround, and misleading after a clobber; docs say "lane count" (#75) — fixed on fix/revise-latest-n-counts-lanes (v1.1.0) -
--revise latest:<non-int>throws a rawValueErrortraceback instead of the clean, actionable error every other bad input gives (#76) — fixed on fix/open-issues-batch - Lane harnesses leak file descriptors —
acp_lane/cli_lanePopen pipes andhttp_lane's retriedHTTPErrorare never closed (28ResourceWarnings, all insideroundtable, none in the stub; found doing #86) (#90) — fixed on fix/lane-fd-leaks (v1.1.1); CI now fails on anyResourceWarning - Env scrub has no test on the
acpharness —child_env()is shared bycliandacp, but onlycliasserts a key stays out; drift reopens #22 (#77) — fixed on fix/open-issues-batch - No per-lane CA bundle — a self-hosted/private-CA endpoint forces a global
TLS bypass that also weakens the OpenRouter lanes
(#78) — fixed on feat/per-lane-ca-bundle (v1.2.0):
ca_bundleand a per-lane-onlyinsecure - Report per-lane
costand run-level spend in--json(and under--diff) — the number cron/CI callers need is computed but not surfaced machine-readably (#79) — fixed on feat/json-spend (v1.3.0); plain--jsonstays a list, see #131 - Plain
--jsonemits an object withresults+spend— a breaking shape change held for 2.0; #79 could not add a top-level key to a bare list (#131) - Discussion mode:
--discusslets lanes share one context and take turns, one lane in flight at a time — the lightweight stand-in for a Buzz-style conversation, no resident runtimes or bus (#93) — shipped in PR #94 -
acplanes: per-laneconfig_optionsapplied viasession/set_config_option(thought level, mode, model) — the agent's CLI flags never reach these; a rejected or unhonoured value fails the lane loudly (#95) — shipped in PR #97 - Tool rounds:
skill/toolpanel.pygiveshttplanes read-only repo tools (read_file/grep/list_dir/git) for grounded reviews; lane settings table and operating rules inskill/SKILL.md(#96) — shipped in PR #97 -
--discuss: a fact first introduced mid-thread propagates unverified — a fabricated "24h expiry" was cited as evidence by the next three turns; both prompts now call an unsourced mid-thread fact a claim, and the skill tells the reader to check it against the code (#98) — shipped in PR #99 - Give a round an identity (
run_id) so--revise latest-runis robust — replaces the fragilelatest:Nfile-count heuristic with a real "round" (#80) — run_id in PR #117,latest-runin PR #122 - Add
--dry-run/--estimate— a priced pre-flight that shows per-lane and panel worst-case cost without dispatching (and surfaces unpriced lanes) (#81) — shipped in PR #133 - Transcripts accumulate forever — add bounded, opt-in retention that never
deletes a
parentof a later round (#82) - TODO: assign mathematical weighting to lanes based on model ability — per-lane weights from each lane's verified track record (CORRECT / PARTLY / WRONG), Beta-smoothed, optionally seeded from benchmark scores. Weights rank findings for verification and annotate the synthesis; they never decide a finding. Motivated by the 2026-09-27 rounds: six lanes agreed on the same wrong "bug" while each real defect came from one lane (#89)
- (feat-req)
--metricsto show lane accuracy metrics — per-lane rounds answered, findings made, verified CORRECT / PARTLY / WRONG counts and rates, lone-claim hit rate, fabrication count, tool-round grounding stats; scoped bylatest:N/ transcript /--lanes,--jsonfor raw numbers, zero network. The read side of the outcome record #89 weights on; never decides a finding (#103) -
synthesize()collects readers serially instead of viaas_completed— free wall-clock win, and removes a serialization trap in the reader loop (#83) — fixed on fix/open-issues-batch -
load_pricingcaches the whole catalog and re-parses it every run — cache the reduced{id: (in, out)}map instead (#84) - No integration tests for
--eachbudget gate, transcript integrity, unknown price, or--diff+--each— the four high-severity behaviours CI doesn't cover (#85) — all four exist:test_each_parallel_writes_one_transcript_per_lane,test_each_gates_the_whole_panel_before_spawning_children,test_budget_refuses_a_lane_it_cannot_price,test_each_with_diff_is_refused_before_anything_runs(stub vendor, no paid calls) - Add Python 3.14 to the CI matrix — the suite already passes on it (verified locally), free signal for no extra dependency (#86) — fixed on fix/open-issues-batch
Epic: encrypted SQLite is the authoritative store for lanes, secrets and (next) metrics — YAML retired (#105)
One encrypted, structured store per host replaces the YAML file: lane config with
an edit history, lane auth secrets (AES-256-GCM, DB key in pass), and the tables
the metrics work (#80, #79, #103, #89, #82, #75) needs a place to write to. Lanes
are edited through subcommands; backups go to telesto. Decisions, definition of
done and risks live on the epic. Children in dependency order:
- Split config validation from YAML parsing; route scripts through
rt.fetch_keys(#106) - Encrypted store:
db init|migrate|doctorandsecrets set|check|list|rotate|rm|rekey(#107) — after #106 - Lane config in the store:
lanes,defaults,globals,lane_versions,db export|import,import-yaml(#108) — after #107 - Run from the store:
load_configreads the DB,fetch_keysreads secrets, YAML retired (#109) — after #108 -
db backup|restoreand the telesto restic job (#110) — after #107 - Fleet rollout notes and the Python 3.11 floor (#111) — after #109
Run before any PR. Several of these are behavioural and were once human-only;
the ones now covered by an automated test name the test, so a green make-style
suite stands in for them. The rest still need a person, because they assert on
the real environment (process table, ps, git, install.sh) rather than a
stub.
-
./roundtable --list— correct route per lane, zero network calls. Automated:test_list_makes_no_network_calls, plus a dedicated CI step that points the process at a dead proxy. - Full run — every active lane answers, exit code
0. Partial delivery must exit non-zero; a silent lane is the bug this tool exists to eliminate. Automated:test_all_answered_exits_zero,test_a_silent_lane_fails_the_run. - Env scrub: run with an invalid
ANTHROPIC_API_KEYexported and confirm aclilane still succeeds via OAuth. If it fails on the bad key, the scrub is broken and that lane has silently moved onto metered billing. Automated:test_ambient_api_key_never_reaches_the_child,test_bedrock_and_vertex_routes_are_scrubbed. Re-check by hand when a new vendor CLI is added — the patterns are broad, but a novel billing route is exactly what they would miss. - Orphan kill: point a
clilane at a hanging command with a short timeout; confirm it fails cleanly andpgrepfinds no survivor. Automated:test_hung_lane_is_killed_and_leaves_no_orphan. - Missing key: point a lane at a secret the store does not hold; confirm
it fails loudly naming the secret, and never calls unauthenticated.
Automated:
test_missing_secret_aborts_the_run,test_no_db_key_and_no_pass_aborts_the_run,test_a_store_secret_reaches_the_authorization_header. - No key in argv: during a run,
ps auxwwshows no key. (human — reads the live process table; keys are held in memory and set as headers only.) - No key in output: transcript and
--jsoncontain noAuthorizationvalue. Partly automated:test_fixtures_carry_no_credentialsscans the golden fixtures; confirm by hand against a real transcript after a run. -
git check-ignore -v roundtable.db roundtable.db-walconfirms a store cannot be committed;git status --porcelainis clean after a run. (human) - Store secrets never in plaintext:
secrets set X --stdin, then the.db,-waland-shmbytes do not contain the value; a positional value is refused; a different DB key dies on the key-check value. Automated:test_plaintext_is_absent_from_db_wal_and_shm,test_a_positional_value_is_refused,test_a_different_key_fails_on_the_key_check_value. On a real host, alsops auxwwduringdb initshows no key (it goes topass inserton stdin). -
./install.shthen./install.sh --uninstallround-trips, and refuses to delete anything that is not a symlink. (human — touches$HOME, no test) -
./install.sh --initwith no store creates one (db doctorpasses), and with a store present leaves it byte-identical. (CI covers both in a throwaway HOME) - Restore drill:
restic restore latestfrom telesto →db restoreinto a scratch--config→db doctor→secrets check→--list, and the scratch store'sdb exportequals the live one. (human — needs telesto and the real DB key; the commands are in the README.) Last run 2026-10-06 on makemake: all steps ok, 9/9 secrets present, export identical. Automated parts:TestBackup.
-
acpharness: a pool-acp lane timed out waiting forsession/promptwhile poolside streamed a full answer — generated, billed, never delivered. The client now answers agent-initiated requests (reject option, notcancelled) and salvages whatever streamed before the deadline, marked truncated (#56) — shipped in PR #61; the reject-vs-cancel refinement followed in PR #63 (#62) - Docs claimed lanes have no tools;
clilanes run the vendor CLI with its full toolset (HAL9000 wrote and deleted a probe test inside a reviewed checkout, 2026-09-05). Tool access is now documented as a property of the harness, not the panel, with revoke-on-cheating and pre-round hygiene (#66) — shipped in PR #67 - Optional second round:
--revise latest:Nreplays a finished run so each lane sees the locked answers (own marked YOURS, peers anonymised) and opens with HOLD or REVISE; report marks the round non-independent (#64) — shipped in PR #65 - A run that dies mid-fan-out lost every answer it already collected
(#58) —
list(pool.map(...))held all answers in memory and wrote the transcript only after the last lane returned, so any interruption discarded completed lanes, paidhttpones included. Now persisted after every lane viaas_completed, atomically, marked"partial": trueuntil the run finishes. - Corrected #46: an empty answer with
finish_reason: lengthis variance, not determinism — reasoning usage ranged 20–21,606 tokens across runs of one identical brief. The retry is restored; the message namingmax_tokensis kept for when retries are exhausted. (#53) -
THE READERS DISAGREEfired when readers merely differed in how liberally they named lanes, not in judgement — a conflict now requires disjoint placements rather than unequal ones. Found on a live run whose own reader reported "no genuine disagreement" while every lane was flagged. (#51) - An empty answer with
finish_reason: lengthis a doomed retry, not a transient one — the model spent its whole budget on reasoning and will do it again. Fails once with an actionable message instead of billingretriesmore certain failures. (#46) -
classify_mentionsanchors sections on the first occurrence anywhere in the text, so a reader restating its instructions collapses every lane into one bucket and manufactures phantomTHE READERS DISAGREErows. Lane matching is unbounded substring. Regression from #16. (#18) -
--diffsynthesis runs outside the budget, semaphores, pacer and deadline. It is the largest request of the run and, since #16, is sent twice. (#19) - A
clilane with neither{prompt}norstdin: truesends no prompt at all, and validation accepts it while its error text promises otherwise. A lane can vote without seeing the question. (#20) - The per-vendor semaphore takes the first lane's
concurrencyrather than the smallest;build_pacersgets the same problem right. (#21) -
SCRUB_ENVcovers API keys but notCLAUDE_CODE_USE_BEDROCK/_VERTEXorANTHROPIC_BASE_URL, which bill a subscription lane with no key involved. (#22) - Nothing detects that two lanes resolved to the same underlying model.
served_byis captured and never used, so an echo can be reported as convergence — the one claim the tool exists to make. (#23) -
acp_lanepasses the realcwdto a coding agent, never drains its stderr, and never reaps the killed process. (#24) -
deadline_secondsbounds nothing: both checks run before the semaphore acquire, so they pass at t=0 and never apply again. (#25) - A config whose lanes are all
active: falsereports0/0 answeredand exits 0, claiming success having asked nobody anything. (#26) - The pre-flight estimate assumes one call per lane, but retries can bill the
prompt up to
(1 + retries)times. (#27) -
cli_lanefilters a hardcoded clock emoji — one machine's shell-hook banner baked into a public tool, silently deleting answer lines. (#28) - A lane named
Off,No,YesorOnis coerced to a YAML 1.1 boolean and fails with a misleading "has no name". (#29) -
Retry-After: 1.5fails.isdigit()and is ignored in favour of the default backoff. (#30) - The stub server in the test suite is hand-written, so its response shape
can drift from what vendors actually return without CI noticing. Add
golden fixtures captured from real responses, and optionally a
workflow_dispatch-only live smoke job. A heavier emulator (LocalAI, Ollama) was considered and rejected: they serve correct responses, while the value here is in the malformed ones. (#13) -
--diffuses two independent readers from different vendors and reports where the readings disagree about a lane's classification, so the synthesizer's own bias is visible rather than invisible. The comparison is mechanical — a third model judging the first two would only move the problem. (#8) - Pre-flight estimate calibrates itself from observed
usage.prompt_tokensinstead of assuming 4 chars/token, falling back to the constant until a model has enough samples. Note the original premise was wrong: 4 chars/token under-estimates a markdown-heavy personality (gpt-oss-120bmeasures 3.30), which is the unsafe direction for a budget check, not a wide-but-safe margin. (#9) - Per-vendor rate-limit awareness — optional
rpmper lane, shared across a vendor group and taking the smallest value, spacing request starts so a per-minute cap is not tripped before any 429 arrives.concurrencybounds how many run at once, which is a different quantity. (#7) - Free-lane retry resilience — a 200 carrying an error body and an empty
answer are both retried instead of ending the lane, and
Retry-Afteris honoured to 90s rather than capped at 30s below the 60s window vendors actually advertise. (#11) - Wrap protocol-over-stdio agents so they can serve as lanes — new
acpharness: initialize → session/new → session/prompt → process-group kill, one turn only. (#2) - Real pricing pulled from the gateway and cached for a day; prompt and completion charged at their separate rates. (#3)
-
--max-spend/budget_usdenforced before dispatch against a worst-case estimate, so an overrun is prevented, not reported. (#4) - Test suite — stub HTTP server and fake CLI binaries, no vendor contacted and nothing spent. Grown from 25 to 134 tests as each defect below landed. (#5)
-
--diffmode reporting AGREED / SPLIT / LONE CLAIMS. (#6) - Direct-call architecture replacing long-lived agent processes — no identity, no message bus, no publish step, nothing left running to bill.
- Secrets via
passentry names only; keys resolved once, held in memory, never inargv, transcripts, or--json. - Env scrub so subscription-backed CLI lanes cannot be silently converted to metered billing by an ambient API key.
- Process-group kill on CLI timeout.
- Per-vendor concurrency caps for single-slot local servers.
- Transcript written per run.
- Truncation surfaced via
finish_reasonrather than passing a clipped answer off as complete. -
extra_bodydenylist so a config typo cannot rewrite the request.
Proudly Made in Nebraska. Go Big Red! 🌽 https://xkcd.com/2347/