Repository navigation
feat(core): add experimental background compaction summaries - #57444
Conversation
Behind experimental.precompute_compaction (off by default). Once the context passes 85% of the compaction ceiling, the prefix the provider just processed is summarized in the background and held; automatic or manual compaction swaps it in with the messages since kept verbatim via keepID, and falls back to a normal summary when the tail outgrew 30k tokens or the conversation, model, or instructions changed. Based on #56845. Co-authored-by: Kit Langton <kit.langton@gmail.com>
8c1fb7f to
88063f0
Compare
![opencode-agent[bot]](https://avatars.githubusercontent.com/in/1549082?s=60&v=4){kind=small}
There was a problem hiding this comment.
The approach looks sound. Checking the held summary when it is swapped in, rather than tracking invalidation events, keeps this small, and the flag-off path is unchanged apart from the extra DB read noted inline. The new tests are focused and readable. I ran the compaction, config and runner suites and the packages/core typecheck, and all pass. I couldn't exercise it end to end: it needs a provider-measured prompt of 64k+ tokens, which the simulated model doesn't produce.
Public API: adds the optional config key experimental.precompute_compaction (boolean, default off) to the config schema, OpenAPI specs and generated client types. This is backwards compatible.
Two inline notes: a held summary is never replaced once it can't be used, and the end-of-turn hook does a DB read on every turn even with the flag off.
There was a problem hiding this comment.
This push is a rebase onto current dev. The change itself is unchanged apart from context, and the compaction, config and runner tests and the packages/core typecheck still pass. Neither earlier point (refreshing a held summary that can no longer be used, and the end-of-turn DB read with the flag off) is addressed yet, so those threads stay open. Nothing new to raise.
Public API: still only adds the optional config key experimental.precompute_compaction (default off), which is backwards compatible.
There was a problem hiding this comment.
Both earlier points are addressed. A held summary that failed or fell behind is now replaced from a newer response, with no repeat for the same response and a cap of two failures. precompute now loads the context only when the setting is on. Publishing the automatic compaction's start before the wait looks right, and it is still published once. The compaction tests (also with --rerun-each 5), the runner and config tests, and the packages/core typecheck pass. I couldn't run it end to end, since it needs a real provider-measured prompt of 64k+ tokens.
Public API: unchanged since the last review. The only addition is still the optional experimental.precompute_compaction config key (default off), which is backwards compatible.
One small test note inline.
| beforeResponse = Deferred.succeed(reached, undefined).pipe(Effect.andThen(Effect.die("provider down"))) | ||
| yield* compaction.precompute(Effect.succeed(loaded(session, messages))) | ||
| yield* Deferred.await(reached) | ||
| yield* Effect.yieldNow |
There was a problem hiding this comment.
This assumes the background fiber has failed after a single yieldNow once the response is reached. If the failure takes another scheduler turn to land, pollUnsafe() in precompute still sees it as running, and the next assertions change. Awaiting the held fiber's exit (or a deferred completed when it fails) would make this deterministic, rather than depending on scheduling.
1 participant
Issue for this PR
Related: #56845 (this is a rework of it on top of #57091 and the
keepIDcheckpoint changes)Type of change
What does this PR do?
Automatic compaction blocks the next step while the summary is written. #57091 made that request almost fully cached, so it's cheap now, but it's still slow: most of the pause is the model generating the summary, which caching doesn't help.
Behind
experimental.precompute_compaction(off by default):/compact), compaction swaps it in without a model call: the summary pluskeepIDpointing at the first message after it, so everything since stays verbatim. If the summary is still being written, compaction waits for it.max(2 × keep, 30k)tokens arrived since (capped at 25% of the budget), or the conversation start, model or instructions changed, the held summary is dropped and compaction summarizes as usual. This avoids the stale-summary failure Claude Code hit (Auto-compaction: two compounding cost bugs — stale precompute keeps ~200k tokens verbatim, and that prefix is repeatedly cache-created instead of cache-read anthropics/claude-code#70459).Compared to #56845 this drops the tail-folding pass, the cache-policy tweaks and the event-based invalidation. Validity is checked when the summary is swapped in instead.
Prior art: Grok Build does a background first pass at ~75% plus a short blocking second pass at 85% (on by default). Claude Code has "precomputed compact" behind a flag. Codex has
async_compaction(off by default, 65%).Known gaps: plugin
session.compactionhooks also see background summaries that may be discarded, and held summaries are in memory only (lost on restart).How did you verify your code works?
bun run checkpasses.packages/core/test/session-compaction.test.ts(off by default, swap with no extra call, waiting on an in-flight summary, fallback on a long tail or a model switch), plus the existing compaction and runner suites. The compaction file passed with--rerun-each 10.Screenshots / recordings
N/A
Checklist
Added by the opencode bot (AI-generated).
Public API: adds optional config key
experimental.precompute_compaction(default off; backwards compatible)