local-plugin v2.0.8: gateway CPU 100% — synchronous full-table vector scan (scanAndTopK) + unbounded trace re-insertion from dedup pagination cap

[tianxin8206]

Summary

Running the OpenClaw local memory plugin (@memtensor/memos-local-plugin v2.0.8), the gateway node process pinned one CPU core at 100% for 40+ minutes on startup, holding 4.2 GB RSS. Investigation traced this to two distinct bugs in the plugin:

1. Vector retrieval does a synchronous O(n) full-table scan on the Node main thread, decoding every vector BLOB and computing cosine in-process — this starves the event loop (100% CPU, GC/IO threads idle).
2. A pagination cap in the capture-side dedup read causes the same trace to be re-inserted indefinitely, bloating the traces table (in our case 518,375 rows of which 437,792 — 84% — were exact duplicates), which is what made bug feat: add internet retrieval and CoT functionality to MemOS #1 catastrophic.

Together: bug #2 grows the table without bound → bug #1 loads the whole thing synchronously on every dreaming/retrieval cycle → CPU melts.

Environment

• @memtensor/memos-local-plugin v2.0.8 (MemOS Local Memory (V7))
• Node 26, better-sqlite3
• dreaming.enabled: true, embedding.enabled: false
• memos.db had grown to 6.8 GB (traces table alone = 5.96 GB, ~4.2 GB of that in vec_summary/vec_action BLOBs)

Evidence — native stack of the 100% CPU main thread (gdb)

#0  pread64
#1  unixRead                         (better_sqlite3.node)
#2  accessPayload                    (better_sqlite3.node)
#3  vdbeColumnFromOverflow           (better_sqlite3.node)
#4  sqlite3VdbeExec
#5  sqlite3_step
#6  Statement::JS_all(...)           <-- synchronous .all()
#7  Builtins_CallApiCallbackOptimizedNoProfiling  (libnode)

strace -c over 3s on the main thread: 56,637 pread64 calls, 99.97% of time — i.e. the thread is doing nothing but synchronously reading BLOB overflow pages off a multi-GB table.

Bug #1 — scanAndTopK blocks the event loop

dist/core/storage/vector.js:149

export function scanAndTopK(db, table, selectExtra, query, k, opts) {
    const { vecColumn, norm2Column, where, params, hardCap } = opts;
    const cols = ["id", vecColumn, ...(norm2Column ? [norm2Column] : []), ...selectExtra];
    const sql = [
        `SELECT ${cols.join(", ")} FROM ${table}`,
        where ? `WHERE ${where}` : "",
        `LIMIT ${hardCap ?? 100000}`,          // default cap = 100,000 rows
    ].filter(Boolean).join(" ");
    const rows = db.prepare(sql).all(params);  // <-- loads up to 100k rows w/ vector BLOBs synchronously
    const decoded = [];
    for (const r of rows) {
        const vec = decodeVector(r[vecColumn]); // <-- decode every BLOB on main thread
        ...
    }
    return topKCosine(query, decoded, k);       // <-- cosine over all of them on main thread
}

Called from dist/core/storage/repos/traces.js:255 (searchByVector).

Problems:

• No ANN/vector index — retrieval is a brute-force O(n) scan over the entire table.
• .all() materializes up to hardCap ?? 100000 rows, each carrying multi-KB vector BLOBs, into JS memory in one synchronous call (hence 4.2 GB RSS and the JS_all stack above).
• Decode + cosine also run synchronously on the main thread.

With a table in the hundreds-of-thousands of rows, every dreaming/retrieval cycle blocks the event loop long enough to peg a core and make the gateway unresponsive.

Suggested fixes: add a real vector index (e.g. sqlite-vec), or at minimum (a) stream/iterate instead of .all(), (b) do decode+cosine off the main thread (worker), (c) pre-filter candidates by cheap SQL predicates before loading BLOBs, (d) make the default hardCap much smaller and explicit.

Bug #2 — dedup pagination cap → unbounded re-insertion

Documented in a code comment at dist/core/storage/repos/traces.js (listAllForEpisode):

/**
 * Full episode-scoped trace fetch with NO pagination cap. The paginated
 * `list({ episodeId })` path silently truncates to 500 rows (default 50),
 * which breaks capture-side dedup when an episode has more than the cap
 * worth of steps — the next runLite/runReflect re-inserts everything past
 * the cap as "novel". Use this for any dedup / reconciliation read.
 */

If any code path still feeds dedup from the paginated list({ episodeId }) (cap 500) instead of listAllForEpisode, then for any episode with >500 steps the dedup check cannot see the older traces, so runLite/runReflect treats them as novel and re-inserts them every cycle.

Observed in our DB — the damage is exactly this shape:

• traces: 518,375 rows, but only 80,583 distinct by (episode_id, turn_id, user_text, agent_text, tool_calls_json) → 437,792 exact duplicates (84%).
• Duplicates concentrated in a few large episodes, e.g. one episode had 49,402 rows but only 153 distinct turn_ids / 588 distinct contents — i.e. the same turns re-inserted hundreds of times.
• Duplicates span the full lifetime of the DB (2026-05-19 → 2026-07-08), consistent with a per-cycle re-insertion loop rather than a one-off.

Suggested fix: ensure all capture-side dedup/reconciliation reads use the uncapped listAllForEpisode (or a keyset-paginated full read), and/or add a UNIQUE constraint / INSERT … ON CONFLICT DO NOTHING on the natural trace identity so re-insertion is idempotent.

Impact

• Gateway CPU pinned at 100%, unresponsive, 4.2 GB RSS.
• DB bloated 8× (≈850 MB → 6.8 GB) purely from duplicate rows + their vectors.

Workaround we applied (data-side only — does not fix the plugin bugs)

Stopped the gateway, backed up, de-duplicated traces by full content identity (518,375 → 80,583 rows), rebuilt the traces_fts FTS5 index, VACUUMed (6.8 GB → 1.3 GB), restarted. CPU settled to ~11%. But both root causes are in plugin code, so this will recur as data grows again.

One operational note that bit us during cleanup: traces has trigram FTS5 triggers (traces_fts_ai/ad/au), so a row-by-row DELETE of 437k rows is pathologically slow (didn't finish in 50 min). Dropping the triggers, doing the bulk delete, then rebuilding the FTS content and recreating the triggers was orders of magnitude faster. Not a bug per se, but worth being aware of for any maintenance tooling.

Happy to provide more detail or test a fix.

[Memtensor-AI]

🤖 AutoDev has picked up this issue and started working on it.

Task ID: 91b3e78270801f79
Working branch: bugfix/autodev-2076-20260708092153967
Target branch: latest dev* branch
Workflow: opsp (analysis → coding → testing → PR)

I will post the PR link here once done. If I need more information, I will ask in the comments.

[Memtensor-AI]

✅ AutoDev task 91b3e78270801f79 completed.

Summary: Fixed two coupled root causes in @memtensor/memos-local-plugin that made the gateway node process pin one CPU core at 100% with 4.2 GB RSS on startup (#2076).

Bug #1 — synchronous full-table vector scan in core/storage/vector.ts. scanAndTopK used db.prepare(sql).all(params), loading up to hardCap (default 100_000) multi-KB vector BLOBs synchronously into JS memory. Fixed by streaming via .iterate() and maintaining the top-K min-heap on hits directly (only k entries + one just-decoded vector live in memory at a time), and lowering the default hardCap to 5_000 as a safer default (all production callers pass their own value explicitly). Peak RSS during a scan drops from O(hardCap × dim) to O(k × dim).

Bug #2 — dedup pagination cap in core/capture/capture.ts. Four dedup call sites (runLite / runLightweight / runReflect / persistRows) used the paginated tracesRepo.list({ episodeId }), which silently caps at 500 rows. Once an episode exceeded 500 traces the older rows became invisible to dedup and were re-inserted every cycle (the reporter's DB had 518_375 rows of which 84% were exact duplicates). Fixed by adding tracesRepo.listAllForEpisode(episodeId), an uncapped read ordered by ts ASC, and switching the four dedup call sites to it. All other paginated list({ episodeId, limit }) callers keep their explicit page-size contract.

Tests: 7 new regression tests (traces-listall.test.ts: 4 cases covering 750-row episode dedup, episode scoping, empty episode, and ts-ordering; vector-stream.test.ts: 3 cases covering top-K correctness parity, streaming under a large cap, and explicit-hardCap truncation). Full plugin unit suite: 1128 pass, 1 skipped. The 2 remaining failures (startup-recovery source-string check, migrator schema-drift traces has no column named role) are pre-existing on base branch dev-v2.0.23 and unrelated to this fix. Type check clean.

Base branch: main
Branch: bugfix/autodev-2076-20260708092153967
Commit: 1b2dfca
PR: #2077
Assigned to: @bittergreen
Reviewers: @bittergreen