Summary
The self-hosted supermemory-server binary (v0.0.7-rc.2) repeatedly crashes and auto-restarts (via our Docker restart policy) during sustained, hours-long ingestion of a large document corpus, with a low-level native allocator panic — not an application-level error. The interval between crashes shrinks each time, suggesting accumulating corruption under load rather than a one-off fault.
Environment
- Version:
0.0.7-rc.2
- Install method: official installer script, pinned version, run inside a Docker container (
node:22-slim base)
- Embeddings: local, CPU-only (
Xenova/bge-base-en-v1.5), SUPERMEMORY_EMBEDDING_RAM_LIMIT=6gb
- Ingest concurrency:
SUPERMEMORY_INGEST_CONCURRENCY=8
taskType: "memory", dreaming: "instant"- Workload: two sequential full-project backfills via
POST /v3/documents, ~1124 and ~2703 documents respectively, each taking 1-3 hours end-to-end due to server-side extraction throughput (~10-15 docs/min)
What happened
Over the course of the two backfills (roughly 10 hours of continuous operation), the container crashed and cleanly auto-restarted 5 times. Each crash produced this exact panic in the container logs, immediately followed by a fresh boot sequence:
[1] pas panic: deallocation did fail at 0x752a962e8a20: Alloc bit not set in pas_segregated_page_deallocate_with_page
(address varies per occurrence). This is a libpas/Bun-runtime-level memory allocator corruption, not a JS exception — no application error was logged before or after it.
Timing pattern
| Crash # |
Time |
Preceded within ~30min by |
Interval since previous crash |
| 1 |
17:54:46 |
[Cron] dreaming-retry sweep |
— |
| 2 |
23:57:34 |
[Cron] dreaming-retry sweep |
~6h |
| 3 |
02:24:18 |
[Cron] dreaming-retry sweep |
~2.5h |
| 4 |
03:25:51 |
[Cron] done (stuck/failed retry) |
~1h |
| 5 |
03:37:03 |
[Cron] dreaming-retry sweep |
~7min |
Two things stand out:
- Every crash was preceded by the periodic
[Cron] stuck-job/dreaming-retry sweep — even though that sweep itself always reported finding nothing to do (0 requeued, 0 workflows triggered, etc.). This may be coincidental (the cron runs on a fixed ~30min cadence and could just be temporally near the crash rather than causing it), but the consistency across all 5 occurrences seemed worth reporting.
- The gap between crashes shrank on each occurrence (~6h → 2.5h → 1h → 7min), which reads like accumulating corruption under sustained load rather than a single transient fault — i.e. it appears to get more likely to crash the longer the process has been running and the more total ingestion it's done, not just under any given moment's concurrency.
Impact
Because our Docker restart policy brings the container back up cleanly each time (and /data — encrypted local storage — survived every restart with correct fact counts on inspection), the service was never down for more than a few seconds and no data was lost that we could detect. But any request in flight during a crash would fail, and the accelerating frequency is concerning for longer-running or larger ingestion jobs than the ones we ran.
Possibly related
#1320 ("Server segfaults under concurrent local-embedding load") looks like the same class of bug — a native memory-safety issue in the local-embedding layer under concurrency, same embedding model (Xenova/bge-base-en-v1.5) — but is a different specific panic (Segmentation fault vs. our pas_segregated_page_deallocate_with_page), on an older version (0.0.5 vs. our 0.0.7-rc.2), with a different trigger pattern (their repro is a short burst of 4 concurrent POSTs + 8 concurrent searches; ours is sustained hours-long ingestion with zero concurrent search load). Flagging as related rather than assuming it's a duplicate, since I can't confirm they share a root cause.
Question for maintainers
Is this a known failure mode of the local-embedding/native layer under sustained load? Any guidance on whether increasing/decreasing SUPERMEMORY_INGEST_CONCURRENCY or SUPERMEMORY_EMBEDDING_RAM_LIMIT would help, or whether this is purely a runtime-level bug that needs a fix upstream?
Summary
The self-hosted
supermemory-serverbinary (v0.0.7-rc.2) repeatedly crashes and auto-restarts (via our Docker restart policy) during sustained, hours-long ingestion of a large document corpus, with a low-level native allocator panic — not an application-level error. The interval between crashes shrinks each time, suggesting accumulating corruption under load rather than a one-off fault.Environment
0.0.7-rc.2node:22-slimbase)Xenova/bge-base-en-v1.5),SUPERMEMORY_EMBEDDING_RAM_LIMIT=6gbSUPERMEMORY_INGEST_CONCURRENCY=8taskType: "memory",dreaming: "instant"POST /v3/documents, ~1124 and ~2703 documents respectively, each taking 1-3 hours end-to-end due to server-side extraction throughput (~10-15 docs/min)What happened
Over the course of the two backfills (roughly 10 hours of continuous operation), the container crashed and cleanly auto-restarted 5 times. Each crash produced this exact panic in the container logs, immediately followed by a fresh boot sequence:
(address varies per occurrence). This is a
libpas/Bun-runtime-level memory allocator corruption, not a JS exception — no application error was logged before or after it.Timing pattern
[Cron]dreaming-retry sweep[Cron]dreaming-retry sweep[Cron]dreaming-retry sweep[Cron]done (stuck/failed retry)[Cron]dreaming-retry sweepTwo things stand out:
[Cron]stuck-job/dreaming-retry sweep — even though that sweep itself always reported finding nothing to do (0 requeued, 0 workflows triggered, etc.). This may be coincidental (the cron runs on a fixed ~30min cadence and could just be temporally near the crash rather than causing it), but the consistency across all 5 occurrences seemed worth reporting.Impact
Because our Docker restart policy brings the container back up cleanly each time (and
/data— encrypted local storage — survived every restart with correct fact counts on inspection), the service was never down for more than a few seconds and no data was lost that we could detect. But any request in flight during a crash would fail, and the accelerating frequency is concerning for longer-running or larger ingestion jobs than the ones we ran.Possibly related
#1320 ("Server segfaults under concurrent local-embedding load") looks like the same class of bug — a native memory-safety issue in the local-embedding layer under concurrency, same embedding model (
Xenova/bge-base-en-v1.5) — but is a different specific panic (Segmentation faultvs. ourpas_segregated_page_deallocate_with_page), on an older version (0.0.5 vs. our 0.0.7-rc.2), with a different trigger pattern (their repro is a short burst of 4 concurrent POSTs + 8 concurrent searches; ours is sustained hours-long ingestion with zero concurrent search load). Flagging as related rather than assuming it's a duplicate, since I can't confirm they share a root cause.Question for maintainers
Is this a known failure mode of the local-embedding/native layer under sustained load? Any guidance on whether increasing/decreasing
SUPERMEMORY_INGEST_CONCURRENCYorSUPERMEMORY_EMBEDDING_RAM_LIMITwould help, or whether this is purely a runtime-level bug that needs a fix upstream?