fix(rag): guarantee forward progress in chunkText

[Agnik47]

Fixes #69

The bug

chunkText (src/providers/rag/index.ts) advances its window with start = breakPoint + 1 - overlap, clamped only by if (start < 0) start = 0. Nothing keeps that value ahead of the previous start.

The half-chunk floor (start + chunkSize * 0.5) guards the ". " and "\n" candidates, but it is not re-checked after the lastIndexOf(" ", end) fallback — which is precisely the branch that produces a break point close to start. The only remaining guard there is breakPoint <= start, so with the shipped CHUNK_SIZE = 1600 / CHUNK_OVERLAP = 320 any break point in (start, start + 319] pushes the next start backwards, and the clamp parks it at 0 forever.

Reproduced against main — one early space followed by an unbroken run:

const text = "x".repeat(10) + " " + "a".repeat(5000)
chunkText(text)   // never returns

iter 0 start 0 breakPoint 10 nextStart 0 chunkLen 10
iter 1 start 0 breakPoint 10 nextStart 0 chunkLen 10
iter 2 start 0 breakPoint 10 nextStart 0 chunkLen 10
...

start never leaves 0, and each iteration pushes a fresh ~10-character slice, so the process spins and grows until the heap is exhausted.

Any ingested session containing a long unbroken token hits this: a URL, a base64 image, a stack trace, a minified payload, or CJK text (lastIndexOf(" ") is whitespace-based). chunkText runs inside RAGProvider.ingest under ConcurrentExecutor with no timeout, and ingest is checkpointed per session — so the hang reproduces at the same session on every resume and the run can never make progress.

The fix

Two changes, both in the loop:

1. Apply the half-chunk floor to the word fallback too. A degenerate break point now falls through to breakPoint = end, producing a genuine full-size chunk instead of a sliver. (The dropped breakPoint <= start clauses were redundant — breakPoint <= start already implies breakPoint < start + chunkSize * 0.5.)
2. Clamp the step forward: start = Math.max(breakPoint + 1 - overlap, start + 1). The floor above already keeps the step positive at the default sizes; this keeps the loop terminating for caller-supplied sizes where overlap exceeds the step, and makes the start < 0 clamp unnecessary.

Normal prose is unaffected. Its break points already clear the floor (>= start + 800), so the step is >= start + 481 and Math.max picks the unchanged value — chunks still end on sentence boundaries with the same overlap.

Verification

• bun test — 8 new tests pass in src/providers/rag/chunking.test.ts: the reported input, full-size chunks rather than slivers, source coverage, an overlap larger than the chunk step, plus the existing sentence-boundary and overlap behaviour as regression guards.
• Fuzz: 3000 randomised inputs (lengths 0–6000, chunkSize 1–2000, overlap 0–2000, alphabets mixing spaces, periods, newlines, unbroken runs and CJK) all terminate with content preserved. Every one of these can hang on main.
• tsc --noEmit clean for src/providers/rag.

One note on the tests: against the unfixed code they hang rather than fail. The loop is synchronous, so bun's per-test timeout cannot interrupt it — that is the bug, not a flaw in the tests.

chunkText is now exported so it can be tested directly; it is otherwise unchanged in signature and behaviour. The diff is scoped to the loop — I left the file's pre-existing prettier --check warning (an over-long logger.info line, present on main) alone.