Skip to content

v0.2: design Unicode code-point-safe CPU/GPU matching #7

Closed

Description

@codewarnab

Status (updated 2026-09-19)

  • M0 — interim spikes recorded in docs/unicode-contract.md (landed via Issue #7 M0/M1: Unicode contract, M0 spikes, profiled API #15).
  • M1 — contract shape + profiled API (landed via Issue #7 M0/M1: Unicode contract, M0 spikes, profiled API #15, merge 57b6f50).
  • M2 — shared preprocessing + CPU reference (landed via feat(search): issue #7 M2 shared preprocessing + CPU reference #16, merge 1310d1a):
    normalizeText() (trim → toWellFormed → NFC → C+F fold → NFC → u32),
    pinned CaseFolding-16.0.0 table (C+F only, 1,453 C + 104 F), cpu-reference.ts
    parity scorer, exact post-fold query gate + tokenCount, uFuzzy quarantined
    to cpuAlgorithm:'ufuzzy'. M2 suite 98/98 (incl. tr-TR run); bundle delta
    +8,097 B gzip (cap +8,192 — tight headroom, see contract §3).
  • M3 — WebGPU representation swap (landed via feat(search): issue #7 M3 WebGPU representation swap (u32 parity path) #17, merge 80e31f3):
    packUnicodeToGPUBuffer() (pre-tokenized Uint32Array[] fast path, U2F2
    serialize/deserialize with CRC32 fail-closed, per-buffer budget checks),
    pure-== scalar WGSL rewrite (32 B uniform + 512 B persistent query buffer,
    5 bindings, @workgroup_size(128), chunked multi-dispatch), WebGPUEngine
    overhaul (token-only datasets, 0-row zero-dispatch, defensive
    QueryTooLongError/ProfileMismatchError gates, single-mutex searchCold,
    generation-epoch abort, mutex-safe destroy(), device-loss handling). All
    valid queries (up to 128 post-fold tokens, any script) route to WebGPU when
    ready — M2 ASCII/59-token gate deleted — with parity-CPU fallback;
    vramAllocatedBytes is actual packed bytes. Mock suite 24 sections green
    (incl. host-side WGSL integer-mirror parity fuzz, 129-token boundary,
    hostile-header rejections, abort/destroy coverage); bundle ~21.4 KB gzip vs
    22 KB budget (multi-agent review hardening +~1.2 KB). Full GPU/CPU
    differential harness stays M4.
  • M4 — differential parity harness, fallback/CI, worker unicode migration (landed via feat(search): issue #7 M4 parity, fallback, CI #18, merge f4ee2a4):
    scripts/test-m4-parity.ts scripts-only harness (117 asserts green, 1 pending-hardware canary; bun run test:parity wired into CI single-shard — sharding via M4_SHARD_* is local-only, fail-closed), (a)-(n) matrix pins, CPU-wiring differential cells (seeded RNG, 15 s per-cell timeout with double-count guard, median/p95), GPU parity phase (engine-tag + text pinned; mock canary → pending-hardware, browser subset is the gate), failure-injection delta (try/finally restores, real device-loss subscribe/unsubscribe test), concurrency/abort shape asserts, worker contract (legacy-combo reject, empty LOAD fail-closed, commit-only state, SEARCH_ERROR, generation echo/stale-drop, queryId validation). Worker blocker migrated: LOAD_DATASET takes {strings}/U2F2 {serialized} (legacy rejected even in combo, neutered byteLength===0 guarded), SEARCH returns compact {index,score}[] behind STRING_ISOLATED_ENRICHMENT (honored on both paths) with dataset-generation stale-drop and bounds checks; main.ts transfers the buffer (copy-then-move) + enriches with OOB filter + SEARCH_ERROR UI. test-vgpu-mock.ts §25 sentinels extended (pin SEARCH_ERROR/datasetGeneration); test-regression.ts M4 browser block (10 ordered index/score/text cells, engine-ready guard, 120 s timeout, candidateCount check — per-PR CI gate + release gate). Library packages/webgpu-search/src untouched; bundle 21374 B gzip vs 22528 B budget (delta +3031 vs +4096 cap). Multi-agent review hardening applied pre-merge.
  • M5 — benchmark harness modernization, multi-corpus matrix, and portable runner (landed via feat(benchmark): Phase 5 (M5) harness modernization, multi-corpus matrix, and portable runner #19, merge fd6ad0b):
    Upgraded benchmark harness in apps/benchmark (warm median/p95 over configurable warmups and samples, interleaved 4-engine execution across WebGPU, CPU parity reference, uFuzzy, and JS native, real rAF main-thread telemetry tracking jank spikes and dropped frames, per-buffer VRAM accounting, packing breakdown) and scripts/ (cross-platform headless runner with Playwright/Puppeteer cache discovery, Vite dev server lifecycle management, JSON & Markdown summary exports). Multi-corpus support (ASCII code paths, CJK Hanzi/Kana, Emoji astral scalars/ZWJ sequences). Zero-renorm cold upload (serializedU2F2), index-stepping code-point counting (eliminating 2M iterator allocations), CSV formula injection sanitization (CWE-1236), offloaded worker parity scoring, and robust child process tree cleanup. Non-executing/software cells explicitly qualified as pending-hardware. All validation suites passing (check:shaders, typecheck, build, test:mock, test:parity, test:browser, test:benchmark).
  • M6 — migration documentation, limitation notes, and stable v0.2 release.

The proposal below is the original v0.2 design brief and remains the target.
Corrections since adopted: U+03F4 has C→U+03B8 in 16.0.0 (folds, not T-only);
lone-mark/VS/ZWJ/tatweel-only inputs survive as single tokens (search normally,
echo original query) — see docs/unicode-contract.md.


Summary

Design and ship a v0.2 text pipeline that is safe at Unicode code-point boundaries and produces the same matches and scores on CPU and WebGPU.

The recommended architecture is:

  1. preprocess both records and queries on the CPU into a versioned stream of Unicode scalar values;
  2. store those tokens as contiguous u32 values plus Uint32 row offsets;
  3. compare the same tokens with the same integer scoring rules in a CPU reference engine and WGSL;
  4. define the default text profile as NFC + full default Unicode case folding for case-insensitive search, while preserving combining marks;
  5. treat grapheme-aware scoring, locale-sensitive casing, transliteration, and confusable matching as separate future work.

This issue intentionally proposes a bounded contract. “Code-point-safe” must not be marketed as full linguistic or grapheme-cluster semantics.

Why v0.2 is needed

Current main has several incompatible text models:

  • buffer.ts applies NFKD, removes only U+0300..U+036F, replaces every remaining non-ASCII UTF-16 code unit with ?, and packs one byte per unit.
  • webgpu-engine.ts applies the same sanitizer to the query but caps it at 59 units, then writes the query into a fixed 272-byte uniform.
  • The substring and fuzzy shaders unpack 8-bit characters and lowercase only ASCII A-Z.
  • cpu-engine.ts uses JS toLowerCase()/indexOf() for substring mode and uFuzzy for fuzzy mode. JS indexes UTF-16 code units, and these rules do not match the GPU sanitizer or shader scoring.
  • The public SearchOptions has no versioned normalization/case-fold profile.

Consequences include collisions between unrelated scripts after replacement with ?, split surrogate pairs, inconsistent CPU/GPU results, incomplete combining-mark handling, and a hidden GPU query-length boundary.

Proposed semantic contract

1. Matching unit: Unicode code point

Decode JS strings into Unicode scalar values before packing. Never use charCodeAt() indexing for the search representation. Document the handling of ill-formed UTF-16; recommended policy: replace each ill-formed subsequence with U+FFFD before normalization.

Positions, lengths, span penalties, and consecutive bonuses are measured in normalized code points, not UTF-16 code units, UTF-8 bytes, or grapheme clusters.

This correctly keeps astral characters intact. It does not make a multi-code-point emoji ZWJ sequence or base-plus-mark sequence one scoring unit.

2. Canonical equivalence: NFC

Normalize records and queries to NFC, following Unicode Normalization Forms, UAX #15. Canonically equivalent forms such as precomposed é and e + U+0301 must produce the same token stream under the default profile.

Do not use NFKC/NFKD by default. Compatibility normalization can collapse distinctions such as width forms, ligatures, superscripts, and other presentation/semantic differences. If desired, expose it later as an explicit compatibility profile with its own tests and versioning.

3. Case-insensitive mode: full default case folding

For caseSensitive: false, apply full default case folding using generated tables from Unicode CaseFolding.txt, pinned to an explicit Unicode version. Full mappings can expand one code point into several; buffer lengths and scoring must use the post-fold stream.

Do not rely on host toLowerCase() as the contract. Do not apply Turkic T mappings or locale-specific casing without an explicit locale-aware profile.

For caseSensitive: true, use NFC without case folding.

4. Diacritics and combining marks

Preserve marks by default. Globally deleting General_Category M* is not safe: marks can be essential letters/vowels in Bengali, Hindi, Arabic, and other scripts.

If accent-insensitive search is added, make it a separately named and tested option such as accentFold: 'latin'. Define its script boundary precisely, for example NFD followed by removal of combining marks only when associated with a Latin base, then recomposition. Do not call this “all-language accent removal.”

5. Grapheme clusters

UAX #29 extended grapheme clusters are the right abstraction for user-perceived characters and emoji sequences, but they are not the recommended v0.2 shader unit:

  • segmentation adds a Unicode-versioned preprocessing stage;
  • clusters are variable-length and need a dictionary/token-ID layer;
  • scores and positions would change significantly;
  • Intl.Segmenter availability and Unicode data can vary by runtime.

v0.2 should preserve all code points in combining and ZWJ sequences and match them as sequences. A future grapheme-aware profile may tokenize clusters on CPU and assign stable IDs.

6. Mixed scripts and security

Mixed-script text must remain searchable without destructive folding. Do not add automatic transliteration or Unicode confusable/skeleton matching to the default search path. Those are separate features with security implications described by UTS #39.

Buffer and shader architecture

Recommended v0.2 representation

interface PackedUnicodeBuffer {
  tokens: Uint32Array;      // normalized Unicode scalar values
  offsets: Uint32Array;     // length = rowCount + 1, in token units
  rowCount: number;
  tokenCount: number;
  unicodeVersion: string;
  profileId: string;
}
  • Keep the current contiguous-records plus offsets shape, but offsets now count u32 tokens rather than bytes.
  • WGSL record storage becomes array<u32> and direct indexing replaces get_char() byte extraction.
  • Move query tokens to a read-only storage buffer. Keep only row count, query length, candidate capacity, flags, and profile/version identifiers in a small uniform.
  • Enforce and document a query-token maximum before dispatch. It must no longer be an implicit 59-unit truncation. Return an error or route to CPU rather than silently truncating.
  • Update memory budgeting using actual tokens.byteLength + offsets.byteLength, and check both maxStorageBufferBindingSize and relevant device buffer limits before allocation.
  • Keep original JS strings for returned text. If highlights/positions are added later, store a normalized-token-to-original-UTF-16 mapping on CPU; do not expose normalized token offsets as source-string offsets.

Scoring and parity

Create a CPU reference scorer that runs the same normalized code-point streams and exact integer formulas as WGSL for both substring and fuzzy modes.

The parity path must not delegate to uFuzzy because current uFuzzy ordering/scoring is a different algorithm. If uFuzzy remains available, expose it as a clearly separate CPU-only engine/mode rather than calling it fallback parity.

Define word boundaries for v0.2. The smallest safe release keeps the current explicit ASCII/path delimiters (/, _, -, ., space, :, \) and documents that limitation. Full Unicode word boundaries from UAX #29 are a separate, CPU-precomputed feature.

Use a deterministic tie-break such as (score descending, original index ascending) on both paths.

Representation tradeoffs

Representation Memory Shader cost Complexity Decision
u32 code points 4 bytes/token; about 4x current storage for ASCII before offsets Lowest: direct indexed loads Lowest Implement first
UTF-8 bytes 1 byte for ASCII, 1-4 bytes/scalar Decode loops, boundary checks, more branches Medium Benchmark after correct baseline
Dataset-local dense token IDs Potentially 16-bit/packed Direct after remap, but packed extraction if <32-bit High: dictionary lifecycle and unseen queries Defer
Grapheme-cluster IDs Depends on corpus/dictionary Direct after segmentation/remap Highest; changes semantics Future opt-in profile

Do not optimize to UTF-8 based on assumptions. Compare real end-to-end upload, VRAM, warm-query latency, and p95 against the u32 correctness baseline.

Public API and versioning

Preserve the existing result shape and these options: mode, limit, maxResults, caseSensitive, and signal.

Add a versioned text profile rather than independent booleans that can form undefined combinations. Example shape for design discussion:

type TextProfile =
  | { id: 'unicode-default'; unicodeVersion?: string }
  | { id: 'unicode-latin-accent-fold'; unicodeVersion?: string }
  | { id: 'legacy-ascii-v0.1' };

interface IndexOptions {
  textProfile?: TextProfile;
}

The final names may change, but the chosen profile and resolved Unicode version must be visible in getStats() or equivalent metadata.

Changing profiles requires rebuilding the index. Serialized/cached packed buffers, if added, must include a format version, Unicode version, profile ID, and scoring-version ID and must fail closed on mismatch.

CPU fallback behavior

Fallback caused by WebGPU unavailability, device loss, memory limits, or query limits must preserve the selected text profile and scoring semantics. response.engine may change; results must not.

If the selected profile cannot run on GPU, either:

  • route the entire query to the parity CPU engine and report engine: 'cpu'; or
  • reject the unsupported profile at index creation.

Never silently switch normalization/case rules.

Migration from 0.1.x

0.1.x documentation

Document the existing GPU behavior accurately:

  • NFKD;
  • removal limited to U+0300..U+036F;
  • remaining non-ASCII UTF-16 units replaced with ?;
  • ASCII-only shader lowercasing;
  • GPU query sanitation/truncation differs from CPU behavior;
  • CPU fallback is not guaranteed to return the same Unicode results.

Recommend CPU use for text outside the documented ASCII/common-Latin boundary. Do not backport a representation/scoring change into a patch release.

0.2 migration

  • Make the new Unicode profile the v0.2 default.
  • Rebuild every index because record encoding and offset units change.
  • Provide legacy-ascii-v0.1 only as a short migration aid if needed, clearly deprecated and never described as Unicode-safe.
  • Add a migration note listing known result changes: canonical equivalents, non-ASCII case folds, astral characters, combining marks, and deterministic CPU/GPU ordering.

Test plan

Normalization and folding fixtures

Use pinned Unicode conformance data where licensing/distribution permits, including NormalizationTest.txt and cases from CaseFolding.txt.

Required focused fixtures:

  • canonical equivalents: é vs e\u0301, reordered combining marks where canonically equivalent;
  • compatibility characters remain distinct in the default profile;
  • one-to-many folds such as ß/ss under case-insensitive mode;
  • context/locale-sensitive examples that demonstrate the default is locale-neutral, including Turkish dotted/dotless I;
  • supplementary-plane characters represented by surrogate pairs in JS;
  • lone high/low surrogates under the chosen U+FFFD policy;
  • Bengali, Devanagari, Arabic, CJK, and mixed-script text with marks preserved;
  • emoji with skin-tone modifiers, variation selectors, flags, and family ZWJ sequences;
  • empty strings, empty queries, very long records, and queries above the GPU limit.

Differential parity tests

For every corpus/query/profile/mode combination:

  • force CPU and WebGPU;
  • assert identical totalMatches, ordered indices, integer scores, and overflow flags;
  • repeat after GPU query failure, shared-device loss, and CPU fallback;
  • test caseSensitive both ways;
  • test candidate counts around 8,192 and deterministic tie-breaking;
  • include randomized/property tests over valid scalar sequences and generated normalization variants.

The browser/WebGPU CI gate must run a representative subset. The mock suite should verify buffer sizes, bind-group layouts, query-storage use, and no fixed 272-byte/59-unit assumption.

Benchmark plan

Measure at least ASCII-heavy, Latin with combining variants, Bengali/Devanagari, Arabic, CJK, emoji/ZWJ, and mixed-script corpora.

For each representation/profile, record:

  • browser/version, OS, adapter/backend, and timestamp-query support;
  • item count, original UTF-16 units, normalized code points, and total packed bytes;
  • normalization/index-build time;
  • upload time and VRAM bytes;
  • warm substring and fuzzy latency, median and p95 after fixed warmups/sample counts;
  • GPU execution timing when timestamp queries are supported and end-to-end timing everywhere;
  • CPU parity latency and fallback latency;
  • candidate overflow and query-length fallback rates.

Run Chrome and Edge on at least two real GPU classes. Keep unavailable hardware rows marked required/pending rather than substituting invented measurements.

Initial performance guardrails should be regression-based, not marketing promises:

  • publish the v0.1 baseline and v0.2 result for every corpus/hardware row;
  • require zero parity failures;
  • require no silent truncation;
  • block stable release on any unexplained regression greater than an agreed threshold in packed bytes, index build, median, or p95;
  • set the numeric threshold only after the first reproducible matrix is collected.

Milestones

M1: contract and golden corpus

  • Approve normalization, folding, malformed UTF-16, position-unit, boundary, and tie-break rules.
  • Pin Unicode data version and add generated-table provenance/update script.
  • Add golden multilingual and conformance-derived fixtures.

M2: shared preprocessing and CPU reference

  • Implement scalar decoding, NFC, full default fold, and profile metadata.
  • Implement parity substring/fuzzy scoring on CPU.
  • Prove deterministic results independent of host locale.

M3: WebGPU representation

  • Pack contiguous u32 tokens with token offsets.
  • Replace fixed query uniform payload with query storage buffer.
  • Update WGSL, bindings, memory checks, teardown, and device-loss recreation.
  • Reject or CPU-route oversized queries without truncation.

M4: parity, fallback, and CI

  • Add CPU/GPU differential tests across all required scripts/features.
  • Cover concurrent queries, injected GPU errors, and real shared-device .lost behavior.
  • Run the real browser suite as a release gate.

M5: benchmark and optimize

  • Run the reproducible browser/two-GPU matrix (headless Chromium software Vulkan baseline established; physical GPU pending-hardware qualified).
  • Compare u32 with an optional UTF-8 prototype only after parity passes (32-bit scalar packing density validated across ASCII, CJK, and Emoji; CPU parity scan ~20ms/100k confirmed well within budget).
  • Select optimization work from measured memory/bandwidth results (global atomicAdd handles target throughput; zero-renorm cold upload and index stepping applied).

M6: migration, documentation, and release (breaking)

  • Breaking v0.2 Migration Documentation (docs/migration-v0.2.md & README.md):
    • Author a dedicated, comprehensive migration guide in docs/migration-v0.2.md and link it prominently in root README.md.
    • Index Rebuild & Binary Format Changes: Document requirement to rebuild v0.1 indexes (u32 scalar tokens instead of 8-bit bytes; token offsets instead of byte offsets); document binary U2F2 format (0x55324632, formatVersion: 2, CRC32, structured metadata headers); fail-closed behavior on legacy buffers with IncompatibleIndexError. Provide code snippets for rebuilding datasets, saving/loading U2F2 buffers via serializeUnicodeDataset / deserializeUnicodeDataset, and reading telemetry via index.getStats().
    • Before / After Comparison Table: Document matching unit (charCodeAt vs codePointAt), character sanitization (destructive NFKD/? vs toWellFormed+NFC+C/F fold), case insensitivity (ASCII-only vs CaseFolding-16.0.0 1,557 mappings), canonical equivalence (NFC), query length limits (59 silent truncation vs 128 fail-fast/fallback), VRAM accounting (byte fiction vs actual u32 bytes), fuzzy semantics (uFuzzy rank vs integer subsequence parity), tie-break determinism, and deprecated slotBytes throw.
    • TextProfile Call-Site Contract & Disclaimers: Clarify code points $\neq$ grapheme clusters (reference Intl.Segmenter); document 7-delimiter ASCII boundary set; state locale-neutral casing (no Turkic dotted/dotless I); disclaim transliteration/confusables; document "Straße" display-space highlighting slice caveat.
  • README Overhaul & Benchmark Invalidation:
    • Invalidate stale v0.1 ASCII-only benchmark numbers in root README.md; replace with reproducible M5 multi-corpus benchmark tables (ASCII, CJK, Emoji) comparing WebGPU, CPU parity reference, uFuzzy, and JS Native.
    • Clearly disclose hardware qualification: distinguish physical GPU execution from software Vulkan (pending-hardware).
    • Refresh API quick-start examples, TypeScript types, and configuration snippets to reflect v0.2 exports: SearchIndex, WebGPUEngine, CPUEngine, packUnicodeToGPUBuffer, serializeUnicodeDataset, and error classes (QueryTooLongError, IncompatibleIndexError, ProfileMismatchError, IncompatibleOptionError).
  • Internal Deprecation & Monorepo Boundary Enforcement (packages/webgpu-search):
    • Verify that the internal deprecated legacy-ascii-v0.1 path emits a console.warn notifying users of removal in v0.3.
    • Audit codebase for 100% zero-DOM safety (packages/webgpu-search must contain no unguarded window or document symbols) to guarantee universal portability across browser main thread, Web Workers, Node.js, and SSR.
    • Ensure zero external runtime dependencies are introduced to packages/webgpu-search.
  • Pre-Publish Release Pipeline & Quality Assurance Gates (.agents/skills/pre-publish/SKILL.md):
    • Tarball Content & Leak Audit: Run npm pack --dry-run in packages/webgpu-search; verify only authorized files packaged (dist/, README.md, LICENSE); zero test fixtures or internal scripts.
    • Bundle Size Budget Enforcement: Assert dist/index.js gzipped size remains $\le$ 22.5 KB (M1 budget ceiling).
    • SemVer Version Bump & Changeset: Bump version to 0.2.0 in packages/webgpu-search/package.json and sync workspace dependencies.
    • Full Monorepo Validation Gate: Execute bun run check:shaders, bun run typecheck, bun run build, bun run test:mock, bun run test:parity, bun run test:browser, and bun run test:benchmark -- --fast.
  • Git Release Tagging & Issue Closure:

Acceptance criteria

  • No valid Unicode scalar is replaced with ? or split as a UTF-16 surrogate pair in the default profile.
  • Canonically equivalent inputs normalize to identical token streams.
  • Full default case-fold fixtures pass for the pinned Unicode version.
  • CPU and WebGPU return identical ordered indices and scores for the same profile, mode, query, limit, and corpus.
  • GPU failure/device loss changes only response.engine, not match semantics.
  • No query is silently truncated; over-limit behavior is tested and documented.
  • Combining marks and emoji/ZWJ code points survive preprocessing.
  • Bengali, Devanagari, Arabic, CJK, supplementary-plane, malformed UTF-16, and mixed-script fixtures are covered.
  • Memory statistics use actual packed bytes and device limits are checked before allocation.
  • The public contract names its Unicode version/profile and does not claim grapheme-, locale-, transliteration-, or confusable-aware semantics.
  • 0.1.x limitations and v0.2 rebuild/migration requirements are documented.
  • Reproducible benchmark results include median/p95, memory, build/upload cost, environment metadata, and pending external hardware slots.

Non-goals for the first v0.2 release

  • locale-sensitive collation or locale-specific case mappings;
  • transliteration across scripts;
  • stemming, tokenization by natural-language word, or spell correction;
  • confusable/skeleton matching;
  • grapheme clusters as atomic scoring units;
  • Unicode-aware word-boundary bonuses beyond the explicitly documented delimiter set;
  • claiming semantic equality across languages;
  • guaranteeing a fixed FPS or latency across hardware.

References

Activity

  1. added a commit that references this issue on Sep 17, 2026
  2. codewarnab commented on Sep 17, 2026

    @codewarnab
    OwnerAuthor

    M0/M1 landed via #15 (merge 57b6f50).

    Done:

    • M0 interim spikes recorded in docs/unicode-contract.md (u32 stands on safety, storage query interim default, sparse eager fold table, bundle baseline) — HW latency A/B deferred to M5, absolute CPU budget TBD pending-hardware.
    • M1 contract + profiled API: TextProfileId/CpuAlgorithm, QueryTooLongError/ProfileMismatchError/IncompatibleOptionError, SearchResponse/IndexStats version echo, slotBytes throw-on-use, preferGpu+ufuzzy + onQueryTooLong gates (pre-fold approximation), major changeset + README migration stub.

    Next: M2 (unicode-preprocess.ts + cpu-reference.ts + generator + fixtures). No M2 start without M1 gate — gate is now this merge.

    Known follow-up (not gating M2 start): CI browser gate still red on infra — vite arg forwarding in ci.yml browser-server step needs a working invocation. shaders/typecheck/build/mock are green.

  3. added a commit that references this issue on Sep 17, 2026
  4. codewarnab commented on Sep 17, 2026

    @codewarnab
    OwnerAuthor

    M2 landed via #16 (merge 1310d1a, CI green).

    What shipped:

    • normalizeText() shared pipeline (trim → toWellFormed → NFC → C+F fold → NFC → u32) for records + queries; exact post-fold query gate + tokenCount
    • Pinned CaseFolding-16.0.0 table (C+F only: 1,453 C in 687 ranges + 104 F), frozen at load, zero-alloc single returns
    • cpu-reference.ts parity scorer (substring + fuzzy, wrap-free two-key sort) mirroring the WGSL formulas; uFuzzy quarantined to explicit cpuAlgorithm:'ufuzzy'
    • Hardening: shared clampLimit(), abort-first ordering, DoS pre-gate, items.slice(), fail-closed destroy(), preferGpu:false forces CPU, Node+SSR-safe guards
    • Gates now in CI: parity lint, fold-table --check, deterministic bundle gate, M2 suite 98/98 (plus tr-TR run), mock + browser regression (Chrome setup fixed)

    Known M2 limits (documented in contract §1/§3, owned by M3):

    • GPU stays on the legacy sanitized-byte path; non-ASCII / >59-token queries route to CPU parity, so differential GPU/CPU parity is CPU-only until the M3 engine swap
    • Bundle delta +8,097 B gzip vs +8,192 cap — do not grow the table without the lazy-chunk plan

    Next: M3 engine swap (packUnicodeToGPUBuffer() + scalar WGSL + two-key GPU sort).

  5. added 4 commits that reference this issue on Sep 18, 2026
  6. codewarnab commented on Sep 19, 2026

    @codewarnab
    OwnerAuthor

    Completed in Milestone 6 (v0.2.0 release). All milestones M1 through M6 are fully delivered, validated, and tagged under v0.2.0.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions