Repository navigation
Description
Status (updated 2026-09-19)
- M0 — interim spikes recorded in
docs/unicode-contract.md(landed via Issue #7 M0/M1: Unicode contract, M0 spikes, profiled API #15). - M1 — contract shape + profiled API (landed via Issue #7 M0/M1: Unicode contract, M0 spikes, profiled API #15, merge
57b6f50). - M2 — shared preprocessing + CPU reference (landed via feat(search): issue #7 M2 shared preprocessing + CPU reference #16, merge
1310d1a):
normalizeText()(trim → toWellFormed → NFC → C+F fold → NFC → u32),
pinnedCaseFolding-16.0.0table (C+F only, 1,453 C + 104 F),cpu-reference.ts
parity scorer, exact post-fold query gate +tokenCount, uFuzzy quarantined
tocpuAlgorithm:'ufuzzy'. M2 suite 98/98 (incl. tr-TR run); bundle delta
+8,097 B gzip (cap +8,192 — tight headroom, see contract §3). - M3 — WebGPU representation swap (landed via feat(search): issue #7 M3 WebGPU representation swap (u32 parity path) #17, merge
80e31f3):
packUnicodeToGPUBuffer()(pre-tokenizedUint32Array[]fast path, U2F2
serialize/deserializewith CRC32 fail-closed, per-buffer budget checks),
pure-==scalar WGSL rewrite (32 B uniform + 512 B persistent query buffer,
5 bindings,@workgroup_size(128), chunked multi-dispatch),WebGPUEngine
overhaul (token-only datasets, 0-row zero-dispatch, defensive
QueryTooLongError/ProfileMismatchErrorgates, single-mutexsearchCold,
generation-epoch abort, mutex-safedestroy(), device-loss handling). All
valid queries (up to 128 post-fold tokens, any script) route to WebGPU when
ready — M2 ASCII/59-token gate deleted — with parity-CPU fallback;
vramAllocatedBytesis actual packed bytes. Mock suite 24 sections green
(incl. host-side WGSL integer-mirror parity fuzz, 129-token boundary,
hostile-header rejections, abort/destroy coverage); bundle ~21.4 KB gzip vs
22 KB budget (multi-agent review hardening +~1.2 KB). Full GPU/CPU
differential harness stays M4. - M4 — differential parity harness, fallback/CI, worker unicode migration (landed via feat(search): issue #7 M4 parity, fallback, CI #18, merge
f4ee2a4):
scripts/test-m4-parity.tsscripts-only harness (117 asserts green, 1 pending-hardware canary;bun run test:paritywired into CI single-shard — sharding viaM4_SHARD_*is local-only, fail-closed), (a)-(n) matrix pins, CPU-wiring differential cells (seeded RNG, 15 s per-cell timeout with double-count guard, median/p95), GPU parity phase (engine-tag + text pinned; mock canary → pending-hardware, browser subset is the gate), failure-injection delta (try/finally restores, real device-loss subscribe/unsubscribe test), concurrency/abort shape asserts, worker contract (legacy-combo reject, empty LOAD fail-closed, commit-only state,SEARCH_ERROR, generation echo/stale-drop, queryId validation). Worker blocker migrated:LOAD_DATASETtakes{strings}/U2F2{serialized}(legacy rejected even in combo, neuteredbyteLength===0guarded), SEARCH returns compact{index,score}[]behindSTRING_ISOLATED_ENRICHMENT(honored on both paths) with dataset-generation stale-drop and bounds checks;main.tstransfers the buffer (copy-then-move) + enriches with OOB filter +SEARCH_ERRORUI.test-vgpu-mock.ts§25 sentinels extended (pinSEARCH_ERROR/datasetGeneration);test-regression.tsM4 browser block (10 ordered index/score/text cells, engine-ready guard, 120 s timeout, candidateCount check — per-PR CI gate + release gate). Librarypackages/webgpu-search/srcuntouched; bundle 21374 B gzip vs 22528 B budget (delta +3031 vs +4096 cap). Multi-agent review hardening applied pre-merge. - M5 — benchmark harness modernization, multi-corpus matrix, and portable runner (landed via feat(benchmark): Phase 5 (M5) harness modernization, multi-corpus matrix, and portable runner #19, merge
fd6ad0b):
Upgraded benchmark harness inapps/benchmark(warm median/p95 over configurable warmups and samples, interleaved 4-engine execution across WebGPU, CPU parity reference, uFuzzy, and JS native, real rAF main-thread telemetry tracking jank spikes and dropped frames, per-buffer VRAM accounting, packing breakdown) andscripts/(cross-platform headless runner with Playwright/Puppeteer cache discovery, Vite dev server lifecycle management, JSON & Markdown summary exports). Multi-corpus support (ASCII code paths, CJK Hanzi/Kana, Emoji astral scalars/ZWJ sequences). Zero-renorm cold upload (serializedU2F2), index-stepping code-point counting (eliminating 2M iterator allocations), CSV formula injection sanitization (CWE-1236), offloaded worker parity scoring, and robust child process tree cleanup. Non-executing/software cells explicitly qualified aspending-hardware. All validation suites passing (check:shaders,typecheck,build,test:mock,test:parity,test:browser,test:benchmark). - M6 — migration documentation, limitation notes, and stable v0.2 release.
The proposal below is the original v0.2 design brief and remains the target.
Corrections since adopted: U+03F4 has C→U+03B8 in 16.0.0 (folds, not T-only);
lone-mark/VS/ZWJ/tatweel-only inputs survive as single tokens (search normally,
echo original query) — seedocs/unicode-contract.md.
Summary
Design and ship a v0.2 text pipeline that is safe at Unicode code-point boundaries and produces the same matches and scores on CPU and WebGPU.
The recommended architecture is:
- preprocess both records and queries on the CPU into a versioned stream of Unicode scalar values;
- store those tokens as contiguous
u32values plusUint32row offsets; - compare the same tokens with the same integer scoring rules in a CPU reference engine and WGSL;
- define the default text profile as NFC + full default Unicode case folding for case-insensitive search, while preserving combining marks;
- treat grapheme-aware scoring, locale-sensitive casing, transliteration, and confusable matching as separate future work.
This issue intentionally proposes a bounded contract. “Code-point-safe” must not be marketed as full linguistic or grapheme-cluster semantics.
Why v0.2 is needed
Current main has several incompatible text models:
buffer.tsapplies NFKD, removes only U+0300..U+036F, replaces every remaining non-ASCII UTF-16 code unit with?, and packs one byte per unit.webgpu-engine.tsapplies the same sanitizer to the query but caps it at 59 units, then writes the query into a fixed 272-byte uniform.- The substring and fuzzy shaders unpack 8-bit characters and lowercase only ASCII A-Z.
cpu-engine.tsuses JStoLowerCase()/indexOf()for substring mode and uFuzzy for fuzzy mode. JS indexes UTF-16 code units, and these rules do not match the GPU sanitizer or shader scoring.- The public
SearchOptionshas no versioned normalization/case-fold profile.
Consequences include collisions between unrelated scripts after replacement with ?, split surrogate pairs, inconsistent CPU/GPU results, incomplete combining-mark handling, and a hidden GPU query-length boundary.
Proposed semantic contract
1. Matching unit: Unicode code point
Decode JS strings into Unicode scalar values before packing. Never use charCodeAt() indexing for the search representation. Document the handling of ill-formed UTF-16; recommended policy: replace each ill-formed subsequence with U+FFFD before normalization.
Positions, lengths, span penalties, and consecutive bonuses are measured in normalized code points, not UTF-16 code units, UTF-8 bytes, or grapheme clusters.
This correctly keeps astral characters intact. It does not make a multi-code-point emoji ZWJ sequence or base-plus-mark sequence one scoring unit.
2. Canonical equivalence: NFC
Normalize records and queries to NFC, following Unicode Normalization Forms, UAX #15. Canonically equivalent forms such as precomposed é and e + U+0301 must produce the same token stream under the default profile.
Do not use NFKC/NFKD by default. Compatibility normalization can collapse distinctions such as width forms, ligatures, superscripts, and other presentation/semantic differences. If desired, expose it later as an explicit compatibility profile with its own tests and versioning.
3. Case-insensitive mode: full default case folding
For caseSensitive: false, apply full default case folding using generated tables from Unicode CaseFolding.txt, pinned to an explicit Unicode version. Full mappings can expand one code point into several; buffer lengths and scoring must use the post-fold stream.
Do not rely on host toLowerCase() as the contract. Do not apply Turkic T mappings or locale-specific casing without an explicit locale-aware profile.
For caseSensitive: true, use NFC without case folding.
4. Diacritics and combining marks
Preserve marks by default. Globally deleting General_Category M* is not safe: marks can be essential letters/vowels in Bengali, Hindi, Arabic, and other scripts.
If accent-insensitive search is added, make it a separately named and tested option such as accentFold: 'latin'. Define its script boundary precisely, for example NFD followed by removal of combining marks only when associated with a Latin base, then recomposition. Do not call this “all-language accent removal.”
5. Grapheme clusters
UAX #29 extended grapheme clusters are the right abstraction for user-perceived characters and emoji sequences, but they are not the recommended v0.2 shader unit:
- segmentation adds a Unicode-versioned preprocessing stage;
- clusters are variable-length and need a dictionary/token-ID layer;
- scores and positions would change significantly;
Intl.Segmenteravailability and Unicode data can vary by runtime.
v0.2 should preserve all code points in combining and ZWJ sequences and match them as sequences. A future grapheme-aware profile may tokenize clusters on CPU and assign stable IDs.
6. Mixed scripts and security
Mixed-script text must remain searchable without destructive folding. Do not add automatic transliteration or Unicode confusable/skeleton matching to the default search path. Those are separate features with security implications described by UTS #39.
Buffer and shader architecture
Recommended v0.2 representation
interface PackedUnicodeBuffer {
tokens: Uint32Array; // normalized Unicode scalar values
offsets: Uint32Array; // length = rowCount + 1, in token units
rowCount: number;
tokenCount: number;
unicodeVersion: string;
profileId: string;
}- Keep the current contiguous-records plus offsets shape, but offsets now count
u32tokens rather than bytes. - WGSL record storage becomes
array<u32>and direct indexing replacesget_char()byte extraction. - Move query tokens to a read-only storage buffer. Keep only row count, query length, candidate capacity, flags, and profile/version identifiers in a small uniform.
- Enforce and document a query-token maximum before dispatch. It must no longer be an implicit 59-unit truncation. Return an error or route to CPU rather than silently truncating.
- Update memory budgeting using actual
tokens.byteLength + offsets.byteLength, and check bothmaxStorageBufferBindingSizeand relevant device buffer limits before allocation. - Keep original JS strings for returned
text. If highlights/positions are added later, store a normalized-token-to-original-UTF-16 mapping on CPU; do not expose normalized token offsets as source-string offsets.
Scoring and parity
Create a CPU reference scorer that runs the same normalized code-point streams and exact integer formulas as WGSL for both substring and fuzzy modes.
The parity path must not delegate to uFuzzy because current uFuzzy ordering/scoring is a different algorithm. If uFuzzy remains available, expose it as a clearly separate CPU-only engine/mode rather than calling it fallback parity.
Define word boundaries for v0.2. The smallest safe release keeps the current explicit ASCII/path delimiters (/, _, -, ., space, :, \) and documents that limitation. Full Unicode word boundaries from UAX #29 are a separate, CPU-precomputed feature.
Use a deterministic tie-break such as (score descending, original index ascending) on both paths.
Representation tradeoffs
| Representation | Memory | Shader cost | Complexity | Decision |
|---|---|---|---|---|
u32 code points |
4 bytes/token; about 4x current storage for ASCII before offsets | Lowest: direct indexed loads | Lowest | Implement first |
| UTF-8 bytes | 1 byte for ASCII, 1-4 bytes/scalar | Decode loops, boundary checks, more branches | Medium | Benchmark after correct baseline |
| Dataset-local dense token IDs | Potentially 16-bit/packed | Direct after remap, but packed extraction if <32-bit | High: dictionary lifecycle and unseen queries | Defer |
| Grapheme-cluster IDs | Depends on corpus/dictionary | Direct after segmentation/remap | Highest; changes semantics | Future opt-in profile |
Do not optimize to UTF-8 based on assumptions. Compare real end-to-end upload, VRAM, warm-query latency, and p95 against the u32 correctness baseline.
Public API and versioning
Preserve the existing result shape and these options: mode, limit, maxResults, caseSensitive, and signal.
Add a versioned text profile rather than independent booleans that can form undefined combinations. Example shape for design discussion:
type TextProfile =
| { id: 'unicode-default'; unicodeVersion?: string }
| { id: 'unicode-latin-accent-fold'; unicodeVersion?: string }
| { id: 'legacy-ascii-v0.1' };
interface IndexOptions {
textProfile?: TextProfile;
}The final names may change, but the chosen profile and resolved Unicode version must be visible in getStats() or equivalent metadata.
Changing profiles requires rebuilding the index. Serialized/cached packed buffers, if added, must include a format version, Unicode version, profile ID, and scoring-version ID and must fail closed on mismatch.
CPU fallback behavior
Fallback caused by WebGPU unavailability, device loss, memory limits, or query limits must preserve the selected text profile and scoring semantics. response.engine may change; results must not.
If the selected profile cannot run on GPU, either:
- route the entire query to the parity CPU engine and report
engine: 'cpu'; or - reject the unsupported profile at index creation.
Never silently switch normalization/case rules.
Migration from 0.1.x
0.1.x documentation
Document the existing GPU behavior accurately:
- NFKD;
- removal limited to U+0300..U+036F;
- remaining non-ASCII UTF-16 units replaced with
?; - ASCII-only shader lowercasing;
- GPU query sanitation/truncation differs from CPU behavior;
- CPU fallback is not guaranteed to return the same Unicode results.
Recommend CPU use for text outside the documented ASCII/common-Latin boundary. Do not backport a representation/scoring change into a patch release.
0.2 migration
- Make the new Unicode profile the v0.2 default.
- Rebuild every index because record encoding and offset units change.
- Provide
legacy-ascii-v0.1only as a short migration aid if needed, clearly deprecated and never described as Unicode-safe. - Add a migration note listing known result changes: canonical equivalents, non-ASCII case folds, astral characters, combining marks, and deterministic CPU/GPU ordering.
Test plan
Normalization and folding fixtures
Use pinned Unicode conformance data where licensing/distribution permits, including NormalizationTest.txt and cases from CaseFolding.txt.
Required focused fixtures:
- canonical equivalents:
évse\u0301, reordered combining marks where canonically equivalent; - compatibility characters remain distinct in the default profile;
- one-to-many folds such as
ß/ssunder case-insensitive mode; - context/locale-sensitive examples that demonstrate the default is locale-neutral, including Turkish dotted/dotless I;
- supplementary-plane characters represented by surrogate pairs in JS;
- lone high/low surrogates under the chosen U+FFFD policy;
- Bengali, Devanagari, Arabic, CJK, and mixed-script text with marks preserved;
- emoji with skin-tone modifiers, variation selectors, flags, and family ZWJ sequences;
- empty strings, empty queries, very long records, and queries above the GPU limit.
Differential parity tests
For every corpus/query/profile/mode combination:
- force CPU and WebGPU;
- assert identical
totalMatches, ordered indices, integer scores, and overflow flags; - repeat after GPU query failure, shared-device loss, and CPU fallback;
- test
caseSensitiveboth ways; - test candidate counts around 8,192 and deterministic tie-breaking;
- include randomized/property tests over valid scalar sequences and generated normalization variants.
The browser/WebGPU CI gate must run a representative subset. The mock suite should verify buffer sizes, bind-group layouts, query-storage use, and no fixed 272-byte/59-unit assumption.
Benchmark plan
Measure at least ASCII-heavy, Latin with combining variants, Bengali/Devanagari, Arabic, CJK, emoji/ZWJ, and mixed-script corpora.
For each representation/profile, record:
- browser/version, OS, adapter/backend, and timestamp-query support;
- item count, original UTF-16 units, normalized code points, and total packed bytes;
- normalization/index-build time;
- upload time and VRAM bytes;
- warm substring and fuzzy latency, median and p95 after fixed warmups/sample counts;
- GPU execution timing when timestamp queries are supported and end-to-end timing everywhere;
- CPU parity latency and fallback latency;
- candidate overflow and query-length fallback rates.
Run Chrome and Edge on at least two real GPU classes. Keep unavailable hardware rows marked required/pending rather than substituting invented measurements.
Initial performance guardrails should be regression-based, not marketing promises:
- publish the v0.1 baseline and v0.2 result for every corpus/hardware row;
- require zero parity failures;
- require no silent truncation;
- block stable release on any unexplained regression greater than an agreed threshold in packed bytes, index build, median, or p95;
- set the numeric threshold only after the first reproducible matrix is collected.
Milestones
M1: contract and golden corpus
- Approve normalization, folding, malformed UTF-16, position-unit, boundary, and tie-break rules.
- Pin Unicode data version and add generated-table provenance/update script.
- Add golden multilingual and conformance-derived fixtures.
M2: shared preprocessing and CPU reference
- Implement scalar decoding, NFC, full default fold, and profile metadata.
- Implement parity substring/fuzzy scoring on CPU.
- Prove deterministic results independent of host locale.
M3: WebGPU representation
- Pack contiguous
u32tokens with token offsets. - Replace fixed query uniform payload with query storage buffer.
- Update WGSL, bindings, memory checks, teardown, and device-loss recreation.
- Reject or CPU-route oversized queries without truncation.
M4: parity, fallback, and CI
- Add CPU/GPU differential tests across all required scripts/features.
- Cover concurrent queries, injected GPU errors, and real shared-device
.lostbehavior. - Run the real browser suite as a release gate.
M5: benchmark and optimize
- Run the reproducible browser/two-GPU matrix (headless Chromium software Vulkan baseline established; physical GPU pending-hardware qualified).
- Compare
u32with an optional UTF-8 prototype only after parity passes (32-bit scalar packing density validated across ASCII, CJK, and Emoji; CPU parity scan ~20ms/100k confirmed well within budget). - Select optimization work from measured memory/bandwidth results (global atomicAdd handles target throughput; zero-renorm cold upload and index stepping applied).
M6: migration, documentation, and release (breaking)
- Breaking v0.2 Migration Documentation (
docs/migration-v0.2.md&README.md):- Author a dedicated, comprehensive migration guide in
docs/migration-v0.2.mdand link it prominently in rootREADME.md. -
Index Rebuild & Binary Format Changes: Document requirement to rebuild v0.1 indexes (
u32scalar tokens instead of 8-bit bytes; token offsets instead of byte offsets); document binaryU2F2format (0x55324632, formatVersion: 2, CRC32, structured metadata headers); fail-closed behavior on legacy buffers withIncompatibleIndexError. Provide code snippets for rebuilding datasets, saving/loading U2F2 buffers viaserializeUnicodeDataset/deserializeUnicodeDataset, and reading telemetry viaindex.getStats(). -
Before / After Comparison Table: Document matching unit (
charCodeAtvscodePointAt), character sanitization (destructive NFKD/? vstoWellFormed+NFC+C/F fold), case insensitivity (ASCII-only vsCaseFolding-16.0.01,557 mappings), canonical equivalence (NFC), query length limits (59 silent truncation vs 128 fail-fast/fallback), VRAM accounting (byte fiction vs actualu32bytes), fuzzy semantics (uFuzzy rank vs integer subsequence parity), tie-break determinism, and deprecatedslotBytesthrow. -
TextProfile Call-Site Contract & Disclaimers: Clarify code points
$\neq$ grapheme clusters (referenceIntl.Segmenter); document 7-delimiter ASCII boundary set; state locale-neutral casing (no Turkic dotted/dotless I); disclaim transliteration/confusables; document"Straße"display-space highlighting slice caveat.
- Author a dedicated, comprehensive migration guide in
- README Overhaul & Benchmark Invalidation:
- Invalidate stale v0.1 ASCII-only benchmark numbers in root
README.md; replace with reproducible M5 multi-corpus benchmark tables (ASCII, CJK, Emoji) comparing WebGPU, CPU parity reference, uFuzzy, and JS Native. - Clearly disclose hardware qualification: distinguish physical GPU execution from software Vulkan (
pending-hardware). - Refresh API quick-start examples, TypeScript types, and configuration snippets to reflect v0.2 exports:
SearchIndex,WebGPUEngine,CPUEngine,packUnicodeToGPUBuffer,serializeUnicodeDataset, and error classes (QueryTooLongError,IncompatibleIndexError,ProfileMismatchError,IncompatibleOptionError).
- Invalidate stale v0.1 ASCII-only benchmark numbers in root
- Internal Deprecation & Monorepo Boundary Enforcement (
packages/webgpu-search):- Verify that the internal deprecated
legacy-ascii-v0.1path emits aconsole.warnnotifying users of removal in v0.3. - Audit codebase for 100% zero-DOM safety (
packages/webgpu-searchmust contain no unguardedwindowordocumentsymbols) to guarantee universal portability across browser main thread, Web Workers, Node.js, and SSR. - Ensure zero external runtime dependencies are introduced to
packages/webgpu-search.
- Verify that the internal deprecated
- Pre-Publish Release Pipeline & Quality Assurance Gates (
.agents/skills/pre-publish/SKILL.md):-
Tarball Content & Leak Audit: Run
npm pack --dry-runinpackages/webgpu-search; verify only authorized files packaged (dist/,README.md,LICENSE); zero test fixtures or internal scripts. -
Bundle Size Budget Enforcement: Assert
dist/index.jsgzipped size remains$\le$ 22.5 KB (M1 budget ceiling). -
SemVer Version Bump & Changeset: Bump version to
0.2.0inpackages/webgpu-search/package.jsonand sync workspace dependencies. -
Full Monorepo Validation Gate: Execute
bun run check:shaders,bun run typecheck,bun run build,bun run test:mock,bun run test:parity,bun run test:browser, andbun run test:benchmark -- --fast.
-
Tarball Content & Leak Audit: Run
- Git Release Tagging & Issue Closure:
- Create annotated Git release tag
v0.2.0. - Update
ISSUE-7-PLAN.mdand GitHub Issue v0.2: design Unicode code-point-safe CPU/GPU matching #7 to 100% complete, closing Issue v0.2: design Unicode code-point-safe CPU/GPU matching #7. - Publish GitHub release notes detailing the breaking Unicode migration, benchmark numbers, and architectural guarantees.
- Create annotated Git release tag
Acceptance criteria
- No valid Unicode scalar is replaced with
?or split as a UTF-16 surrogate pair in the default profile. - Canonically equivalent inputs normalize to identical token streams.
- Full default case-fold fixtures pass for the pinned Unicode version.
- CPU and WebGPU return identical ordered indices and scores for the same profile, mode, query, limit, and corpus.
- GPU failure/device loss changes only
response.engine, not match semantics. - No query is silently truncated; over-limit behavior is tested and documented.
- Combining marks and emoji/ZWJ code points survive preprocessing.
- Bengali, Devanagari, Arabic, CJK, supplementary-plane, malformed UTF-16, and mixed-script fixtures are covered.
- Memory statistics use actual packed bytes and device limits are checked before allocation.
- The public contract names its Unicode version/profile and does not claim grapheme-, locale-, transliteration-, or confusable-aware semantics.
- 0.1.x limitations and v0.2 rebuild/migration requirements are documented.
- Reproducible benchmark results include median/p95, memory, build/upload cost, environment metadata, and pending external hardware slots.
Non-goals for the first v0.2 release
- locale-sensitive collation or locale-specific case mappings;
- transliteration across scripts;
- stemming, tokenization by natural-language word, or spell correction;
- confusable/skeleton matching;
- grapheme clusters as atomic scoring units;
- Unicode-aware word-boundary bonuses beyond the explicitly documented delimiter set;
- claiming semantic equality across languages;
- guaranteeing a fixed FPS or latency across hardware.
References
- Unicode normalization, UAX Issue #7 M0/M1: Unicode contract, M0 spikes, profiled API #15: https://unicode.org/reports/tr15/
- Unicode full/default case-fold data: https://www.unicode.org/Public/UCD/latest/ucd/CaseFolding.txt
- Unicode text segmentation, UAX feat(filter): issue #10 M2 columnar attribute index & structured pre-filtering #29: https://www.unicode.org/reports/tr29/
- Unicode security and mixed-script/confusable guidance, UTS Fix audit findings: GPU pool, index integrity, worker/snapshot, API, CI #39: https://www.unicode.org/reports/tr39/
- ECMAScript text processing: https://tc39.es/ecma262/2026/multipage/text-processing.html
String.prototype.normalize(): https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/String/normalizeIntl.Segmenter: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Intl/Segmenter- WGSL: https://www.w3.org/TR/WGSL/
- Current CPU dependency source, uFuzzy: https://github.com/leeoniya/uFuzzy/blob/main/src/uFuzzy.js
- Example of diacritic-folding edge cases in another fuzzy-search library: feat: add option to ignore diacritics krisk/Fuse#773 and krisk/Fuse@5a01f29
Activity
Merge pull request #15 from codewarnab/feat/issue-7-m0-m1-contract
codewarnab commentedon Sep 17, 2026
M0/M1 landed via #15 (merge 57b6f50).
Done:
Next: M2 (unicode-preprocess.ts + cpu-reference.ts + generator + fixtures). No M2 start without M1 gate — gate is now this merge.
Known follow-up (not gating M2 start): CI browser gate still red on infra — vite arg forwarding in ci.yml browser-server step needs a working invocation. shaders/typecheck/build/mock are green.
Merge pull request #16 from codewarnab/feat/issue-7-m2-preprocess
codewarnab commentedon Sep 17, 2026
M2 landed via #16 (merge 1310d1a, CI green).
What shipped:
normalizeText()shared pipeline (trim → toWellFormed → NFC → C+F fold → NFC → u32) for records + queries; exact post-fold query gate +tokenCountCaseFolding-16.0.0table (C+F only: 1,453 C in 687 ranges + 104 F), frozen at load, zero-alloc single returnscpu-reference.tsparity scorer (substring + fuzzy, wrap-free two-key sort) mirroring the WGSL formulas; uFuzzy quarantined to explicitcpuAlgorithm:'ufuzzy'clampLimit(), abort-first ordering, DoS pre-gate,items.slice(), fail-closeddestroy(),preferGpu:falseforces CPU, Node+SSR-safe guards--check, deterministic bundle gate, M2 suite 98/98 (plus tr-TR run), mock + browser regression (Chrome setup fixed)Known M2 limits (documented in contract §1/§3, owned by M3):
Next: M3 engine swap (
packUnicodeToGPUBuffer()+ scalar WGSL + two-key GPU sort).fix(search): issue #7 M3 review hardening (fail-closed trust boundari…
Merge pull request #17 from codewarnab/feat/issue-7-m3-webgpu-swap
fix(search): issue #7 M4 review hardening (harness, worker, browser g…
Merge pull request #18 from codewarnab/feat/issue-7-m4-parity
codewarnab commentedon Sep 19, 2026
Completed in Milestone 6 (v0.2.0 release). All milestones M1 through M6 are fully delivered, validated, and tagged under v0.2.0.