remove hardcoded file names and allow arbitrary models to be used

[BobdenOs]

Refactor: Replace Custom Tokenizer with HuggingFace Tokenizers and Support Dynamic Model File Discovery

This PR refactors the vector embedding pipeline to remove hardcoded file names, replace the hand-rolled WordPiece tokenizer with @huggingface/tokenizers, and switch the default model to perplexity-ai/pplx-embed-v1-0.6b.

Key Changes

model-utils.js

• Added @huggingface/tokenizers import; loadModelAndVocab now creates a proper Tokenizer instance instead of manually parsing a vocabulary map.
• Removed all custom tokenization helpers (preTokenize, wordPieceTokenize, validateTokenIds).
• Added fetchModelFiles(): dynamically discovers the required model files by first downloading onnx/model.onnx, then parsing the ONNX protobuf binary to detect any external data file references, and returning the full list of files to download (including tokenizer.json / tokenizer_config.json).
• downloadModelIfNeeded and forceRedownloadModel now preserve directory structure (use full relative paths instead of path.basename).
• Downloads inside downloadModelIfNeeded are now parallelized via Promise.all.
• downloadFile now creates intermediate directories automatically.

embedding.js

• Default model changed from Xenova/all-MiniLM-L6-v2 to perplexity-ai/pplx-embed-v1-0.6b.
• Removed the hardcoded FILES constant; file list is now fetched dynamically via fetchModelFiles().
• Replaced wordPieceTokenizer with tokenizeWithChunks, which delegates to the HuggingFace Tokenizer instance.
• vocab variable replaced with tokenizer.
• initializeModelAndVocab now accepts the files list as a parameter.

InferenceSession.js

• InferenceSession.create() now accepts a file path string in addition to Uint8Array, enabling the session to load the model directly from disk rather than reading it into memory first.

package.json

• Added @huggingface/tokenizers: ^0.1.3 as a dependency.

Category: Refactor 🔧

• Added relevant entry to the change log?

• 🔄 Regenerate and Update Summary

PR Bot Information

Version: 1.29.26

• LLM: anthropic--claude-4.6-sonnet
• File Content Strategy: Full file content
• Output Template: Repository PR Template
• Correlation ID: d8a8ac30-97df-11f1-834d-db338db31e22
• Summary Prompt: Default Prompt
• Event Trigger: pull_request.opened