[kerry0520]

Description

Legacy Huggingface text vocabularies use raw lines for tokens and merge rules. On POSIX, getline removes LF but leaves a CRLF record's CR in both fields. The factory can return an object, then encoding fails because the expected vocabulary keys are missing. Windows text-mode streams already handle these record endings.

For example, this small legacy vocabulary reproduces the issue when loaded by the production factory on Android ARM64:

from pathlib import Path
rows = [b"430 3", b"0 0 0", b"", b"3 1", b"h", b"e", b"he", b"h e"]
Path("hf-crlf.txt").write_bytes(b"\r\n".join(rows) + b"\r\n")

Encoding he should return [2]. The current Android parser retains the record CRs and aborts during encoding; the LF control and Windows parser return [2].

Use the numeric count header to identify consistent CRLF records, then remove exactly one terminal record CR from vocabulary and merge lines. Keep LF-mode content and the last unterminated record unchanged. This only changes HuggingfaceTokenizer::load_vocab; it keeps the existing text stream, factory failure behavior, other formats and public interfaces. It is independent of #5034 and does not change the Pipeline binary-loading fix in #4691.

Validation:

• Compiled the production tokenizer sources with MSVC x64 and Android ARM64 NDK r30, linked against existing MNN core libraries. These are tokenizer/factory tests, not a fresh full-engine build or model-inference test.
• Reproduced the existing CRLF Qwen vocabulary failure on Android; fixed CRLF/LF inputs produce identical IDs and decoded bytes on both platforms for ASCII, Chinese, embedded CR and trailing CR inputs.
• Checked small HF fixtures for LF/CRLF, payload CR, CRCRLF, unterminated final records and merge behavior. Windows baseline/fixed results remain identical. Raw-token storage is checked by a local diagnostic; it does not add a public API or new decoding capability.
• Limited Sentencepiece special-token, Tiktoken, BERT and Pipeline smoke controls pass. This is not full coverage of those formats.

The protocol assumes consistent record endings. Mixed endings, general malformed-resource validation, I/O faults, macOS/Linux desktop builds, model quality and performance are not covered. No new test-framework changes or temporary reproduction tools are included.

Module

LLM

Type

• Feature
• Bugfix
• Perf
• Refact
• Style
• Doc
• Test
• Chore

Checklist

• Commit message follows [Module:Type] Description format
• Code compiles without errors
• Tested on relevant platform(s)
• No unrelated format or style changes included