Skip to content

fix(chat-template): bind pad_token into chat template context - #2433

Open
ermolushka wants to merge 1 commit into
EricLBuehler:masterfrom
ermolushka:fix/pad-token-chat-template-context
Open

ermolushka wants to merge 1 commit into
EricLBuehler:masterfrom
ermolushka:fix/pad-token-chat-template-context

Conversation

@ermolushka

Copy link
Copy Markdown

Problem

ChatTemplate deserializes pad_token from tokenizer_config.json, but only
bos_token, eos_token, and unk_token are forwarded into the MiniJinja
render context in apply_chat_template. Any chat template that references
pad_token (e.g. {% if pad_token is defined %}...{% endif %}) always sees
it as undefined, diverging from Transformers, which binds it.

Fixes #2432.

  • Added a pad_tok() accessor on ChatTemplate, mirroring the existing
    bos_tok() / eos_tok() / unk_tok().
  • Added a pad_tok parameter to apply_chat_template_to and bound
    it into the MiniJinja context as pad_token, together with the other special
    tokens, in both the tools and no-tools render branches.
  • Extracted pad_tok from chat_template.pad_token in processing.rs
    the same way the other three tokens already are.

Testing

  • Added pad_token_is_bound_into_template_context and
    pad_token_is_bound_in_chat_template_context (the latter matching the
    regression test from the issue) in mistralrs-core/src/pipeline/chat_template.rs.
  • cargo test -p mistralrs-core --lib pipeline::chat_template (22/22 passing)
  • cargo clippy --workspace --tests --examples -- -D warnings clean
  • cargo fmt --all -- --check clean

@github-actions

Copy link
Copy Markdown
Code Metrics Report
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Language              Files        Lines         Code     Comments       Blanks
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 C Header                 46         9600         7138         1327         1135
 C++                       1           36           30            1            5
 C++ Header                6         4022         3237          399          386
 CSS                       5         1069          918            6          145
 CUDA                    143        38519        32957         1775         3787
 Dockerfile                1           35           19            9            7
 HTML                      3          125          119            0            6
 JavaScript                8         1241         1186           11           44
 Jinja2                    7          694          656            5           33
 JSON                     34        27568        27565            0            3
 Makefile                  1           18           16            0            2
 MDX                      37         6974            0         5156         1818
 Metal Shading Lan|       37        14422        11414         1136         1872
 PowerShell                1          657          571           31           55
 Python                  161        34938        31448          487         3003
 Shell                     3         1071          852          115          104
 Plain Text               54        10714            0         9231         1483
 TOML                     28         1399         1212           43          144
 TypeScript               11         1658         1418           66          174
 YAML                      3           25           23            2            0
─────────────────────────────────────────────────────────────────────────────────
 Jupyter Notebooks         3          122           83           23           16
 |- Markdown               1           60           30           22            8
 |- Python                 1          122          113            1            8
 (Total)                              304          226           46           32
─────────────────────────────────────────────────────────────────────────────────
 Markdown                283        13044            0         9784         3260
 |- BASH                  27          349          268           48           33
 |- Dockerfile             2           14           12            0            2
 |- JSON                   6          292          292            0            0
 |- PowerShell             1            1            1            0            0
 |- Python               135         7349         6119          306          924
 |- Rust                  63         3873         2873          398          602
 |- TOML                   7          116           97            0           19
 (Total)                            25038         9662        10536         4840
─────────────────────────────────────────────────────────────────────────────────
 Rust                    772       500427       456580         5582        38265
 |- Markdown             443        10396          452         8755         1189
 (Total)                           510823       457032        14337        39454
─────────────────────────────────────────────────────────────────────────────────
 Svelte                   19         1974         1832           50           92
 |- CSS                    1            4            4            0            0
 |- JavaScript            19          928          773           25          130
 (Total)                             2906         2609           75          222
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Total                  1667       693856       590308        44794        58754
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

pad_token from tokenizer_config.json is parsed but never bound into the chat-template context

1 participant