Skip to content

fix(cli): don't auto-attach an architecture-incompatible GGUF projector - #2439

Open
ermolushka wants to merge 1 commit into
EricLBuehler:masterfrom
ermolushka:fix/gguf-mmproj-arch-mismatch-2421
Open

ermolushka wants to merge 1 commit into
EricLBuehler:masterfrom
ermolushka:fix/gguf-mmproj-arch-mismatch-2421

Conversation

@ermolushka

Copy link
Copy Markdown

Problem

Directory-scan mmproj auto-discovery in serve/run only matched on filename ("looks like a projector"), never on whether that projector's architecture actually pairs with the model being served. A non-related mmproj*.gguf belonging to a different (vision) model in the same flat directory would silently get attached to any unrelated text-only GGUF, and the load would then hard-fail deep inside the multimodal loader instead of just serving the text model. With multiple unrelated projectors present, discovery hard-errored as "ambiguous" instead of ignoring them.

Fixes #2421.

  • Added mistralrs_core::gguf_architecture_accepts_projector(architecture, vision_projector_type, audio_projector_type), reusing the same acceptance logic (multimodal_vision_registry::family_from_names, widened to pub(crate), plus the direct Gemma4/Qwen-VL dispatch architectures) the native multimodal loader itself uses, so the prediction can't drift from what the loader actually accepts.
  • Added filter_projector_candidates in the CLI's GGUF discovery: reads the local model's own general.architecture and each candidate projector's clip.vision/audio.projector_type header (metadata-only, no weight data), and drops incompatible candidates before projector resolution runs. Fails open (previous behavior) when the model isn't local or its architecture can't be read.
  • Wired the filter into apply_quant_resolution, skipped when ModelType::Multimodal was explicitly requested (the user already asserted multimodal intent). Logs a warn! naming how many candidates were ignored.
  • Added --mmproj none as an explicit opt-out, via a new mmproj_disabled flag on FormatOptions.

Testing

  • Added local_gguf_ignores_unrelated_projector_architecture, local_gguf_still_attaches_a_genuinely_compatible_projector, local_gguf_incompatible_projector_does_not_cause_ambiguity, and mmproj_none_disables_auto_selection_even_with_a_compatible_projector in mistralrs-cli/src/commands/serve.rs, using real (metadata-only) GGUF files written via candle's own GGUF writer, not filename stubs.
  • Added projector_compat_tests (4 tests) for gguf_architecture_accepts_projector in mistralrs-core/src/gguf/mod.rs.
  • Verified the new tests are working by temporarily disabling the filter and confirming both regressed to the issue's exact symptoms (wrong auto-attach; "ambiguous" hard error), then restored.
  • Reproduced end-to-end against the real built binary with real (metadata-only) GGUF files on disk, reproducing the issue's exact command and log lines pre-fix, and confirming post-fix it warns and proceeds text-only.
  • cargo test -p mistralrs-core -p mistralrs-cli (1519 passing)
  • cargo clippy -p mistralrs-core -p mistralrs-cli --tests -- -D warnings clean
  • cargo fmt --all -- --check clean
  • Regenerated CLI reference docs (serve.md, run.md, tune.md, bench.md) for the updated --mmproj help text

@github-actions

Copy link
Copy Markdown
Code Metrics Report
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Language              Files        Lines         Code     Comments       Blanks
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 C Header                 46         9600         7138         1327         1135
 C++                       1           36           30            1            5
 C++ Header                6         4022         3237          399          386
 CSS                       5         1069          918            6          145
 CUDA                    143        38519        32957         1775         3787
 Dockerfile                1           35           19            9            7
 HTML                      3          125          119            0            6
 JavaScript                8         1241         1186           11           44
 Jinja2                    7          694          656            5           33
 JSON                     34        27568        27565            0            3
 Makefile                  1           18           16            0            2
 MDX                      37         6974            0         5156         1818
 Metal Shading Lan|       37        14422        11414         1136         1872
 PowerShell                1          657          571           31           55
 Python                  161        34938        31448          487         3003
 Shell                     3         1071          852          115          104
 Plain Text               54        10714            0         9231         1483
 TOML                     28         1399         1212           43          144
 TypeScript               11         1658         1418           66          174
 YAML                      3           25           23            2            0
─────────────────────────────────────────────────────────────────────────────────
 Jupyter Notebooks         3          122           83           23           16
 |- Markdown               1           60           30           22            8
 |- Python                 1          122          113            1            8
 (Total)                              304          226           46           32
─────────────────────────────────────────────────────────────────────────────────
 Markdown                283        13044            0         9784         3260
 |- BASH                  27          349          268           48           33
 |- Dockerfile             2           14           12            0            2
 |- JSON                   6          292          292            0            0
 |- PowerShell             1            1            1            0            0
 |- Python               135         7349         6119          306          924
 |- Rust                  63         3873         2873          398          602
 |- TOML                   7          116           97            0           19
 (Total)                            25038         9662        10536         4840
─────────────────────────────────────────────────────────────────────────────────
 Rust                    772       500427       456580         5582        38265
 |- Markdown             443        10396          452         8755         1189
 (Total)                           510823       457032        14337        39454
─────────────────────────────────────────────────────────────────────────────────
 Svelte                   19         1974         1832           50           92
 |- CSS                    1            4            4            0            0
 |- JavaScript            19          928          773           25          130
 (Total)                             2906         2609           75          222
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
 Total                  1667       693856       590308        44794        58754
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━

@santanu20

Copy link
Copy Markdown

Tested on the exact repro from #2421: text-only qwen2.5-0.5b-instruct GGUF + an unrelated Qwen3.5-9B mmproj-F16.gguf (clip) in the same flat directory, built from this branch (Linux, CPU path). Pre-fix this hard-crashes in the multimodal loader; with this PR the projector is skipped with the new warning ("…ignoring them and loading text-only. Pass --mmproj <file>…"), the model loads through the native qwen2 text pipeline, the server comes up, and chat completions work. Fix confirmed — thanks!

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

serve: projector auto-discovery binds the directory's only mmproj to ANY model — crashes text-only GGUFs (arch not supported by multimodal loader)

2 participants