fix(cli): don't auto-attach an architecture-incompatible GGUF projector - #2439
Open
ermolushka wants to merge 1 commit into
Open
ermolushka wants to merge 1 commit into
ermolushka wants to merge 1 commit into
Conversation
Code Metrics Report━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Language Files Lines Code Comments Blanks ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ C Header 46 9600 7138 1327 1135 C++ 1 36 30 1 5 C++ Header 6 4022 3237 399 386 CSS 5 1069 918 6 145 CUDA 143 38519 32957 1775 3787 Dockerfile 1 35 19 9 7 HTML 3 125 119 0 6 JavaScript 8 1241 1186 11 44 Jinja2 7 694 656 5 33 JSON 34 27568 27565 0 3 Makefile 1 18 16 0 2 MDX 37 6974 0 5156 1818 Metal Shading Lan| 37 14422 11414 1136 1872 PowerShell 1 657 571 31 55 Python 161 34938 31448 487 3003 Shell 3 1071 852 115 104 Plain Text 54 10714 0 9231 1483 TOML 28 1399 1212 43 144 TypeScript 11 1658 1418 66 174 YAML 3 25 23 2 0 ───────────────────────────────────────────────────────────────────────────────── Jupyter Notebooks 3 122 83 23 16 |- Markdown 1 60 30 22 8 |- Python 1 122 113 1 8 (Total) 304 226 46 32 ───────────────────────────────────────────────────────────────────────────────── Markdown 283 13044 0 9784 3260 |- BASH 27 349 268 48 33 |- Dockerfile 2 14 12 0 2 |- JSON 6 292 292 0 0 |- PowerShell 1 1 1 0 0 |- Python 135 7349 6119 306 924 |- Rust 63 3873 2873 398 602 |- TOML 7 116 97 0 19 (Total) 25038 9662 10536 4840 ───────────────────────────────────────────────────────────────────────────────── Rust 772 500427 456580 5582 38265 |- Markdown 443 10396 452 8755 1189 (Total) 510823 457032 14337 39454 ───────────────────────────────────────────────────────────────────────────────── Svelte 19 1974 1832 50 92 |- CSS 1 4 4 0 0 |- JavaScript 19 928 773 25 130 (Total) 2906 2609 75 222 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ Total 1667 693856 590308 44794 58754 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ |
|
Tested on the exact repro from #2421: text-only |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Directory-scan mmproj auto-discovery in serve/run only matched on filename ("looks like a projector"), never on whether that projector's architecture actually pairs with the model being served. A non-related mmproj*.gguf belonging to a different (vision) model in the same flat directory would silently get attached to any unrelated text-only GGUF, and the load would then hard-fail deep inside the multimodal loader instead of just serving the text model. With multiple unrelated projectors present, discovery hard-errored as "ambiguous" instead of ignoring them.
Fixes #2421.
mistralrs_core::gguf_architecture_accepts_projector(architecture, vision_projector_type, audio_projector_type), reusing the same acceptance logic (multimodal_vision_registry::family_from_names, widened topub(crate), plus the direct Gemma4/Qwen-VL dispatch architectures) the native multimodal loader itself uses, so the prediction can't drift from what the loader actually accepts.filter_projector_candidatesin the CLI's GGUF discovery: reads the local model's owngeneral.architectureand each candidate projector'sclip.vision/audio.projector_typeheader (metadata-only, no weight data), and drops incompatible candidates before projector resolution runs. Fails open (previous behavior) when the model isn't local or its architecture can't be read.apply_quant_resolution, skipped whenModelType::Multimodalwas explicitly requested (the user already asserted multimodal intent). Logs awarn!naming how many candidates were ignored.--mmproj noneas an explicit opt-out, via a new mmproj_disabled flag on FormatOptions.Testing
local_gguf_ignores_unrelated_projector_architecture,local_gguf_still_attaches_a_genuinely_compatible_projector,local_gguf_incompatible_projector_does_not_cause_ambiguity, andmmproj_none_disables_auto_selection_even_with_a_compatible_projectorinmistralrs-cli/src/commands/serve.rs, using real (metadata-only) GGUF files written via candle's own GGUF writer, not filename stubs.projector_compat_tests(4 tests) forgguf_architecture_accepts_projectorinmistralrs-core/src/gguf/mod.rs.cargo test -p mistralrs-core -p mistralrs-cli(1519 passing)cargo clippy -p mistralrs-core -p mistralrs-cli --tests -- -D warningscleancargo fmt --all -- --checkcleanserve.md,run.md,tune.md,bench.md) for the updated--mmprojhelp text