[Feature Request] Auto-switch to a vision-capable model (e.g. `gemini-3.5`) when an image is uploaded

[Rosecheng7]

Harness: dsh 0.1.0-rc.6 (dsh web profile)
Environment: macOS (Apple Silicon), Node v25.9.0

Summary

When a user uploads an image in the web UI, the harness does not automatically switch to a vision-capable model. The current model (e.g. a text-only model) stays selected, so any subsequent request that references the image either fails or produces a useless response — the user has to manually switch to a model that can see images (e.g. gemini-3.5) before the image is actually usable.

Current behavior

1. User is on a text-only model
2. User uploads an image via the UI.
3. The model stays unchanged; the image attachment is effectively ignored (or the request errors out).
4. User must notice the failure, manually open the model picker, and select a vision-capable model like gemini-3.5, then retry.

Expected behavior

• When an image is uploaded, the harness automatically switches to a vision-capable model (e.g. gemini-3.5) — or at minimum shows a clear, one-click prompt: "This model can't see images. Switch to gemini-3.5?"
• The switch should be non-destructive to the current conversation: keep the conversation history and the uploaded image, just swap the model for subsequent turns.
• If multiple vision-capable models are configured, pick the first available one (or the one the user last used), and surface a hint in the UI showing which model was auto-selected.
• Conversation history and the uploaded image are preserved across the switch.
• The UI indicates which model is now active after the auto-switch.
• Users can opt out of auto-switching if they prefer manual control.

[VickyXAI]

Real problem, but the half of it that belongs to this plugin is already shipped, and the other half is the harness's model picker rather than the route.

What this plugin owns: telling the truth about which models can see. Every model this route publishes carries inputModalities, and the image modality is declared from a measured list, not the gateway's vision tag. Ten tagged models were sent the same inline PNG: Google's and Moonshot's answered correctly, OpenAI's returned HTTP 400 after taking payment, xAI's returned 503, and Anthropic's charged and streamed nothing at all. So the verified set is:

google/gemini-2.5-flash
google/gemini-3.5-flash
google/gemini-3.6-flash
moonshot/kimi-k3

(visionModels widens this without waiting for a release, as the gateway improves.) Note that gemini-3.5 is not a model id here — it is google/gemini-3.5-flash.

What this plugin cannot do: change the session's model. Selecting a model is a DSH decision, and a plugin reaching across to swap it mid-conversation would be the wrong place for that logic. The information the picker needs is already on the models it lists, so the UI can gate the upload or offer the switch without any further work on this side.

On auto-switching itself, I would push back on the "automatically" part. Changing the model changes the price, the tone, the tool behaviour, and the reasoning support of every subsequent turn — silently, in response to a drag-and-drop. Your fallback wording is the better default: "This model can't see images. Switch to google/gemini-3.5-flash?" — one click, conversation and attachment preserved, and the active model shown afterwards. That is exactly what your own expected-behaviour list asks for at minimum, and I think it should be the maximum too, with opt-in auto-switch for people who want it.

The right home for the switch is the DSH web UI (deepseek-ai/deepseek-harness), since it owns the picker, the attachment surface, and the session state. Worth filing there — the modality data it would need is already published by this route, which makes it a UI change rather than a capability one. Happy to be pointed at that issue if you open it.

[VickyXAI]

Followed up upstream, and checked the harness source while I was there — the current behaviour is better than described, but your ask still stands.

DSH does not silently ignore the attachment. It refuses the send:

• packages/host/apiproxy/src/api-proxy.ts:2488 returns attachment-error / MODEL_DOES_NOT_SUPPORT_IMAGES when the active model's inputModalities lacks image
• packages/host/apiproxy/src/api-proxy.ts:2299 returns model-unavailable if you switch to a text-only model while the session already holds images
• packages/fs/tool-fs/src/read-image.ts:73 refuses the read_image tool on the same basis

So the image is never quietly dropped. What is missing is exactly the second half of your request: the refusal never names a model that would work, and there is no one-click switch — which is why it reads as "the harness did nothing useful."

Filed there: deepseek-ai/deepseek-harness#2911 — proposing the refusal carry candidate image-capable models in its details, the web UI render that as "switch to X and resend" with history and attachment preserved, and the model picker mark image-capable entries. Your issue is linked in it. I argued against silent auto-switching as the default for the reason in my earlier comment; a one-click confirm gets you the same result without changing price and behaviour behind a drag-and-drop.

Leaving this open until the upstream thread lands. The data side is already done on this route: inputModalities is declared from the measured set, so anything the picker needs is published today.

[VickyXAI]

Re-measured today, and the half of this that lives in the plugin had gone stale. Fixed in dsh-clawrouter@0.10.2.

The same inline PNG, sent through the gateway to every chat+vision-tagged model, three runs each: 31 of 37 answer correctly now, against 4 of 10 on 2026-08-16. Every Anthropic, Google, Moonshot, xAI and Z.ai model works, and so does OpenAI's non-pro line plus gpt-5.6-sol-pro / gpt-5.6-terra-pro. The gateway fixed all three failure modes from the first probe.

Since the harness refuses an attachment for any model whose inputModalities lacks image, the four-model default is exactly what made "only four models can see" true for you. 0.10.2 declares image input on all thirty-one; visionModels still narrows or widens it per install.

Still excluded, because each failure is billed: openai/gpt-5.2-pro, gpt-5.4-pro, gpt-5.5-pro drop the image and answer as if none was sent, gpt-5.6-luna-pro returns 500 after payment, and both NVIDIA Nemotron entries answer wrongly or rate-limit. Filed against the gateway as BlockRunAI/blockrun#443.

npm install dsh-clawrouter@0.10.2

The switch-on-upload itself is still the harness's (deepseek-ai/deepseek-harness discussion #2911, open). Leaving this open until that lands.

[VickyXAI]

Re-measured again today. 34 of 40 now, up from 31 of 37 on 2026-08-30 — shipped in dsh-clawrouter@0.10.5. More usefully, the measurement is a script now, so you can check it yourself against your own wallet rather than waiting for me:

npm run probe:vision              # measures the tagged models not yet in the list
npm run probe:vision -- --all     # re-measures everything

It prints the visionModels array ready to paste. Free models cost nothing to probe; paid ones are a fraction of a cent each and it prints the estimate before sending anything.

Newly working, now offered image input: deepseek/deepseek-v4-flash-vision-exp, xiaomi/mimo-v2.5 (both new to the catalog), and openai/gpt-5.6-luna-pro — which used to return HTTP 500 after taking payment and which the gateway has fixed.

The three gpt-*-pro entries still cannot take an image, but they now fail INVALID_REQUEST instead of silently dropping it and billing you for a plausible answer. That is the more important half of the change for anyone who was hitting them: a refusal you can see beats a wrong answer you cannot.

Two things I got wrong or could not see before, both worth stating since this issue is where the list's accuracy lives:

1. The NVIDIA models are not the problem — we are. I had written nvidia/llama-3.2-11b-vision up as a model that cannot see, on the strength of it answering "you didn't provide an image" three times out of three. That answer is simply correct: the gateway strips every image part on both its NVIDIA paths before dispatch, unconditionally, for every NVIDIA model, under a comment asserting NVIDIA models have no vision. Probed directly against NVIDIA, that model and nemotron-3-nano-omni both name a test image's colour correctly. The gateway maintainers have the fix deploying now. Those two stay out of the default until it lands — this list is what works through the gateway — but the fault was mis-attributed, and the streaming path drops the image with no log line at all, which is why it took reading the source to see.

2. A model can answer as a different model, which looks exactly like a flaky vision model. nvidia/nemotron-3-nano-omni-30b-a3b-reasoning passed my three-colour test on some runs and failed it on others. The failures were not the model: the gateway's fallback answered as nvidia/nemotron-3-nano-30b, which has no vision at all. Same for qwen/qwen3.8-flash, answered by qwen3.7-flash. Since 0.10.4 this route reads the served model off each response chunk, so /spend now names the substitute under the model you asked for:

nvidia/nemotron-3-nano-omni-30b-a3b-reasoning  $0  3 calls  (free tier — no payment was signed)
    answered by nvidia/nemotron-3-nano-30b on 1 of 3 — the gateway substituted a different model

The probe applies the same rule: a correct answer from a substituted model is not a pass, because it says nothing about the model being measured. Tracked upstream as BlockRunAI/blockrun#450.

Also, since it bears on trying any of this: as of 0.10.3 the models the catalog bills as free need no wallet at all — the gateway serves them unpaid, so this route no longer asks for a key before sending one. Previously it did, which made "no accounts, no API keys" false for exactly the models that deliver it. nemotron-3-nano-omni is the free tier's only vision model, which is why its substitution matters more than the others.

The harness half of your original request — auto-switching, or a refusal that names a model which would work — is still upstream at deepseek-ai/deepseek-harness#2911 and unchanged. Everything above only makes the refusal correct more often, not friendlier.

[VickyXAI]

One correction to the comment above, because it sets the wrong expectation for the three models still excluded.

I wrote that openai/gpt-5.6-luna-pro "used to return HTTP 500 after taking payment and the gateway has fixed it." It does work now, but that was never an image failure. It was unroutable: the catalog priced it at $0.10/$0.60 after OpenAI moved it to $0.20/$1.20, and the gateway derives its upstream cost ceiling from the catalog price — so every request matched no endpoint at all, and since that model declares no fallback, the call failed outright. Repricing it to the real upstream numbers fixed it. The image had been working the whole time.

So gpt-5.2-pro, gpt-5.4-pro and gpt-5.5-pro are not waiting on the same thing, and should not be assumed fixed alongside it. Their image handling is a genuinely separate change which is still open upstream. When it lands, npm run probe:vision -- --models=openai/gpt-5.2-pro,openai/gpt-5.4-pro,openai/gpt-5.5-pro re-measures all three for about a cent, and if they pass they go straight into visionModels without waiting for a release here — that config key exists precisely so you do not have to.

Worth knowing generally, since it explains a shape you may hit again: a model can be listed, tagged correctly, and completely unreachable, because the price in the catalog is what decides which upstream providers the gateway is allowed to use. Nothing in the error says so.