Intern-Decision vision model on an M2: 0.9s per 1080p screenshot, and 4-bit wasn’t faster than bf16

[dex0shubham]

Intern-Decision-0.8B is Shanghai AI Lab’s multimodal decision model, where screenshots as well as typed questions go in and calibrated answers are produced. This repo runs it locally on Apple Silicon through mlx-vlm, with no change to the model.

My motivation to build this was to have a practical way to run a small vision decision model locally on Mac. Instead of treating it as a general-purpose chat model, I targeted useful confidence estimates and a simple interface.

pip install intern-decision-mlx
intern-decision-mlx decide --brief    # first run downloads 1.7 GB of weights

What I found while making the port

1. Image size → speed Image resolution matters a lot. A full-size 1080p screenshot becomes 2,357 tokens and takes about 3.6 s, while shrinking the long edge to 1024 px gives 893 tokens and about 0.9 s on an 8 GB M2 MacBook Air.
2. 4-bit isn’t automatically faster On a 512×384 screenshot with 3 questions, 4-bit took 466 ms versus 468 ms for bf16 on an M2. It also flipped a near-tie answer, so I kept bf16 as the default.
3. Only compute logits where they’re needed I use skip_logits=True, return_hidden=True, then run speculative_logits_from_hidden only at the positions where decisions are actually read. That gives the same probabilities while making inference about 1.2–1.4× faster.
4. The MLX port matches the reference Against the lab’s PyTorch reference, the port produced the same answer on all 67 fields, with a maximum probability difference of 0.023. The reference itself differs by 0.049 between bf16 and fp32, which puts the observed difference in the range expected from bf16 rounding.
5. Memory stays manageable Peak Metal memory was about 2.1 GB. The server also caps the MLX cache at 1 GB with mx.set_cache_limit.

Limitation:
The 0.8B model’s judgement can be weak: for instance, it said that a “Delete account” button was not destructive, so it should not be used as a safety gate. Currently, this implementation supports only the 0.8B model.

What I’d like to learn next:

What are the bounds of this small local decision model or how far it can be pushed before a larger model becomes necessary?

Thanks to Shanghai AI Lab for the model, and to the mlx-vlm project for making the Apple Silicon port possible.

• Repo: https://github.com/dex0shubham/intern-decision-mlx
• PyPI: https://pypi.org/project/intern-decision-mlx/
• Model: https://huggingface.co/internlm/Intern-Decision-0.8B
• mlx-vlm: https://github.com/Blaizzy/mlx-vlm