[Bug][NPU] u2 (2-bit) weights: MatMul wrong for 2+ activation rows on NPU40XX/50XX (breaks 2-bit LLM prefill) - root cause and one-line fix

OpenVINO Version

2026.5.0 (2026.5.0-23160-6db412d0a27), NPU compiler 8.4 (pip) and npu_compiler 20e7912 (8.3)

Operating System

Windows 11 (Panther Lake, NPU 5010, driver 1005540); Ubuntu 26.04 (Lunar Lake, NPU 4000, driver 1.38.0)

Device used for inference

NPU

Describe the problem

On NPU40XX / NPU50XX a MatMul with u2 (2-bit) constant weights is right for one activation row and wrong for two
or more rows (zeros / garbage). It breaks every 2-bit LLM prefill on the NPU (e.g. Ternary Bonsai 2 from Prism ML),
while CPU and GPU already handle u2 weights.

Max rel. error against fp32, y = x[M, 4096] @ W^T, W = (u2 codes - 1) * scale, Panther Lake:

	M = 1	M = 2	M = 4	M = 16
stock compiler, scale per row	0.0004	1.0175	1.0000	23179.9
stock compiler, scale per group of 128	0.0005	1.0000	1.0000	26074.1
with the fix below, scale per row	0.0004	0.0004	0.0003	0.0003
with the fix below, scale per group of 128	0.0005	0.0014	0.0019	0.0016

Repro script and details: openvinotoolkit/npu_compiler#355 (no reply there yet, so we are raising it here too).

Root cause and a one-line fix

finishDynamicDequantizeRewrite() in the IE pass weights_dequantize_to_dynamic_dequantize.cpp marks the
DynamicDequantize for the DynDeq -> FakeQuantize bridge only when isKVcachedPattern() || !zpElemType.isInteger(2).
A 2-bit zero point in a multi-row (prefill) graph stays unmarked and runs on the dynamic_dequantize SHAVE kernel,
which returns zeros for u2 inputs. Marking it always sends it down the same FakeQuantize -> group unroll -> palletized
LUT path as one-row graphs (still 2-bit storage): correct for every M, and M = 64 is as fast as INT4.

The change (plus the LIT expectation of U2WeightsConstPrefillPatternToDynamicDequantize) is on
ryugyosoft-max/npu_compiler@develop...ryugyosoft-max:fix-u2-multirow-dyndeq (6 lines,
on top of 20e7912). That test pins the unmarked route, so the SHAVE route for u2 prefill looks intentional - if the
kernel is meant to handle u2 (newer kernel / driver?), please say so; with the kernels shipped today it gives wrong
results.

Verified on a real model: Ternary Bonsai 2 4B / 9B, 64-token prefill graphs vs token-by-token decoding over 256
positions: mean KL 0.0005, top-1 agreement 98.4%, the same NLL, on both NPUs; prefill 25 -> ~300 tok/s (4B, Panther
Lake). It has shipped in onw (an LLM runtime for Intel NPUs) since 3.0.

Related NPU compiler reports (openvinotoolkit/npu_compiler, unanswered so far)

fix log error #379 group-wise scales cost a 16-byte weight-table row per output channel and group (1 bit per weight per graph):
a 2-bit 27B needs 13.8 GB of NPU memory with decode + prefill graphs
How do you build inference engine dll in such a way all its dependencies are in one folder? In other words, I won't need to run setupvars.bat everytime I want to use the dll... #376 weightless compilation fails or keeps the weights for NF4, INT4 and 2-bit models
Device with "CPU" name is not registered in the InferenceEngine #375, YOLOv3: Loading Network to the Device #374, AWS IoT Greengrass cannot find myriad #373, Fix error (libpng-0) #378 compile-time issues (2-bit 64-token part: 335 s -> ~20 s with fixes, same binary)
Memory leak when using Python API #377 NPU 3720 weightless INT4

Thank you!

@ryugyosoft-max

Hi @ryugyosoft-max

I'm using Lunar Lake to test and was able to see the same output:

OpenVINO 2026.5.0-23569-178d03e2104 | NPU 4000 | compiler 0x80003 | driver 1789049546
scale per row          M= 1  max rel err 0.0004
scale per row          M= 2  max rel err 1.0000   <-- wrong
scale per row          M= 4  max rel err 1.0000   <-- wrong
scale per row          M=16  max rel err 23179.8281   <-- wrong
scale per group of 128  M= 1  max rel err 0.0005
scale per group of 128  M= 2  max rel err 1.0000   <-- wrong
scale per group of 128  M= 4  max rel err 1.0000   <-- wrong
scale per group of 128  M=16  max rel err 26074.1016   <-- wrong

The u2-weight MatMul produces near-correct results for a single activation row (M=1) but incorrect numerical results when processing multiple rows (M=2, M=4, and M=16).

I'll run another test using the patch and let you know the result.

Thanks for reproducing it! The fix is the one-line patch 01_u2_multirow_zeros.patch in the gist (against npu_compiler 20e7912): https://gist.github.com/ryugyosoft-max/bb2f223987db3dae11d680bf5421cd50#file-01_u2_multirow_zeros-patch - with it, the same script gives max rel. errors around 0.0003-0.002 for every M (table in the issue). openvinotoolkit/npu_compiler#355 is the same bug, so this issue can be the one to track it.

Some context on why we care: we are working on running 2-bit LLMs (Ternary Bonsai 2 from Prism ML, 4B / 9B / 27B) on the NPU, and this bug blocks their prompt processing. Even with the fix, though, there is a second problem in the way: openvinotoolkit/npu_compiler#379. With group-wise scales, every compiled graph carries a weight table of 16 bytes per output channel and group of 128 (1 bit per weight, per graph), which weightless compilation cannot separate or share. As soon as the decode and prefill graphs are both loaded, the NPU memory is about twice the 2-bit weights:

Ternary Bonsai 2	2-bit codes + scales	weight tables (1-token + 64-token graphs)	NPU memory
9B	~2.1 GB	~2.2 GB	4.47 GB
27B	~6.8 GB	~6.8 GB	13.84 GB

So the 27B does not fit a 16 GB laptop with prompt graphs, although its weights are 6.8 GB (details and what we tried in #379). If possible, could someone on the compiler side take a look at that one too? Thanks!

@ryugyosoft-max

Hi @ryugyosoft-max
The latest test results show that the previously observed multirow numerical error is no longer reproducible in the tested environment. Both scale configurations produce results close to the reference across all tested matrix sizes.

The results support marking the numerical reproduction as successful for this test configuration. I refer this case to the dev team for further validation.

Great, thanks for verifying it and passing it on to the dev team!

@ryugyosoft-max

Hi @ryugyosoft-max
Since you have identified a potential solution, would you be willing to submit a pull request (PR) with the proposed fix?

This would let the dev team review the implementation, validate the results, and work with you on any necessary changes before merging.

We appreciate your contribution and look forward to your PR.

Sure - here it is: openvinotoolkit/npu_compiler#390

It marks constant 2-bit weights for the FakeQuantize bridge in multi-row graphs too (the same result as the patch you
tested), and it keeps the weights-as-inputs prefill pattern as it is, so the existing LIT tests still pass. Thanks!

Navigation Menu

Repository navigation

[Bug][NPU] u2 (2-bit) weights: MatMul wrong for 2+ activation rows on NPU40XX/50XX (breaks 2-bit LLM prefill) - root cause and one-line fix #39010

OpenVINO Version

Operating System

Device used for inference

Describe the problem

Root cause and a one-line fix

Related NPU compiler reports (openvinotoolkit/npu_compiler, unanswered so far)

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Bug][NPU] u2 (2-bit) weights: MatMul wrong for 2+ activation rows on NPU40XX/50XX (breaks 2-bit LLM prefill) - root cause and one-line fix #39010

Description

OpenVINO Version

Operating System

Device used for inference

Describe the problem

Root cause and a one-line fix

Related NPU compiler reports (openvinotoolkit/npu_compiler, unanswered so far)

Activity

Zulkifli-Intel commented on Oct 26, 2026

ryugyosoft-max commented on Oct 26, 2026

Zulkifli-Intel commented on Oct 27, 2026

ryugyosoft-max commented on Oct 28, 2026

Zulkifli-Intel commented on Oct 28, 2026

ryugyosoft-max commented on Oct 28, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions