[ryugyosoft-max]

Details

Fixes the multi-row 2-bit weights bug of openvinotoolkit/openvino#39010 (also reported as #355).

WeightsDequantizeToDynamicDequantize marks the new IE.DynamicDequantize for the DynDeq -> FakeQuantize bridge pass
only for the KV-cached (one-row) pattern when the weights are 2-bit. A constant 2-bit weight used by a MatMul /
FullyConnected with M >= 2 rows (the prompt / prefill graph of an LLM) is therefore left unmarked and is dequantized
by the DynamicDequantize SW kernel, which returns zeros for these weights on NPU4000 and NPU5010: the layer output is
wrong (max. relative error 1.0 to over 20000 with the reproducer in #39010), while the same weights in a one-row graph are correct.

With this change constant 2-bit weights are marked in the multi-row pattern as well, so they take the same path as in
the one-row graphs (FakeQuantize -> group unroll -> palletized weights on the DPU; the weights stay 2-bit). The
weights-as-inputs (block argument) prefill pattern is not changed (DontBlockArgPrefillPatternToDynamicDequantize
still holds).

Tests

• weights_dequantize_to_dynamic_dequantize.mlir: U2WeightsConstPrefillPatternToDynamicDequantize now expects the
  marker; new I2WeightsConstPrefillPatternToDynamicDequantize covers signed 2-bit weights without a zero point.
• LIT, run by hand on a Windows Release build (vpux-opt + FileCheck, the RUN line of each file):
  weights_dequantize_to_dynamic_dequantize.mlir passes for NPU3720, NPU4000 and NPU5010, and
  weights_dequantize_to_dynamic_dequantize_50XX+.mlir passes for NPU5010. The same file fails with the unpatched
  pass (both 2-bit const prefill tests unmarked). It also fails if every 2-bit DynamicDequantize is marked, because
  that breaks DontBlockArgPrefillPatternToDynamicDequantize; this PR therefore limits the change to constant
  weights.

The branch is based on 20e7912, one commit behind develop. The commit it is missing (#386) only changes CI files:
our token cannot push workflow files to the fork.

Validation

• Reproducer from #39010 (y = x[M, 4096] @ W^T, u2 weights, scale per row or per group of 128, NPU5010): max.
  relative error 0.0003-0.0019 for M = 1, 2, 4, 16 with the fix (before: 1.0 or more for every M >= 2).
• Real model: Ternary Bonsai 2 4B / 9B, 64-token prefill graphs vs token-by-token decoding over 256 positions: mean KL
  0.0005, top-1 agreement 98.4 %, the same NLL (NPU5010 and NPU4000).
• Verified independently by @Zulkifli-Intel on OpenVINO 2026.4.0-22470-d8047fb380b, NPU 4000:
  [Bug][NPU] u2 (2-bit) weights: MatMul wrong for 2+ activation rows on NPU40XX/50XX (breaks 2-bit LLM prefill) - root cause and one-line fix openvino#39010 (comment)
• Speed: the 2-bit prefill FullyConnected compiles to the same palletized path as INT4 (M = 64, 6 x 4096 x 4096:
  3.4 ms on NPU5010).

Tickets

• [Bug][NPU] u2 (2-bit) weights: MatMul wrong for 2+ activation rows on NPU40XX/50XX (breaks 2-bit LLM prefill) - root cause and one-line fix openvino#39010
• u2 (2-bit) constant weights: MatMul is correct for one activation row, wrong for M ≥ 2 (NPU40XX, NPU50XX) #355

🤖 Generated with Claude Code

[nikolaygorb]

I read openvinotoolkit/openvino#39010 and I'd like to proceed with internal testig with your changes.

Just curious: how it was discovered? User-scenario I mean.

Thank you for your PR!

[ryugyosoft-max]

Thanks for picking it up!

Some background: the existing LLM stacks for NPUs tend to lag behind the newest models, so I have been writing my own
apps with Claude that run recent LLMs entirely on the NPU, and publishing them on Hugging Face:

• onw (Intel NPU): https://huggingface.co/ryugyosoft/onw
• onwx (AMD Ryzen AI NPU): https://huggingface.co/ryugyosoft/onwx

A big goal is Ternary Bonsai 2 (4B / 9B / 27B, ternary weights stored as u2 with a scale per group of 128). I think
models like this can make the NPU much more useful in the age of agentic AI: even if it is slower than a GPU, it runs at
very low power, and for jobs that run overnight the speed hardly matters.

On AMD we program the NPU at a low level, so the Bonsai port went smoothly. On Intel the model has to go through the
compiler, and on the way we hit a few problems - the weights ending up twice in memory, slow compiles on Panther Lake,
and this one.

How it showed up: token-by-token decoding (one-row graphs) was fine, but when we added a prompt (prefill) graph that
handles 64 tokens at once, the output became garbage on NPU 4000 / NPU 5010, while NPU 3720 was fine. We compared the
prefill logits against decoding the same tokens one by one, then narrowed it down layer by layer to a single
FullyConnected with u2 weights and M >= 2 rows - the reproducer in #39010.

For now onw ships its own compiler build with this fix for NPU 4000+, so having it upstream would let the 2-bit models
use the stock compiler again.

[nikolaygorb]

Thank you for sharing detailed info.

Could you please share a more details there?
Reproducer profiler details? Benchmark details, arguments, output?

Kindly appreciated.

[ryugyosoft-max]

Sure. Each of the other problems has its own report with measurements, reproducers or profiles, and a reference patch
(all patches and scripts: https://gist.github.com/ryugyosoft-max/bb2f223987db3dae11d680bf5421cd50).

Weights twice in memory (a 1-token decode graph and a 64-token prefill graph over the same weights)

• [NPU compiler] Group-wise scales cost one weight-table row per output channel and group: 1 bit per weight per graph #379 - group-wise scales cost a 16-byte weight-table row per output channel and group, i.e. 1 bit per weight per
  graph, and the tables are not separated by weightless compilation. Ternary Bonsai 2 on NPU 5010, the 2-bit codes kept
  once for both graphs: 4B 2.56 GB, 9B 4.47 GB, 27B 13.84 GB of NPU memory for 0.9 / 2.1 / 6.8 GB of weights. This is
  the one that still blocks the 27B on a 16 GB laptop, and the tables are also ~1/3 of the bytes read per decoded token.
• openvino#39007 - weightless blobs keep the rebuilt weights in their own buffer, so every graph holds another full
  copy; our plugin prototype shares the matching init outputs (llm-jp-4.1-8B NF4, six graphs: +5.7 GB instead of
  ~29 GB, bit-identical outputs). Reproducer: repro_ws_share.py in the gist.
• [NPU compiler] Weightless compilation (weights separation) fails or keeps the weights for NF4, INT4 and 2-bit models #376 - weightless compilation fails or keeps the weights for NF4, INT4 and 2-bit models (the list of failures and
  fixes is in the issue).

Slow compiles on Panther Lake (Bonsai 2 4B first load ~25 min with the pip compiler -> 285 s with our fixes; one
64-token part 335 s -> ~20 s; blobs byte-identical)

• [NPU compiler] Multi-cluster strategy search serialises on the Windows heap lock (TensorDistributionMap) #373 - MultiClusterStrategyAssignment waits on the Windows heap lock (TensorDistributionMap): found with an
  out-of-process stack sampler; 3x slower multi-threaded than single-threaded.
• [NPU compiler] ContentAttr::fold() re-folds a whole concatenated weight for every DPU tile (O(tiles x weight)) #374 - ContentAttr::fold() re-folds a whole concatenated weight for every DPU tile (Granite 8B 1-token part 84.9 s
  with pip 8.4 -> 3.3 s).
• [NPU compiler] Strategy search and vertical fusion recompute the same results for every weight group (2-bit / grouped models) #375 - strategy search and vertical fusion recompute the same results for every weight group (Bonsai 9B 64-token
  part 122-160 s -> 65-90 s, then 335 s -> 60 s for the 4B).
• [NPU compiler] MatMul with a constant matrix on the LEFT of a batched input is unrolled into one small matmul per batch #378 - a MatMul with a constant matrix on the left of a batched input is unrolled per batch.
• openvino#39009 - compile_model calls from several threads of one process run one after another (reproducer
  repro_mt_compile.py).

This PR: python repro_u2_mgt1.py (the gist, also inline in #39010) prints the device / compiler / driver
versions and the max. relative error for M = 1, 2, 4, 16; the table in #39010 is its output on Panther Lake before and
after the fix. For the model check we compared the 64-token prefill logits against token-by-token decoding of the same
256 positions (mean KL 0.0005, top-1 agreement 98.4 %, same NLL).

If you meant something else by "there" - or want a specific log or profile attached - just tell me which, and I will
add it.