potatomaxx: reorder MoE experts on disk so llama.cpp reads fewer, larger blocks

[rsh1k]

CORRECTION (see comment below): the claim below that this helps "fine-grained MoE" is wrong. I surveyed seven real MoE checkpoints and none has expert slices under 256 KiB — the smallest, granite, is 288 KiB; DeepSeek-V2-Lite is 1584 KiB. Reordering experts on disk is correct and weight-preserving but helps no model that currently exists. What survives is the queue-depth finding (33×), per-expert precision, and the dequantiser validation. Leaving the original text intact below rather than rewriting it.

---

A tool that repacks a GGUF MoE checkpoint so experts that fire together sit
adjacent on disk. Output is a drop-in GGUF — same tensor names, same shapes,
byte-identical weights — so llama.cpp reads it unchanged. Nothing to patch.

https://github.com/Quilzo/potatomaxx — Rust, zero dependencies,
GPL-2.0-or-later.

I'm posting partly to share it and partly because I'd value a sanity check on the
correctness argument from people who know this format far better than I do.

Why bother

A big MoE checkpoint is streamed from storage while it runs, and storage cares
enormously how you ask. On my laptop (i5-1235U, 7.6 GB RAM, no GPU) the same NVMe
delivers 0.099 GB/s at queue depth 1 and 3.291 GB/s at depth 16. A factor of
33, from nothing but concurrency.

A GGUF MoE layer stacks its experts into a few tensors with the expert index as
the last axis, so fetching top-k means k slices out of each — k × n_tensors
scattered reads. If the experts a token wants are adjacent, those become a few
contiguous runs instead.

The correctness argument, which is what I'd like checked

Permuting the expert axis is a relabelling. If new slot j holds old expert
perm[j], and you permute the router's weight rows by the same perm, then the
logit for slot j is exactly the logit the original model computed for expert
perm[j]. Top-k selects the same real experts, from the same bytes.

The file-level half relies on GGUF locating tensor data solely through the
offset in its tensor_info, with physical order unconstrained and inter-tensor
padding explicitly allowed. That's my reading of the spec — if that's wrong,
I'd rather hear it now.

potatomaxx verify proves it after the fact by reading both files in full and
comparing every expert slice. It's strict enough to reject a single flipped byte,
and CI asserts that it does:

$ potatomaxx verify --model orig.gguf --repacked packed.gguf --plan p.pmxplan
  13 tensors compared, 1 byte-identical, 12 matched as a permutation
  12.10 MiB of weights confirmed unchanged
OK — the repacked file holds exactly the original weights.

Every tensor carrying a layer's expert axis is permuted together, including the
router. Routers are never requantised — quantisation error there perturbs
expert selection (the "expert shift" problem), which would invalidate the very
routing trace the plan was derived from.

Honest scope: layout is the secondary lever

I originally built this around layout and the measurements disagreed. Once a
layer's top-k reads are issued concurrently — which #25294's expert streaming
does — reordering buys 1.1–1.6×, and only for slices in a particular size
band. Above ~256 KiB slices the bandwidth surface plateaus and layout barely
matters.

So this is not a big win for Mixtral-shaped models with large experts. It's for
fine-grained MoE — hundreds of small experts per layer — where slices land in the
16–128 KiB range where request size still matters a lot. potatomaxx analyse
tells you which case you're in, and says plainly when the answer is "leave it
alone":

 layer   experts    req/token        after   speedup  verdict
     0       128        23.37         7.48     3.01x  repack

It needs a routing trace

The whole thing keys off which experts your workload actually selects. The text
format is one line per (token, layer):

0 0 12 44 7 91
0 1 3 55 8 12

This is the ask. A small patch to dump router top-k would make this usable
against real models, and would be independently useful for anyone studying expert
locality. I'd be glad to write it if there's interest in the shape it should
take — or to be told the routing data is already reachable somewhere I've missed.

Also possibly of interest: two I/O findings

From potatomaxx kio, which compares read paths on your device. Both are relevant
to #25294:

• How you drive io_uring matters more than using it. Submitting a batch and
  waiting for all of it drains the ring between batches; that peaked at QD8 and
  then declined. A sliding window — replace each completed read immediately —
  gained +148% at QD8.
• RWF_DONTCACHE (6.14) measured 49% slower and is worth having anyway. It
  left 7.4% page-cache residency instead of filling RAM, and cachestat() shows
  the plain path thrashing. It's not a throughput optimisation; it's a decision
  not to evict everyone else's working set to hold 40 GB you'll read once.

Caveat on all of it: measured under WSL2, so the block path is virtualised, and a
thread pool actually beat my io_uring at QD16 (3.29 vs 1.54 GB/s). I don't have
bare-metal access to check whether that's a virtualisation artefact, so please
treat the directions as more trustworthy than the absolute numbers. potatomaxx kio reproduces the table in about a minute if anyone wants to correct me.

Try it without downloading anything

cargo build --release
potatomaxx synth      # synthetic MoE checkpoint + trace
potatomaxx kio        # which read path is fastest here
potatomaxx analyse --model synth.gguf --trace synth.pmxtrace --probe surf.json

Happy to be told the layout argument is wrong, or that the gain isn't worth the
repack for real checkpoints. That's most of why I'm posting.

[rsh1k]

Correction — the layout claim above is wrong, and I'd rather say so here than quietly edit it.

I wrote that this targets "fine-grained MoE — hundreds of small experts per layer — where slices land in the 16–128 KiB range". I then went and checked whether any such model actually exists. It doesn't.

One expert contributes hidden_size × intermediate_size weights to each of its three matrices, and each matrix is read as one slice, so a model's config.json settles it for a couple of kilobytes. Surveying seven real checkpoints:

model	experts	hidden	inter	slice @ Q4_K
granite-3.0-1b-a400m	32	1024	512	288 KiB
OLMoE-1B-7B	64	2048	1024	1152 KiB
Qwen3-30B-A3B	128	2048	768	864 KiB
DeepSeek-V2-Lite	64	2048	1408	1584 KiB
Mixtral-8x7B	8	4096	14336	32256 KiB
Phi-3.5-MoE	16	4096	6400	14400 KiB

Zero of seven are under 256 KiB, which is where my measured bandwidth surface stops rewarding larger requests. Even DeepSeek-V2-Lite — the architecture usually described as fine-grained — is 1584 KiB, because "fine-grained" refers to expert count, not to matrix width. At Q2_K only granite crosses the line.

So: reordering experts on disk is correct and provably weight-preserving, and useless on every MoE checkpoint that currently exists. If you were considering trying it for the layout benefit, don't — there isn't one. My original post oversold it and I'm sorry for the noise.

Confirmed empirically too: run against real Granite 3.0 1b-a400m, analyse reported "leave alone" on all 24 layers. Repack + verify did work correctly — 96 permuted real Q4_K/Q6_K tensors, all 782 MiB confirmed byte-identical — so the machinery is sound. It just has no problem to solve.

What does survive, and is the part I'd now actually ask about:

• Queue depth is worth 33× on my device — 0.099 GB/s at QD1 vs 3.291 at QD16. That dwarfs everything else and is a runtime property, so it's relevant to llama : stream MoE routed experts from disk  #25294's expert streaming. Related: driving io_uring as batch-and-drain peaked at QD8 and then declined; a sliding window gained +148% at QD8.
• Per-expert precision reduces bytes moved regardless of slice size. Granite showed nothing only because it's already Q4_K; an F16 or Q8 checkpoint has room. This can't be expressed in GGUF, since a tensor carries one type.
• The dequantiser validation may be independently useful: potatomaxx compare --a f16.gguf --b q4km.gguf dequantises the same tensor from two quantisations and reports correlation. Against the F16 build of granite: Q4_K 0.997089 (7.63% rel. error), Q6_K 0.999822 (1.89%) — the ratio their bit-widths predict. It's a cheap way to catch a decoder regression, and needs only one tensor from each file.

Still after a router top-k dump, which remains the one thing I can't produce myself and the reason routing traces here are synthetic. Details: https://github.com/Quilzo/potatomaxx/blob/main/docs/REAL-MODEL-RESULTS.md