pub fn moe_grouped_decode_min_rows() -> usizeExpand description
Optional kernel lookup: KernelHandle(0) instead of an error.
#[track_caller] so the audit names the DISPATCH SITE — this helper stands
between ~500 call sites and GpuBackend::kernel, and without it every
optional lookup in the binary would be reported against this one line.
A zero handle is a SILENT slower path, so a lookup that lands here for a
model that genuinely needs the kernel is a bug. Either gate the call on the
model’s config so it is never issued, or declare it in the target’s
MODEL.toml [expected_absent] with a reason; the boot gate
(kernel_audit::classify_failures) fails closed on anything else.
Minimum rows in flight for the grouped-GEMM MoE decode arm. SSOT for the
SSM stack (qwen3_ssm::trait_decode_multi_seq) and the attention layers
(qwen3_attention::…::multi_seq::ffn), which must agree — they are the
same trade on the same weights.
The arm reads each routed expert ONCE instead of once per token, so it wins when there are enough tokens to amortise the expert sort/permute launch overhead, and loses when there are not. Both ends are measured:
| n | verdict | measurement |
|---|---|---|
| 4 | LOSS | 31 vs 56 tok/s on Holo — the fixed per-layer sort/permute dominates at small N |
| >=16 | WIN | SSM-side alone C=32 172.7 -> 216.2 tok/s (+25%); #415’s attention-side extension +7.9% at C=32 / +9.7% at C=64 on Qwen3.6-35B-A3B-NVFP4, paired gsm8k n=200 strict 0.960 vs 0.900 baseline, zero regressions |
16 is the smallest width measured on the winning side. n=5..15 is UNMEASURED, not a known win — it sits on the losing side of this gate on purpose, because the one thing we know about the gap is that the loss at n=4 is large (-45%) and the win at n=16 is smaller (+25%).