pub const DENSE_GEMV_BATCHM_MAX_M: u32 = 16;Expand description
Dense BF16 batched GEMV (M rows): C[t] = A[t] @ B^T for t in [0, M).
The M-row generalisation of dense_gemv_batch2. Reads the BF16 weight
matrix ONCE for all M rows instead of M times, which is the whole point:
at decode the BF16 projections (q/k/v/o + shared expert) are pure weight
streaming, so M separate M=1 GEMVs make the step scale linearly with the
number of concurrent sequences.
Bit-identical to M separate dense_gemv calls (same K-iteration order and
reduction tree per row; the kernel dir builds with –fmad=false).
input: [M, K] BF16 contiguous. output: M rows at
output + t * out_stride (BF16 elements). Caller must pass m <= 8
(MAX_M in the kernel); larger batches should use a tiled GEMM.
Kernel: dense_gemv_bf16_batchm(A, B, C, M, N, K, out_stride)
Grid: (ceil(N/4), 1, 1) Block: (256, 1, 1)
Mirror of MAX_M in kernels/gb10/common/dense_gemv_bf16_batchm.cu.
The kernel clamps silently above this, so the Rust side must refuse.
🔴 16 since 2026-09-02. The old 8 was the kernel’s compiled row array, never an
arithmetic boundary: each row is an independent FP32 accumulator over the same kv
order, m appears in no row’s operand sequence, and the fold is per-row. So every
width up to MAX_M is bit-identical both to the narrower tier and to M serial
dense_gemv_bf16 calls. Verified on the 12 real GLM-5.3 prefill shapes with cold
weights, including the regression direction that matters — m <= 8 byte-unchanged,
because decode, the MTP verify arm and the BF16 lm_head arm all run m <= 8 on this
same kernel (scripts/glm53-dense-bf16/bench_m16.cu, spark-bench).
🪤 This constant is load-bearing OUTSIDE the GEMV: it gates the lm_head batched arm
(model/impl_a3.rs), the MTP row dispatch (layers/mtp_head/row_dispatch.rs) and it
sizes verify_k for the KDA/DSA/MLP workspaces (weight_loader/glm5_next_load.rs).
Raising it widens those arms and grows per-layer scratch — a memory-budget change, not
only a kernel one.