pub fn lora_gemv_max_m() -> u32Expand description
Row count at or below which the delta runs as m row-wise GEMVs instead
of one GEMM (ATLAS_LORA_GEMV_MAX_M, default 8).
dense_gemm_tc tiles 16 rows. At m=2 it does the FULL B-matrix traffic —
B is [n_out, max_rank] and independent of m — to produce 2 useful rows out
of 16, at one row-block of grid. Measured on qwen3.8-27B + an r=64 adapter:
the FFN deltas cost ~8% of a decode step at m=1 (GEMV) and ~64% at m=2
(GEMM). Same work, 8x the price, purely from crossing this boundary.
The GEMV loop is also the canonical form: apply_lora_bgmv documents
itself as byte-identical to n single-row apply_lora_delta calls, so
this makes the small-m path agree with that oracle rather than diverge
from it.
Default 48 covers the plain decode ladder (C=1..8) AND the speculative verify shapes, which present n*k rows — up to 4 seqs x k=9 = 36 under cross-sequence batched DFlash verify. 8 left those on the GEMM: raising it took DFlash+LoRA from 34.8 to 48.6 tok/s at C=2 and 47.6 to 54.2 at C=4, with accepts and output unchanged. Prefill’s m is orders of magnitude larger and keeps the GEMM.