dense_gemv_batch2

Function dense_gemv_batch2 

Source
pub fn dense_gemv_batch2(
    gpu: &dyn GpuBackend,
    kernel: KernelHandle,
    input: DevicePtr,
    weight: &DenseWeight,
    output: DevicePtr,
    n: u32,
    k: u32,
    out_stride: u32,
    stream: u64,
) -> Result<()>
Expand description

Dense BF16 GEMV, batched over 2 rows (M=2): one pass over the weight produces both output rows, halving weight bandwidth vs two dense_gemv launches. Bit-identical to two M=1 dense_gemv calls — each row’s accumulator follows the same K-iteration/reduction order.

input: [2, K] BF16 (contiguous); output: two rows at output + t * out_stride (BF16 elements). Used by the K=2 MTP verify path for the GDN in_proj_qkvz (dequant-to-BF16 on FP8 checkpoints), which otherwise re-read the full projection weight once per verify token.

Kernel: dense_gemv_bf16_batch2(A, B, C, N, K, out_stride)