quant_gemm

Function quant_gemm 

Source
pub fn quant_gemm(
    gpu: &dyn GpuBackend,
    gemm_nvfp4: KernelHandle,
    gemm_fp8: KernelHandle,
    gemm_dense: KernelHandle,
    input: DevicePtr,
    weight: &QuantWeight,
    output: DevicePtr,
    m: u32,
    n: u32,
    k: u32,
    stream: u64,
) -> Result<()>
Expand description

Unified GEMM dispatch: select kernel based on weight quantization format.

For M>1 prefill projections (Q/K/V/O). Falls back to dense GEMM for BF16.