q2_0_gemv_vec

spark_model::layers::ops

Function q2_0_gemv_vec 

Source 
pub fn q2_0_gemv_vec(
    gpu: &dyn GpuBackend,
    kernel: KernelHandle,
    input: DevicePtr,
    weight: &PackedQ2Weight,
    output: DevicePtr,
    stream: u64,
) -> Result<()>
Expand description

Q2_0 GEMV (M=1 decode), CANDIDATE B: C[1,N] = A[1,K] @ dequant(B).

Vectorized code loads (one uint32 = 16 ternary codes per lane) + shared- memory activation staging. A BF16 [1,K], B raw block_q2_0, C BF16 [1,N]. Dequant (code-1)*d happens inside the dot-product.

Kernel: q2_0_gemv_vec(A, B, C, N, K, group) Grid: (ceil(N/8),1,1) Block: (256,1,1)