quantize_to_fp8

Function quantize_to_fp8 

Source
pub fn quantize_to_fp8(
    bf16_weight: &DenseWeight,
    n: usize,
    k: usize,
    gpu: &dyn GpuBackend,
    quantize_kernel: KernelHandle,
    stream: u64,
) -> Result<Fp8DenseWeight>
Expand description

Quantize a BF16 dense weight to FP8 E4M3 on GPU.

Allocates FP8 weight buffer + per-row scale buffer, runs GPU quantization kernel. Called once at model load time (not on the hot path).