Enum QuantFormat 

pub enum QuantFormat {
    Nvfp4,
    Fp8,
}

Runtime quantization format for weight dispatch.

Determines which GEMV/GEMM kernels are used for decode, prefill, and MTP verify. Adding a new quant format requires:

1. Add variant here
2. Add kernel dispatch in the layer forward paths
3. Add weight loading logic in load_moe_qwen35 / attention loader

Variants

Nvfp4

NVFP4 E2M1 — default, highest throughput. Uses w4a16 kernels.

Fp8

FP8 E4M3 block-scaled — native FP8 serving. Uses w8a16 kernels.

Implementations

impl QuantFormat

pub fn peak_memory_multiplier(&self) -> f64

Peak GPU memory multiplier for OOM pre-flight estimation.

Accounts for model-building overhead on top of raw weight bytes:

• NVFP4: 1.3x (weight pointers aliased, transposed copies + predequant)
• FP8: 1.5x (zero-copy weights, transposed attention copies, FP8 pointer tables)

Adding a new format: set the multiplier based on empirical peak/on-disk ratio.

Trait Implementations

impl Clone for QuantFormat

fn clone(&self) -> QuantFormat

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for QuantFormat

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl PartialEq for QuantFormat

fn eq(&self, other: &QuantFormat) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for QuantFormat

impl Eq for QuantFormat

impl StructuralPartialEq for QuantFormat