Expand description
Per-quant weight abstraction.
Plug-in trait the vendor-agnostic forward modules (e.g.
super::qwen3_5) call instead of reaching into a concrete weight
type like MlxInt8Weight. Each backend’s weight loader implements
QuantWeights and overrides whichever fused variants it ships
kernels for; the rest fall back to default impls that compose the
unfused primitives or fail loudly.
Convention: gemv is mandatory (every backend ships one). Fused
variants (gemv_silu_gate, gemv_silu_gate_resid,
gemv_gate_up_with) are advisory — backends override only the
ones they have fused kernels for. Defaults either fall back to a
correct-but-slower composition (gemv_gate_up_with) or error
loudly so the caller can choose between hard-requiring the fused
kernel and degrading to a manual silu+mul+gemv pipeline.
Traits§
- Quant
Weights - A quantised weight tensor that can drive matvec / matmul ops on a
GpuBackend. Implementations live with each backend’s weight loader (Metal:MlxInt8Weight; future CUDA:Nvfp4Weight,Fp8DenseWeight, …).