Enum KvCacheDtype 

pub enum KvCacheDtype {
    Bf16,
    Fp8,
    Nvfp4,
    Turbo4,
    Turbo3,
    Turbo2,
    Turbo8,
    Turbo4KTurbo3V,
    Turbo4KTurbo8V,
    Turbo3KTurbo8V,
    Bf16KTurbo4V,
    Bf16KTurbo3V,
    Fp8KTurbo4V,
    Fp8KTurbo3V,
    Bf16KTurbo2V,
    Fp8KTurbo2V,
}

KV cache quantization dtype.

Variants

Bf16

2 bytes per element.

Fp8

1 byte per element (FP8 E4M3 with per-tensor scale).

Nvfp4

0.5 bytes data + per-group FP8 scale (E2M1 packed nibbles).

Turbo4

4-bit WHT + Lloyd-Max quantization (TurboQuant). Same byte layout as NVFP4 but with Walsh-Hadamard rotation and optimal Gaussian codebook for ~2x lower MSE at the same bit rate.

Turbo3

3-bit WHT + Lloyd-Max (8 levels). 22% smaller than turbo4.

Turbo2

2-bit WHT + Lloyd-Max (4 levels). 6.4x compression vs bf16 (3 bits/elem total: 2 b data + 0.5 b scale + 0.5 b layout overhead). Full write + paged-decode + chunked-prefill kernel coverage. 2-bit keys cannot sustain tool-grammar constrained decoding with the standard boundary policy; requires the higher auto high-precision-layer default (see auto_high_precision_layers) validated on the GB10 flagship.

Turbo8

WHT + FP8 E4M3. Same memory as FP8 but with outlier suppression. Enables FP8-level memory for models with large RMS norm weights.

Turbo4KTurbo3V

TurboQuant+ asymmetric: K stored at turbo4 (4-bit), V at turbo3 (3-bit). K dominates attention score precision; V tolerates lower precision per turboquant_plus/docs/papers/asymmetric-kv-compression.md. Saves ~14% bandwidth at decode (4.5 b/elem K + 3.375 b/elem V vs 4.5 + 4.5 symmetric turbo4). Decode kernel dispatch needs a new paged_decode_attn_turbo4k_turbo3v variant; write kernel forks reshape_and_cache_flash_turbo4 for K and ..._turbo3 for V on the same launch.

Turbo4KTurbo8V

K=turbo4, V=turbo8. K=4-bit codebook; V=FP8. ~11% bandwidth saving vs pure turbo8 symmetric. Same dispatch-table follow-up applies.

Turbo3KTurbo8V

K=turbo3, V=turbo8. Smallest K (3-bit) with V=FP8 retention.

Bf16KTurbo4V

TurboQuant+ safer-asym: K stored at BF16 baseline (full precision), V compressed to turbo4 4-bit codebook. Preserves K’s attention-score fidelity completely while compressing V which dominates KV bandwidth at long context.

Bf16KTurbo3V

K=bf16, V=turbo3 (3-bit). Aggressive V compression with full-precision K.

Fp8KTurbo4V

K=fp8 (1 byte/elem with per-tensor scale), V=turbo4. K kept at the usual fp8 quality; V at 4-bit codebook. Middle ground between bf16/turbo and pure turbo8.

Fp8KTurbo3V

K=fp8, V=turbo3. Smallest combo retaining fp8 K precision.

Bf16KTurbo2V

K=bf16 baseline, V=turbo2 (2-bit). Most aggressive V compression with full-precision K. Per asymmetric-kv-compression.md: symmetric turbo2/ turbo2 collapses quality (+58.5% PPL); this asym preserves K and only pays the +9.5% V-side cost — 6× better quality at the same V compression.

Fp8KTurbo2V

K=fp8, V=turbo2. The canonical “asymmetric rescue” config (analog of llama-cpp-turboquant’s q8_0/turbo2). Best compression-to-quality ratio for turbo2 V on tested models.

Implementations

impl KvCacheDtype

pub const ALL: [KvCacheDtype; 16]

Every variant, in the order the enum declares them.

Extend this together with KvCacheDtype::name below — the non-exhaustive-match error a new variant raises there points here.

pub const fn name(self) -> &'static str

The canonical --kv-cache-dtype spelling. Display delegates here, so the string a picker offers is byte-for-byte the string the flag parser reads back — the round trip the tests below pin.

impl KvCacheDtype

pub fn kv_pair(self) -> (KvCacheDtype, KvCacheDtype)

Returns the (K_dtype, V_dtype) pair. For symmetric variants both elements are identical. For asymmetric variants the pair differs.

pub fn is_wht_rotated(self) -> bool

True for the symmetric turbo dtypes whose cache contents are stored in the WHT-rotated basis (the write path applies wht_bf16_inplace before quantizing). Gates the WHT(Q) / iWHT(out) attention bookends — call on the K or V side of kv_pair(), not on the combined variant. Turbo2 is rotated by the write path like the rest; omitting it here is what desynced the decode bookends from the write path.

pub fn is_asymmetric(self) -> bool

True if K and V use different storage layouts.

Trait Implementations

impl Clone for KvCacheDtype

fn clone(&self) -> KvCacheDtype

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for KvCacheDtype

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Display for KvCacheDtype

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl FromStr for KvCacheDtype

type Err = Error

The associated error which can be returned from parsing.

fn from_str(s: &str) -> Result<Self>

Parses a string s to return a value of this type.

impl PartialEq for KvCacheDtype

fn eq(&self, other: &KvCacheDtype) -> bool

Tests for self and other values to be equal, and is used by ==.

fn ne(&self, other: &Rhs) -> bool

Tests for !=. The default implementation is almost always sufficient, and should not be overridden without very good reason.

impl Copy for KvCacheDtype

impl Eq for KvCacheDtype

impl StructuralPartialEq for KvCacheDtype