Struct KvCacheConfig 

pub struct KvCacheConfig {
    pub block_size: usize,
    pub num_kv_heads: usize,
    pub head_dim: usize,
    pub num_layers: usize,
    pub dtype: KvCacheDtype,
    pub layer_dtypes: Vec<KvCacheDtype>,
    pub layer_dims: Vec<(usize, usize)>,
    pub cache_blocks_per_seq: Option<u32>,
}

Configuration for the paged KV cache.

Fields

block_size: usize

Tokens per block.

num_kv_heads: usize

Number of KV heads.

head_dim: usize

Dimension per head.

num_layers: usize

Number of attention layers (only full_attention layers have KV cache).

dtype: KvCacheDtype

Quantization dtype for cache storage (uniform fallback).

layer_dtypes: Vec<KvCacheDtype>

Per-layer KV cache dtype override. When non-empty, layer_dtypes[i] specifies the dtype for attention layer i. When empty, all layers use the uniform dtype field (backward compatible).

layer_dims: Vec<(usize, usize)>

Per-layer (num_kv_heads, head_dim) overrides for heterogeneous attention models (e.g. Gemma-4 with sliding 16×256 vs full 4×512). When non-empty, layer_dims[i] specifies the (nkv, hd) for layer i; allocation and kernel stride computations use these per-layer values so writes/reads land at the correct offsets. When empty, all layers use the uniform num_kv_heads/head_dim (backward compatible — homogeneous models need no change).

cache_blocks_per_seq: Option<u32>

--high-speed-swap HBM-shrink knob (Phase 6.1). When Some(N), each sequence is capped at N HBM-resident blocks; older blocks are evicted to disk via HighSpeedSwap and read back on demand. None (default) preserves the existing behavior — sequences hold every block in HBM forever and rely on --swap-space-gb for admission control. The try_evict_oldest_for_seq helper below is only valid when this is Some.

Implementations

impl KvCacheConfig

pub fn dtype_for_layer(&self, layer_idx: usize) -> KvCacheDtype

Resolve the effective dtype for a given attention layer index.

pub fn v_block_bytes_dims( &self, dtype: KvCacheDtype, nkv: usize, hd: usize, ) -> usize

V-side bytes per block for asymmetric dtypes; equals block_bytes_dims for symmetric dtypes. Use this when allocating the V pool separately from K. Once real asym kernels land, callers should switch to: k_size = block_bytes_dims(kv_pair().0, …) v_size = block_bytes_dims(kv_pair().1, …) and allocate the two pools independently.

pub fn k_block_bytes_dims( &self, dtype: KvCacheDtype, nkv: usize, hd: usize, ) -> usize

K-side bytes per block for a specific (asym-aware) dtype and dims. For symmetric dtypes, returns the same value as block_bytes_dims. For asymmetric, returns the K-component (e.g. Bf16KTurbo3V → bf16 bytes).

pub fn k_block_bytes_for_layer(&self, layer_idx: usize) -> usize

K-side bytes per block for a specific attention layer. Replaces the legacy single-stride view for asym dtypes by routing through the K component of the dtype pair.

pub fn v_block_bytes_for_layer(&self, layer_idx: usize) -> usize

V-side bytes per block for a specific attention layer. For symmetric dtypes this equals k_block_bytes_for_layer.

pub fn block_bytes(&self) -> usize

Bytes per block per layer (K or V, not both), using the uniform dtype.

pub fn dims_for_layer(&self, layer_idx: usize) -> (usize, usize)

(num_kv_heads, head_dim) for a specific attention layer. Falls back to the global values when no per-layer override is set.

pub fn block_bytes_for_layer(&self, layer_idx: usize) -> usize

Bytes per block for a specific attention layer (K or V, not both). Uses per-layer (nkv, hd) from layer_dims when set — this lets heterogeneous layers (Gemma-4 sliding vs full) allocate tight, correctly-sized pools so the kernel’s per-layer stride matches the allocation layout.

pub fn block_bytes_kv(&self) -> usize

Bytes per block per layer (K + V combined).

pub fn block_bytes_kv_all_layers(&self) -> usize

Sum of K+V block bytes across all layers for one block slot. Accounts for mixed dtypes when layer_dtypes is set AND asym K/V splits.

pub fn cache_stride_elements(&self) -> usize

Cache stride in elements (for FP8/BF16 kernels). Elements per block = block_size * num_kv_heads * head_dim.

pub fn nvfp4_data_bytes(&self) -> usize

NVFP4 data section bytes per block (packed E2M1 nibbles).

pub fn nvfp4_scale_bytes(&self) -> usize

NVFP4 scale section bytes per block (FP8 per-group scales).

pub fn turbo4_data_bytes(&self) -> usize

Turbo4 data section bytes (same layout as NVFP4: 4-bit packed).

pub fn turbo4_scale_bytes(&self) -> usize

Turbo4 scale section bytes (same layout as NVFP4: FP8 per-group).

pub fn turbo3_data_bytes(&self) -> usize

Turbo3 data section bytes (3-bit packed: 8 values in 3 bytes).

pub fn turbo3_scale_bytes(&self) -> usize

Turbo3 scale section bytes (FP8 per-group, same as turbo4).

pub fn turbo2_data_bytes(&self) -> usize

Turbo2 data section bytes (2-bit packed: 4 values per byte).

pub fn turbo2_scale_bytes(&self) -> usize

Turbo2 scale section bytes (FP8 per-group, same as turbo3/turbo4).

pub fn turbo8_data_bytes(&self) -> usize

Turbo8 data section bytes (FP8 E4M3: 1 byte per element).

pub fn turbo8_scale_bytes(&self) -> usize

Turbo8 scale section bytes — BF16 per-group scales (2 bytes each, vs the 1-byte FP8 scales NVFP4/Turbo3/Turbo4 use). The BF16 upgrade is what makes Turbo8 viable across many-layer models like MiniMax M2.7 (58 Turbo8 layers under auto HP=2). Returns (num_groups) * 2 bytes total.