Struct LayerKvCache 

pub struct LayerKvCache {
    pub k: DevicePtr,
    pub v: DevicePtr,
    pub capacity: u32,
    pub dtype: MetalKvDtype,
    pub k_scales: Option<DevicePtr>,
    pub v_scales: Option<DevicePtr>,
}

Per-layer KV cache for a full-attention layer (single-batch).

dtype selects the storage format; for the turbo formats k/v hold packed quantized data in the WHT-rotated basis and scales holds the per-16-element group scales (bf16 for Turbo8, FP8 E4M3 bytes for Turbo4). The forward routes appends and attention through the matching kernels with WHT(Q)/iWHT(out) bookends.

Fields

k: DevicePtr

v: DevicePtr

capacity: u32

Capacity in tokens — caller pre-allocates max_seq_len * KV_DIM.

dtype: MetalKvDtype

k_scales: Option<DevicePtr>

Per-side group-scale buffers — Some only for quantized sides (both for symmetric turbo dtypes, V-only for the safer-asym Bf16K+TurboNV family, neither for Bf16).

v_scales: Option<DevicePtr>

Implementations

impl LayerKvCache

pub fn alloc( gpu: &dyn GpuBackend, dtype: MetalKvDtype, max_seq: u32, kv_dim: u32, ) -> Result<Self>

Allocate a cache in the given storage format.