q2_dequant_scratch_bytes

Function q2_dequant_scratch_bytes 

Source
pub fn q2_dequant_scratch_bytes(config: &ModelConfig) -> usize
Expand description

Bytes for the native keep-packed Q2_0 prefill transient-dequant scratch: the LARGEST keep-packed projection [N, K] expanded to BF16 (2 bytes/elem). The prefill dequant writes N*K BF16 elements into this buffer, which is then consumed by the same-stream GEMM and reused by the next projection.

Every keep-packed projection has exactly one dimension equal to hidden_size (FFN gate/up [inter, h], FFN down [h, inter], attention q/k/v/o [·, h] or [h, ·], fused GDN in_proj_qkvz [qkvz, h]), so N*K = max_other_dim * hidden — an EXACT bound, not an over-estimate, for the covered families (the k/v and gated-q terms are safe upper bounds). Independent of batch tokens (this dequants the WEIGHT, not activations).