Module buffers

Expand description

Pre-allocated GPU buffer arena for intermediate tensors.

All buffer sizes derive from ModelConfig (SSOT). The arena is allocated once during initialization and reused across decode steps.

Re-exports§

decode_meta: Fixed-stride batched-decode metadata layout, derived from the serve max_batch_size (SSOT — consumed by sizes.rs for the scratch envelope and by spark-model’s upload_batch_metadata_fixed/_at for the upload offsets, replacing the former hardcoded 0/128/256/512/768 gaps that fit exactly 32 rows).

Q12_SIZING_STREAMS: Streams assumed when provisioning scratch for the Q12 kernel-batched prefill path. The per-stream metadata region scales with N, so the scratch buffer must be sized for a realistic max concurrent batched streams. Beyond this, check_kernel_batched_eligible falls the dispatch back to the per-stream path (which respects the same arena cap), so this bound only governs how often the fast path is available — never safety.

q2_dequant_scratch_bytes: Bytes for the native keep-packed Q2_0 prefill transient-dequant scratch: the LARGEST keep-packed projection [N, K] expanded to BF16 (2 bytes/elem). The prefill dequant writes N*K BF16 elements into this buffer, which is then consumed by the same-stream GEMM and reused by the next projection.
q12_batched_scratch_bytes: Exact scratch footprint (bytes) of the Q12 kernel-batched prefill staging for n streams of chunk_len tokens each. SSOT for both scratch sizing (BufferSizes::from_config) and the pre-flight eligibility check (check_kernel_batched_eligible), so the two can never disagree about whether a batch fits. Mirrors the staging layout in batch_kernel.rs (MoE topk area + N per-stream meta blocks) and stage_batched.rs (stacked positions ×(3 if MRoPE) + slots + block/seq_len pointer arrays) plus the per-SSM-layer h_state_ptrs JIT slot.
q12_batched_scratch_bytes_varlen: Exact scratch footprint for a ragged Q12 batch.