per_sequence_state_bytes

Function per_sequence_state_bytes 

Source
pub fn per_sequence_state_bytes(
    config: &ModelConfig,
    max_seq_len: usize,
    spec_on: bool,
) -> Result<PerSequenceState>
Expand description

Per-sequence owned device state for config, at a context of max_seq_len tokens.

Returns zeros for any model that owns no per-sequence device state outside the pools — which is every model except GLM-5.3 today. spec_on gates the proposer term: with speculation off no proposer is constructed and no proposer state is ever allocated.