Struct AttentionWeights 

pub struct AttentionWeights {
    pub q_proj: DenseWeight,
    pub k_proj: DenseWeight,
    pub v_proj: DenseWeight,
    pub o_proj: QuantizedWeight,
    pub q_norm: DenseWeight,
    pub k_norm: DenseWeight,
    pub q_norm_full: Option<DenseWeight>,
    pub k_norm_full: Option<DenseWeight>,
    pub k_scale: f32,
    pub v_scale: f32,
}

Full attention layer weights (12 layers in Qwen3-Next).

Fields

q_proj: DenseWeight

Q projection: [hidden_size, num_heads * head_dim] BF16.

k_proj: DenseWeight

K projection: [hidden_size, num_kv_heads * head_dim] BF16.

v_proj: DenseWeight

V projection: [hidden_size, num_kv_heads * head_dim] BF16.

o_proj: QuantizedWeight

O projection: [num_heads * head_dim, hidden_size] NVFP4.

q_norm: DenseWeight

Q RMS norm weight: [head_dim] BF16 (per-head Qwen3-family convention).

k_norm: DenseWeight

K RMS norm weight: [head_dim] BF16 (per-head Qwen3-family convention).

q_norm_full: Option<DenseWeight>

MiniMax-style full-hidden Q RMSNorm weight: [num_heads * head_dim] BF16.

Set to Some(..) for models that apply RMSNorm over the concatenated Q projection output (MiniMax M2) before the view-into-heads and before RoPE. Mathematically different from the per-head q_norm above (MiniMax normalizes by the global hidden-dim RMS; Qwen3 normalizes per-head). Attention forward branches on .is_some() to pick which pre-RoPE norm to apply. Default None keeps all existing models on the per-head q_norm path — behavior-preserving for every non- MiniMax loader.

k_norm_full: Option<DenseWeight>

MiniMax-style full-hidden K RMSNorm weight: [num_kv_heads * head_dim] BF16.

k_scale: f32

K scale for FP8 KV cache.

v_scale: f32

V scale for FP8 KV cache.

Trait Implementations

impl Clone for AttentionWeights

fn clone(&self) -> AttentionWeights

Returns a duplicate of the value.

fn clone_from(&mut self, source: &Self)

Performs copy-assignment from source.

impl Debug for AttentionWeights

fn fmt(&self, f: &mut Formatter<'_>) -> Result

Formats the value using the given formatter.

impl Copy for AttentionWeights