Struct MoeWeights 

pub struct MoeWeights {
    pub gate: DenseWeight,
    pub shared_expert: ExpertWeight,
    pub shared_expert_gate: DenseWeight,
    pub experts: Vec<ExpertWeight>,
    pub router_pre_norm: Option<DenseWeight>,
    pub correction_bias: Option<DenseWeight>,
}

MoE layer weights.

Fields

gate: DenseWeight

Router gate: [hidden_size, num_experts] BF16.

shared_expert: ExpertWeight

Shared expert (always active).

shared_expert_gate: DenseWeight

Shared expert gate sigmoid weight: [1] BF16.

experts: Vec<ExpertWeight>

Per-expert weights: 512 experts.

router_pre_norm: Option<DenseWeight>

Optional router pre-normalization weight. Set for Gemma-4 MoE where the HF reference applies a pure RMSNorm to the router input followed by a per-dim scale multiplication: router_input = rms_norm(x) * scale * hidden_size^(-0.5) Stored as a BF16 [hidden_size] vector containing scale * root_size so the existing rms_norm kernel (output = x/rms(x) * weight) applies both steps in one pass. None for models that feed the router from the raw post-attention residual.

correction_bias: Option<DenseWeight>

Optional expert correction bias: [num_experts] F32.

Set for models using the DeepSeek-V3 / MiniMax-M2 loss-free-balancing routing trick: the bias is added to sigmoid(gate_logits) only for top-k selection; gathered dispatch weights come from the unbiased sigmoid scores. Consumed by moe_topk_sigmoid kernel via its bias argument.

None for softmax-routed Qwen/Gemma MoE. Nemotron-H carries its own bias in NemotronMoeWeights::e_score_correction_bias because its MoE is a separate layer type (Mamba-2 interleaved) — those paths don’t touch this struct.