Module nemotron_mamba2

Module nemotron_mamba2 

Source
Expand description

Nemotron-H Mamba-2 SSM layer implementing TransformerLayer.

Standalone SSM layer (no FFN component). Forward pass:

  1. RMS norm (standard weight*x scaling)
  2. in_proj GEMV → [z, xBC, dt]
  3. Conv1d update on xBC (WITH bias, fused SiLU)
  4. Split xBC_out → x, B, C
  5. Mamba-2 SSM decode (state update + output)
  6. Gated RMS norm: rms_norm(y, ssm_norm) * silu(z)
  7. out_proj GEMV
  8. Residual add

Structs§

NemotronMamba2Layer