Expand description
Nemotron-H Mamba-2 SSM layer implementing TransformerLayer.
Standalone SSM layer (no FFN component). Forward pass:
- RMS norm (standard weight*x scaling)
- in_proj GEMV → [z, xBC, dt]
- Conv1d update on xBC (WITH bias, fused SiLU)
- Split xBC_out → x, B, C
- Mamba-2 SSM decode (state update + output)
- Gated RMS norm: rms_norm(y, ssm_norm) * silu(z)
- out_proj GEMV
- Residual add