Expand description
RMS norms (RmsNorm QK-norm + RmsNormGated), fp16-safe.
Modules§
- rms_
norm - Root-mean-square normalisation (last-dim, fp16-safe); also the Mamba-3 QK-Norm. Root-mean-square normalisation over the last dimension.
- rms_
norm_ gated - RMSNorm followed by a SiLU(z) gate (Mamba-2 output norm). RMS normalisation fused with a SiLU(z) gate — the Mamba-2 output norm.