Skip to main content

Module norm

burn_mamba::modules

Module norm 

Source
Expand description

RMS norms (RmsNorm QK-norm + RmsNormGated), fp16-safe.

Modules§

rms_norm
Root-mean-square normalisation (last-dim, fp16-safe); also the Mamba-3 QK-Norm. Root-mean-square normalisation over the last dimension.
rms_norm_gated
RMSNorm followed by a SiLU(z) gate (Mamba-2 output norm). RMS normalisation fused with a SiLU(z) gate — the Mamba-2 output norm.