pub(crate) fn safe_norm<const D: usize>(t: Tensor<D>) -> Tensor<D>Expand description
Euclidean norm over the last axis, formed scale-free so it cannot overflow: the components are divided by their (detached) largest magnitude before squaring, and the result is scaled back.
Squaring the raw components is the obvious way and the wrong one here,
because the inputs are raw in-projection channels: ‖r‖² overflows f32 at
|r| ≈ 2e19 and f16 at |r| ≈ 250, which is an ordinary activation.
The overflow does not announce itself — ∞ divides back to 0, so a very
large generator would silently produce no rotation, the exact opposite of
the intended “turn as far as the bound allows”. Same trick, same reason, as
RmsNorm’s fp16 path.
The sum of squares is floored by div_eps before the sqrt, so a zero
vector lands in clamp_min’s flat region and backprops to a finite 0
instead of sqrt’s singular 1/(2·0).
§Shapes
t:[..., n]- out :
[..., 1]