Expand description
Custom activations (fp16-stable silu / softplus / log_sigmoid).
Modules§
- log_
sigmoid - Numerically-stable log-sigmoid (fp16-aware).
Numerically-stable log-sigmoid:
log σ(x) = −log(1 + e^−x). - silu
- SiLU activation (fp16-aware).
SiLU (a.k.a. swish) activation:
silu(x) = x · sigmoid(x). - softplus
- Softplus activation (fp16-aware).
Softplus activation:
softplus(x) = log(1 + eˣ), a smooth ReLU.