Activations
Mathematics, derivatives, and defaults for 11 activations.
01Elementwise transformations#
Shape is preserved. PReLU learns a single shared alpha, not a per-channel slope. Other activations have no parameters. GELU defaults to the exact erf form; true selects the tanh approximation.
| API | Forward | Local derivative |
|---|---|---|
| ReLU() | max(0,x) | x>0 ? 1 : 0 |
| LeakyReLU(0.01f) | x>0 ? x : αx | x>0 ? 1 : α |
| PReLU(0.25f) | x>0 ? x : αx | dα += Σ(x≤0) dy·x |
| ELU(1.f) | x>0 ? x : α(exp(x)−1) | x>0 ? 1 : y+α |
| GELU(false) | x·Φ(x) | Φ(x)+x·φ(x) |
| GELU(true) | 0.5x[1+tanh(c(x+0.044715x³))] | 0.5(1+tanh u)+0.5x(1−tanh²u)c(1+0.134145x²); u=c(x+0.044715x³) |
| Sigmoid() | σ=1/(1+exp(−x)) | σ(1−σ) |
| Tanh() | tanh(x) | 1−y² |
| SiLU() | x·σ(x) | σ+xσ(1−σ) |
| Mish() | x·tanh(softplus(x)) | tanh(s)+xσ(x)(1−tanh²(s)) |
02Softmax#
Subtract the row maximum and exponentiate on the final axis. Rows sum approximately to 1. Backward uses a dot product instead of a full Jacobian. An all−∞ row is not a valid distribution.
03LogSoftmax#
Uses stable x−logsumexp(x) on the final axis. Outputs are log probabilities, computed directly instead of first forming very small probabilities and then taking their logarithm.
04Zero and testing#
ReLU chooses derivative 0 at zero. LeakyReLU/PReLU use the negative branch. Skip kink points in finite-difference checks. Use tolerances rather than bitwise equality for SIMD and float32 differences.