Normalization

BatchNorm, LayerNorm, GroupNorm, and InstanceNorm.

2 min readv1.0.0C++20

From the implementationinclude/nexus_model/layers/normalization.hpp

01Common transformation#

Mean and variance are computed over selected axes. Epsilon provides stability, gamma is learnable scale, beta is shift. Grouping axes define the statistical meaning of each variant.

\hat x=\frac{x-\mu}{\sqrt{\sigma^2+\epsilon}},\qquad y=\gamma\hat x+\beta

BatchNorm(num_features,eps=1e-5,momentum=0.1,affine=true). 1D/2D share the base, with channel axis 1. Training reduces batch and spatial axes. Normalization uses /m variance; running_var uses the /(m−1) correction when m>1. Eval uses running_mean/running_var, which are saved in state_dict.

\mu_{run}\leftarrow(1-\alpha)\mu_{run}+\alpha\mu_{batch}

03LayerNorm#

LayerNorm(normalized_size,eps=1e-5) normalizes only the final axis, which must equal normalized_size. It is batch-independent with no running statistics. weight starts at 1, bias at 0; there is no affine-disable option.

1LayerNorm norm(64);2auto y=norm.forward(Tensor::zeros({4,32,64}));

04GroupNorm / InstanceNorm#

GroupNorm(num_groups,num_channels,eps=1e-5,affine=true) groups channels equally within each sample. Channels must divide by groups. InstanceNorm(num_channels,eps=1e-5,affine=true) is GroupNorm with groups=C. Neither keeps running statistics.

05Normalization derivative#

For a group, define u=dy⊙gamma. dgamma=Σ(dy⊙x̂), dbeta=Σdy accumulate over appropriate axes. In eval BatchNorm, statistics are constant, so backward applies direct scaling.

dx=\frac{mu-\sum u-\hat x\sum(u\hat x)}{m\sqrt{\sigma^2+\epsilon}}

◆Read alongside the source. API behavior reflects the current implementation.

01Common transformation#

Mean and variance are computed over selected axes. Epsilon provides stability, gamma is learnable scale, beta is shift. Grouping axes define the statistical meaning of each variant.

\hat x=\frac{x-\mu}{\sqrt{\sigma^2+\epsilon}},\qquad y=\gamma\hat x+\beta

02BatchNorm1D / BatchNorm2D#

\mu_{run}\leftarrow(1-\alpha)\mu_{run}+\alpha\mu_{batch}

05Normalization derivative#

For a group, define u=dy⊙gamma. dgamma=Σ(dy⊙x̂), dbeta=Σdy accumulate over appropriate axes. In eval BatchNorm, statistics are constant, so backward applies direct scaling.

dx=\frac{mu-\sum u-\hat x\sum(u\hat x)}{m\sqrt{\sigma^2+\epsilon}}