Weight initialization
Kaiming, Xavier, orthogonal, and reproducibility.
01Seeds and primitives#
init::manual_seed(uint64_t) seeds a thread-local mt19937; the seed is cast to result_type. constant_, zeros_, ones_, uniform_(low,high), normal_(mean,stddev) fill in place. Set the seed before creating parameters.
02Fan-in and fan-out#
For [out,in,spatial...], fan_in=in·kernel_volume and fan_out=out·kernel_volume. For rank<2, both equal max(numel,1). Initialization aims to reduce growth or decay of activation and gradient scales with depth.
03Kaiming#
kaiming_uniform_/kaiming_normal_(tensor,negative_slope=0,fan_out=false). Gain √(2/(1+a²)); normal std=gain/√fan, uniform bound=√3·gain/√fan. Linear and Conv default to uniform.
04Xavier and orthogonal#
xavier_uniform_/xavier_normal_(tensor,gain=1) use fan_in+fan_out. orthogonal_(tensor,gain=1) accepts rank-2 and applies Gram–Schmidt to normal samples. Recurrent defaults use gain=1; verify other gains numerically.