Linear
Fully connected layers, weight layout, and analytic derivatives.
01Constructor and parameters#
Linear(in_features,out_features,use_bias=true). Feature counts must be positive. weight() returns an [out,in] Parameter; bias() returns an [out] Parameter or nullptr when disabled. Weights start with Kaiming uniform, biases with zeros.
Linear layer(64,128,true);auto x=Tensor::zeros({4,16,64});auto y=layer.forward(x); // [4,16,128]02Forward pass#
The final axis must match in_features. Earlier axes are flattened into rows and preserved in the output. W=[out,in] layout avoids physically transposing weights.
03Analytic backward#
dY=[M,O], X=[M,I], W=[O,I]. The layer clears dX; dW/db add to existing accumulations. requires_grad=false disables only the corresponding parameter contribution.
04Compute cost#
Forward multiplication takes approximately 2MIO FLOPs; weights occupy 4IO bytes excluding bias. Training adds gradients and caches. A new forward on the same module replaces the previous backward cache.