RNN, LSTM & GRU
Gate equations and backpropagation through time.
01Shared interface#
RNN/LSTM/GRU(input_size,hidden_size,num_layers=1,truncate_bptt=0). Inputs [B,T,F], outputs [B,T,H]; unidirectional and batch-first. No separate initial/final-state API. Every forward starts from zeros. Callers must ensure valid dimensions, positive T, and positive layer count.
LSTM sequence(32,64,2);auto x=Tensor::zeros({4,20,32});auto y=sequence.forward(x); // [4,20,64]02Tanh RNN#
Input weights use Kaiming, recurrent weights use orthogonal initialization. Per-layer keys are lN.weight_ih, lN.weight_hh, lN.bias_ih, lN.bias_hh.
03LSTM gates#
Order i,f,g,o: input, forget, candidate, output. The forget block of the input bias starts at 1. Backward propagates both hidden and cell gradients through time.
04GRU gates#
r and z use sigmoid; n uses tanh. Reset is applied to the recurrent candidate projection including bias, before adding the input projection. Check GRU variant compatibility when transferring weights.
05Truncated BPTT#
truncate_bptt=k>0 differentiates only the last k steps: t0=max(0,T−k). Earlier inputs keep zero gradients. Historical states still affect forward, but gradients stop at the cutoff. This is not automatic sliding-window training.
06Expanded LSTM gate equations#
The projection a_t=W_ih x_t+b_ih+W_hh h_(t−1)+b_hh splits into four H-wide blocks. i/f/o use sigmoid, g uses tanh. Sigmoid gates scale information flow in [0,1]; the additive cell path carries long-term derivatives.
07Cell and gate derivatives#
d c_t combines the future cell derivative with the h_t path. It propagates to the previous cell through f_t. Apply sigmoid derivative q(1−q) to i, f, o and 1−g² to g, then differentiate the Linear projections.