Validation & limitations

Gradient checks, memory tests, and usage contracts.

2 min readv1.0.0C++20

From the implementationtests/test_model.cpp

01Correctness checks#

Tests cover Linear/Conv references, central finite differences, activations, normalization, recurrent/attention, state_dict, SIMD, and hot-path allocations. Kink points of piecewise functions are skipped.

g_i\approx\frac{L(\theta_i+\varepsilon)-L(\theta_i-\varepsilon)}{2\varepsilon}

02Choosing tolerances#

Too-small epsilon suffers float32 rounding; too-large epsilon increases approximation error. Use absolute and relative errors. Disable dropout or control its mask.

e_{rel}=\frac{|g_a-g_n|}{\max(1,|g_a|,|g_n|)}

03Allocation claim#

Tests check allocation_count stays unchanged on warmed fixed-shape paths after output headers are released. This does not imply no heap use across the application; the counter covers AlignedStorage only.

04Caller responsibilities#

Ensure positive dimensions, correct backward shapes, valid indices, head divisibility, and memory lifetime. Not every kernel performs full validation. Do not run concurrent forward on one module. Keep learnable PositionalEncoding within max_len.

05Troubleshooting#

Large gradients: check zero_grad and batch averaging. Shape errors: check the last Linear axis and Conv NCHW. New allocations: are old outputs still alive? CPU use with CUDA enabled: host Module intentionally uses the CPU path.

◆Read alongside the source. API behavior reflects the current implementation.

01Correctness checks#

g_i\approx\frac{L(\theta_i+\varepsilon)-L(\theta_i-\varepsilon)}{2\varepsilon}

02Choosing tolerances#

Too-small epsilon suffers float32 rounding; too-large epsilon increases approximation error. Use absolute and relative errors. Disable dropout or control its mask.

e_{rel}=\frac{|g_a-g_n|}{\max(1,|g_a|,|g_n|)}