From mathematics to training
A numerical derivative walkthrough and a complete program learning y=2x+1.
01Compute one example by hand#
With x=[1,2], W=[3,4], b=0, a single-output Linear gives y=11. Let target t=10 and L=(y−t)²/2; loss is 0.5 and external dy=1. Passing 1 to backward is correct for this particular loss.
02What the gradient means#
dW=[1,2], db=1, dx=[3,4]. SGD with η=0.1 changes weights to [2.9,3.8] and bias to −0.1. The same input now gives y=10.4 and loss 0.08. A derivative describes the direction in which a small parameter change affects loss.
03Batch averaging#
The program below uses MSE=(1/B)Σ(y−t)², so it needs a 2/B factor unlike the half-square example above. Apply batch averaging exactly once. zero_grad clears accumulation at the start of each step.
04Complete training example#
This uses NexusModel alone, with MSE and simple SGD written explicitly for teaching. Use NexusOptim integration for general optimizer features. The program exits successfully when loss is below 1e−3 after 200 steps.
#include <nexus_model/nexus_model.hpp>#include <iostream>using namespace nexus_model;int main() { init::manual_seed(42); Linear model(1,1); auto x=Tensor::from_values({5,1},{-2,-1,0,1,2}); auto target=Tensor::from_values({5,1},{-3,-1,1,3,5}); auto dy=Tensor::zeros({5,1}); float loss=0; for(int step=0;step<200;++step) { model.zero_grad(); auto y=model.forward(x); loss=0; for(std::size_t i=0;i<y.numel();++i) { float e=y[i]-target[i]; loss+=e*e/5.f; dy[i]=2.f*e/5.f; } model.backward(dy); // Educational SGD; use NexusOptim for production optimizers. for(auto* p:model.parameters()) for(std::size_t i=0;i<p->data.numel();++i) p->data[i]-=0.03f*p->grad[i]; } std::cout<<"MSE: "<<loss<<"\n"; return loss<1e-3f ? 0 : 1;}