Training loop

Loss, gradient accumulation, and NexusOptim.

2 min readv1.0.0C++20

From the implementationexamples/train_end_to_end_example.cpp

01Order of a step#

zero_grad → forward → loss.backward → model.backward → optimizer.step. Gradients accumulate across backward calls. This enables micro-batch accumulation, but requires clearing between independent steps.

\theta_{t+1}=\theta_t-\eta\nabla_\theta L

02Borrowed parameter bridge#

as_param_group passes data and gradient pointers only for requires_grad=true parameters. Do not replace parameter storage or destroy the model while the optimizer is alive. The adapter is included separately from the core.

1#include <nexus_model/adapters/nexus_optim.hpp>2#include <nexus_optim/algorithms/adamw.hpp>3nexus_optim::ParamGroupOptions group;4group.learning_rate = 1e-2;5nexus_optim::AdamW<float>::Options options;6options.lr = 1e-2;7options.weight_decay = 1e-2;8nexus_optim::AdamW<float> optimizer(9  {nexus_model::as_param_group(model, group)}, options);

03From logits to gradients#

The example calls CrossEntropyLoss<float> per row. Copy backward output before the next row and divide by batch size. Pass logits to CrossEntropy; do not prepend Softmax.

L=-\frac1B\sum_b\log p_{b,y_b},\qquad \frac{\partial L}{\partial z_{b,c}}=\frac{p_{b,c}-\mathbf1[c=y_b]}B

04End-to-end example#

NexusData, NexusLoss, and NexusOptim must be sibling directories. The example uses 96 synthetic samples, 8 features, 3 classes, and 8 AdamW epochs. This is an integration check.

1cmake -S . -B build-e2e -DNEXUS_MODEL_BUILD_EXAMPLES=ON2cmake --build build-e2e --config Release3.\build-e2e\Release\train_end_to_end.exe

◆Read alongside the source. API behavior reflects the current implementation.