MicroTape
A small reverse-mode graph for attention.
From the implementationinclude/nexus_model/core/micro_tape.hpp
01Lifetime#
reset clears the used-slot count and operation list while keeping capacity. load copies a tensor into a slot and returns an int ID. The next reset invalidates old graph IDs. This is not global autograd.
02Operations#
matmul and matmul_bt use rank-3 batched matrices. add/mul are elementwise same-shape operations; add_matrix adds a repeated constant mask without differentiating it. Do not assume general broadcasting.
| Operation | Rule |
|---|---|
| matmul | [B,M,K] × [B,K,N] → [B,M,N] |
| matmul_bt | [B,M,K] × [B,N,K]ᵀ → [B,M,N] |
| scale | factor · a |
| add / mul | a+b / a⊙b |
| add_matrix | [B,M,N] + [M,N] |
| softmax | last axis |
03External gradient#
backward(root,grad) zeros all used-slot derivatives, copies the seed gradient to the root, and reverses the operations. value(id) and grad(id) access results.
MicroTape tape;auto q = Tensor::full({1,2,4}, 0.1f);auto k = Tensor::full({1,3,4}, 0.2f);int iq=tape.load(q), ik=tape.load(k);int p=tape.softmax(tape.scale(tape.matmul_bt(iq,ik),0.5f));tape.backward(p, Tensor::full({1,2,3},1.f));◆Read alongside the source. API behavior reflects the current implementation.