Transformer blocks

Positional encoding, post-norm encoder, and decoder.

2 min readv1.0.0C++20

From the implementationinclude/nexus_model/layers/attention.hpp

01PositionalEncoding#

PositionalEncoding(embed,max_len=512,learnable=false) adds sine/cosine values to [B,T,E]. Fixed tables grow, but learnable parameter tables retain their initial size; maintain T≤max_len. Learnable tables start sinusoidal and accumulate gradients over the batch.

PE_{p,2i}=\sin(p/10000^{2i/E}),\quad PE_{p,2i+1}=\cos(p/10000^{2i/E})

02TransformerEncoderLayer#

TransformerEncoderLayer(embed,heads,ff=0,dropout=0.1); ff≤0 uses 4·embed. Self-attention→dropout→residual→LayerNorm; Linear→GELU→Linear→dropout→residual→LayerNorm. This is post-norm.

u=LN(x+D(A(x))),\qquad y=LN(u+D(FFN(u)))

1Sequential encoder({2  std::make_shared<PositionalEncoding>(64,128),3  std::make_shared<TransformerEncoderLayer>(64,4,256,0.1f)4});5auto y=encoder.forward(Tensor::zeros({4,32,64}));

03TransformerDecoderLayer#

TransformerDecoderLayer(embed,heads,ff=0,dropout=0.1) contains masked self-attention, cross-attention, and FFN. Call forward(target,memory) first. Single-argument forward requires previously cached memory and throws ModelError on the first call.

04Memory-gradient limitation#

Decoder backward returns only target gradients. Inner cross-attention dk/dv are not included or exposed by the decoder public API. Full encoder-decoder training requires an extension that propagates them or an explicit cross-attention pipeline.

◆Read alongside the source. API behavior reflects the current implementation.

01PositionalEncoding#

PE_{p,2i}=\sin(p/10000^{2i/E}),\quad PE_{p,2i+1}=\cos(p/10000^{2i/E})

02TransformerEncoderLayer#

u=LN(x+D(A(x))),\qquad y=LN(u+D(FFN(u)))

1Sequential encoder({2  std::make_shared<PositionalEncoding>(64,128),3  std::make_shared<TransformerEncoderLayer>(64,4,256,0.1f)4});5auto y=encoder.forward(Tensor::zeros({4,32,64}));