Dropout

Inverted dropout, seeds, and train/eval behavior.

2 min readv1.0.0C++20

From the implementationinclude/nexus_model/layers/dropout.hpp

01Constructor#

Dropout(probability=0.5f,seed=0xD1CEU), probability ∈ [0,1]. Masks change on each training forward. Same seed and call sequence give a repeatable series. init::manual_seed does not change this separate seed field.

02Preserve the expected value#

Retained values scale by 1/(1−p). At p=1 output and derivative are zero; eval or p=0 applies the identity.

m_i\sim Bernoulli(1-p),\qquad y_i=\frac{x_im_i}{1-p},\qquad dx_i=\frac{g_im_i}{1-p}

03Recorded forward mode#

training_at_forward is saved; backward uses that forward mask and mode. Attention dropout is implemented separately and requires p<1.

◆Read alongside the source. API behavior reflects the current implementation.

NEXUSMODEL

DocsLayer reference

NEXUSMODEL / 16

Dropout

Inverted dropout, seeds, and train/eval behavior.

2 min readv1.0.0C++20

From the implementationinclude/nexus_model/layers/dropout.hpp

01Constructor#

02Preserve the expected value#

Retained values scale by 1/(1−p). At p=1 output and derivative are zero; eval or p=0 applies the identity.

m_i\sim Bernoulli(1-p),\qquad y_i=\frac{x_im_i}{1-p},\qquad dx_i=\frac{g_im_i}{1-p}

03Recorded forward mode#

training_at_forward is saved; backward uses that forward mask and mode. Attention dropout is implemented separately and requires p<1.

◆Read alongside the source. API behavior reflects the current implementation.