Dropout and Stacking Preserve the Interface

Inverted dropout changes branch values during training but becomes the identity at evaluation. Stacked blocks preserve model width while ordinarily using distinct parameters at each depth.

Dropout changes values during training without changing tensor shape. Stacking then passes the model-width output of one block to the next block. These two facts preserve the interface $(B,T,d_{model})$ , but they do not make repeated blocks identical.

Training and Evaluation Calculate Different Functions

Under inverted dropout with drop probability $p$ , each branch coordinate uses

\operatorname{Dropout}(z_i)=\frac{m_i z_i}{1-p}, \qquad m_i\sim\operatorname{Bernoulli}(1-p)

during training. During evaluation, dropout returns $z_i$ unchanged. The factor $1/(1-p)$ keeps the training-time expectation equal to $z_i$ under the mask distribution.

For $z=[2,4]$ , $p=0.5$ , and mask $m=[1,0]$ , training returns $[4,0]$ . Evaluation returns $[2,4]$ .

Place Dropout Precisely

One explicit pre-norm residual branch is

y=x+\operatorname{Dropout}(F(N(x))).

Moving dropout before $F$ , after the residual addition, or into attention weights defines another computation. A reproducible account names the exact placement and the training/evaluation mode.

A Stack Reuses Shapes, Not Parameters

If each block maps $(B,T,d_{model})$ to the same shape, $L$ blocks can be composed:

x^{(\ell+1)}=\operatorname{Block}^{(\ell)}(x^{(\ell)}).

Ordinary Transformer stacks use different learned parameters at different depths. Sharing parameters across layers is possible, but it is a separate architectural choice.

Reproducible Dropout Checks

Fix a random seed when comparing exact training outputs.
Test evaluation mode separately and expect deterministic values.
Verify that dropout changes no dimensions.
Compare averages over many masks when testing the expectation, not one mask.
Record probability and placement with the architecture.

Q1. Apply inverted dropout

With $p=0.25$ , input coordinate $z_i=3$ , and retained mask value $m_i=1$ , what training-time value is returned?

Your answer

Compute it first, then check your number.

Hint

Use

m_i z_i/(1-p)

Solution

3/(1-0.25)=4