Review

A Transformer block preserves a model-width residual stream while two branches update it. Under the pre-norm convention used in this subject,

What You Should Be Able to Reconstruct

• Residual stream: distinguish the persistent stream from temporary branch updates and calculate both additions.
• LayerNorm: calculate per-token mean and population variance, normalize the feature axis, then apply learned gain and bias.
• RMSNorm: scale by the root mean square without subtracting the mean.
• Operation order: write and calculate both pre-norm and post-norm forms.
• Position-wise MLP: derive projection shapes, parameter counts, and the independence of different sequence positions.
• Activation or gate: name the exact nonlinear operation and compare it under a common input or parameter budget.
• Dropout: separate training from evaluation and state probability and placement.
• Stacking: preserve $(B,T,d_{model})$ while using distinct parameters at different depths unless sharing is explicit.

One Compact Numerical Check

For $r=[1,2,3,4]$:

• LayerNorm has mean $2.5$ and population variance $1.25$;
• RMSNorm has root mean square $\sqrt{7.5}\approx2.738613$;
• the normalized outputs differ because only LayerNorm subtracts the mean.

For any residual branch $F$, remember that

in general. Shape equality does not imply functional equality.

Audit Before You Trust a Block

1. Write the equations in execution order.
2. Label every intermediate shape.
3. Check normalization one token record at a time.
4. Change one position and test MLP independence.
5. Zero one branch at a time and inspect the changed downstream input.
6. Test dropout separately in training and evaluation modes.
7. Count parameters from declared projection shapes and bias conventions.

The next exercises require these calculations without relying on a framework block whose internal choices remain hidden.