Review

Review the frozen token-to-logit chain, block and readout shapes, tied and untied parameter ledgers, target alignment, stable loss, causal intervention, and architecture audit order.

The frozen decoder maps IDs to vocabulary logits through one explicit chain:

I\rightarrow E[I]+P[:T]\rightarrow X^{(0)} \rightarrow X^{(1)}\rightarrow X^{(2)} \rightarrow N_f(X^{(2)})\rightarrow Z.

Every decoder block preserves $(B,T,4)$ and applies its own causal mask. Final LayerNorm acts across the feature axis. Tied unembedding uses $W_U=E^\top$ to produce $Z:(B,T,8)$ .

Frozen Configuration

$V$	$C$	$d_{model}$	$L$	$h$	$d_h$	$d_{ff}$
8	4	4	2	2	2	8

The deterministic audit uses learned absolute positions, pre-LayerNorm blocks, ReLU, final LayerNorm, zero dropout, attention matrices without bias, biased MLPs, no output bias, and tied input-output weights.

Reconstruct the Parameter Count

embedding: 32;
position table: 16;
each block: $64+76+16=156$ ;
two blocks: 312;
final LayerNorm: 8;
tied unembedding: 0 additional entries.

The tied total is 368. An independent $4\times8$ unembedding raises it to 400.

Reconstruct the Target Contract

For input <bos> A B C, the target is A B C <eos>. Each logits row predicts the token one step to its right. The causal mask prevents later input positions from affecting earlier rows even though all known training positions are calculated in parallel.

For target $y$ and logit row $z$ :

\ell=-z_y+\operatorname{logsumexp}(z).

Average only included target positions. Stable log-sum-exp subtracts the row maximum before exponentiation.

Audit Order

ID and context ranges;
lookup and position rows;
block shapes and causal interventions;
final-normalization axis;
tied parameter identity and vocabulary logits;
input-target shift;
vocabulary-axis cross-entropy and valid denominator;
complete parameter ledger.

The next chapter keeps these contracts fixed while introducing trainable parameters, batches, gradients, optimization, validation, and checkpoints.