Review

Review encoder-only, decoder-only, and encoder-decoder sources, masks, objectives, tensor shapes, readouts, target shifts, cross-attention, and architecture audits.

Transformer families reuse attention blocks under different information-flow contracts. The decisive details are the query, key, and value sources; the visibility mask; the prediction targets; and the output interface.

Reconstruct the Three Families

Encoder-only

One input stream supplies self-attention queries, keys, and values.
Every non-padding position may usually read every non-padding input position.
The output is one contextual record per input position.
A separate readout chooses token-level, pooled, or designated-position use.

Decoder-only

One token stream supplies self-attention queries, keys, and values.
A causal mask preserves prefix-only dependence.
Input and target tokens are shifted for next-token prediction.
Known training sequences can be processed in parallel; generation still reveals new tokens sequentially.

Encoder-decoder

Encoder self-attention contextualizes the source.
Decoder self-attention is causal over the target prefix.
Cross-attention uses decoder queries and encoder keys and values.
The target stream is shifted so the current answer is not present in its prediction input.

Preserve the Shape Ledger

For source length $T_s$ and target length $T_t$ :

Quantity	Shape
encoder state	$(B,T_s,d_{model})$
decoder state	$(B,T_t,d_{model})$
encoder self-attention scores	$(B,h,T_s,T_s)$
decoder self-attention scores	$(B,h,T_t,T_t)$
cross-attention scores	$(B,h,T_t,T_s)$
target vocabulary logits	$(B,T_t,V)$

The cross-attention score axes are a useful source audit: target queries form rows, and encoded source keys form columns.

Keep Four Distinctions Visible

A visibility mask permits routes; learned attention weights choose among permitted routes.
Input corruption for masked-token prediction is not an attention mask.
Parallel computation over a known training sequence does not permit future information under a correct causal mask.
Architecture family does not by itself determine objective, readout, data, scale, post-training, or capability.

Audit Before Comparing Results

Write clean source and target text, actual model inputs, every visibility matrix, Q/K/V sources, logits-target alignment, padding behavior, and excluded loss positions. Then test a forbidden dependence by changing information that should be unavailable. A model name is not a substitute for this evidence.

Chapter 6 now fixes one decoder-only convention and follows it from token IDs through repeated blocks to vocabulary logits and next-token loss.