The Transformer began as an encoder-decoder model for sequence-to-sequence tasks. Later systems retained only the encoder, retained only the decoder, or adapted both sides under new training objectives. These families reuse many operations from the previous chapters, but they do not expose the same information to each position.
This chapter compares three common families:
- an encoder-only model lets each non-padding input position read the whole input sequence;
- a decoder-only model uses a causal mask so each position reads only the allowed prefix;
- an encoder-decoder model encodes a source sequence, then uses causal target self-attention and cross-attention to the encoded source.
The comparison holds the tokens small. We will use source tokens A B C and,
when a separate target is needed, target tokens X Y. Exact visibility
matrices will show permitted routes before any model learns attention weights.
Four Questions Define the Comparison
For every architecture, ask:
- Which sequence supplies the query?
- Which sequence supplies the keys and values?
- Which pairs of positions are permitted by the mask?
- Which targets and loss positions define learning?
These questions are more reliable than choosing an architecture from its name. Two systems may use similar blocks but different masks, objectives, readouts, or source-target interfaces.