Encoder-Decoder Models Separate Source and Target Streams

An encoder-decoder Transformer maintains two sequences. The encoder turns a source sequence into contextual records. The decoder builds target records causally while consulting those encoder records through cross-attention.

For source length $T_s$ and target-input length $T_t$:

Each decoder layer ordinarily contains causal target self-attention, source-target cross-attention, and an MLP, with residual and normalization details stated separately.

Shift the Target Stream

Suppose source A B C should produce target X Y <eos>. A teacher-forced training example may use:

The encoder sees A B C. The decoder never receives the token it is currently asked to predict at the same prediction step. A missing or inconsistent shift can leak the answer while leaving dimensions valid.

Three Routes Serve Different Purposes

• Encoder self-attention exchanges information within the source.
• Decoder self-attention exchanges information within the allowed target prefix.
• Cross-attention lets a target query retrieve from encoded source records.

Removing any one route changes the function. Cross-attention does not replace causal target self-attention: generated target tokens also provide relevant context for later target predictions.

Conditional Generation Is Broader Than Translation

The original Transformer studied translation, and encoder-decoder models are natural when one sequence conditions another. The same information flow can support summarization and other structured source-to-target tasks. Performance still depends on objective, data, model size, decoding, and evaluation.

References

• Ashish Vaswani et al., Attention Is All You Need, 2017.
• Colin Raffel et al., Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer, 2019.