Review

Review the complete single-head self-attention layer, its tensor shapes, projection roles, mask rules, output interface, costs, interpretation limits, and a practical audit order.

The Complete Layer

For batch-major input

X\in\mathbb R^{B\times T\times d_{model}},

one single-head self-attention layer without projection biases is

\begin{aligned} Q&=XW_Q, & W_Q&:d_{model}\times d_k,\\ K&=XW_K, & W_K&:d_{model}\times d_k,\\ V&=XW_V, & W_V&:d_{model}\times d_v,\\ S&=\frac{QK^\top}{\sqrt{d_k}}, & S&:B\times T\times T,\\ A&=\operatorname{softmax}_{last}(S+M), & A&:B\times T\times T,\\ H&=AV, & H&:B\times T\times d_v,\\ Y&=HW_O, & W_O&:d_v\times d_{model}. \end{aligned}

The transpose swaps the final two axes of $K$ within each batch item. The mask is applied before softmax. Softmax normalizes key-position columns within every query row. The output projection returns the feature axis to $d_{model}$ .

Distinctions to Keep

Self-attention and cross-attention

In self-attention, Q, K, and V are projected from the same source sequence. In cross-attention, queries come from one sequence and keys and values from another. The mask is a separate choice: self-attention may be causal or bidirectional.

Score, weight, value, and contribution

$S_{ij}$ is an unnormalized query-key match. $A_{ij}$ is its row-normalized weight. $v_j$ is the content vector at source position $j$ . $A_{ij}v_j$ is that source's vector contribution to the reading at position $i$ . None of these quantities can be substituted for the others.

Causal and padding masks

A causal mask removes real future positions. A padding mask removes artificial positions added to make sequence lengths equal in a batch. They may be combined, but they express different rules.

An attention layer and a Transformer block

This chapter's layer ends at $Y=HW_O$ . It still lacks position information, multiple heads, a residual update, normalization, and a position-wise MLP. Those components change the architecture and cannot be inferred from the word attention.

What the Projection Ablations Establish

Changing $W_Q$ changes what each position requests.
Changing $W_K$ changes how each position can be matched.
Changing $W_V$ changes routed content without directly changing attention weights.
Changing $W_O$ changes how weighted value features are written into model width without changing the attention weights.
Setting $W_O=0$ makes this branch's output zero for the current forward pass; it does not by itself measure how an entire trained network would compensate.

These are controlled mathematical statements. Semantic labels for trained features require further evidence.

Benefit, Cost, and Limits

One self-attention layer permits a direct information path between allowed positions. Dense attention also creates $T^2$ query-key scores per sequence and head. Known training tokens can be processed together under a causal mask, but autoregressive generation still waits for each newly selected token.

An attention heat map records weights in one operation. It is not a complete causal explanation of a model output because values, projections, residual paths, later computation, and alternative weight patterns also matter.

Audit Order

When an implementation fails, inspect the first incorrect object in dependency order:

input and projection shapes;
Q, K, and V shapes and selected rows;
score shape and scale;
allowed and forbidden mask entries;
finite softmax values and row sums;
exact zeroes at forbidden positions;
weighted value shape and selected rows;
output-projection orientation and final width;
independence between batch items.

A passing final shape is weak evidence. Many axis mistakes still return a finite tensor with the expected dimensions.

If a Step Is Unclear

Review Queries Compare with Keys and Read Values for scores, scaling, and masks.
Review Attention Returns a Weighted Vector Sum for the four-token numerical trace.
Review Softmax Turns Scores into a Soft Alignment for stable normalization and interpretation limits.
Return to Add the Batch Axis Without Mixing Sequences when the formulas are clear but tensor axes are not.

Check Yourself

Can you derive every intermediate shape from $B,T,d_{model},d_k,d_v$ ?
Can you explain why Q and K widths must match while V may differ?
Can you identify the softmax axis from the meaning of one query row?
Can you explain the output projection without calling it merely a reshape?
Can you distinguish batch sharing of parameters from mixing batch data?
Can you state the direct-path benefit together with quadratic dense cost?
Can you separate an observed attention weight from a causal claim?

One head produces one learned routing-and-content computation. The next chapter asks what changes when several heads operate in parallel subspaces and their results must be joined without losing track of any axis.