One Sequence Supplies Queries, Keys, and Values

Self-attention projects queries, keys, and values from the same source sequence, while the mask separately controls which positions may be read. Contrast this source rule with cross-attention and keep the three projected roles distinct.

The word self in self-attention describes where queries, keys, and values come from. They are projected from the same input sequence:

Q=XW_Q,\qquad K=XW_K,\qquad V=XW_V.

It does not mean that a position may attend only to itself. Depending on the mask, a query may read every position, only its causal prefix, or another allowed subset.

The Three Roles Remain Different

Consider a row $x_i$ from $X$ . The same row creates three records:

q_i=x_iW_Q,\qquad k_i=x_iW_K,\qquad v_i=x_iW_V.

Their jobs are not interchangeable:

$q_i$ represents what position $i$ is looking for;
$k_i$ represents how position $i$ can be matched;
$v_i$ represents what position $i$ can contribute.

The query at position $i$ is compared with keys. The resulting weight on column $j$ multiplies $v_j$ , not $k_j$ . A high query-key score therefore routes the corresponding value; it does not copy the key into the result.

Self-Attention and Cross-Attention Differ by Source

The following table separates three choices that are often mixed together:

Operation	Query source	Key/value source	Typical visibility
Bidirectional self-attention	sequence $X$	the same $X$	all unpadded positions
Causal self-attention	sequence $X$	the same $X$	current and earlier positions
Cross-attention	sequence $X$	another sequence $Z$	allowed positions in $Z$

For cross-attention,

Q=XW_Q,\qquad K=ZW_K,\qquad V=ZW_V.

The query length and key-value length may now differ. If $X$ has $T_q$ positions and $Z$ has $T_k$ positions, the score matrix has shape $(T_q,T_k)$ . Chapter 5 will place this operation inside an encoder-decoder architecture. Here it is enough to see that self versus cross and causal versus bidirectional answer different questions.

A Two-Position Example

Let

X=\begin{bmatrix}1&0\\0&1\end{bmatrix},\qquad W_Q=I,\qquad W_K=\begin{bmatrix}1&1\\1&-1\end{bmatrix},\qquad W_V=\begin{bmatrix}2&0\\0&3\end{bmatrix}.

Then

Q=\begin{bmatrix}1&0\\0&1\end{bmatrix},\quad K=\begin{bmatrix}1&1\\1&-1\end{bmatrix},\quad V=\begin{bmatrix}2&0\\0&3\end{bmatrix}.

The first query has the same dot product, 1, with both keys. If both positions are visible, softmax assigns equal weights and the reading is

\tfrac12[2,0]+\tfrac12[0,3]=[1,1.5].

The keys decided that the match was tied. The values decided that the reading contained 1 in its first coordinate and 1.5 in its second. Replacing $W_V$ would change the reading without changing any attention weight.

Q1. Separate source from visibility

A layer forms Q, K, and V from the same sequence, but a mask lets each query read only its causal prefix. Which description is exact?

Choose one

Causal self-attention

Cross-attention

Attention only to the same position

Select one choice, then check.

Hint

Ask two questions: are the source tensors the same, and what does the mask allow?

Solution

It is causal self-attention. Q, K, and V share the same source, while the mask restricts each query to its causal prefix.

Use Two Questions When Reading an Architecture

For any attention operation, first identify the source of Q and the source of K/V. Then identify the visibility mask. This two-question method is more reliable than inferring behavior from the word attention alone.