Model Width, Head Count, and Head Width

Multi-head attention uses several widths with different jobs:

• $d_{model}$ is the feature width entering and leaving the layer;
• $h$ is the number of attention heads;
• $d_k$ is the query and key width inside each head;
• $d_v$ is the value and head-output width inside each head.

These symbols describe an architecture. They are not automatically linked by one equation.

The Common Equal-Split Configuration

Many Transformers choose

Then the concatenated head width is

and $d_{model}$ must be divisible by $h$. For example, with $d_{model}=512$ and $h=8$, each head commonly uses $d_k=d_v=64$.

This is a design convention, not the definition of multi-head attention. A valid layer may use $hd_v\ne d_{model}$ and let $W_O$ map the concatenated width back to $d_{model}$. Query-key width and value width may also differ.

Separate Projection Shapes

For each head $r$,

The head result has shape

Concatenating $h$ results along their feature axes gives

so the output projection has shape

Compare Two Valid Configurations

Both are mathematically valid. The second uses a larger combined value width and therefore a larger $W_V$ total projection and output projection. Calling both “three-head attention” does not make their parameter counts equal.

Divisibility Depends on the Implementation Choice

If an implementation reshapes one $d_{model}$-wide vector directly into $h$ equal heads, then $d_{model}/h$ must be an integer. If it first projects into an explicit total width $hd_k$, only that total projected width needs to split evenly under the chosen equal-head convention.

Framework APIs often impose the common equal split for convenience. When an API rejects a configuration, distinguish a mathematical incompatibility from an interface constraint.

Write the Configuration Before the Shapes

A shape trace is ambiguous if $h,d_k,d_v$, projection convention, and axis order are missing. State those choices first. Only then is a reshape such as $(B,T,hd_k)\to(B,T,h,d_k)$ justified.