Count Parameters and Attention Work

The phrase “more heads” does not by itself determine parameter count or computation. We must also know the per-head widths, total projected widths, bias convention, sequence length, and which tensors are materialized.

General Bias-Free Parameter Count

With equal widths across heads, the projection matrices contain

The total is

Separate per-head matrices and combined matrices have the same number of entries when they represent the same widths. Combining is an implementation arrangement, not parameter sharing among heads.

Fixed Total Width Makes the Standard Count Independent of Head Count

Under the common convention

the count becomes

Changing $h$ while reducing each head width to keep the total widths fixed does not change this projection count. For $d=512$, one head of width 512 and eight heads of width 64 both use $4(512)^2=1{,}048{,}576$ bias-free projection parameters.

They do not compute the same function: the one-head version has one softmax distribution, while the eight-head version has eight.

If an implementation uses projection biases, add their exact lengths. Under one common convention, Q, K, V, and output biases add $hd_k+hd_k+hd_v+d_{model}$ entries. Some models omit selected biases, so the convention must be stated rather than assumed.

Dense Scores Depend on Head Count and Sequence Length

Each head forms $T^2$ scores per batch item. Across $B$ items and $h$ heads, the score tensor contains

entries. The approximate multiply-add work for scores and weighted values is

When $hd_k$ and $hd_v$ remain fixed, these expressions are approximately independent of how the total width is split among heads. The number of separate score matrices still grows with $h$, which can affect kernels, memory layout, and overhead even when the leading arithmetic count is similar.

Compare Counts Under Explicit Conventions

Let $d_{model}=12$, $T=100$, $B=2$, with no biases.

The score entry count triples because there are three separate attention maps. Each map's score dot products are only one third as wide, so the leading score arithmetic remains comparable under this fixed-total-width setup. Entry count, arithmetic, memory, and wall-clock time are related but not interchangeable. The first row uses $4(12)^2=576$ projection parameters and $2(1)(100)^2=20{,}000$ score entries. Replacing one head with three gives $2(3)(100)^2=60{,}000$ score entries while leaving the projection count fixed.

Never Compare Head Counts Without Widths

“Eight heads cost eight times as much as one” is false under the common fixed total width. “Eight heads are free” is also false: eight dense maps are stored, and implementation overhead changes. A useful comparison states parameter widths, tensor entries, arithmetic, and measured runtime separately.