Unembedding Produces One Logit per Vocabulary Token

Unembedding maps each final model-width record to one score for every vocabulary token. With $H:(B,T,d_{model})$ and $W_U:(d_{model},V)$,

The values in $Z$ are logits. They are not probabilities and do not need to sum to 1.

Calculate One Tied Logit Row

Under weight tying, $W_U=E^\top$. Use

The logit for token C is the dot product with its embedding row $E_C=[0,0,0,1]$:

The A logit is $-0.852803$. The <eos> embedding is $[1,0,0,1]$, so

Calculating every token gives approximately

C has the largest logit in this illustrative row. These sparse untrained weights provide arithmetic, not a claim about learned language behavior.

Softmax Uses the Vocabulary Axis

For each $(b,t)$ row, softmax normalizes across $V=8$. Normalizing across time would make positions compete with one another and would not define a next-token distribution. Stable implementations subtract the row maximum before exponentiation.

Top-logit selection can be performed without materializing probabilities, but probabilities or log-probabilities are required for calibrated sampling and cross-entropy calculations.