Interactive explanation

How a language model processes text

Follow one small decoder-only transformer from text to a next-token probability distribution. Begin with the complete architecture, then open the token, vector, attention, residual, and output calculations.

The complete decoder-only path

Begin with the whole model, then open the calculations below. The diagram uses the visual structure of the canonical Transformer while showing the decoder-only path used by many text-generating LLMs.

Decoder-only transformer architecturePrompt text is tokenized, converted to token and position vectors, passed through repeated pre-normalization causal-attention and feed-forward sublayers with residual paths, and projected into next-token probabilities.Prompt textAttention connects words to contextTokenizertoken strings → vocabulary IDsToken embeddingID selects one learned rowPosition informationencodes token order+Decoder-only transformer block× NInput vector XLayer normalizationCausal self-attentionQ, K, VprojectionsQKᵀ/√dscoresMaskcausal + softmaxWeights × Vcontexteach token sees only itself and earlier tokens+residualLayer normalizationFeed-forward networkup projection → activation → down projectionthe same learned transformation is applied at every position+residualFinal normalization + vocabulary projectionhidden vector → logitsSoftmax → next-token probabilities
A modern decoder-only language model. The original 2017 Transformer also included an encoder and cross-attention for sequence-to-sequence tasks. Normalization order and position methods vary across models.

The trace updates while you type. Update link makes the current text part of the page URL so that you can share the same example.

  1. 1

    Convert text into tokens and token IDs

    A tokenizer splits text into units from a vocabulary. Each token has an integer ID: its address in that vocabulary. The number is an index, not a measure of the token's meaning.

    position 0TheID 28position 1modelID 90position 2learnsID 146position 3fromID 117position 4dataID 165
    How this teaching tokenizer differs from a production tokenizer

    Production tokenizers usually split unfamiliar words into reusable subword pieces and look them up in a fixed vocabulary containing thousands of entries. To accept arbitrary text without shipping such a vocabulary, this page separates words and punctuation and maps them deterministically into a fixed table of 256 rows. The later matrix operations are real, but these teaching IDs are not GPT token IDs.

  2. 2

    Look up token vectors and add position information

    The token ID selects one row from an embedding table. That row is the token vector. A separate position vector records where the token occurs. Both vectors have the same shape, so the model can add them element by element.

    Embedding lookup and position addition for each token
    Token and IDEmbedding rowPosition vectorInput to the block
    The · 28[-0.33, 0.32, 0.58, 0.15][0.00, 0.34, 0.00, 0.34][-0.33, 0.66, 0.58, 0.49]
    model · 90[0.37, -0.57, 0.31, 0.39][0.29, 0.18, 0.03, 0.34][0.66, -0.39, 0.35, 0.73]
    learns · 146[-0.57, -0.19, -0.32, -0.56][0.31, -0.14, 0.07, 0.33][-0.26, -0.33, -0.25, -0.22]
    from · 117[-0.54, -0.47, -0.39, -0.51][0.05, -0.34, 0.10, 0.32][-0.49, -0.81, -0.29, -0.18]
    data · 165[-0.39, 0.30, 0.50, -0.58][-0.26, -0.22, 0.13, 0.31][-0.65, 0.07, 0.63, -0.27]
    How the position vector is generated

    This teaching model uses four sinusoidal values at positionp: sin(p), cos(p), sin(p/10), and cos(p/10), each scaled by 0.34. The original Transformer used sinusoidal encodings with many frequencies. Other models use learned position embeddings or rotary position encodings. These methods differ, but each gives the model information about token order.

  3. 3

    Use causal self-attention to collect context

    After normalization, three learned matrices project every input vector into a query, key, and value. A query is compared with the visible keys. Softmax converts the comparison scores into weights, and those weights combine the value vectors.

    Attention weights for one teaching head. Rows query; columns provide keys and values.
    Query ↓ / Key →Themodellearnsfromdata
    The100.0%maskedmaskedmaskedmasked
    model19.4%80.6%maskedmaskedmasked
    learns11.1%47.5%41.3%maskedmasked
    from8.5%33.0%30.6%27.8%masked
    data14.3%17.6%21.4%23.5%23.1%

    Darker cells have larger weights. The upper-right cells are masked because a decoder cannot use future tokens when predicting the next token. This is one head in one teaching block; real models contain many heads and layers. Attention weights show the weighting operation, not a complete explanation of model reasoning.

    Inspect the query, key, and value vectors
    One query, key, and value vector for every position
    TokenQuery, QKey, KValue, V
    The[-1.21, 0.30, 0.91, 0.13][-1.04, 0.82, -0.06, 0.27][-1.36, 0.85, 0.25, 0.18]
    model[0.89, -0.82, -0.30, 0.92][0.33, -1.32, 0.51, 0.32][0.35, -1.21, 0.23, 0.44]
    learns[0.62, -0.91, 0.09, 1.05][-0.04, -1.21, 0.66, 0.36][-0.07, -1.11, 0.48, 0.45]
    from[0.38, -0.91, 0.34, 1.09][-0.30, -1.06, 0.71, 0.38][-0.38, -0.96, 0.61, 0.46]
    data[-0.65, -0.27, 1.27, -0.21][-1.06, 0.62, 0.52, -0.46][-0.91, 0.46, 0.97, -0.53]
    Follow the final token's attention calculation

    For four-number queries and keys, the scale is √4 = 2. The final row therefore computes softmax(QKᵀ / 2) over every visible position. Its weights sum to one.

    Final query [-0.65, -0.27, 1.27, -0.21]

    Attention weights [0.14, 0.18, 0.21, 0.24, 0.23]

    Weighted context [-0.45, -0.45, 0.55, 0.19]

  4. 4

    Complete the transformer block

    The attention result is projected back to the model-vector shape and added to the block input through a residual path. The model normalizes that sum, applies a feed-forward network, and adds a second residual update. This page uses a pre-normalization decoder block; other Transformer variants place normalization in a different order.

    1. InputX
    2. NormalizeNorm(X)
    3. Causal attentionAttention(Q, K, V)
    4. ResidualX + attention
    5. Normalize + MLPlinear → ReLU → linear
    6. ResidualX + MLP
    Inspect the final position after each operation

    Attention update [-0.17, -0.39, 0.40, 0.23]

    After the first residual [-0.82, -0.31, 1.04, -0.04]

    Input to the feed-forward network [-1.16, -0.41, 1.58, -0.00]

    Feed-forward update [0.03, 0.27, 0.49, 0.08]

    Block output [-0.79, -0.04, 1.53, 0.04]

  5. 5

    Project the result into vocabulary probabilities

    After the final normalization, an output projection produces one score, called a logit, for every token in the output vocabulary. Softmax converts the logits into probabilities that sum to one.

    Final hidden vector [-1.16, -0.27, 1.60, -0.17]

    data86.6%
    .6.2%
    examples3.0%
    tokens2.8%
    patterns1.0%
    text0.4%
    Inspect the logits before softmax
    data
    2.812
    .
    0.171
    examples
    -0.536
    tokens
    -0.613
    patterns
    -1.647
    text
    -2.689
  6. 6

    Select one token and repeat

    A decoding rule selects from the probability distribution. If this teaching model chooses its highest-probability token, the prompt becomes The model learns from data data. The model then processes that longer sequence to produce another token. Repeating this loop produces text one token at a time.

    Production systems may adjust the distribution with temperature, top-k, top-p, repetition penalties, or other decoding rules. Those rules change selection; they do not retrain the model.

References and further exploration

  • Vaswani et al., Attention Is All You Need, for the original encoder–decoder Transformer, scaled dot-product attention, multi-head attention, and sinusoidal position encoding.
  • Brandon Rohrer, Transformers from Scratch, for a visual construction of the architecture from smaller mathematical ideas.
  • Cho et al., Transformer Explainer, for an interactive GPT-2 visualization that moves between model structure and lower-level calculations.

Learn the ideas behind each step

LLM Primer develops the programming and mathematics needed to replace this toy trace with complete implementations.

Start with PythonView curriculum