Weight Tying Reuses the Token Embedding Matrix

The token embedding maps vocabulary IDs to model-width records. Unembedding maps model-width records back to vocabulary logits. Because their shapes are transposes, one parameter table can serve both roles:

This is weight tying. The two operations use the same learned entries, not two matrices that merely happened to start equal.

Trace Both Directions

For token C, ID 3 selects row

At output, the same row acts as the vocabulary classifier vector for C:

An update to this shared row affects both future token lookups for C and the readout score assigned to C. Tying constrains parameterization; it does not force an input occurrence and an output score to have the same value.

Compare Parameter Counts

The embedding already contributes $Vd_{model}=32$ entries. An independent unembedding would add another $d_{model}V=32$. Tying adds no second matrix.

For the frozen model:

• tied total: 368 entries;
• untied total: $368+32=400$ entries.

If an output bias were included, tying would not remove that separate vector. This model declares no output bias.

Equality Is Not Sharing

A useful audit changes one embedding entry and checks both lookup and output logits. If only one side changes, the implementation copied values instead of sharing the parameter. Serialization and optimizer state must also refer to one shared parameter rather than two independently updated arrays.

Weight tying was studied before Transformer language models. The mechanism transfers cleanly; empirical improvements remain tied to the models and tasks that were evaluated.

References

• Ofir Press and Lior Wolf, Using the Output Embedding to Improve Language Models, 2016.
• Hakan Inan, Khashayar Khosravi, and Richard Socher, Tying Word Vectors and Word Classifiers, 2016.