Masked-Token Prediction Uses Corrupted Inputs

Masked-token prediction creates targets by hiding or changing selected input tokens. An encoder reads the corrupted sequence with left and right context, then predicts the original token at selected positions.

Separate Clean Text, Model Input, and Target

Start with the clean token sequence:

A B C

Select position 2 and present the corrupted input:

A <mask> C

The target at position 2 is B. If the vocabulary logits at that position are $z_2:(V)$, the selected loss is

The model is permitted to use both A and C because its self-attention is not causal.

Loss Selection Is Not Attention Masking

The selected prediction position still participates in attention. “Masked” in masked-token prediction describes input corruption and target selection; it is not the same operation as filling an attention-score entry with $-\infty$.

Keep three ledgers:

This separation also exposes accidental leakage, such as leaving B unchanged while treating the prediction as if it had been hidden under every corruption policy.

BERT Is One Historical Design, Not the Family Definition

BERT used a particular corruption procedure and combined masked-language-model pretraining with a next-sentence objective in its reported experiments. Later encoder models changed objectives, corruption, data, and training. Encoder-only means the architecture retains the encoder information flow; it does not mean every model must reproduce all BERT training choices.

A Small Loss Calculation

If the model assigns target probability $p(B)=0.25$, the selected-token loss is

using natural logarithms. Unselected positions contribute nothing under a selected-only mean, though another implementation may report additional diagnostics.

Reference

• Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova, BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding, 2018.