Encoder-Only Models Build Contextual Input Records

An encoder-only Transformer maps an input sequence to a sequence of contextual records. Under full self-attention, every non-padding output position may depend on every non-padding input position.

For input representations $X:(B,T_s,d_{model})$, an encoder block returns

The sequence length and model width are preserved. What changes is the content of each record: $H_{b,t}$ can combine information routed from multiple input positions across multiple blocks.

Full Visibility Is Not Uniform Attention

For A B C, full visibility permits all nine query-source pairs. Softmax still produces a separately learned distribution for each query and head. A mask says what is possible; scores and learned projections determine the actual weights.

This distinction prevents a common error: “bidirectional” means that both left and right context are permitted, not that the model uses them equally.

Padding Must Not Become Evidence

Suppose a batch contains:

example 1: A B C
example 2: D E <pad>

A source padding mask should prevent queries from reading the <pad> key-value record. Many applications also exclude padded query positions from their loss or readout. These are two decisions:

• key masking controls what other positions may read;
• query/loss masking controls whether a padded output is used.

Confusing them can preserve all tensor shapes while letting padding influence real token representations or metrics.

Choose a Readout for the Task

The encoder returns one record per position. A later component may:

• classify every token from its corresponding record;
• pool several records for sequence classification;
• use a designated record under a documented convention;
• expose all records as memory for a decoder.

The encoder architecture does not make one readout mandatory.