Match Architecture, Objective, and Interface to a Task

Architecture family is one design choice among several. A useful selection starts from the information available at prediction time, the desired output interface, and the training signal—not from a model name alone.

“Natural starting family” does not mean “only possible family.” Prompting, special tokens, pooling, adapters, and alternative masks can let a family serve other tasks. Such adaptations must still be described and evaluated.

Match the Objective Too

The same architecture can be trained under different objectives. Conversely, related objectives can be implemented under different interfaces. Record at least:

• model input and any corruption;
• visibility mask;
• prediction targets and their shift;
• positions included in the loss;
• readout used for evaluation;
• inference procedure.

Without this ledger, “BERT-like,” “GPT-like,” or “T5-like” hides the details needed to reproduce the computation.

Compare Costs Under Declared Shapes

Full encoder self-attention stores $T_s^2$ score entries per head. Decoder self-attention also forms a $T_t^2$ dense matrix in a common implementation, even though its upper triangle is forbidden. Cross-attention adds $T_tT_s$ score entries per head and its own projections in each decoder layer that contains it.

These counts do not by themselves determine runtime or model quality. Kernels, sparsity, caching, widths, layer counts, and hardware matter.

Keep Capability Claims Bounded

A family determines permitted information paths. It does not guarantee that a trained model learns a desired skill, uses a particular route, remains factual, or generalizes outside its evaluation. Attach capability claims to a concrete model, data and adaptation procedure, task, metric, and comparison.