Multimodal Transformers Need an Input and Fusion Contract

“Multimodal Transformer” does not identify one architecture. First ask how each modality becomes records and where information from different modalities can mix.

Tokenize an Image into Patches

An image of height $H$, width $W$, and $C$ channels can be divided into non-overlapping $P\times P$ patches. The number of patch tokens is

when both dimensions are divisible by $P$. Each flattened patch has $P^2C$ values before projection to $d_{model}$.

For a $32\times32$ RGB image and $P=8$:

This changes the input contract. The resulting records can still pass through ordinary Transformer blocks. Vision Transformer is one influential instance of this approach under its stated pretraining and transfer settings.

Choose Where Modalities Meet

Image and text records can be concatenated into one sequence, processed by separate encoders and joined by cross-attention, or connected through learned adapter/resampler modules. These are different fusion contracts.

Flamingo provides one image-language fusion design. It does not define all multimodal systems.

Do Not Hide the Data Contract

Architecture alone does not establish grounding or cross-modal understanding. Alignment data, objectives, negative examples, resolution, augmentation, and evaluation are part of the evidence. Their full treatment belongs to a later multimodal subject.