Activations and Gates Change the MLP

The operation between the MLP projections is part of the architecture. ReLU, GELU, and gated variants can have the same input and output shapes while calculating different functions.

ReLU and GELU Treat Negative Inputs Differently

ReLU uses $\max(0,x)$. GELU weights an input by the standard normal cumulative distribution:

For $x=-1,0,1$, exact GELU is approximately

whereas ReLU returns $[0,0,1]$. GELU can retain a small negative value and smoothly changes near zero. An implementation should also state whether it uses exact GELU or an approximation.

A Gate Uses Two Expanded Projections

A gated MLP can be written as

where $\odot$ is coordinate-wise multiplication. One projection decides how strongly each expanded coordinate opens; the other supplies the value being gated.

If the gate preactivation is $[1,-1]$, the value branch is $[2,3]$, and $\phi$ is ReLU, then

SwiGLU uses the SiLU activation in the gate and is one member of this broader family. Names alone are not enough: record the activation and every projection shape.

Compare Under a Parameter Budget

A gated MLP has an additional input projection. Keeping the same $d_{ff}$ therefore increases parameters and computation relative to a two-projection MLP. Fair comparisons often reduce the gated hidden width, but the exact ratio depends on biases and conventions. State the actual shapes instead of relying on “same size.”

References

• Dan Hendrycks and Kevin Gimpel, Gaussian Error Linear Units (GELUs), 2016.
• Noam Shazeer, GLU Variants Improve Transformer, 2020.