Mixture of Experts Separates Total and Active Parameters
Trace top-1 routing and keep total stored parameters, selected expert capacity, routing work, communication, capacity, and load balance distinct.
A mixture-of-experts (MoE) layer stores several MLP experts and uses a router to select a small number for each token. This can increase total parameter capacity without applying every expert to every token.
Trace a Top-1 Layer
For each token record :
- the router computes one score per expert;
- a routing rule chooses one expert;
- the token is dispatched to that expert;
- the expert MLP transforms it;
- the result returns to the token's original position.
The router and dispatch are real work. “One active expert” does not mean the other system costs vanish.
Keep a Parameter Ledger
Take , , four experts, and two biased linear maps per expert. One expert has
parameters. A bias-free router has 16. Therefore:
| Quantity | Count |
|---|---|
| one dense MLP / one expert | 76 |
| four experts plus router, total stored | |
| active expert parameters per top-1 token | 76 |
The final row describes selected expert capacity, not complete per-token cost. It excludes routing, dispatch, gather, communication, capacity padding, dropped tokens, and load imbalance.
Capacity and Balance Can Fail
If many tokens choose one expert, that expert can become overloaded while others are idle. Implementations may cap expert capacity, drop or reroute tokens, and add load-balancing objectives. These choices affect training and serving behavior.
Q1. Separate total and active parameters
Three identical experts contain 50 parameters each. A top-1 router contains 12 parameters. Report total stored parameters and active expert parameters for one token, excluding router work.
Answer it first, then check.
Hint
Solution
Switch Transformer studies one top-1 sparse expert design under stated models, data, precision, hardware, and tasks. Its measurements should retain those conditions.