Mixture of Experts

Also known as: MoE

Mixture of Experts is an architecture that replaces the dense feed-forward layer in a transformer block with a collection of parallel expert sub-networks and a learned routing function. For each token, the router activates only a small subset of experts — typically two out of dozens or hundreds — so total parameter count is large while the compute per token remains comparable to a much smaller dense model. MoE architectures like Mixtral and GPT-4 achieve high capacity with favorable inference economics.