A Mixture of Experts model replaces a single dense feed-forward layer with a set of parallel sub-networks called experts. A small router network looks at each token and picks the two or three experts best suited to it. The rest stay idle for that token.
That selectivity is the whole point. A dense model with 400 billion parameters runs all 400 billion for every token it produces. An MoE model of the same total size might activate 30 billion, which cuts compute per token by an order of magnitude while keeping the capacity that large parameter counts buy.
The trade is memory and balance. All experts must sit in memory even though most are idle, so serving costs stay high. Routers also drift toward a few favourite experts during training, which wastes capacity, so training adds a load-balancing term that pushes traffic across the pool.
Several frontier models released since 2024 use this design. When a provider quotes "total" and "active" parameter counts separately, the model is almost certainly an MoE.