The Cost of Expertise: Understanding MoE Decode Performance

Mixture-of-experts (MoE) architectures have emerged as a popular strategy for scaling large language models (LLMs), enabling substantial increases in parameter counts without proportional growth in per-token FLOPs. Existing analyses often model MoE inference as an attention backbone followed by a routing layer that is used to route tokens to multiple Feed-Forward Network (FFN). MoE-based LLMs expand the design space across sparsity and scaling dimensions, such as top-k routing, expert intermediate size and prompt-dependent expert activation whose systems implications remain underexplored especially for the decode-phase, despite decode dominating interactive inference workloads. We systematically study how MoE design choices influence inference latency during the decode phase, which is the primary latency bottleneck in LLM workloads. We analyze latency behavior across varying batch size, expert intermediate sizes, and numbers of activated experts. Our results reveal that expert intermediate size, batching behavior, and expert activation patterns interact in non-trivial ways: latency scales near-linearly with expert width, batching gains plateau for wider experts due to early GPU saturation, and activating additional experts increases latency through memory pressure rather than compute overhead. These findings expose critical system-level trade-offs in MoE design that are invisible to FLOP-centric analyses. CCS Concepts • Computing methodologies → Artificial intelligence.