Training and inference efficiency of large language models

Large Language Models (LLMs) have become central to modern artificial intelligence, enabling high-quality reasoning and interaction across a wide range of tasks. However, the current paradigm of "brute force" scaling---increasing performance strictly by scaling compute, data, and parameters---is reaching critical asymptotes. The exhaustion of high-quality data, the compute and memory costs inherent to the transformer architecture, and the associated massive energy footprint for training and inference create a regime of diminishing returns, where the resource cost to achieve the next increment of performance becomes prohibitively high. This thesis argues that the next frontier of performance lies not in unconstrained growth, but in computational efficiency across the entire model lifecycle.

We investigate efficiency along three axes: optimization, architecture, and inference dynamics. First, we address the computational bottlenecks of long-context modeling. We introduce a hardware-aware sparse attention mechanism that handles complex dynamic sparsity patterns with significant wall-clock acceleration on GPUs. Second, we target sample efficiency during training. We propose AdEMAMix, a novel optimizer that better leverage momentum to enable significantly faster convergence with fewer training tokens. Third, we improve architectural information flow. We present DenseFormer, which utilizes weighted depth-wise residual connections to mitigate signal degradation, resulting in higher performance per parameter. Finally, we challenge the static compute paradigm of standard Transformers. We introduce CoTFormer, an architecture that bridges the gap between model depth and Chain-of-Thought reasoning. By treating recurrent layer applications as the generation of intermediate tokens, CoTFormer solves the attention alignment problem of variable-depth models. This allows for adaptive inference, where the model automatically allocates more compute to complex tokens while saving compute on simpler ones.

Together, these contributions demonstrate that algorithmic innovation can yield performance gains equivalent to significant scaling, offering a path toward more sustainable, accessible, and capable language models.