ESLM: Risk-Averse Selective Language Modeling with Hierarchical Batch Selection

Large language model pretraining is compute-intensive, yet many tokens contribute marginally to learning, resulting in inefficiency. We introduce Efficient Selective Language Modeling (Eslm), an online, risk-aware batch selection algorithm that improves training efficiency and distributional robustness. Eslm operates in two phases: (i) instance-level selection via a shallow early-exit model pass that computes proxy per-instance statistics (e.g., loss or entropy) and retains data points using value-at-risk thresholding; and (ii) loss shaping with token-level selection via risk-aware thresholding on per-token scores. This data-centric mechanism reshapes the training objective, prioritizing high-risk tokens and eliminating redundant gradient computation. Framed as a bilevel game, the model competes with a masking adversary selecting worst-case token subsets, recovering conditional value-at-risk minimization, and linking selective pretraining to distributionally robust optimization. We extend our approach to Ada-Eslm, which adaptively tunes selection confidence during training. Experiments on GPT-2 pretraining show that Eslm consistently improves FLOPs-to-target efficiency, reducing effective training FLOPs while matching or improving validation perplexity and maintaining competitive downstream performance across multiple model scales, dataset mixtures, and knowledge distillation settings.