The efficient training of large-scale deep neural networks currently relies on an ecosystem of optimization heuristics, including weight decay, learning rate warmup, and specific parameterizations. Despite their ubiquity, the fundamental mechanisms driving their success remain poorly understood. Through applied analysis and large-scale experimentation, this thesis establishes that these key methods share a common underlying benefit: they balance the speed at which internal representations change across varying network layers, training phases, and model scales.
First, this work challenges the widespread belief that weight decay acts primarily as an explicit regularizer. Instead, we demonstrate that it drives the update dynamics of individual neurons toward a steady equilibrium state. This equilibrium balances the relative weight updates across the network, fundamentally controlling how fast neurons learn the direction of their weight vectors. This framework resolves the long-standing question of why AdamW outperforms Adam with L2-regularization, demonstrating that AdamW effectively balances learning speeds across layers while L2-regularization induces harmful imbalances. By isolating this underlying mechanism, we propose training on the sphere with a fixed weight norm as a principled alternative. Explicitly controlling relative updates through spherical training provides the benefits of weight decay while eliminating complex interactions with other hyperparameters.
Focusing next on the temporal dynamics of training, we demonstrate that learning rate warmup fundamentally balances the effective update size across time. Early in optimization, a small critical batch size, characterized by highly correlated sample gradients, combines with initial weight norm dynamics to produce disproportionately large relative representation changes. These outsized early updates can destabilize the network or cause lasting degradation. While lowering the overall learning rate avoids this instability, it results in overly small later updates that severely limit long-term learning. Warmup resolves this trade-off: explicitly suppressing the learning rate during the initial phase prevents detrimental early updates while preserving the high peak rates necessary for efficient long-term learning.
Finally, the thesis evaluates representation learning dynamics when transferring hyperparameters from small to large models. We critically examine the Maximal Update Parameterization (muP), demonstrating that its assumptions regarding geometric alignment of weight updates and layer inputs hold only briefly at the onset of training. In practical scenarios like Large Language Model (LLM) training, it is weight decay, rather than muP, that predominantly balances the update dynamics of internal representations across varying widths by explicitly counteracting muP's prescribed scaling later in training. This reframes muP as a form of implicit learning rate warmup, fundamentally challe
