Feature Learning with Gradient-Based Methods: Insights from Solvable Models of Neural Networks

Understanding why gradient-trained neural networks succeed requires theoretical models that capture their ability to learn internal representations from high-dimensional data. Despite the empirical dominance of large-scale deep learning, even small neural networks remain analytically elusive: their learning dynamics and representational structure are not yet fully understood. This thesis develops solvable models of shallow neural networks in high dimensions, providing a rigorous framework for feature learning, i.e., the acquisition of low-dimensional structure embedded in high-dimensional inputs through gradient-based training.

The first part of the thesis investigates the minimal setting in which neural networks surpass fixed-feature or kernel methods. We show that a single gradient update to the first layer of a two-layer network suffices to break the Gaussian Universality characterizing fixed-feature methods.

The second part studies progressive feature learning when using stochastic gradient descent (SGD) training. We show that simple modification to vanilla SGD, including data repetition, moves the dynamics from a hardness landscape governed by information and leap exponents to a more permissive regime dictated by a novel generative exponent. Moreover, we discuss first attempts to extend these ideas to deeper networks.

Together, these contributions develop a solvable theory of representation formation in shallow networks. They quantify when feature learning occurs, how much improvement it offers over fixed-feature baselines, and which structural aspects of data and training most strongly influence learnability.