Despite impressive widespread success, deep learning remains largely empirical, expensive to train, difficult to control, and susceptible to hidden bias. This thesis develops a theory-driven framework that links data composition, architectural choices, and gradient-based optimization to early predictive behavior and training efficiency. Moreover, we clarify how these facets are intertwined. Through theoretical analysis, we identify the governing mechanisms and translate them into actionable design levers.
We begin with class-imbalanced data and analyze early updates through a per-class gradient decomposition. At the onset of training, unequal frequencies make majority-class gradients dominate the update, yielding an early drop in minority performance. We derive geometric conditions for simultaneous progress and propose Per-Class Normalized Gradient Descent (PCNGD), which equalizes per-class contributions and removes this effect in full-batch settings. The same analysis delineates both the limits and the potential of balancing approaches: in mini-batch SGD, minority gradients are estimated from fewer examples and thus exhibit larger directional noise, attenuating their effective contribution even after norm equalization; per-class normalization alone is therefore insufficient, whereas oversampling can restore balance under identifiable conditions.
This raises a natural question: can imbalance arise even without differences between classes in the data? We show that it can. Initial Guessing Bias (IGB) quantifies how architecture and initialization alone induce systematic, seed-dependent skew at time zero, even on perfectly balanced data. We identify design elements (activations, normalization placement, pooling, depth) that steer models toward neutrality or prejudice, and establish a direct link between initial predictive behavior and trainability; notably, the highly trainable edge-of-chaos regime coincides with transient, strongly skewed initial predictions. Coupling IGB with mean-field analyses yields phase diagrams encoding information about the emergence of bias in DNN outputs and gradient stability at initialization, providing practical guidance for initialization and model design. Moreover, IGB can induce per-class gradient-norm disparities at initialization, fitting model-induced effects into the Part-I gradient-imbalance lens.
Finally, we examine efficiency in diffusion models, where sampling typically demands many iterative denoising steps. Extending the framework to Lévy-driven processes, we analyze the minimum diffusion window needed to erase data structure (the speciation time) and show, via a score-based analysis, that in high dimensions this window still grows with dimension, as in the Gaussian case. Heavy-tailed noise alters dynamics but does not remove the fundamental scaling barrier, clarifying the limits of noise design as an efficiency lever.
Taken together, these results reveal shared mechanisms linking bias and efficiency a
