Widespread adoption of deep neural networks (DNNs), coupled with the trillion-parameter scale of modern models, has pushed compute, memory, and power demands of training and inference to their limits, making DNN compression optimization essential for continued AI progress. Microscaling numerical formats have recently emerged as a powerful approach for low-precision training and inference, using block-wise representations that share an exponent across small groups of low-bitwidth elements. This thesis studies low-precision microscaling formats and shows how sensitivity-aware precision allocation and sparsity can make them practical for optimized training and inference.
We first analyze how quantization sensitivity varies across layers and training phases, and introduce a DNN training recipe that uses mixed-mantissa MXINT formats.
We show that moderate-sized DNNs, including CNNs and Transformer-based models, can be trained using 4-bit mantissas for over 99% of training operations and using 6 bits for the remaining, more sensitive layers and epochs. We then leverage insights from recent studies and extend our method to LLM pretraining by using MXINT4 in the backward pass and combining it with stochastic rounding and block-wise Randomized Hadamard Transforms to handle heavy-tailed activations. We show that MXINT4 can achieve MXFP4-level perplexities.
We next turn to inference and show that quantization sensitivity in large language models is highly non-uniform across layers, columns, and value distributions. Building on this observation, we develop MXSens, a sensitivity-aware post-training quantization method that assigns {4, 6, 8}-bit mantissas at column and layer granularity based on empirical Hessian- and output-sensitivity metrics. MXSens enables nearâ 4-bit average bitwidth while preserving accuracy across LLaMA, Qwen, Mistral, and Gemma models, outperforming state-of-the-art rotation-based and mixed-precision baselines under strict W4A4KV4 configurations.
Finally, we provide the first theoretical and empirical analysis of the interaction between block-wise quantization and sparsity, proving that the two techniques are fundamentally non-orthogonal and that applying sparsity before quantization is the optimal ordering. Our analysis yields upper bounds on the error introduced by suboptimal ordering, and our experiments across CNNs, ViTs, and LLMs show that combining quantization and sparsity can introduce up to 13% additional perplexity error if not applied optimally.
Together, these contributions establish a principled foundation for low-bitwidth microscaling formats. The thesis shows that with sensitivity-driven precision allocation, unbiased rounding, outlier handling, and mathematically grounded sparsityâ quantization coordination, 4-bit fixed-point microscaling becomes practical for both training and inference, enabling scalable and energy-efficient deployment of modern AI systems.
