Concept

Batch normalization

Related publications (30)

Learning of Continuous and Piecewise-Linear Functions With Hessian Total-Variation Regularization

Michaël Unser, Shayan Aziznejad, Joaquim Gonçalves Garcia Barreto Campos

We develop a novel 2D functional learning framework that employs a sparsity-promoting regularization based on second-order derivatives. Motivated by the nature of the regularizer, we restrict the search space to the span of piecewise-linear box splines shi ...

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC2022

Disparity Between Batches as a Signal for Early Stopping

Patrick Thiran, Mahsa Forouzesh

We propose a metric for evaluating the generalization ability of deep neural networks trained with mini-batch gradient descent. Our metric, called gradient disparity, is the l2 norm distance between the gradient vectors of two mini-batches drawn from the t ...

Springer2021

Gradient-based Methods for Deep Model Interpretability

Suraj Srinivas

In this dissertation, we propose gradient-based methods for characterizing model behaviour for the purposes of knowledge transfer and post-hoc model interpretation. Broadly, gradients capture the variation of some output feature of the model upon unit vari ...

EPFL2021

Large Steps in Inverse Rendering of Geometry

Wenzel Alban Jakob, Baptiste Alexandre Marie Philippe Claude Nicolet

Inverse reconstruction from images is a central problem in many scientific and engineering disciplines. Recent progress on differentiable rendering has led to methods that can efficiently differentiate the full process of image formation with respect to mi ...

ASSOC COMPUTING MACHINERY2021

Neural network surrogate of QuaLiKiz using JET experimental data to populate training space

Henri Weisen, Yann Camenen

Within integrated tokamak plasma modeling, turbulent transport codes are typically the computational bottleneck limiting their routine use outside of post-discharge analysis. Neural network (NN) surrogates have been used to accelerate these calculations wh ...

AMER INST PHYSICS2021

On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Them

Sabine Süsstrunk, Mathieu Salzmann, Tao Lin, Chen Liu

We analyze the influence of adversarial training on the loss landscape of machine learning models. To this end, we first provide analytical studies of the properties of adversarial loss functions under different adversarial budgets. We then demonstrate tha ...

2021

Learning Lipschitz-Controlled Activation Functions in Neural Networks for Plug-and-Play Image Reconstruction Methods

Michaël Unser, Dimitris Perdios, Alexis Marie Frederic Goujon, Pakshal Narendra Bohra

Ill-posed linear inverse problems are frequently encountered in image reconstruction tasks. Image reconstruction methods that combine the Plug-and-Play (PnP) priors framework with convolutional neural network (CNN) based denoisers have shown impressive per ...

2021

STORM+: Fully Adaptive SGD with Momentum for Nonconvex Optimization

Volkan Cevher, Ali Kavis

In this work we investigate stochastic non-convex optimization problems wherethe objective is an expectation over smooth loss functions, and the goal is to find an approximate stationary point. The most popular approach to handling such problems is varianc ...

2021

Differentially Private Stochastic Coordinate Descent

Rachid Guerraoui, Georgios Damaskinos

In this paper we tackle the challenge of making the stochastic coordinate descent algorithm differentially private. Compared to the classical gradient descent algorithm where updates operate on a single model vector and controlled noise addition to this ve ...

AIAA2021

Subquadratic Overparameterization for Shallow Neural Networks

Volkan Cevher, Armin Eftekhari, Thomas Michaelsen Pethick, Chaehwan Song

Overparameterization refers to the important phenomenon where the width of a neural network is chosen such that learning algorithms can provably attain zero loss in nonconvex training. The existing theory establishes such global convergence using various i ...

2021