Recherche

Afficher tous les résultats pour

Accueil

Personne

Sebastian Urban Stich

Cette personne a quitté l’EPFL

À propos
Confidentialité
Mentions légales

Graph Chatbot

Publications associées (27)

Page 1 sur 2

Accelerated Stochastic Matrix Inversion: General Theory and Speeding up BFGS Rules for Faster Second-Order Optimization

Sebastian Urban Stich

We present the first accelerated randomized algorithm for solving linear systems in Euclidean spaces. One essential problem of this type is the matrix inversion problem. In particular, our algorithm can be specialized to invert positive definite matrices i ...

2018

Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data

Martin Jaggi, Sebastian Urban Stich, Tao Lin, Sai Praneeth Reddy Karimireddy

Decentralized training of deep learning models is a key element for enabling data privacy and on-device learning over networks. In realistic learning scenarios, the presence of heterogeneity across different clients' local datasets poses an optimization ch ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2021

Ensemble Distillation for Robust Model Fusion in Federated Learning

Martin Jaggi, Sebastian Urban Stich, Tao Lin

Federated Learning (FL) is a machine learning setting where many devices collaboratively train a machine learning model while keeping the training data decentralized. In most of the current training schemes the central model is refined by averaging the par ...

2020

LENA: Communication-Efficient Distributed Learning with Self-Triggered Gradient Uploads

Martin Jaggi, Sebastian Urban Stich

In distributed optimization, parameter updates from the gradient computing node devices have to be aggregated in every iteration on the orchestrating server. When these updates are sent over an arbitrary commodity network, bandwidth and latency can be limi ...

MICROTOME PUBLISHING2021

Stochastic distributed learning with gradient quantization and double-variance reduction

Sebastian Urban Stich

We consider distributed optimization over several devices, each sending incremental model updates to a central server. This setting is considered, for instance, in federated learning. Various schemes have been designed to compress the model updates in orde ...

TAYLOR & FRANCIS LTD2022

Critical Parameters for Scalable Distributed Learning with Large Batches and Asynchronous Updates

Martin Jaggi, Sebastian Urban Stich, Amirkeivan Mohtashami

It has been experimentally observed that the efficiency of distributed training with stochastic gradient (SGD) depends decisively on the batch size and—in asynchronous implementations—on the gradient staleness. Especially, it has been observed that the spe ...

MICROTOME PUBLISHING2021

On Matching Pursuit and Coordinate Descent

Martin Jaggi, Sebastian Urban Stich, Sai Praneeth Reddy Karimireddy

Two popular examples of first-order optimization methods over linear spaces are coordinate descent and matching pursuit algorithms, with their randomized variants. While the former targets the optimization by moving along coordinates, the latter considers ...

2018

The Error-Feedback Framework: Better Rates for SGD with Delayed Gradients and Compressed Updates

Sebastian Urban Stich, Sai Praneeth Reddy Karimireddy

We analyze (stochastic) gradient descent (SGD) with delayed updates on smooth quasi-convex and non-convex functions and derive concise, non-asymptotic, convergence rates. We show that the rate of convergence in all cases consists of two terms: (i) a stocha ...

Microtome Publishing2020

Efficient Greedy Coordinate Descent for Composite Problems

Martin Jaggi, Sebastian Urban Stich, Sai Praneeth Reddy Karimireddy, Anastasiia Koloskova

Coordinate descent with random coordinate selection is the current state of the art for many large scale optimization problems. However, greedy selection of the steepest coordinate on smooth problems can yield convergence rates independent of the dimension ...

2019

Decentralized deep learning with arbitrary communication compression

Martin Jaggi, Sebastian Urban Stich, Tao Lin, Anastasiia Koloskova

Decentralized training of deep learning models is a key element for enabling data privacy and on-device learning over networks, as well as for efficient scaling to large compute clusters. As current approaches are limited by network bandwidth, we propose t ...

2019

Consensus Control for Decentralized Deep Learning

Martin Jaggi, Sebastian Urban Stich, Tao Lin, Anastasiia Koloskova

Decentralized training of deep learning models enables on-device learning over networks, as well as efficient scaling to large compute clusters. Experiments in earlier works reveal that, even in a data-center setup, decentralized training often suffers fro ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2021

Semantic Perturbations with Normalizing Flows for Improved Generalization

Martin Jaggi, Sebastian Urban Stich, Tatjana Chavdarova

Data augmentation is a widely adopted technique for avoiding overfitting when training deep neural networks. However, this approach requires domain-specific knowledge and is often limited to a fixed set of hard-coded transformations. Recently, several work ...

IEEE2021

Masked Training of Neural Networks with Partial Gradients

Martin Jaggi, Sebastian Urban Stich, Amirkeivan Mohtashami

State-of-the-art training algorithms for deep learning models are based on stochastic gradient descent (SGD). Recently, many variations have been explored: perturbing parameters for better accuracy (such as in Extra-gradient), limiting SGD updates to a sub ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2022

Error Feedback Fixes SignSGD and other Gradient Compression Schemes

Martin Jaggi, Sebastian Urban Stich, Sai Praneeth Reddy Karimireddy, Quentin Rebjock

Sign-based algorithms (e.g. signSGD) have been proposed as a biased gradient compression technique to alleviate the communication bottleneck in training large neural networks across multiple workers. We show simple convex counter-examples where signSGD doe ...

PMLR2019

Approximate Steepest Coordinate Descent

Martin Jaggi, Sebastian Urban Stich

We propose a new selection rule for the coordinate selection in coordinate descent methods for huge-scale optimization. The efficiency of this novel scheme is provably better than the efficiency of uniformly random selection, and can reach the efficiency o ...

PMLR2017

Adaptive balancing of gradient and update computation times using global geometry and approximate subproblems

Martin Jaggi, Sebastian Urban Stich, Sai Praneeth Reddy Karimireddy

2018

Decentralized Stochastic Optimization and Gossip Algorithms with Compressed Communication

Martin Jaggi, Sebastian Urban Stich, Anastasiia Koloskova

We consider decentralized stochastic optimization with the objective function (e.g. data samples for machine learning task) being distributed over n machines that can only communicate to their neighbors on a fixed communication graph. To reduce the communi ...

2019

A Unified Theory of Decentralized SGD with Changing Topology and Local Updates

Martin Jaggi, Sebastian Urban Stich, Anastasiia Koloskova

Decentralized stochastic optimization methods have gained a lot of attention recently, mainly because of their cheap per iteration cost, data locality, and their communication-efficiency. In this paper we introduce a unified convergence analysis that cover ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2020

Local SGD Converges Fast and Communicates Little

Sebastian Urban Stich

Mini-batch stochastic gradient descent (SGD) is state of the art in large scale distributed training. The scheme can reach a linear speedup with respect to the number of workers, but this is rarely seen in practice as the scheme often suffers from large ne ...

2019

Sparsified SGD with Memory

Martin Jaggi, Sebastian Urban Stich, Jean-Baptiste Francis Marie Juliette Cordonnier

Huge scale machine learning problems are nowadays tackled by distributed optimization algorithms, i.e. algorithms that leverage the compute power of many devices for training. The communication overhead is a key bottleneck that hinders perfect scalability. ...

2018