Publication

WAFFLE: Weighted Averaging for Personalized Federated Learning

2021

Abstract

In federated learning, model personalization can be a very effective strategy to deal with statistical heterogeneity across clients. We introduce WAFFLE (Weighted Averaging For Federated LEarning): a personalized collaborative machine learning algorithm based on SCAFFOLD. SCAFFOLD uses stochastic control variates to converge towards a model close to the globally optimal model even in classification tasks where the marginal distribution of labels across clients is highly skewed. However, WAFFLE uses the Euclidean distance between clients’ updates to weigh their contributions and thus minimize the trained model’s loss on one specific agent. Through a series of experiments, we compare our proposed new method to two recent personalized federated learning methods, Weight Erosion and APFL, as well as two global methods, federated averaging and SCAFFOLD. We evaluate our method using two categories of non-identical client distributions (concept shift and label skew) on two benchmarked image data sets, MNIST and CIFAR10. Our experiments demonstrate the effectiveness of WAFFLE compared with other methods, as it achieves or improves accuracy with faster convergence.

Official source

https://infoscience.epfl.ch/entities/publication/900d2434-8834-4278-b11d-aaaaaccbf1ee

About this result

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Graph Chat

Chat with Graph Search

Ask any question about EPFL courses, lectures, exercises, research, news, etc. or try the example questions below.

DISCLAIMER: The Graph Chatbot is not programmed to provide explicit or categorical answers to your questions. Rather, it transforms your questions into API requests that are distributed across the various IT services officially administered by EPFL. Its purpose is solely to collect and recommend relevant references to content that you can explore to help you answer your questions.

2021

Abstract

Official source

https://infoscience.epfl.ch/entities/publication/900d2434-8834-4278-b11d-aaaaaccbf1ee

About this result

Related concepts (31)

Language model

A language model is a probabilistic model of a natural language that can generate probabilities of a series of words, based on text corpora in one or multiple languages it was trained on. Large language models, as their most advanced form, are a combination of feedforward neural networks and transformers. They have superseded recurrent neural network-based models, which had previously superseded the pure statistical models, such as word n-gram language model.

Euclidean distance

In mathematics, the Euclidean distance between two points in Euclidean space is the length of a line segment between the two points. It can be calculated from the Cartesian coordinates of the points using the Pythagorean theorem, therefore occasionally being called the Pythagorean distance. These names come from the ancient Greek mathematicians Euclid and Pythagoras, although Euclid did not represent distances as numbers, and the connection from the Pythagorean theorem to distance calculation was not made until the 18th century.

Euclidean space

Euclidean space is the fundamental space of geometry, intended to represent physical space. Originally, that is, in Euclid's Elements, it was the three-dimensional space of Euclidean geometry, but in modern mathematics there are Euclidean spaces of any positive integer dimension n, which are called Euclidean n-spaces when one wants to specify their dimension. For n equal to one or two, they are commonly called respectively Euclidean lines and Euclidean planes.

Related publications (32)

WAFFLE: Weighted Averaging for Personalized Federated Learning

Chat with Graph Search

When Weak LLMs Speak with Confidence, Preference Alignment Gets Stronger

DEPO: Dual-Efficiency Preference Optimization for LLM Agents

ProteomeLM: A proteome-scale language model enables accurate and rapid prediction of protein–protein interactions and gene essentiality across taxa