Publication

AGGREGATHOR: Byzantine Machine Learning via Robust Gradient Aggregation

Rachid Guerraoui, El Mahdi El Mhamdi, Sébastien Louis Alexandre Rouault, Georgios Damaskinos, Arsany Hany Abdelmessih Guirguis
2019
Conference Papers

Abstract

We present AGGREGATHOR, a framework that implements state-of-the-art robust (Byzantine-resilient) distributed stochastic gradient descent. Following the standard parameter server model, we assume that a minority of worker machines can be controlled by an adversary and behave arbitrarily. Such a setting has been theoretically studied with several of the existing approaches using a robust aggregation of the workers’ gradient estimations. Yet, the question is whether a Byzantine-resilient aggregation can leverage more workers to speedup learning. We answer this theoretical question, and implement these state-of-the-art theoretical approaches on AGGREGATHOR, to assess their practical costs. We built AGGREGATHOR around TensorFlow and introduce modifications for vanilla TensorFlow towards making it usable in an actual Byzantine setting. AGGREGATHOR also permits the use of unreliable gradient transfer over UDP to provide further speed-up (without losing the accuracy) over the native communication protocols (TCP-based) of TensorFlow in saturated networks. We quantify the overhead of Byzantine resilience of AGGREGATHOR to 19% and 43% (to ensure weak and strong Byzantine resilience respectively) compared to vanilla TensorFlow.

Official source

https://infoscience.epfl.ch/entities/publication/d0591e21-def6-403c-b349-8089de58a513

About this result

This page is automatically generated and may contain information that is not correct, complete, up-to-date, or relevant to your search query. The same applies to every other page on this website. Please make sure to verify the information with EPFL's official sources.

Rachid Guerraoui, El Mahdi El Mhamdi, Sébastien Louis Alexandre Rouault, Georgios Damaskinos, Arsany Hany Abdelmessih Guirguis
2019
Conference Papers

Abstract

Official source

https://infoscience.epfl.ch/entities/publication/d0591e21-def6-403c-b349-8089de58a513

About this result

Related concepts (24)

Byzantine army

The Byzantine army was the primary military body of the Byzantine armed forces, serving alongside the Byzantine navy. A direct continuation of the Eastern Roman army, shaping and developing itself on the legacy of the late Hellenistic armies, it maintained a similar level of discipline, strategic prowess and organization. It was among the most effective armies of western Eurasia for much of the Middle Ages. Over time the cavalry arm became more prominent in the Byzantine army as the legion system disappeared in the early 7th century.

Internet protocol suite

The Internet protocol suite, commonly known as TCP/IP, is a framework for organizing the set of communication protocols used in the Internet and similar computer networks according to functional criteria. The foundational protocols in the suite are the Transmission Control Protocol (TCP), the User Datagram Protocol (UDP), and the Internet Protocol (IP). Early versions of this networking model were known as the Department of Defense (DoD) model because the research and development were funded by the United States Department of Defense through DARPA.

Artificial neural network

Artificial neural networks (ANNs, also shortened to neural networks (NNs) or neural nets) are a branch of machine learning models that are built using principles of neuronal organization discovered by connectionism in the biological neural networks constituting animal brains. An ANN is based on a collection of connected units or nodes called artificial neurons, which loosely model the neurons in a biological brain. Each connection, like the synapses in a biological brain, can transmit a signal to other neurons.

Related publications (30)

Statistical Inference for Inverse Problems: From Sparsity-Based Methods to Neural Networks

Pakshal Narendra Bohra

In inverse problems, the task is to reconstruct an unknown signal from its possibly noise-corrupted measurements. Penalized-likelihood-based estimation and Bayesian estimation are two powerful statistical paradigms for the resolution of such problems. They ...

EPFL2024

Error assessment of an adaptive finite elements-neural networks method for an elliptic parametric PDE

Alexandre Caboussat, Marco Picasso, Maude Girardin

We present a finite elements-neural network approach for the numerical approximation of parametric partial differential equations. The algorithm generates training data from finite element simulations, and uses a data -driven (supervised) feedforward neura ...

Elsevier Science Sa2024

Random matrix methods for high-dimensional machine learning models

Antoine Philippe Michel Bodin

In the rapidly evolving landscape of machine learning research, neural networks stand out with their ever-expanding number of parameters and reliance on increasingly large datasets. The financial cost and computational resources required for the training p ...

EPFL2024