Publications related to Empirical risk minimization

Understanding generalization and robustness in modern deep learning

In this thesis, we study two closely related directions: robustness and generalization in modern deep learning. Deep learning models based on empirical risk minimization are known to be often non-robust to small, worst-case perturbations known as adversari ...

EPFL2024

On the Generalization of Stochastic Gradient Descent with Momentum

Volkan Cevher, Kimon Antonakopoulos

While momentum-based accelerated variants of stochastic gradient descent (SGD) are widely used when training machine learning models, there is little theoretical understanding on the generalization error of such methods. In this work, we first show that th ...

Microtome Publishing2024

End-to-End Learning for Stochastic Optimization: A Bayesian Perspective

Daniel Kuhn, Tobias Sutter, Yves Rychener

We develop a principled approach to end-to-end learning in stochastic optimization. First, we show that the standard end-to-end learning algorithm admits a Bayesian interpretation and trains a posterior Bayes action map. Building on the insights of this an ...

2023

Knowledge Distillation for 6D Pose Estimation by Aligning Distributions of Local Predictions

Mathieu Salzmann, Shuxuan Guo, Yinlin Hu

Knowledge distillation facilitates the training of a compact student network by using a deep teacher one. While this has achieved great success in many tasks, it remains completely unstudied for image-based 6D object pose estimation. In this work, we intro ...

Ieee Computer Soc2023

When do Minimax-fair Learning and Empirical Risk Minimization Coincide?

Volkan Cevher

Minimax-fair machine learning minimizes the error for the worst-off group. However, empirical evidence suggests that when sophisticated models are trained with standard empirical risk minimization (ERM), they often have the same performance on the worst-of ...

2023

Universal and adaptive methods for robust stochastic optimization

Ali Kavis

Within the context of contemporary machine learning problems, efficiency of optimization process depends on the properties of the model and the nature of the data available, which poses a significant problem as the complexity of either increases ad infinit ...

EPFL2023

Impact of Redundancy on Resilience in Distributed Optimization and Learning

Nirupam Gupta, Shuo Liu

This paper considers the problem of resilient distributed optimization and stochastic learning in a server-based architecture. The system comprises a server and multiple agents, where each agent has its own local cost function. The agents collaborate with ...

Assoc Computing Machinery2023

End-to-end kernel learning via generative random Fourier features

Fanghui Liu, Jie Yang

Random Fourier features (RFFs) provide a promising way for kernel learning in a spectral case. Current RFFs-based kernel learning methods usually work in a two-stage way. In the first-stage process, learn-ing an optimal feature map is often formulated as a ...

ELSEVIER SCI LTD2023

RMAML: Riemannian meta-learning with orthogonality constraints

Soumava Kumar Roy

Meta-learning is the core capability that enables intelligent systems to rapidly generalize their prior ex-perience to learn new tasks. In general, the optimization-based methods formalize the meta-learning as a bi-level optimization problem, that is a nes ...

ELSEVIER SCI LTD2023

Asymptotic Errors for Teacher-Student Convex Generalized Linear Models (Or: How to Prove Kabashima’s Replica Formula)

Florent Gérard Krzakala, Alia Abbara

There has been a recent surge of interest in the study of asymptotic reconstruction performance in various cases of generalized linear estimation problems in the teacher-student setting, especially for the case of i.i.d standard normal matrices. Here, we g ...

2022

Byzantine Machine Learning Made Easy By Resilient Averaging of Momentums

Rachid Guerraoui, John Stephan, Sadegh Farhadkhani, Nirupam Gupta, Rafaël Benjamin Pinot

Byzantine resilience emerged as a prominent topic within the distributed machine learning community. Essentially, the goal is to enhance distributed optimization algorithms, such as distributed SGD, in a way that guarantees convergence despite the presence ...

PMLR2022

Local plasticity rules can learn deep representations using self-supervised contrastive predictions

Wulfram Gerstner, Jean Ventura, Bernd Albert Illing

Learning in the brain is poorly understood and learning rules that respect biological constraints, yet yield deep hierarchical representations, are still unknown. Here, we propose a learning rule that takes inspiration from neuroscience and recent advances ...

2021

NUQSGD: Provably Communication-efficient Data-parallel SGD via Nonuniform Quantization

Dan Alistarh, Fartash Faghri

As the size and complexity of models and datasets grow, so does the need for communication-efficient variants of stochastic gradient descent that can be deployed to perform parallel model training. One popular communication-compression method for data-para ...

Microtome Publishing2021

Disparity Between Batches as a Signal for Early Stopping

Patrick Thiran, Mahsa Forouzesh

We propose a metric for evaluating the generalization ability of deep neural networks trained with mini-batch gradient descent. Our metric, called gradient disparity, is the l2 norm distance between the gradient vectors of two mini-batches drawn from the t ...

Springer2021

Byzantine Fault-Tolerant Distributed Machine Learning with Norm-Based Comparative Gradient Elimination

Nirupam Gupta, Shuo Liu

This paper considers the Byzantine fault-tolerance problem in distributed stochastic gradient descent (D-SGD) method - a popular algorithm for distributed multi-agent machine learning. In this problem, each agent samples data points independently from a ce ...

IEEE COMPUTER SOC2021

Chaining Meets Chain Rule: Multilevel Entropic Regularization and Training of Neural Networks

Emmanuel Abbé

We derive generalization and excess risk bounds for neural networks using a family of complexity measures based on a multilevel relative entropy. The bounds are obtained by introducing the notion of generated hierarchical coverings of neural networks and b ...

Microtome Publishing2020

Wasserstein Distributionally Robust Learning

Soroosh Shafieezadeh Abadeh

Many decision problems in science, engineering, and economics are affected by uncertainty, which is typically modeled by a random variable governed by an unknown probability distribution. For many practical applications, the probability distribution is onl ...

EPFL2020

Extrapolation for Large-batch Training in Deep Learning

Martin Jaggi, Sebastian Urban Stich, Tao Lin, Lingjing Kong

Deep learning networks are typically trained by Stochastic Gradient Descent (SGD) methods that iteratively improve the model parameters by estimating a gradient on a very small fraction of the training data. A major roadblock faced when increasing the batc ...

2020

Risk prediction error signaling: A two-component response?

Michael Herzog, Ophélie Gladys Favrod, Kerstin Preuschoff, Marc Michael Lauffs, Sophie Alberta Catherine Geoghan

Organisms use rewards to navigate and adapt to (uncertain) environments. Error-based learning about rewards issupported by the dopaminergic system, which is thought to signal reward prediction errors to make adjustmentsto past predictions. More recently, t ...

2020

A new spatial count data model with Bayesian additive regression trees for accident hot spot identification

Prateek Bansal, Rico Krüger

The identification of accident hot spots is a central task of road safety management. Bayesian count data models have emerged as the workhorse method for producing probabilistic rankings of hazardous sites in road networks. Typically, these methods assume ...

2020