Publications by Bruno Loureiro

A High Dimensional Statistical Model for Adversarial Training: Geometry and Trade-Offs

This work investigates adversarial training in the context of margin-based linear classifiers in the high-dimensional regime where the dimension d and the number of data points n diverge with a fixed ratio α = n/d. We introduce a tractable mathematical mod ...

2025

Analysis of Bootstrap and Subsampling in High-dimensional Regularized Regression

Adrien Vandenbroucque, Florent Gérard Krzakala, Lenka Zdeborová, Bruno Loureiro, Lucas Andry Clarte

We investigate popular resampling methods for estimating the uncertainty of statistical models, such as subsampling, bootstrap and the jackknife, and their performance in high-dimensional supervised regression tasks. We provide a tight asymptotic descripti ...

ACM2024

Asymptotics of feature learning in two-layer networks after one gradient-step

Yue Lu, Hugo Chao Cui, Florent Gérard Krzakala, Lenka Zdeborová, Bruno Loureiro, Luca Pesce

In this manuscript, we investigate the problem of how two-layer neural networks learn features from data, and improve over the kernel regime, after being trained with a single gradient descent step. Leveraging the insight from (Ba et al., 2022), we model t ...

2024

Universality laws for Gaussian mixtures in generalized linear models

Florent Gérard Krzakala, Lenka Zdeborová, Bruno Loureiro, Ludovic Théo Stephan

A recent line of work in high-dimensional statistics working under the Gaussian mixture hypothesis has led to a number of results in the context of empirical risk minimization, Bayesian uncertainty quantification, separation of kernel methods and neural ne ...

2024

Gaussian universality of perceptrons with random labels

Florent Gérard Krzakala, Lenka Zdeborová, Bruno Loureiro, Ludovic Théo Stephan

While classical in many theoretical settings-and in particular in statistical physics-inspired works-the assumption of Gaussian i.i.d. input data is often perceived as a strong limitation in the context of statistics and machine learning. In this study, we ...

Amer Physical Soc2024

How Two-Layer Neural Networks Learn, One (Giant) Step at a Time

Florent Gérard Krzakala, Bruno Loureiro, Luca Pesce, Ludovic Théo Stephan

For high-dimensional Gaussian data, we investigate theoretically how the features of a two-layer neural network adapt to the structure of the target function through a few large batch gradient descent steps, leading to an improvement in the approximation c ...

2024

Online Learning and Information Exponents: On The Importance of Batch size, and Time/Complexity Tradeoffs

Florent Gérard Krzakala, Bruno Loureiro, Luca Pesce, Ludovic Théo Stephan

We study the impact of the batch

n_b

on the iteration time

T

of training two-layer neural networks with one-pass stochastic gradient descent (SGD) on multi-index target functions of isotropic covariates. We characterize the optimal batch size minimizin ...

JMLR2024

Fluctuations, bias, variance and ensemble of learners: exact asymptotics for convex losses in high-dimension

Florent Gérard Krzakala, Bruno Loureiro

From the sampling of data to the initialisation of parameters, randomness is ubiquitous in modern Machine Learning practice. Understanding the statistical fluctuations engendered by the different sources of randomness in prediction is therefore key to unde ...

Iop Publishing Ltd2023

On double-descent in uncertainty quantification in overparametrized models

Florent Gérard Krzakala, Lenka Zdeborová, Bruno Loureiro, Lucas Andry Clarte, Bruno Loureiro

Uncertainty quantification is a central challenge in reliable and trustworthy machine learning. Naive measures such as last-layer scores are well-known to yield overconfident estimates in the context of overparametrized neural networks. Several methods, ra ...

PMLR Proceedings of Machine Learning Research2023

Expectation consistency for calibration of neural networks

Florent Gérard Krzakala, Lenka Zdeborová, Bruno Loureiro, Lucas Andry Clarte, Bruno Loureiro

Despite their incredible performance, it is well reported that deep neural networks tend to be overoptimistic about their prediction confidence. Finding effective and efficient calibration methods for neural networks is therefore an important endeavour tow ...

2023

Error scaling laws for kernel classification under source and capacity conditions

Hugo Chao Cui, Florent Gérard Krzakala, Lenka Zdeborová, Bruno Loureiro, Bruno Loureiro

In this manuscript we consider the problem of kernel classification. While worst-case bounds on the decay rate of the prediction error with the number of samples are known for some classifiers, they often fail to accurately describe the learning curves of ...

2023

Learning curves for the multi-class teacher-student perceptron

Elisabetta Cornacchia, Francesca Mignacco, Lenka Zdeborová, Bruno Loureiro, Bruno Loureiro

One of the most classical results in high-dimensional learning theory provides a closed-form expression for the generalisation error of binary classification with a single-layer teacher-student perceptron on i.i.d. Gaussian inputs. Both Bayes-optimal (BO) ...

IOP Publishing Ltd2023

Generalization error rates in kernel regression: the crossover from the noiseless to noisy regime*

Hugo Chao Cui, Florent Gérard Krzakala, Lenka Zdeborová, Bruno Loureiro

In this manuscript we consider kernel ridge regression (KRR) under the Gaussian design. Exponents for the decay of the excess generalization error of KRR have been reported in various works under the assumption of power-law decay of eigenvalues of the feat ...

2022

Subspace clustering in high-dimensions: Phase transitions & Statistical-to-Computational gap

Florent Gérard Krzakala, Lenka Zdeborová, Bruno Loureiro, Luca Pesce, Bruno Loureiro

A simple model to study subspace clustering is the high-dimensional k -Gaussian mixture model where the cluster means are sparse vectors. Here we provide an exact asymptotic characterization of the statistically optimal reconstruction error in this model i ...

2022

Fluctuations, Bias, Variance & Ensemble of Learners: Exact Asymptotics for Convex Losses in High-Dimension

Florent Gérard Krzakala, Bruno Loureiro

From the sampling of data to the initialisation of parameters, randomness is ubiquitous in modern Machine Learning practice. Understanding the statistical fluctuations engendered by the different sources of randomness in prediction is therefore key to unde ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2022

Secure Coding via Gaussian Random Fields

Florent Gérard Krzakala, Bruno Loureiro

Inverse probability problems whose generative models are given by strictly nonlinear Gaussian random fields show the all-or-nothing behavior: There exists a critical rate at which Bayesian inference exhibits a phase transition. Below this rate, the optimal ...

IEEE2022

Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks

Florent Gérard Krzakala, Lenka Zdeborová, Bruno Loureiro, Bruno Loureiro, Ludovic Théo Stephan

Despite the non-convex optimization landscape, over-parametrized shallow networks are able to achieve global convergence under gradient descent. The picture can be radically different for narrow net-works, which tend to get stuck in badly-generalizing loca ...

2022

The Gaussian equivalence of generative models for learning with shallow neural networks

Florent Gérard Krzakala, Lenka Zdeborová, Bruno Loureiro

Understanding the impact of data structure on the computational tractability of learning is a key challenge for the theory of neural networks. Many theoretical works do not explicitly model training data, or assume that inputs are drawn component-wise inde ...

2021

Learning Gaussian Mixtures with Generalized Linear Models: Precise Asymptotics in High-dimensions

Florent Gérard Krzakala, Lenka Zdeborová, Bruno Loureiro, Gabriele Sicuro

Generalised linear models for multi-class classification problems are one of the fundamental building blocks of modern machine learning tasks. In this manuscript, we characterise the learning of a mixture of K Gaussians with generic means and covariances v ...

2021

Generalization Error Rates in Kernel Regression: The Crossover from the Noiseless to Noisy Regime

Hugo Chao Cui, Florent Gérard Krzakala, Lenka Zdeborová, Bruno Loureiro

In this manuscript we consider Kernel Ridge Regression (KRR) under the Gaussian design. Exponents for the decay of the excess generalization error of KRR have been reported in various works under the assumption of power-law decay of eigenvalues of the feat ...

2021