Publications de Florent Gérard Krzakala

A phase transition between positional and semantic learning in a solvable model of dot-product attention

Freya Behrens, Hugo Chao Cui, Florent Gérard Krzakala, Lenka Zdeborová

Many empirical studies have provided evidence for the emergence of algorithmic mechanisms (abilities) in the learning of language models, that lead to qualitative improvements in the model capabilities. Yet, a theoretical characterization of how such mecha ...

2025

Estimating Rank-One Matrices with Mismatched Prior and Noise: Universality and Large Deviations

Florent Gérard Krzakala, Lenka Zdeborová

We prove a universality result that reduces the free energy of rank-one matrix estimation problems in the setting of mismatched prior and noise to the computation of the free energy for a modified Sherrington–Kirkpatrick spin glass. Our main result is an a ...

2025

Phase Diagram of Compressed Sensing with l0-norm Regularization

Florent Gérard Krzakala, Lenka Zdeborová

Noiseless compressive sensing is a two-step setting that allows for undersampling a sparse signal and then reconstructing it without loss of information. The LASSO algorithm, based on l(1) regularization, provides an efficient and robust way to address thi ...

IOP Publishing Ltd2025

Bayes-optimal learning of deep random networks of extensive-width*

Hugo Chao Cui, Florent Gérard Krzakala, Lenka Zdeborová

We consider the problem of learning a target function corresponding to a deep, extensive-width, non-linear neural network with random Gaussian weights. We consider the asymptotic limit where the number of samples, the input dimension and the network width ...

IOP Publishing Ltd2025

Low-rank matrix estimation with inhomogeneous noise

Florent Gérard Krzakala, Lenka Zdeborová

We study low-rank matrix estimation for a generic inhomogeneous output channel through which the matrix is observed. This generalizes the commonly considered spiked matrix model with homogeneous noise to include for instance the dense degree-corrected stoc ...

2025

A High Dimensional Statistical Model for Adversarial Training: Geometry and Trade-Offs

Florent Gérard Krzakala, Bruno Loureiro

This work investigates adversarial training in the context of margin-based linear classifiers in the high-dimensional regime where the dimension d and the number of data points n diverge with a fixed ratio α = n/d. We introduce a tractable mathematical mod ...

2025

Analysis of Bootstrap and Subsampling in High-dimensional Regularized Regression

Adrien Vandenbroucque, Florent Gérard Krzakala, Lenka Zdeborová, Bruno Loureiro, Lucas Andry Clarte

We investigate popular resampling methods for estimating the uncertainty of statistical models, such as subsampling, bootstrap and the jackknife, and their performance in high-dimensional supervised regression tasks. We provide a tight asymptotic descripti ...

ACM2024

Asymptotics of feature learning in two-layer networks after one gradient-step

Yue Lu, Hugo Chao Cui, Florent Gérard Krzakala, Lenka Zdeborová, Bruno Loureiro, Luca Pesce

In this manuscript, we investigate the problem of how two-layer neural networks learn features from data, and improve over the kernel regime, after being trained with a single gradient descent step. Leveraging the insight from (Ba et al., 2022), we model t ...

2024

Universality laws for Gaussian mixtures in generalized linear models

Florent Gérard Krzakala, Lenka Zdeborová, Bruno Loureiro, Ludovic Théo Stephan

A recent line of work in high-dimensional statistics working under the Gaussian mixture hypothesis has led to a number of results in the context of empirical risk minimization, Bayesian uncertainty quantification, separation of kernel methods and neural ne ...

2024

The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponents

Florent Gérard Krzakala, Lenka Zdeborová, Emanuele Troiani, Luca Pesce

We investigate the training dynamics of two-layer neural networks when learning multi-index target functions. We focus on multi-pass gradient descent (GD) that reuses the batches multiple times and show that it significantly changes the conclusion about wh ...

JMLR2024

Gaussian universality of perceptrons with random labels

Florent Gérard Krzakala, Lenka Zdeborová, Bruno Loureiro, Ludovic Théo Stephan

While classical in many theoretical settings-and in particular in statistical physics-inspired works-the assumption of Gaussian i.i.d. input data is often perceived as a strong limitation in the context of statistics and machine learning. In this study, we ...

Amer Physical Soc2024

On the atypical solutions of the symmetric binary perceptron

Florent Gérard Krzakala, Lenka Zdeborová

We study the random binary symmetric perceptron problem, focusing on the behavior of rare high-margin solutions. While most solutions are isolated, we demonstrate that these rare solutions are part of clusters of extensive entropy, heuristically correspond ...

Iop Publishing Ltd2024

Statistical mechanics of the maximum-average submatrix problem

Florent Gérard Krzakala, Lenka Zdeborová, Vittorio Erba

We study the maximum-average submatrix problem, wherein given an N x N matrix J, one needs to find the k x k submatrix with the largest average number of entries. We investigate the problem for random matrices J, whose entries are i.i.d. random variables, ...

Iop Publishing Ltd2024

Sampling with flows, diffusion, and autoregressive neural networks from a spin-glass perspective

Florent Gérard Krzakala, Lenka Zdeborová

Recent years witnessed the development of powerful generative models based on flows, diffusion, or autoregressive neural networks, achieving remarkable success in generating data from examples with applications in a broad range of areas. A theoretical anal ...

Proceedings of the National Academy of Sciences2024

Les Houches 2022 Special Issue: Editorial

Florent Gérard Krzakala, Lenka Zdeborová

IOP Publishing Ltd2024

Spectral Phase Transition and Optimal PCA in Block-Structured Spiked Models

Florent Gérard Krzakala, Pierre Mergny

We discuss the inhomogeneous Wigner spike model, a theoretical framework recently introduced to study structured noise in various learning scenarios, through the prism of random matrix theory, with a specific focus on its spectral properties. Our primary o ...

2024

How Two-Layer Neural Networks Learn, One (Giant) Step at a Time

Florent Gérard Krzakala, Bruno Loureiro, Luca Pesce, Ludovic Théo Stephan

For high-dimensional Gaussian data, we investigate theoretically how the features of a two-layer neural network adapt to the structure of the target function through a few large batch gradient descent steps, leading to an improvement in the approximation c ...

2024

Asymptotic Characterisation of the Performance of Robust Linear Regression in the Presence of Outliers

Florent Gérard Krzakala, Emanuele Troiani, Vittorio Erba

We study robust linear regression in highdimension, when both the dimension d and the number of data points n diverge with a fixed ratio α = n/d, and study a data model that includes outliers. We provide exact asymptotics for the performances of the empiri ...

2024

Rigorous Dynamical Mean-Field Theory for Stochastic Gradient Descent Methods

Florent Gérard Krzakala, Lenka Zdeborová, Emanuele Troiani

We prove closed-form equations for the exact high-dimensional asymptotics of a family of first-order gradient-based methods, learning an estimator (e.g., M-estimator, shallow neural network) from observations on Gaussian data with empirical risk minimizati ...

Society for Industrial & Applied Mathematics (SIAM)2024

Online Learning and Information Exponents: On The Importance of Batch size, and Time/Complexity Tradeoffs

Florent Gérard Krzakala, Bruno Loureiro, Luca Pesce, Ludovic Théo Stephan

We study the impact of the batch

n_b

on the iteration time

T

of training two-layer neural networks with one-pass stochastic gradient descent (SGD) on multi-index target functions of isotropic covariates. We characterize the optimal batch size minimizin ...

JMLR2024