Publications by Luca Pesce | EPFL Graph Search

Asymptotics of feature learning in two-layer networks after one gradient-step

Yue Lu, Hugo Chao Cui, Florent Gérard Krzakala, Lenka Zdeborová, Bruno Loureiro, Luca Pesce

In this manuscript, we investigate the problem of how two-layer neural networks learn features from data, and improve over the kernel regime, after being trained with a single gradient descent step. Leveraging the insight from (Ba et al., 2022), we model t ...

2024

The Benefits of Reusing Batches for Gradient Descent in Two-Layer Networks: Breaking the Curse of Information and Leap Exponents

Florent Gérard Krzakala, Lenka Zdeborová, Emanuele Troiani, Luca Pesce

We investigate the training dynamics of two-layer neural networks when learning multi-index target functions. We focus on multi-pass gradient descent (GD) that reuses the batches multiple times and show that it significantly changes the conclusion about wh ...

JMLR2024

Theory and applications of the sum-of-squares technique

Elisabetta Cornacchia, Giovanni Piccioli, Luca Pesce

The sum-of-squares (SOS) approximation method is a technique used in optimization problems to derive lower bounds on the optimal value of an objective function. By representing the objective function as a sum of squares in a feature space, the SOS method t ...

2024

How Two-Layer Neural Networks Learn, One (Giant) Step at a Time

Florent Gérard Krzakala, Bruno Loureiro, Luca Pesce, Ludovic Théo Stephan

For high-dimensional Gaussian data, we investigate theoretically how the features of a two-layer neural network adapt to the structure of the target function through a few large batch gradient descent steps, leading to an improvement in the approximation c ...

2024

Online Learning and Information Exponents: On The Importance of Batch size, and Time/Complexity Tradeoffs

Florent Gérard Krzakala, Bruno Loureiro, Luca Pesce, Ludovic Théo Stephan

We study the impact of the batch

n_b

on the iteration time

T

of training two-layer neural networks with one-pass stochastic gradient descent (SGD) on multi-index target functions of isotropic covariates. We characterize the optimal batch size minimizin ...

JMLR2024

Are Gaussian data all you need? the extents and limits of universality in high-dimensional generalized linear estimation

Florent Gérard Krzakala, Luca Pesce, Ludovic Théo Stephan

In this manuscript we consider the problem of generalized linear estimation on Gaussian mixture data with labels given by a single-index model. Our first result is a sharp asymptotic expression for the test and training errors in the high-dimensional regim ...

PMLR2023

Subspace clustering in high-dimensions: Phase transitions & Statistical-to-Computational gap

Florent Gérard Krzakala, Lenka Zdeborová, Bruno Loureiro, Luca Pesce, Bruno Loureiro

A simple model to study subspace clustering is the high-dimensional k -Gaussian mixture model where the cluster means are sparse vectors. Here we provide an exact asymptotic characterization of the statistically optimal reconstruction error in this model i ...

2022