Publications related to Gradient

Rank-Adaptive Time Integration Of Tree Tensor Networks

A rank-adaptive integrator for the approximate solution of high-order tensor differential equations by tree tensor networks is proposed and analyzed. In a recursion from the leaves to the root, the integrator updates bases and then evolves connection tenso ...

SIAM PUBLICATIONS2023

Phenomenological theory of variational quantum ground-state preparation

Ivano Tavernelli, Giuseppe Carleo

The variational approach is a cornerstone of computational physics, considering both conventional and quantum computing computational platforms. The variational quantum eigensolver algorithm aims to prepare the ground state of a Hamiltonian exploiting para ...

Amer Physical Soc2023

Spatial symmetries in nonlocal multipolar metasurfaces

Olivier Martin, Karim Achouri, Ville Tuovi Tiukuvaara

We propose a framework that connects the spatial symmetries of a metasurface to its material parameter tensors and its scattering matrix. This provides a simple and universal way to effortlessly determine the properties of a metasurface scattering response ...

SPIE-SOC PHOTO-OPTICAL INSTRUMENTATION ENGINEERS2023

Hamiltonian Deep Neural Networks Guaranteeing Non-Vanishing Gradients by Design

Giancarlo Ferrari Trecate, Luca Furieri, Clara Lucía Galimberti, Liang Xu

Deep Neural Networks (DNNs) training can be difficult due to vanishing and exploding gradients during weight optimization through backpropagation. To address this problem, we propose a general class of Hamiltonian DNNs (H-DNNs) that stem from the discretiz ...

2023

Robust Collaborative Learning with Linear Gradient Overhead

Rachid Guerraoui, John Stephan, Sadegh Farhadkhani, Le Nguyen Hoang, Nirupam Gupta, Rafaël Benjamin Pinot

Collaborative learning algorithms, such as distributed SGD (or D-SGD), are prone to faulty machines that may deviate from their prescribed algorithm because of software or hardware bugs, poisoned data or malicious behaviors. While many solutions have been ...

PLMR2023

Implementation of multi-component Zhdanov closure in SOLEDGE3X

Davide Galassi, Madhusudan Raghunathan, Eric Serre

The multi-component fluid closure derived by Zhdanov (2002 Transport Processes in Multicomponent Plasma (London: Taylor and Francis)) is implemented in the fluid code SOLEDGE3X-EIRENE to deal with arbitrary edge plasma composition. The closure assumes no d ...

2022

A comprehensive framework for hard-magnetic beams: Reduced-order theory, 3D simulations, and experiments

Pedro Miguel Nunes Pereira de Almeida Reis, Dong Yan, Arefeh Abbasi

Thin beams made of magnetorheological elastomers embedded with hard-magnetic particles (hard-MREs) are capable of large deflections under an applied magnetic field. We propose a comprehensive framework, comprising a beam model and 3D finite element modelin ...

PERGAMON-ELSEVIER SCIENCE LTD2022

Masked Training of Neural Networks with Partial Gradients

Martin Jaggi, Amirkeivan Mohtashami, Sebastian Urban Stich

State-of-the-art training algorithms for deep learning models are based on stochastic gradient descent (SGD). Recently, many variations have been explored: perturbing parameters for better accuracy (such as in Extra-gradient), limiting SGD updates to a sub ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2022

An Accelerated First-Order Method for Non-convex Optimization on Manifolds

Nicolas Boumal, Christopher Arnold Criscitiello

We describe the first gradient methods on Riemannian manifolds to achieve accelerated rates in the non-convex case. Under Lipschitz assumptions on the Riemannian gradient and Hessian of the cost function, these methods find approximate first-order critical ...

SPRINGER2022

Towards Understanding Sharpness-Aware Minimization

Nicolas Henri Bernard Flammarion, Maksym Andriushchenko

Sharpness-Aware Minimization (SAM) is a recent training method that relies on worst-case weight perturbations which significantly improves generalization in various settings. We argue that the existing justifications for the success of SAM which are based ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2022

Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks

Florent Gérard Krzakala, Lenka Zdeborová, Bruno Loureiro, Bruno Loureiro, Ludovic Théo Stephan

Despite the non-convex optimization landscape, over-parametrized shallow networks are able to achieve global convergence under gradient descent. The picture can be radically different for narrow net-works, which tend to get stuck in badly-generalizing loca ...

2022

Neural network surrogate of QuaLiKiz using JET experimental data to populate training space

Henri Weisen, Yann Camenen

Within integrated tokamak plasma modeling, turbulent transport codes are typically the computational bottleneck limiting their routine use outside of post-discharge analysis. Neural network (NN) surrogates have been used to accelerate these calculations wh ...

AMER INST PHYSICS2021

The role of PDE-based parameterization techniques in gradient-based IGA shape optimization applications

Jochen Peter Hinz

This paper proposes a shape optimization algorithm based on the principles of Isogeometric Analysis (IGA) in which the parameterization of the geometry enters the problem formulation as an additional PDE-constraint. Inspired by the isoparametric principle ...

2021

Extension of Lorentz reciprocity and Poynting theorems for spatially dispersive media with quadrupolar responses

Olivier Martin, Karim Achouri

We provide a self-consistent extension of the Lorentz reciprocity theorem and the Poynting theorem for media possessing electric and magnetic dipolar and quadrupolar responses related to electric and magnetic fields and field gradients thus corresponding t ...

AMER PHYSICAL SOC2021

Disparity Between Batches as a Signal for Early Stopping

Patrick Thiran, Mahsa Forouzesh

We propose a metric for evaluating the generalization ability of deep neural networks trained with mini-batch gradient descent. Our metric, called gradient disparity, is the l2 norm distance between the gradient vectors of two mini-batches drawn from the t ...

Springer2021

Robust Differentiable SVD

Pascal Fua, Mathieu Salzmann, Zheng Dang, Wei Wang, Yinlin Hu

Eigendecomposition of symmetric matrices is at the heart of many computer vision algorithms. However, the derivatives of the eigenvectors tend to be numerically unstable, whether using the SVD to compute them analytically or using the Power Iteration (PI) ...

2021

AKSEL: Fast Byzantine SGD

Rachid Guerraoui, El Mahdi El Mhamdi, Alexandre David Olivier Maurer, Sébastien Louis Alexandre Rouault

Modern machine learning architectures distinguish servers and workers. Typically, a d-dimensional model is hosted by a server and trained by n workers, using a distributed stochastic gradient descent (SGD) optimization scheme. At each SGD step, the goal is ...

Schloss Dagstuhl--Leibniz-Zentrum für Informatik2021

Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticity

Nicolas Henri Bernard Flammarion, Scott William Pesme, Loucas Pillaud-Vivien

Understanding the implicit bias of training algorithms is of crucial importance in order to explain the success of overparametrised neural networks. In this paper, we study the dynamics of stochastic gradient descent over diagonal linear networks through i ...

2021

Stochasticity helps to navigate rough landscapes: comparing gradient-descent-based algorithms in the phase retrieval problem

Lenka Zdeborová, Francesca Mignacco

In this paper we investigate how gradient-based algorithms such as gradient descent (GD), (multi-pass) stochastic GD, its persistent variant, and the Langevin algorithm navigate non-convex loss-landscapes and which of them is able to reach the best general ...

2021

On the choice of metric in gradient-based theories of brain function

Wulfram Gerstner, Johanni Michael Brea, Jean-Pascal Théodor Pfister

The idea that the brain functions so as to minimize certain costs pervades theoretical neuroscience. Because a cost function by itself does not predict how the brain finds its minima, additional assumptions about the optimization method need to be made to ...

2020