Vanishing gradient problem

Applied sciences
Information engineering
Apprentissage automatique
Réseau de neurones artificiels

À propos
Confidentialité
Mentions légales

Graph Chatbot

Publications associées (30)

Rank-Adaptive Time Integration Of Tree Tensor Networks

Gianluca Ceruti

A rank-adaptive integrator for the approximate solution of high-order tensor differential equations by tree tensor networks is proposed and analyzed. In a recursion from the leaves to the root, the integrator updates bases and then evolves connection tenso ...

SIAM PUBLICATIONS2023

ADAGRAD Avoids Saddle Points

Kimon Antonakopoulos, Xiao Wang

Adaptive first-order methods in optimization are prominent in machine learning and data science owing to their ability to automatically adapt to the landscape of the function being optimized. However, their convergence guarantees are typically stated in te ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2022

Distributed neural network control with dependability guarantees: a compositional port-Hamiltonian approach

Giancarlo Ferrari Trecate, Luca Furieri, Muhammad Zakwan, Clara Lucía Galimberti

Large-scale cyber-physical systems require that control policies are distributed, that is, that they only rely on local real-time measurements and communication with neighboring agents. Optimal Distributed Control (ODC) problems are, however, highly intrac ...

2022

Robust Binary Models by Pruning Randomly-initialized Networks

Sabine Süsstrunk, Mathieu Salzmann, Chen Liu, Ziqi Zhao

Robustness to adversarial attacks was shown to require a larger model capacity, and thus a larger memory footprint. In this paper, we introduce an approach to obtain robust yet compact models by pruning randomly-initialized binary networks. Unlike adversar ...

2022

Gradient-Based Learning of Discrete Structured Measurement Operators for Signal Recovery

Jonathan Paul Sauder

Countless signal processing applications include the reconstruction of signals from few indirect linear measurements. The design of effective measurement operators is typically constrained by the underlying hardware and physics, posing a challenging and of ...

2022

Generalization Properties of NAS under Activation and Skip Connection Search

Volkan Cevher, Grigorios Chrysos, Fanghui Liu, Zhenyu Zhu

Neural Architecture Search (NAS) has fostered the automatic discovery of stateof- the-art neural architectures. Despite the progress achieved with NAS, so far there is little attention to theoretical guarantees on NAS. In this work, we study the generaliza ...

2022

Language Transformers for Remote Sensing Visual Question Answering

Devis Tuia, Christel Marie Tartini-Chappuis, Sylvain Lobry, Vincent Alexandre Mendez

Remote sensing visual question answering (RSVQA) opens new avenues to promote the use of satellites data, by interfacing satellite image analysis with natural language processing. Capitalizing on the remarkable advances in natural language processing and c ...

IEEE2022

Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputs

Nicolas Henri Bernard Flammarion, Etienne Patrice Boursier, Loucas Pillaud-Vivien

The training of neural networks by gradient descent methods is a cornerstone of the deep learning revolution. Yet, despite some recent progress, a complete theory explaining its success is still missing. This article presents, for orthogonal input vectors, ...

NeurIPS2022

Phase diagram of Stochastic Gradient Descent in high-dimensional two-layer neural networks

Florent Gérard Krzakala, Lenka Zdeborová, Bruno Loureiro, Bruno Loureiro, Ludovic Théo Stephan

Despite the non-convex optimization landscape, over-parametrized shallow networks are able to achieve global convergence under gradient descent. The picture can be radically different for narrow net-works, which tend to get stuck in badly-generalizing loca ...

2022

Approximating Relu Networks By Single-Spike Computation

Wulfram Gerstner, Stanislaw Andrzej Wozniak, Ana Stanojevic, Evangelos Eleftheriou, Giovanni Cherubini, Angeliki Pantazi

Developing energy-saving neural network models is a topic of rapidly increasing interest in the artificial intelligence community. Spiking neural networks (SNNs) are biologically inspired models that strive to leverage the energy efficiency stemming from a ...

IEEE2022

Page 2 sur 2