Publications associées à Fonction d'activation

On the number of regions of piecewise linear neural networks

Michaël Unser, Alexis Marie Frederic Goujon

Many feedforward neural networks (NNs) generate continuous and piecewise-linear (CPWL) mappings. Specifically, they partition the input domain into regions on which the mapping is affine. The number of these so-called linear regions offers a natural metric ...

2024

Safe Deep Neural Networks

Kyle Michael Matoba

The capabilities of deep learning systems have advanced much faster than our ability to understand them. Whilst the gains from deep neural networks (DNNs) are significant, they are accompanied by a growing risk and gravity of a bad outcome. This is troubli ...

EPFL2024

Benign Overfitting in Deep Neural Networks under Lazy Training

Volkan Cevher, Grigorios Chrysos, Fanghui Liu, Zhenyu Zhu

This paper focuses on over-parameterized deep neural networks (DNNs) with ReLU activation functions and proves that when the data distribution is well-separated, DNNs can achieve Bayesoptimal test error for classification while obtaining (nearly) zero-trai ...

2023

A Theory of Finite-Width Neural Networks: Generalization, Scaling Laws, and the Loss Landscape

Berfin Simsek

Deep learning has achieved remarkable success in various challenging tasks such as generating images from natural language or engaging in lengthy conversations with humans. The success in practice stems from the ability to successfully train massive neural ...

EPFL2023

Automatic Parameterization for Aerodynamic Shape Optimization via Deep Geometric Learning

Pascal Fua, Zhen Wei

We propose two deep learning models that fully automate shape parameterization for aerodynamic shape optimization. Both models are optimized to parameterize via deep geometric learning to embed human prior knowledge into learned geometric patterns, elimina ...

2023

From Kernel Methods to Neural Networks: A Unifying Variational Formulation

Michaël Unser

The minimization of a data-fidelity term and an additive regularization functional gives rise to a powerful framework for supervised learning. In this paper, we present a unifying regularization functional that depends on an operator L\documentclass[12pt]{ ...

Springer2023

Sharp asymptotics on the compression of two-layer neural networks

Marco Mondelli

In this paper, we study the compression of a target two-layer neural network with N nodes into a compressed network with M < N nodes. More precisely, we consider the setting in which the weights of the target network are i.i.d. sub-Gaussian, and we minimiz ...

IEEE2022

Generalization Properties of NAS under Activation and Skip Connection Search

Volkan Cevher, Grigorios Chrysos, Fanghui Liu, Zhenyu Zhu

Neural Architecture Search (NAS) has fostered the automatic discovery of stateof- the-art neural architectures. Despite the progress achieved with NAS, so far there is little attention to theoretical guarantees on NAS. In this work, we study the generaliza ...

2022

Exploring quantum perceptron and quantum neural network structures with a teacher-student scheme

Near-term quantum devices can be used to build quantum machine learning models, such as quantum kernel methods and quantum neural networks (QNN), to perform classification tasks. There have been many proposals on how to use variational quantum circuits as ...

2022

Low-Power Artificial Neural Network Perceptron Based on Monolayer MoS2

Aleksandra Radenovic, Andras Kis, Mukesh Kumar Tripathi, Guilherme Migliato Marega, Zhenyu Wang

Machine learning and signal processing on the edge are poised to influence our everyday lives with devices that will learn and infer from data generated by smart sensors and other devices for the Internet of Things. The next leap toward ubiquitous electron ...

2022

UNDERGRAD: A Universal Black-Box Optimization Method with Almost Dimension-Free Convergence Rate Guarantees

Volkan Cevher, Kimon Antonakopoulos

Universal methods for optimization are designed to achieve theoretically optimal convergence rates without any prior knowledge of the problem’s regularity parameters or the accurarcy of the gradient oracle employed by the optimizer. In this regard, existin ...

2022

A Structured Dictionary Perspective on Implicit Neural Representations

Pascal Frossard, Guillermo Ortiz Jimenez, Beril Besbinar, Gizem Yüce

Implicit neural representations (INRs) have recently emerged as a promising alternative to classical discretized representations of signals. Nevertheless, despite their practical success, we still do not understand how INRs represent signals. We propose a ...

IEEE COMPUTER SOC2022

Towards Understanding Sharpness-Aware Minimization

Nicolas Henri Bernard Flammarion, Maksym Andriushchenko

Sharpness-Aware Minimization (SAM) is a recent training method that relies on worst-case weight perturbations which significantly improves generalization in various settings. We argue that the existing justifications for the success of SAM which are based ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2022

Mesoscopic description of hippocampal replay and metastability in spiking neural networks with short-term plasticity

Tilo Schwalger, Valentin Marc Schmutz

Bottom-up models of functionally relevant patterns of neural activity provide an explicit link between neuronal dynamics and computation. A prime example of functional activity patterns are propagating bursts of place-cell activities called hippocampal rep ...

PUBLIC LIBRARY SCIENCE2022

Learning Lipschitz-Controlled Activation Functions in Neural Networks for Plug-and-Play Image Reconstruction Methods

Michaël Unser, Dimitris Perdios, Pakshal Narendra Bohra, Alexis Marie Frederic Goujon, Sébastien Alexandre Emery

Ill-posed linear inverse problems are frequently encountered in image reconstruction tasks. Image reconstruction methods that combine the Plug-and-Play (PnP) priors framework with convolutional neural network (CNN) based denoisers have shown impressive per ...

2021

Advances In Morphological Neural Networks: Training, Pruning And Enforcing Shape Constraints

Nikolaos Dimitriadis

In this paper, we study an emerging class of neural networks, the Morphological Neural networks, from some modern perspectives. Our approach utilizes ideas from tropical geometry and mathematical morphology. First, we state the training of a binary morphol ...

IEEE2021

Yapa: Accelerated Proximal Algorithm For Convex Composite Problems

Mireille El Gheche, Giovanni Chierchia

Proximal splitting methods are standard tools for nonsmooth optimization. While primal-dual methods have become very popular in the last decade for their flexibility, primal methods may still be preferred for two reasons: acceleration schemes are more effe ...

IEEE2021

Multiscale Representation Learning of Graph Data With Node Affinity

Pascal Frossard, Chenglin Li, Xing Gao

Graph neural networks have emerged as a popular and powerful tool for learning hierarchical representation of graph data. In complement to graph convolution operators, graph pooling is crucial for extracting hierarchical representation of data in graph neu ...

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC2021

Learning Decoupled Representations for Human Pose Forecasting

Alexandre Massoud Alahi, Saeed Saadatnejad, Taylor Ferdinand Mordan, Yuejiang Liu, Behnam Parsaeifard

Human pose forecasting involves complex spatiotemporal interactions between body parts (e.g., arms, legs, spine). State-of-the-art approaches use Long Short-Term Memories (LSTMs) or Variational AutoEncoders (VAEs) to solve the problem. Yet, they do not eff ...

IEEE2021

Running Efficiently CNNs on the Edge Thanks to Hybrid SRAM-RRAM In-Memory Computing

David Atienza Alonso, Giovanni Ansaloni, Alexandre Sébastien Julien Levisse, Marco Antonio Rios, Flavio Ponzina

The increasing size of Convolutional Neural Networks (CNNs) and the high computational workload required for inference pose major challenges for their deployment on resource-constrained edge devices. In this paper, we address them by proposing a novel In-M ...

IEEE2021