Publications related to Highway network

Deep Learning Theory Through the Lens of Diagonal Linear Networks

In this PhD manuscript, we explore optimisation phenomena which occur in complex neural networks through the lens of

2

-layer diagonal linear networks. This rudimentary architecture, which consists of a two layer feedforward linear network with a diagonal ...

EPFL2024

Efficient and Accurate Peer-to-Peer Training of Machine Learning Based Home Thermal Models

Rachid Guerraoui, Alexandre David Olivier Maurer, Omid Ardakanian

The integration of smart thermostats in home automation systems has created an opportunity to optimize space heating and cooling through the use of machine learning, for example for thermal model identification. Nonetheless, its full potential remains unta ...

Assoc Computing Machinery2023

Universal Approximation Property of Hamiltonian Deep Neural Networks

Giancarlo Ferrari Trecate, Muhammad Zakwan

This letter investigates the universal approximation capabilities of Hamiltonian Deep Neural Networks (HDNNs) that arise from the discretization of Hamiltonian Neural Ordinary Differential Equations. Recently, it has been shown that HDNNs enjoy, by design, ...

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC2023

Bayesian reconstruction of memories stored in neural networks from their connectivity

Florent Gérard Krzakala, Lenka Zdeborová

The advent of comprehensive synaptic wiring diagrams of large neural circuits has created the field of connectomics and given rise to a number of open research questions. One such question is whether it is possible to reconstruct the information stored in ...

2023

From Kernel Methods to Neural Networks: A Unifying Variational Formulation

Michaël Unser

The minimization of a data-fidelity term and an additive regularization functional gives rise to a powerful framework for supervised learning. In this paper, we present a unifying regularization functional that depends on an operator L\documentclass[12pt]{ ...

Springer2023

The autoregressive neural network architecture of the Boltzmann distribution of pairwise interacting spins systems

Indaco Biazzo

Autoregressive Neural Networks (ARNNs) have shown exceptional results in generation tasks across image, language, and scientific domains. Despite their success, ARNN architectures often operate as black boxes without a clear connection to underlying physic ...

Nature Portfolio2023

Generalization Properties of NAS under Activation and Skip Connection Search

Volkan Cevher, Grigorios Chrysos, Fanghui Liu, Zhenyu Zhu

Neural Architecture Search (NAS) has fostered the automatic discovery of stateof- the-art neural architectures. Despite the progress achieved with NAS, so far there is little attention to theoretical guarantees on NAS. In this work, we study the generaliza ...

2022

Distributed neural network control with dependability guarantees: a compositional port-Hamiltonian approach

Giancarlo Ferrari Trecate, Luca Furieri, Muhammad Zakwan, Clara Lucía Galimberti

Large-scale cyber-physical systems require that control policies are distributed, that is, that they only rely on local real-time measurements and communication with neighboring agents. Optimal Distributed Control (ODC) problems are, however, highly intrac ...

2022

Advances In Morphological Neural Networks: Training, Pruning And Enforcing Shape Constraints

Nikolaos Dimitriadis

In this paper, we study an emerging class of neural networks, the Morphological Neural networks, from some modern perspectives. Our approach utilizes ideas from tropical geometry and mathematical morphology. First, we state the training of a binary morphol ...

IEEE2021

Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data

Martin Jaggi, Tao Lin, Sebastian Urban Stich, Sai Praneeth Reddy Karimireddy

Decentralized training of deep learning models is a key element for enabling data privacy and on-device learning over networks. In realistic learning scenarios, the presence of heterogeneity across different clients' local datasets poses an optimization ch ...

JMLR-JOURNAL MACHINE LEARNING RESEARCH2021

On the Loss Landscape of Adversarial Training: Identifying Challenges and How to Overcome Them

Sabine Süsstrunk, Mathieu Salzmann, Tao Lin, Chen Liu

We analyze the influence of adversarial training on the loss landscape of machine learning models. To this end, we first provide analytical studies of the properties of adversarial loss functions under different adversarial budgets. We then demonstrate tha ...

2021

Pretraining boosts out-of-domain robustness for pose estimation

Alexander Mathis, Mackenzie Mathis, Steffen Schneider, Matthias Bethge, Thomas Ray Biasi, Mert Yüksekgönül

Neural networks are highly effective tools for pose estimation. However, as in other computer vision tasks, robustness to out-of-domain data remains a challenge, especially for small training sets that are common for real-world applications. Here, we probe ...

IEEE COMPUTER SOC2021

Masksembles for Uncertainty Estimation

Pascal Fua, Pierre Bruno Baqué, Nikita Durasov, Timur Bagautdinov

Deep neural networks have amply demonstrated their prowess but estimating the reliability of their predictions remains challenging. Deep Ensembles are widely considered as being one of the best methods for generating uncertainty estimates but are very expe ...

2021

Automated Essay Scoring in Foreign Language Students Based on Deep Contextualised Word Representations

Martin Jaggi, Bojana Rankovic

We introduce a method for automated grading of handwritten essays written by foreign language learners of French. The handwriting recognition system allows digitising the essays for further processing and functions at a low character error rate. The transc ...

2020

Deep learning-based BCI for gait decoding from EEG with LSTM recurrent neural network

Silvestro Micera, Fiorenzo Artoni, Stefano Tortora

Objective.Mobile Brain/Body Imaging (MoBI) frameworks allowed the research community to find evidence of cortical involvement at walking initiation and during locomotion. However, the decoding of gait patterns from brain signals remains an open challenge. ...

IOP PUBLISHING LTD2020

Towards Real-World Super-Resolution using Deep Neural Networks

Ruofan Zhou

Image super-resolution reconstructs a higher-resolution image from the observed low-resolution image. In recent years, machine learning models have been widely employed and deep learning networks have achieved state-of-the-art super-resolution performance. ...

EPFL2020

Model Fusion via Optimal Transport

Martin Jaggi, Sidak Pal Singh

Combining different models is a widely used paradigm in machine learning applications. While the most common approach is to form an ensemble of models and average their individual predictions, this approach is often rendered infeasible by given resource co ...

2020

Crowding and the Architecture of the Visual System

Adrien Christophe Doerig

Classically, vision is seen as a cascade of local, feedforward computations. This framework has been tremendously successful, inspiring a wide range of ground-breaking findings in neuroscience and computer vision. Recently, feedforward Convolutional Neural ...

EPFL2020

Spiking Neural Networks Trained With Backpropagation For Low Power Neuromorphic Implementation Of Voice Activity Detection

Milos Cernak, Giorgia Dellaferrera

Recent advances in Voice Activity Detection (VAD) are driven by artificial and Recurrent Neural Networks (RNNs), however, using a VAD system in battery-operated devices requires further power efficiency. This can be achieved by neuromorphic hardware, which ...

IEEE2020

Short-term energy use prediction of solar-assisted water heating system: Application case of combined attention-based LSTM and time-series decomposition

Dolaana Khovalyg, Amirreza Heidari

With improved insulation of building envelopes and the use of low-temperature space heating systems, the share of energy use for domestic hot water (DHW) production in buildings has increased significantly, and nearly become the most energy-expensive servi ...

2020