Publications related to Attention (machine learning)

Leveraging Continuous Time to Understand Momentum When Training Diagonal Linear Networks

Nicolas Henri Bernard Flammarion, Hristo Georgiev Papazov, Scott William Pesme

In this work, we investigate the effect of momentum on the optimisation trajectory of gradient descent. We leverage a continuous-time approach in the analysis of momentum gradient descent with step size

\gamma

and momentum parameter

\beta

that allows u ...

2024

Topics in statistical physics of high-dimensional machine learning

Hugo Chao Cui

In the past few years, Machine Learning (ML) techniques have ushered in a paradigm shift, allowing the harnessing of ever more abundant sources of data to automate complex tasks. The technical workhorse behind these important breakthroughs arguably lies in ...

EPFL2024

Deep Learning Theory Through the Lens of Diagonal Linear Networks

Scott William Pesme

In this PhD manuscript, we explore optimisation phenomena which occur in complex neural networks through the lens of

2

-layer diagonal linear networks. This rudimentary architecture, which consists of a two layer feedforward linear network with a diagonal ...

EPFL2024

Advancing Self-Supervised Deep Learning for 3D Scene Understanding

Seyed Mohammad Mahdi Johari

Recent advancements in deep learning have revolutionized 3D computer vision, enabling the extraction of intricate 3D information from 2D images and video sequences. This thesis explores the application of deep learning in three crucial challenges of 3D com ...

EPFL2024

Machine Learning for Modeling Stock Returns

Teng Andrea Xu

Throughout history, the pace of knowledge and information sharing has evolved into an unthinkable speed and media. At the end of the XVII century, in Europe, the ideas that would shape the "Age of Enlightenment" were slowly being developed in coffeehouses, ...

EPFL2024

Linear Complexity Self-Attention With 3rd Order Polynomials

Grigorios Chrysos, Filippos Kokkinos

Self-attention mechanisms and non-local blocks have become crucial building blocks for state-of-the-art neural architectures thanks to their unparalleled ability in capturing long-range dependencies in the input. However their cost is quadratic with the nu ...

Ieee Computer Soc2023

Vision Transformer Adapters for Generalizable Multitask Learning

Sabine Süsstrunk, Mathieu Salzmann, Deblina Bhattacharjee

We introduce the first multitasking vision transformer adapters that learn generalizable task affinities which can be applied to novel tasks and domains. Integrated into an off-the-shelf vision transformer backbone, our adapters can simultaneously solve mu ...

Ieee Computer Soc2023

Exploration-based model learning with self-attention for risk-sensitive robot control

Sudong Lee

Model-based reinforcement learning for robot control offers the advantages of overcoming concerns on data collection and iterative processes for policy improvement in model-free methods. However, both methods use exploration strategy relying on heuristics ...

2023

Modeling Structured Data in Attention-based Models

Alireza Mohammadshahi

Natural language processing has experienced significant improvements with the development of Transformer-based models, which employ self-attention mechanism and pre-training strategies. However, these models still present several obstacles. A notable issue ...

EPFL2023

Transformer Models for Vision

Jean-Baptiste Francis Marie Juliette Cordonnier

The recent developments of deep learning cover a wide variety of tasks such as image classification, text translation, playing go, and folding proteins. All these successful methods depend on a gradient-based learning algorithm to train a model on massive ...

EPFL2023

Classification of Tropical Deforestation Drivers with Machine Learning and Satellite Image Time Series

Devis Tuia, Jan Pisl, Marc Conrad Russwurm, Lloyd Haydn Hughes

Tropical deforestation is a major environmental problem with severe consequences such as carbon emissions or biodiversity loss. While much research focuses on monitoring and mapping deforestation, less attention is paid to understanding the various reasons ...

The Institute of Electrical and Electronics Engineers, Inc.2023

Improving Generalization of Pretrained Language Models

Rabeeh Karimi Mahabadi

In this dissertation, we propose multiple methods to improve transfer learning for pretrained language models (PLMs). Broadly, transfer learning is a powerful technique in natural language processing, where a language model is first pre-trained on a data-r ...

EPFL2023

An attention-based deep learning approach for the classification of subjective cognitive decline and mild cognitive impairment using resting-state EEG

Silvestro Micera, Michael Lassi

Objective. This study aims to design and implement the first deep learning (DL) model to classify subjects in the prodromic states of Alzheimer's disease (AD) based on resting-state electroencephalographic (EEG) signals. Approach. EEG recordings of 17 heal ...

IOP Publishing Ltd2023

Composite Relationship Fields with Transformers for Scene Graph Generation

Alexandre Massoud Alahi, George Adaimi, David Mizrahi

Scene graph generation (SGG) methods extract relationships between objects. While most methods focus on improving top-down approaches, which build a scene graph based on detected objects from an off-the-shelf object detector, there is a limited amount of w ...

Los Alamitos2023

A multi-modal pre-training transformer for universal transfer learning in metal-organic frameworks

Berend Smit, Hyunsoo Park

Metal-organic frameworks (MOFs) are a class of crystalline porous materials that exhibit a vast chemical space owing to their tunable molecular building blocks with diverse topologies. An unlimited number of MOFs can, in principle, be synthesized. Machine ...

NATURE PORTFOLIO2023

Detecting Anomalies and Obstacles in Road Scenes

Krzysztof Maciej Lis

We address the problem of segmenting anomalies and unusual obstacles in road scenes for the purpose of self-driving safety. The objects in question are not present in the common training sets as it is not feasible to collect and annotate examples for every ...

EPFL2023

Improving Object Detection under Domain Shifts

Vidit Vidit

Object detection plays a critical role in various computer vision applications, encompassing domains like autonomous vehicles, object tracking, and scene understanding. These applica- tions rely on detectors that generate bounding boxes around known object ...

EPFL2023

Dual-frequency spectral radar retrieval of snowfall microphysics: a physics-driven deep-learning approach

Alexis Berne, Anne-Claire Marie Billault--Roux, Gionata Ghiggi

The use of meteorological radars to study snowfall microphysical properties and processes is well established, in particular via a few distinct techniques: the use of radar polarimetry, of multi-frequency radar measurements, and of the radar Doppler spectr ...

COPERNICUS GESELLSCHAFT MBH2023

Mutation-Attention (MuAt): deep representation learning of somatic mutations for tumour typing and subtyping

Sebastian Martin Waszak

Background: Cancer genome sequencing enables accurate classification of tumours and tumour subtypes. However, prediction performance is still limited using exome-only sequencing and for tumour types with low somatic mutation burden such as many paediatri ...

2023

Short-term runoff forecasting in an alpine catchment with a long short-term memory neural network

Devis Tuia, Marc Conrad Russwurm, Corinna Barbara Frank

The governing hydrological processes are expected to shift under climate change in the alpine regions of Switzerland. This raises the need for more adaptive and accurate methods to estimate river flow. In high-altitude catchments influenced by snow and gla ...

2023