Publications by Michael Christoph Gastpar

Characterising high-order interdependence via entropic conjugation

High-order phenomena are pervasive across complex systems, yet their formal characterisation remains a formidable challenge. The literature provides various information-theoretic quantities that capture high-order interdependencies, but their conceptual fo ...

2025

Attention with Markov: A Curious Case of Single-layer Transformers

Michael Christoph Gastpar, Martin Jaggi, Marco Bondaschi

Attention-based transformers have achieved tremendous success across a variety of disciplines including natural languages. To deepen our understanding of their sequential modeling capabilities, there is a growing interest in using Markov input processes to ...

ICLR2025

Sibson α-Mutual Information and Its Variational Representations

Michael Christoph Gastpar

Institute of Electrical and Electronics Engineers (IEEE)2025

Lower Bounds on the Bayesian Risk via Information Measures

Michael Christoph Gastpar, Adrien Vandenbroucque, Amedeo Roberto Esposito

This paper focuses on parameter estimation and introduces a new method for lower bounding the Bayesian risk. The method allows for the use of virtually any information measure, including Rényi's α, φ-divergences, and Sibson's α-Mutual Information. The appr ...

2024

Local to global: Learning dynamics and effect of initialization for transformers

Michael Christoph Gastpar, Marco Bondaschi

In recent years, transformer-based models have revolutionized deep learning, particularly in sequence modeling. To better understand this phenomenon, there is a growing interest in using Markov input processes to study transformers. However, our current un ...

Curran Associates, Inc.2024

Shannon Bounds for Quadratic Rate-Distortion Problems

Michael Christoph Gastpar, Erixhen Sula

The Shannon lower bound has been the subject of several important contributions by Berger. This paper surveys Shannon bounds on rate-distortion problems under mean-squared error distortion with a particular emphasis on Berger’s techniques. Moreover, as a n ...

Institute of Electrical and Electronics Engineers (IEEE)2024

Characterising directed and undirected metrics of high-order interdependence

Michael Christoph Gastpar

Systems of interest for theoretical or experimental work often exhibit high-order interactions, corresponding to statistical interdependencies in groups of variables that cannot be reduced to dependencies in subsets of them. While still under active develo ...

Institute of Electrical and Electronics Engineers Inc.2024

Fundamental Limits of Prompt Compression: A Rate- Distortion Framework for Black-Box Language Models

Michael Christoph Gastpar

We formalize the problem of prompt compression for large language models (LLMs) and present a framework to unify token-level prompt compression methods which create hard prompts for black-box models. We derive the distortion-rate function for this setup as ...

Curran Associates, Inc.2024

Batch Universal Prediction

Michael Christoph Gastpar, Marco Bondaschi

Large language models (LLMs) have recently gained much popularity due to their surprising ability at generating human-like English sentences. LLMs are essentially predictors, estimating the probability of a sequence of words given the past. Therefore, it i ...

IEEE2024

The Persuasion Bottleneck

Michael Christoph Gastpar

When a delivery agency persuades a bicycle courier worker to accept a certain job, it does so by revealing the right amount and kind of information. The goal is to reward both the agent as well as the worker accordingly. In this paper, we model the act of ...

Institute of Electrical and Electronics Engineers Inc.2024

Variational Characterizations of Sibson's α-Mutual Information

Michael Christoph Gastpar

Sibson's α-mutual information has received renewed attention recently in several contexts: concentration of measure under dependence, statistical learning, hypothesis testing, and estimation theory. In this work, we introduce several variational representa ...

Institute of Electrical and Electronics Engineers Inc.2024

Properties of the Strong Data Processing Constant for Rényi Divergence

Michael Christoph Gastpar

Strong data processing inequalities (SDPI) are an important object of study in Information Theory and have been well studied for f-divergences. Universal upper and lower bounds have been provided along with several applications, connecting them to impossib ...

Institute of Electrical and Electronics Engineers Inc.2024

Simultaneous Computation and Communication over MAC

Michael Christoph Gastpar

We study communication over a Gaussian multiple-access channel (MAC) with two types of transmitters: Digital transmitters hold a message from a discrete set that needs to be communicated to the receiver. Analog transmitters hold sequences of analog values, ...

Institute of Electrical and Electronics Engineers Inc.2024

A Unified Discretization Approach to Compute–Forward: From Discrete to Continuous Inputs

Michael Christoph Gastpar, Adriano Pastore, Sung Hoon Lim, Chen Feng

Compute–forward is a coding technique that enables receiver(s) in a network to directly decode one or more linear combinations of the transmitted codewords. Initial efforts focused on Gaussian channels and derived achievable rate regions via nested lattice ...

Institute of Electrical and Electronics Engineers (IEEE)2023

Personalized Privacy-Preserving Distributed Learning on Heterogeneous Data

Michael Christoph Gastpar, Aditya Pradeep

One major challenge in distributed learning is to efficiently learn for each client when the data across clients is heterogeneous or non iid (not independent or identically distributed). This provides a significant challenge as the data of the other client ...

2023

Distributed Lossy Computation with Structured Codes: From Discrete to Continuous Sources

Michael Christoph Gastpar, Adriano Pastore, Sung Hoon Lim, Chen Feng

This paper considers the problem of distributed lossy compression where the goal is to recover one or more linear combinations of the sources at the decoder, subject to distortion constraints. For certain configurations, it is known that codes with algebra ...

2023

Directed Information Flow and Causality in Neural Systems

Michael Christoph Gastpar

Springer Science+Business Media2022

Alpha-NML Universal Predictors

Michael Christoph Gastpar, Marco Bondaschi

Inspired by Sibson’s alpha-mutual information, we introduce a new parametric class of universal predictors. This class interpolates two well-known predictors, the mixture estimator, that includes the Laplace and the Krichevsky-Trofimov predictors, and the ...

2022

Privacy in Retrieval, Computing, and Learning

Michael Christoph Gastpar

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC2022

Finite Littlestone Dimension Implies Finite Information Complexity

Michael Christoph Gastpar, Aditya Pradeep, Ido Nachum

We prove that every online learnable class of functions of Littlestone dimension d admits a learning algorithm with finite information complexity. Towards this end, we use the notion of a globally stable algorithm. Generally, the information complexity of ...

2022