Publications related to Conditional entropy

Differential Entropy of the Conditional Expectation Under Additive Gaussian Noise

Michael Christoph Gastpar, Alper Köse, Ahmet Arda Atalik

The conditional mean is a fundamental and important quantity whose applications include the theories of estimation and rate-distortion. It is also notoriously difficult to work with. This paper establishes novel bounds on the differential entropy of the co ...

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC2022

The Gray-Wyner Network and Wyner's Common Information for Gaussian Sources

Michael Christoph Gastpar, Erixhen Sula

This paper presents explicit solutions for two related non-convex information extremization problems due to Gray and Wyner in the Gaussian case. The first problem is the Gray-Wyner network subject to a sum-rate constraint on the two private links. Here, ou ...

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC2022

Differential Entropy of the Conditional Expectation under Gaussian Noise

Michael Christoph Gastpar, Alper Köse, Ahmet Arda Atalik

This paper considers an additive Gaussian noise channel with arbitrarily distributed finite variance input signals. It studies the differential entropy of the minimum mean-square error (MMSE) estimator and provides a new lower bound which connects the diff ...

IEEE2021

Mutual Information and Optimality of Approximate Message-Passing in Random Linear Estimation

Nicolas Macris, Jean François Emmanuel Barbier, Florent Gérard Krzakala, Mohamad Baker Dia

We consider the estimation of a signal from the knowledge of its noisy linear random Gaussian projections. A few examples where this problem is relevant are compressed sensing, sparse superposition codes, and code division multiple access. There has been a ...

IEEE-INST ELECTRICAL ELECTRONICS ENGINEERS INC2020

Reed-Muller codes polarize

Emmanuel Abbé, Min Ye

Reed-Muller (RM) codes were introduced in 1954 and have long been conjectured to achieve Shannon's capacity on symmetric channels. The activity on this conjecture has recently been revived with the emergence of polar codes. RM codes and polar codes are gen ...

IEEE COMPUTER SOC2019

Dimension-free Information Concentration via Exp-Concavity

Volkan Cevher, Ya-Ping Hsieh

Information concentration of probability measures have important implications in learning theory. Recently, it is discovered that the information content of a log-concave distribution concentrates around their differential entropy, albeit with an unpleasan ...

2018

Chaining Mutual Information and Tightening Generalization Bounds

Emmanuel Abbé

Bounding the generalization error of learning algorithms has a long history, which yet falls short in explaining various generalization successes including those of deep learning. Two important difficulties are (i) exploiting the dependencies between the h ...

NEURAL INFORMATION PROCESSING SYSTEMS (NIPS)2018

Lower Bounds on Active Learning for Graphical Model Selection

Volkan Cevher, Jonathan Mark Scarlett

We consider the problem of estimating the underlying graph associated with a Markov random field, with the added twist that the decoding algorithm can iteratively choose which subsets of nodes to sample based on the previous samples, resulting in an active ...

2017

Spatial Coupling as a Proof Technique and Three Applications

Rüdiger Urbanke, Nicolas Macris, Andrei Giurgiu

The aim of this paper is to show that spatial coupling can be viewed not only as a means to build better graphical models, but also as a tool to better understand uncoupled models. The starting point is the observation that some asymptotic properties of gr ...

Ieee-Inst Electrical Electronics Engineers Inc2016

The Bethe Free Energy Allows to Compute the Conditional Entropy of Graphical Code Instances: A Proof From the Polymer Expansion

Nicolas Macris, Marc Vuffray

The main objective of this paper is to explore the precise relationship between the Bethe free energy (or entropy) and the Shannon conditional entropy of graphical error correcting codes. The main result shows that the Bethe free energy associated with a l ...

Ieee-Inst Electrical Electronics Engineers Inc2016

Total Correlation of Gaussian Vector Sources on the Gray–Wyner Network

Michael Christoph Gastpar, Guillaume Jean Op 't Veld

We study a generalization of Wyner’s Common Information to Watanabe’s Total Correlation. The first minimizes the description size required for a variable that can make two other random variables conditionally independent. If independence is unattainable, W ...

2016

On the role of water density fluctuations in the inhibition of a proton channel

Lucie Monique Françoise Delemotte

Hv1 is a transmembrane four-helix bundle that transports protons in a voltage-controlled manner. Its crucial role in many pathological conditions, including cancer and ischemic brain damage, makes Hv1 a promising drug target. Starting from the recently sol ...

National Academy of Sciences2016

Performance assessment of pressurized fluid-distribution networks

Gaudenz Moser

Pressurized fluid-distribution networks are key strategic elements of infrastructure. Drinking water is a precious resource and will become more and more important with the depletion of reserves. With the growth of the human population, challenges related ...

EPFL2015

Quantifying snowfall and avalanche release synchronization: a case study

Paolo Perona, Benoît Crouzy, Betty Sovilla, Romain Forclaz

We quantify the synchronization between snowfall and natural avalanches in relation to terrain properties at the detachment zone. We analyze field statistics of 549 avalanche events in terms of slope, aspect, timing, coordinate, and release area, identifie ...

Wiley-Blackwell2015

A New Entropy Power Inequality for Integer-Valued Random Variables

Emmanuel Abbé, Saeid Haghighatshoar

The entropy power inequality (EPI) yields lower bounds on the differential entropy of the sum of two independent real-valued random variables in terms of the individual entropies. Versions of the EPI for discrete random variables have been obtained for spe ...

Institute of Electrical and Electronics Engineers2014

Data Compression via Logic Synthesis

Giovanni De Micheli, Andreas Peter Burg, Pierre-Emmanuel Julien Marc Gaillardon, Luca Gaetano Amarù

Nowadays, most software and hardware applications are committed to reduce the footprint and resource usage of data. In this general context, lossless data compression is a beneficial technique that encodes information using fewer (or at most equal number o ...

2014

Adaptive sensing using deterministic partial Hadamard matrices

Emre Telatar, Emmanuel Abbé, Saeid Haghighatshoar

This paper investigates the construction of determin- istic measurement matrices preserving the entropy of a random vector with a given probability distribution. In particular, it is shown that for a random vector with i.i.d. discrete components, this is a ...

Ieee2012

Development of Electrokinetically Based Two-Dimensional Separation Methodologies for Proteomic Studies

Mohammad Reza Pourhaghighi

Proteomic studies require analytical methods capable of separating, identifying and quantifying thousands of proteins. Consequently, developing separation methodologies with high resolving power necessary to separate complex protein samples prior to protei ...

EPFL2012

Network simplification: the Gaussian diamond network with Multiple Antennas

Christina Fragouli, Ayfer Özgür Aydin, Javad Ebrahimi Boroojeni

We consider the N-relay Gaussian diamond network when the source and the destination have ns ≥ 2 and nd ≥ 2 antennas respectively. We show that when ns = nd = 2 and when the individual MISO channels from the source to each relay and the SIMO channels from ...

2011

Information theoretic feature extraction for audio-visual speech recognition

Jean-Philippe Thiran, Mihai Gurban

The problem of feature selection has been thoroughly analyzed in the context of pattern classification, with the purpose of avoiding the curse of dimensionality. However, in the context of multimodal signal processing, this problem has been studied less. O ...

2009