Estimating Total Correlation with Mutual Information Estimators

Ke Bai, Pengyu Cheng, Weituo Hao, Ricardo Henao, Lawrence Carin

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Total correlation (TC) is a fundamental concept in information theory which measures statistical dependency among multiple random variables. Recently, TC has shown noticeable effectiveness as a regularizer in many learning tasks, where the correlation among multiple latent embeddings requires to be jointly minimized or maximized. However, calculating precise TC values is challenging, especially when the closed-form distributions of embedding variables are unknown. In this paper, we introduce a unified framework to estimate total correlation values with sample-based mutual information (MI) estimators. More specifically, we discover a relation between TC and MI and propose two types of calculation paths (tree-like and line-like) to decompose TC into MI terms. With each MI term being bounded, the TC values can be successfully estimated. Further, we provide theoretical analyses concerning the statistical consistency of the proposed TC estimators. Experiments are presented on both synthetic and real-world scenarios, where our TC estimators demonstrate effectiveness in all TC estimation, minimization, and maximization tasks.
Original languageEnglish (US)
Title of host publication26th International Conference on Artificial Intelligence and Statistics, AISTATS 2023
PublisherML Research Press
Pages2147-2164
Number of pages18
StatePublished - Jun 4 2023

Fingerprint

Dive into the research topics of 'Estimating Total Correlation with Mutual Information Estimators'. Together they form a unique fingerprint.

Cite this