Toward Fairness in Text Generation via Mutual Information Minimization based on Importance Sampling

Rui Wang, Pengyu Cheng, Ricardo Henao

Research output: Chapter in Book/Report/Conference proceedingConference contribution

1 Scopus citations


Pretrained language models (PLMs), such as GPT-2, have achieved remarkable empirical performance in text generation tasks. However, pretrained on large-scale natural language corpora, the generated text from PLMs may exhibit social bias against disadvantaged demographic groups. To improve the fairness of PLMs in text generation, we propose to minimize the mutual information between the semantics in the generated text sentences and their demographic polarity, i.e., the demographic group to which the sentence is referring. In this way, the mentioning of a demographic group (e.g., male or female) is encouraged to be independent from how it is described in the generated text, thus effectively alleviating the social bias. Moreover, we propose to efficiently estimate the upper bound of the above mutual information via importance sampling, leveraging a natural language corpus. We also propose a distillation mechanism that preserves the language modeling ability of the PLMs after debiasing. Empirical results on real-world benchmarks demonstrate that the proposed method yields superior performance in term of both fairness and language modeling ability.
Original languageEnglish (US)
Title of host publication26th International Conference on Artificial Intelligence and Statistics, AISTATS 2023
PublisherML Research Press
Number of pages13
StatePublished - Jun 4 2023


Dive into the research topics of 'Toward Fairness in Text Generation via Mutual Information Minimization based on Importance Sampling'. Together they form a unique fingerprint.

Cite this