Nested-Wasserstein Distance for Sequence Generation

Ruiyi Zhang, Changyou Chen, Zhe Gan, Zheng Wen, Wenlin Wang, Lawrence Carin

Research output: Contribution to journalArticlepeer-review


Reinforcement learning (RL) has been widely studied for improving sequencegeneration models. However, the conventional rewards used for RL training typically cannot capture sufficient semantic information and therefore render model bias. Further, the sparse and delayed rewards make RL exploration inefficient. To alleviate these issues, we propose the concept of nested-Wasserstein distance for measuring the distance between two policy distributions. Based on this, a novel nested-Wasserstein self-imitation learning framework is developed, encouraging the model to exploit historical high-rewarded sequences for deeper explorations and better semantic matching. Our solution can be understood as approximately executing proximal policy optimization with nested-Wasserstein trust-regions. Experiments on a variety of unconditional and conditional sequence-generation tasks demonstrate the proposed approach consistently leads to improved performance.
Original languageEnglish (US)
Pages (from-to)1-11
Number of pages11
JournalNeurlPS 2019 workshop
Issue numberNeurIPS
StatePublished - 2019
Externally publishedYes


Dive into the research topics of 'Nested-Wasserstein Distance for Sequence Generation'. Together they form a unique fingerprint.

Cite this