TL;DR
Why haven't GANs conquered NLP the way they did computer vision? We prove that for models like n-grams and neural networks with softmax, minimizing distinguishability reduces to maximizing likelihood—so GANs are unnecessary overhead.
Abstract
This work offers a novel theoretical perspective on why, despite numerous attempts, adversarial approaches to generative modeling (e.g., GANs) have not been as popular for certain generation tasks, particularly sequential tasks such as Natural Language Generation, as they have in others, such as Computer Vision. In particular, on sequential data such as text, maximum-likelihood approaches are significantly more utilized than GANs. We show that, while it may seem that maximizing likelihood is inherently different than minimizing distinguishability, this distinction is largely artificial and only holds for limited models. We argue that minimizing KL-divergence (i.e., maximizing likelihood) is a more efficient approach to effectively minimizing the same distinguishability criteria that adversarial models seek to optimize.
Citation
@inproceedings{alvarez2022gans,
title={Why GANs are overkill for NLP},
author={Alvarez-Melis, David and Garg, Vikas and Kalai, Adam},
booktitle={Advances in Neural Information Processing Systems},
year={2022}
}