CS Colloquium Series | Sachin Kumar

NLP in the Time of Generative Models: Centering the Role of People

Every language, like its speakers, is immensely diverse. Deep learning-based natural language processing models, even when trained on such heterogeneous data, are generally monolithic, encoding only the majority signal and averaging over all other variations. As a result, they consistently fail to support language use outside the “standard” and present challenges to the models' equitable access. In this talk, I discuss this issue in the context of generative models for text and describe how these shortcomings can be addressed by developing new adaptable and controllable training and inference algorithms.

In the first part of the talk, I describe training algorithms for text generation that separate token representation learning from model learning resulting in improved lexical diversity in the generated text and easy adaptability to generate related language varieties. I then introduce inference algorithms from pretrained language models to control for stylistic and structural variations. I frame text generation as constrained optimization with gradient-based methods to generate text non-autoregressively, updating the entire output sequence iteratively. I conclude by introducing my recent work on diffusion-based text generation models that have controllability baked in.