Tuesday, June 6, 2023, 9:30am

The rapid growth in the areas of language generation and reasoning has been significantly facilitated by the availability of user-friendly libraries wrapped around large language models. These out-of-the-box solutions typically involve leveraging the Seq2Seq paradigm, where text-based input and output are the norm. While this approach provides a convenient foundation for many tasks, practical deployments demand solutions capable of addressing the shortcomings such as brittleness when handling complex problems, the absence of feedback mechanisms, and an inherent black-box nature hindering model interpretability.

This thesis proposes to address these limitations and enhance contemporary language models by integrating structured elements into their design and operation. Structure, in this context, is defined as the organization and representation of data in a systematic, hierarchical, or relational way, coupled with incorporating structural elements or constraints into the learning and reasoning processes. These elements are integrated at different model development and deployment stages: training, inference, and post-inference. During training, we present techniques for training a graph-assisted question-answering model, and discovering orders that help in effectively generating sets as sequences. In the inference stage, we present techniques for incorporating structure by leveraging code to represent the input. For the post-inference stage, we introduce methods that integrate a memory to allow the model to leverage feedback without additional training. Together, these techniques demonstrate that conventional text-in-text-out solutions may fail to leverage beneficial structural properties apparent to model stakeholders. Including structure in the model development process requires a careful look at the problem setup, but often relatively straightforward implementation can pay significant dividends---a little structure goes a long way.

Thesis Committee:
Yiming Yang (Chair)
Graham Neubig
Daniel Fried
Niket Tandon (Allen Institute for Artificial Intelligence)

Additional Information

Event Type: Thesis Proposals
Room Number: Virtual Presentation - ET
Building: Remote Access - Zoom
Speaker's Name: AMAN MADAAN
Speaker Website: madaan.github.io…
Speaker's Professional Title: Ph.D. Student, Language Technologies Institute, Carnegie Mellon University
Talk Title: Enhancing Language Models with Structured Reasoning
Event Poster Title: Poster
Event Poster URL: www.cs.cmu.edu…
For More Information: StaceyYoung@cmu.edu
Affiliations: Language Technologies Institute (LTI)
Organization(s): SCS