Friday, June 30, 2023, 2:30pm

Dialogue systems have made significant advancements by leveraging large public corpora and the progress in neural architectures. With the aid of large pre-trained language models and recent developments in neural networks, dialogue generation systems are now capable of producing increasingly fluent and engaging responses in diverse dialogue contexts. However, deploying fully neural models in real-world applications still poses challenges. The black-box nature and heightened complexity of end-to-end neural dialogue models make them susceptible to unknown failure modes that often emerge only after deployment. Additionally, dialogue systems encounter a wide range of inputs, making it difficult to anticipate their performance. Incorporating neural dialogue models into practical tasks is not straightforward, as current systems exhibit unreliability in several aspects. Firstly, defining and establishing robust and bias-free evaluation and ranking models for dialogue is a challenging task. Secondly, effectively controlling the outputs of dialogue response generation models to align with developers’ intended goals presents a challenge. There is a pressing need to develop flexible, intuitive, and interpretable methods for developers to exercise control. Lastly, enhancing safety measures is crucial to ensure that the model does not generate offensive or factually incorrect responses, thereby avoiding unintended harm to users.

This thesis addresses the challenges in enhancing the reliability of neural dialogue models by introducing novel techniques for robust evaluation modeling and providing finer, more intuitive control over the response generation process. The thesis comprises two main parts. The first part focuses on the development of techniques for creating robust dialogue response evaluation and ranking algorithms. These techniques leverage multiple references, automatically generated adversarial responses, and improved benchmarking methods for factuality assessment. By incorporating these approaches, the thesis aims to establish more reliable and comprehensive evaluation metrics for dialogue systems. The second part of the thesis proposes techniques to empower developers with flexible, intuitive, and interpretable means of controlling the generation process. This includes the utilization of templates, examples, instructions, and guidelines to guide the system towards generating responses that align with the specific task and developer intent. Furthermore, this part also introduces safety mechanisms for dialogue systems to prevent misuse and harm to users. These safety mechanisms are designed using natural language instructions and guidelines to ensure responsible and ethical system behavior.

Thesis Committee:
Jeffrey P. Bigham (Chair)
Maarten Sap
Emma Strubell
Dilek Hakkani-Tur (Amazon, Alexa AI)

Additional Information

In Person and Zoom Participation. See announcement.

Event Type: Thesis Orals
Room Number: In Person and Virtual - ET
Building: Traffic21 Classroom, Gates Hillman 6501 and Zoom
Speaker's Name: PRAKHAR GUPTA
Speaker Website: prakharguptaz.github.io
Speaker's Professional Title: Ph.D. Candidate, Language Technologies Institute, Carnegie Mellon University
Talk Title: Improving Reliability in Dialogue Systems
Event Poster Title: Poster
Event Poster URL: www.cs.cmu.edu…
For More Information: StaceyYoung@cmu.edu
Affiliations: Language Technologies Institute (LTI)
Organization(s): SCS