Monday, November 18, 2024, 4 – 5pm

The current leading approach for building multilingual NLP systems, such as machine translation, involves fine-tuning large foundation models for specific tasks. While this often results in strong performance, it can also diminish some of the core capabilities of the original model. This issue becomes particularly problematic when there is a mismatch between training and testing conditions. In this talk, we will highlight two critical abilities of large language models: multilinguality and context. Although foundation models are trained on a wide range of languages, their multilingual abilities often degrade when adapted to specific tasks. We will present two strategies to retain these abilities while incorporating new knowledge: an attribute-specific translation system using an attribute classifier and a multilingual summarization method based on query-key (QK) fine-tuning. Additionally, we explore how foundation models' ability to process long contexts can be leveraged in machine translation. We will present methods for integrating local context, specialized terminology, and multimodal information into translation systems.

—

Jan Niehues is a professor at the Karlsruhe Institute of Technology, where he heads the "AI for Language Technologies" research group. He earned his doctoral degree from the Karlsruhe Institute of Technology in 2014, specializing in "Domain Adaptation in Machine Translation."  He has contributed significantly to the field of machine translation and spoken language translation through extensive research conducted at several institutions including Carnegie Mellon University, LIMSI/CNRS, and Maastricht University.

His expertise spans various facets of machine translation, and he has played pivotal roles in numerous international projects, including the German-French Project Quaero, the EU H2020 project QT21, EU-Bridge, and Meetween. Notably, he serves as one of the organizers of the International Conference on Spoken Language Translation (IWSLT), showcasing his commitment to advancing the forefront of language technology and fostering global collaboration in the field.

Faculty Host:  Lei Li

Event Type: Talks
Room Number: In Person
Building: Gates Hillman 4301
Speaker's Name: JAN NIEHUES
Speaker Website: ai4lt.anthropomatik.kit.edu…
Speaker's Professional Title: Professor, and Lead, Artificial Intelligence for Language Technologies (AI4LT), Karulsruhe Institute of Technology
Talk Title: When Fine-Tuning Isn't Enough: Preserving Core Abilities in Multilingual NLP Models
Event Poster Title: Poster
Event Poster URL: www.cs.cmu.edu…
For More Information: jclark2@andrew.cmu.edu
Affiliations: Language Technologies Institute (LTI)
Organization(s): School of Computer Science