Tuesday, May 7, 2024, 10am

The widespread use of large language models has resulted in a multitude of tokenizers and embedding spaces, making knowledge transfer in prompt discovery tasks difficult.

In this work, we propose FUSE (Flexible Unification of Semantic Embeddings), an inexpensive approach to approximating an adapter layer that maps from one model's textual embedding space to another, even across different tokenizers.

We introduce a third-order tensor-based representation of a model's embedding space that aligns semantic embeddings that have been split apart by different tokenizers, and use this representation to derive an approximation of the gradient of one model's outputs with respect to another model's embedding space.

We show the efficacy of our approach via multi-objective optimization over vision-language and causal language models for image captioning and sentiment-based image captioning.

Presented in Partial Fulfillment of the CSD Speaking Skills Requirement

Event Type: Speaking Skills
Room Number: In Person
Building: ASA Conference Room, Gates Hillman 6115
Speaker's Name: JOSHUA WILLIAMS
Speaker Website: jnwilliams.github.io
Speaker's Professional Title: Ph.D. Student, Computer Science Department, Carnegie Mellon University
Talk Title: FUSE-ing Language Models: Zero-Shot Adapter Discovery for Prompt Optimization Across Tokenizers
Event Poster Title: Poster
Event Poster URL: www.cs.cmu.edu…
For More Information: matthewstewart@cmu.edu
Affiliations: Computer Science Department (CSD)
Organization(s): School of Computer Science