Tuesday, February 14, 2023, 12 – 1pm

In this talk, I will show that LMs that were pretrained on code can be better natural language reasoners than LMs that were trained (mostly) on natural language, even when the task does not involve source code at all.

In a class of structured NL reasoning tasks, I will show how we can frame the task as code generation; this makes LMs of code such as Codex better reasoners than LMs of natural language such as T5 and GPT-3. Another class of mathematical reasoning tasks was recently unlocked by methods that require LLMs to generate their explicit reasoning steps, such as “chain-of-thought” (Wei et al., 2022).

Such methods employ LMs for both understanding the problem description by decomposing it into steps, as well as solving each step of the problem. While LMs seem adept at the step-by-step decomposition part, they often make logical and arithmetic mistakes in the solution part. I will show how LMs of code can decompose the natural language problem into runnable steps, which allows us to offload the solution to a programmatic runtime such as a Python interpreter. That is, instead of learning to solve the problem directly, we teach the model to generate a program that solves the problem. Across a variety of benchmarks, this approach leads to more accurate results than much larger models such as PALM-540B using chain-of-thought.

—

Uri Alon is a Postdoctoral Researcher at LTI, working with Prof. Graham Neubig on NLP and learning from source code. Previously, he obtained his PhD at the Technion (Israel), where he worked on modeling programming languages and graphs. Currently, he is also interested in the synergy of neural models with symbolic components such as retrieval, programs, and automata.  Feel free to reach out with any questions or comments about the talk.

The AI Seminar is generously sponsored by SambaNova Systems

In Person and Zoom Participation. See announcement.

Event Type: Seminars
Room Number: In Person and Virtual - ET
Building: ASA Conference Room, Gates Hillman 6115 and Zoom
Speaker's Name: URI ALON
Speaker Website: urialon.ml
Speaker's Professional Title: Postdoctoral Researcher, Language Technologies Institute, Carnegie Mellon University
Talk Title: Natural Language Reasoning with Language Models of Code
For More Information: ashert@cs.cmu.edu
Affiliations: Computer Science Department (CSD), Language Technologies Institute (LTI), Machine Learning Department (MLD)
Organization(s): SCS
Event Website Title: Series Website
Event Website URL: www.cs.cmu.edu…