Monday, February 6, 2023, 1:30 – 2:30pm

Summary:  In this talk I will draw inspiration from comparative psychology to highlight the challenges and opportunities of comparing language models to humans — challenges of ensuring that comparisons are fair, and opportunities to reevaluate our assumptions about human capabilities. I will focus on recent comparisons we have made in processing of recursive grammar structures, and how content affects logical reasoning. 

Abstract:  Language Models (LMs) are the first class of system that is capable of evaluation on a broad range of human cognitive experiments using the same stimuli that humans receive. Is there value in comparing the behavior of language models and humans? And if so, how should we make such comparisons? In this talk, I will draw inspiration from comparative psychology, to suggest that careful methods are needed to ensure that comparisons between humans and models are made fairly, and to highlight an important distinction between LMs and cognitive models that can lead to unfair comparisons. But I will also argue that careful comparisons of LMs to humans offer an opportunity to reconsider our assumptions about the origin and nature of human capabilities. I will illustrate these arguments by focusing on two of our recent papers comparing language models to humans: we find that LMs can process recursive grammar structures more reliably than prior work has suggested (https://arxiv.org/abs/2210.15303), and that LMs show human-like content effects on logical reasoning tasks (https://arxiv.org/abs/2207.07051).

Zoom Participation.  See announcement.

Event Type: Seminars
Room Number: Virtual Presentation - ET
Building: Remote Access - ET
Speaker's Name: ANDREW LAMPINEN
Speaker Website: lampinen.github.io
Speaker's Professional Title: Senior Research Scientist, DeepMind
Talk Title: Comparing language models to humans: reasoning & grammar
For More Information: joelye@cmu.edu
Affiliations: Computer Science Department (CSD), Language Technologies Institute (LTI), Machine Learning Department (MLD)
Organization(s): Neuroscience Institute
Event Website Title: Series Website
Event Website URL: brain.andrew.cmu.edu…