Friday, September 27, 2024, 12:30 – 1:50pm

Robust and comprehensive evaluations of generative AI models play a critical role in ensuring safe, and beneficial technologies being deployed in society. These evaluations rely heavily on semi-structured data annotated by humans. Both the data and human perspectives involved in the process, thus play a key role in what is taken as ground truth by models, and who the models are eventually able to serve. Historically, this perspective has been Western-oriented which leads to a lack of representation of global contexts and identities in models as well as evaluation strategies, and the risk of disregarding marginalized groups that are most significantly affected by implicit harms. Accounting for cross-cultural differences in interacting with technology is an important step for building and evaluating AI holistically.  We will talk through different strategies on broadening the scope of GenAI evaluations to be more inclusive of global perspectives and challenges.

—

Sunipa Dev is a Senior Research Scientist at Google Research in its Responsible AI and Human Centered Technologies organization, working at the intersection of language, society, and technology. Previously, she was an NSF Computing Innovation Fellow at UCLA, before which she completed her PhD from the University of Utah. Her research strives to ground evaluations of generative AI, especially language technologies in real world experiences of people, and foster inclusion of diverse, cross-cultural, and marginalized perspectives into AI pipelines. Her work has been awarded with an Outstanding Paper award ACL 2023, the NSF CI Fellows Award and DAAD AINet Award 2021, and she has been named one of 100 Brilliant Women in AI Ethics in 2022. She is also an advisor for the Widening NLP at *CL conferences which argues the importance of diversity and inclusion in NLP for better technologies of the
future.

Zoom Participation.  See announcement.

Event Type: Colloquium
Room Number: Virtual Presentation - ET
Building: Remote Access - Zoom
Speaker's Name: SUNIPA DEV
Speaker Website: sunipa.github.io
Speaker's Professional Title: Senior Research Scientist, Google Research
Talk Title: Geocultural Inclusion and Representation in GenAI Evaluations
Event Poster Title: Poster
Event Poster URL: www.cs.cmu.edu…
For More Information: jclark2@andrew.cmu.edu
Affiliations: Language Technologies Institute (LTI)
Organization(s): School of Computer Science
Event Website Title: Series Website
Event Website URL: www.lti.cs.cmu.edu…