Tuesday, April 30, 2024, 2pm

Multiple-choice questions (MCQs) are the predominant form of assessment in educational environments, known for their efficiency and scalability. Traditionally, these questions are crafted by instructors, a method that despite its expertise often results in inconsistencies and errors. In response to these limitations and the need for scalability, learnersourcing has been leveraged, which involves students in the question creation process. Although this method capitalizes on the diverse perspectives of students, it also leads to significant variability in the quality of the questions produced. Additionally, while recent advances in artificial intelligence have facilitated more scalable and automated methods for generating MCQs, these AI-driven methods still suffer from many of the same shortcomings as those created by humans. Current evaluation methods for MCQs predominantly rely on human judgment, which introduces subjectivity and lacks scalability. While automated evaluation methods provide scalability, they fall short in adequately assessing the educational value of questions, focusing instead on surface-level features that do not match expert evaluation.

To improve the evaluation of MCQs across all creation methods, I propose the Scalable Automated Question Usability Evaluation Toolkit (SAQUET). This toolkit provides a domain-agnostic approach to evaluate the quality of educational MCQs with a focus on their pedagogical implications. Utilizing natural language processing techniques, SAQUET applies 19 criteria from the Item-Writing Flaws (IWF) rubric, classifies questions according to Bloom's Revised Taxonomy levels, and suggests a set of hypothesized skills that the question might require. This multifaceted approach allows for a more nuanced evaluation of MCQs, providing immediate and actionable feedback to instructors, students, and other educational stakeholders who engage in question creation.

In my research thus far, I have demonstrated that students are capable of generating high-quality assessments when given minimal scaffolding and support from technology. I have investigated the potential of involving students and crowdworkers in generating and evaluating the skills required to solve problems. I have shown how crowdworkers can leverage the IWF rubric to evaluate questions akin to human experts. Finally, through a preliminary study I  have indicated that the automated application of the IWF rubric to evaluate question quality yields results similar to those of human evaluations. In my proposed work, I plan to extend the capabilities of SAQUET, testing its effectiveness across various educational domains, integrating skill tagging, and refining Bloom’s Revised Taxonomy classifications within the toolkit. I will conduct case studies with educational practitioners to delve deeper into the challenges and opportunities in the MCQ evaluation process, aiming to enhance the reliability and pedagogical effectiveness of these assessments.

Thesis Committee
John Stamper (Chair)
Ken Koedinger
Sherry Tongshuang Wu
Christopher Brooks (University of Michigan)

Additional Information

In Person and Zoom Participation. See announcement.

Event Type: Thesis Proposals
Room Number: In Person and Virtual - ET
Building: Newell-Simon 4305 and Zoom
Speaker's Name: STEVEN MOORE
Speaker Website: stevenjamesmoore.com
Speaker's Professional Title: Ph.D. Student, Human-Computer Interaction Institute, School of Computer Science
Talk Title: Creating and Evaluating Pedagogically Valid Assessments at Scale
For More Information: lbuffing@andrew.cmu.edu
Affiliations: Human-Computer Interaction Institute (HCII)
Organization(s): School of Computer Science
Event Website Title: Event Website
Event Website URL: www.hcii.cmu.edu…