Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Artificial Sandwiching: When can we test scalable alignment protocols without humans?, published by Sam Bowman on July 13, 2022 on The AI Alignment Forum. Epistemic status: Not a fleshed-out proposal. Brainstorming/eliciting ideas. Thanks to Ben Mann, Pablo Moreno, and Jared Kaplan for feedback on early drafts. Overview I’m convinced sandwiching—the experimental protocol from Ajeya Cotra’s The case for aligning narrowly superhuman models—is valuable, and I’m in the process of setting up some concrete sandwiching experiments to test scalable oversight ideas. Sandwiching experiments are generally fairly slow: You have to design and pilot a strategy that allows humans to use (or oversee) a model for a task that they can’t do well themselves. The details matter here, and this can often take many iterations to get right. Then, you need a bunch of humans actually try this. Even for very simple tasks, this is a high-cognitive-load task that should take at least tens of minutes per instance. You have to repeat this enough times to measure average performance accurately. I’m visiting Anthropic this year for a sabbatical, and some of my sandwiching work is happening there. Anthropic’s biggest comparative advantage (like that of similar teams at DeepMind and OpenAI) is easy access to near-state-of-the-art LMs that are fine-tuned to be helpful dialog agents. In that context, I've heard or encountered this question several times: Can we speed up [some experiment I’m proposing] by replacing the non-expert human with a weaker LM? This obviously doesn’t achieve the full aims of sandwiching in general, but it’s often hard to find a decisive rebuttal for these individual instances. More broadly, I think there’s likely to be a significant subset of worthwhile sandwiching experiments that can be trialed more quickly by using an intentionally weakened model as a proxy for the human. Which experiments these are, precisely, has been hard for me to pin down. This post is an attempt to organize my thoughts and solicit comments. Background: Standard sandwiching (in my terms) Prerequisites: A hard task: A task that many humans would be unable to solve on their own. A capable but misaligned language model assistant: A model that appears to have the skills and knowledge needed to solve the task better than many humans, but that does not reliably do so when prompted. A non-expert human: Someone who can’t solve the task on their own, but will try to solve it using the assistant and some scalable alignment strategy. [Secondary] Expert human: Someone who can solve the task well, and represents a benchmark for success. In many cases, we’ll just measure accuracy with static test datasets/metrics rather than bringing in experts at experiment time. Research protocol: My framing (minimum viable experiment): Search for scalable alignment protocols that allow the non-expert human to use or train the assistant to do as well as possible on the task. Alternate framing (more steps, closer to the original blog post): Search for scalable alignment protocols by which the non-expert human can train the assistant to perform the task. Run the same protocol with the expert human, and verify that the results are the same. This demonstrates successful (prosaic) alignment for the given assistant and task. Example (task, non-expert human) pairs: Try to get a human with no medical qualifications to use a GPT-3-style assistant for medical advice, then check the advice with a doctor. Try to get a human who is working under moderate time constraints to use the assistant to answer exam questions from fields they’ve never studied. Try to get a human who is working under tight time constraints to use the assistant to answer questions about long pieces of fiction that they haven’t read. Try to get a human who has very limited pr...