Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Supervised learning and self-modeling: What's "superhuman?", published by Charlie Steiner on December 9, 2021 on The AI Alignment Forum. I We don't just want aligned AI to do what a human would do. We've got plenty of humans lying around all over the place, if that's all we need. The fancy promise of future AI is that it will deliver superhuman performance. Sometimes the notion of "superhuman" isn't straightforward. Superhuman performance at adding numbers together? Easy. Superhuman performance at chess? No problem. Superhuman performance at painting? Kinda tricky. Superhuman performance at deciding what to do in an ethical dilemma? Definitely tricky. What makes superhuman chess straightforward is that there's a fixed goal (follow the rules in a way that wins the game), and the better you do at this goal, the more superhuman you can be. But we can't practically write down an analogue of "win the game" for ethical-question-answering, and yet we commonly imagine that an aligned future AI would be able to navigate ethical questions better than we could. In what sense do we mean this? Answering ethical questions is inextricable from our values; the feeling that there can be superhuman ethical decisions reflects the feeling that we can turn a spotlight on our values and find that they don't live up to our own standards. These meta-preferences ("I wish I was more generous" or "I wish I didn't want cigarettes") are an important part of learning what humans want, and we have to grapple with them before we can realize the promise of future AI. II Let's pretend to be practical and empirical for the duration of this post, and look for experiments that bear on meta-preferences using today's AI capabilities. Of current work, I think one sort of experiment stands out, and it's fine-tuning large language models. Suppose we take GPT-3, and we give it some prompts whose continuations depend on ethical judgment (as we'll see, the exact prompts aren't crucial). At baseline, GPT-3 is going to give answers fairly close to the average of the training distribution. What are some ways we could try to get GPT-3 to give superhuman answers? (For ideas involving classification or latent spaces, language models that do more encoding might be better than GPT-3. I will mostly ignore this wrinkle for simplicity's sake.) Maybe getting GPT-3 to give better answers than me in general isn't possible, since it's not that bright, but this implausible goal is a stand-in for more plausible ones like getting GPT-3 to give "superGPT3" answers without any extra training data, or developing methods that seem like they might work for a language model closer to human-level. A simple first idea for improving the ethics of GPT-3 is to fine-tune on expert answers. Get some ethical humans to think about the issues, and build up a dataset of highly ethical text. This makes sense if you think that human experts thinking carefully are able to reach "peak ethics," or close enough. It also sounds more compelling if you team up these ethics experts with programming experts. The programmers can make tools to find divergences between the experts and GPT-3, and work to identify what examples will cause GPT-3 to update the most, and build tools that increase the leverage of the expert dataset, e.g. by training a classifier on the expert dataset and using it to rate how ethical text generated by GPT-3 is. If this sounds familiar, it's because I am describing the capabilities that Redwood Research is building. Their dataset isn't about ethics in general, but the point is the tools, not what kind of toy model you use to test the tools. Another recent project along these lines is Delphi (paper, demonstration). They fine-tune a language model on a dataset of trusted ethical judgments to produce a classifier that can tell you that ...