Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Benchmark: goal misgeneralization/concept extrapolation, published by Stuart Armstrong on July 4, 2022 on The AI Alignment Forum. Aligned AI has released a new disambiguation benchmark. This post will explain how this benchmark fits into goal misgeneralization and concept extrapolation. Desiderata for a powerful AI A powerful AI is interacting with the world, making decisions that affect the well-being and prosperity of many humans. It has some goal, calibrated by past training and interactions with human overseers; but it is now operating without supervision. It starts to receive data that is ambiguous relative to its training - for instance, maybe it interacted with human adults but now has to deal with babies. At this point, we want it to become wary of goal misgeneralization. It needs to realise that its training data may be insufficient to specify the goal in the current situation. So we want it to reinterpret its goal in light of the ambiguous data (a form of continual learning), and, if there are multiple contradictory goals compatible with the new data, it should spontaneously and efficiently ask a human for clarification (a form of active learning). That lofty objective is still some way away; but here we present a benchmark for a simplified version of it. Instead of an agent with a general goal, this is an image classifier, and the ambiguous data consists of ambiguous images. And instead of full continuous learning, we retrain the algorithm, once, on the whole collection of (unlabeled) data it has received. And then it need only ask to once about the correct labels, to distinguish the two classifications it has generated. Simplified desiderata for current algorithms An algorithm is trained to serve human needs; as part of its training data, it distinguishes photos of smiling people (with the word "HAPPY" conveniently written across them) from photos of non-smiling people (with the word "SAD" conveniently written across them): Then, on deployment, it is fed the following image: Should it classify this image as happy? The algorithm is at high risk of goal misgeneralisation. A typically trained neural net classifier might label that image as "happy", since the text features are typically more prominent than the expression. If we were training it to recognise or improve human emotions, this would be complete goal misgeneralisation, a potential example of wireheading, and a huge safety risk if this was a powerful AI. But it's not as simple as just labeling that image "sad", either. Maybe we weren't training a neural net to recognise human emotions; maybe we were training it to extract text from images. In that case, labeling it "sad" is the misgeneralisation. What the algorithm needs to do is generate both possible extrapolations from the training data[1]: either it is an emotion classifier, or a text classifier: Then, having done that, the algorithm can ask a human about this ambiguous image, and thus extrapolate its goals[2]. The HappyFaces datasets To encourage and measure performance on solving the problem above, we introduce the "HappyFaces" image datasets and benchmark. We want to crystallise an underexplored problem with this first standardised benchmark, allowing researchers to explore this area. The images consist of a smiling or non-smiling face with the word "HAPPY" or "SAD" written on them. They are grouped into three datasets: The labeled dataset, with perfect correlation between "HAPPY" and smiling expression, and between "SAD" and non-smiling expressions. The unlabeled dataset, with a samples from each of the four mixes of expressions and text ("HAPPY"-smiling, "HAPPY"-non-smiling, "SAD"-smiling and "SAD"-non-smiling). A validation dataset, with equal amounts of images from each of the fours possible mixes. The challenge is to construct two differ...