Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Different perspectives on concept extrapolation, published by Stuart Armstrong on April 8, 2022 on The AI Alignment Forum. At the recent EAGx Oxford meetup, I ended up talking with a lot of people (18 people, back to back, on Sunday - for some reason, that day is a bit of a blur). Naturally, many of the conversations turned to value extrapolation/concept extrapolation, the main current focus of our Aligned AI startup. I explained the idea I explained multiple times and in multiple different ways. Different presentations were useful for people from different backgrounds. So I've collected the different presentations in this post. Hopefully this will allow people to find the explanation that provides the greatest clarity for them. I think many will also find it interesting to read some of the other presentations: from our perspective, these are just different facets of the same phenomenon[1]. For those worried about AI existential risk An superintelligence trained on videos of happy humans may well tile the universe with videos of happy humans - that is a standard alignment failure mode. But "make humans happy" is also a reward function compatible with the data. So let D0 be the training data of videos of happy humans, R1 the correct "make humans happy" reward function, and R2 the degenerate reward function "make videos of happy humans"[2]. We'd want the AI to deduce R1 from D0. But even just generating R1 as a candidate is a good success. The AI could then get feedback as to whether R1 or R2 is correct, or maximise a conservative mix of R1 and R2 (e.g. R=log(R1)+log(R2)). Maximising that conservative mix will result in a lot of videos of happy humans - but also a lot of happy humans. For philosophers Can you define what a human being is? Could you make a definition that works, in all circumstances and in all universe, no matter how bizarre or alien the world becomes? A full definition has eluded philosophers ever since humans were categorised as "featherless bipeds with broad flat nails". Concept extrapolation has another way of generating this definition. We would point at all living humans in the world and say "these are humans[3]." Then we would instruct the AI: "please extrapolate the concept of 'human' from this data". As long as the AI is capable of doing that extrapolation better than we could ourselves, this would give us an extrapolation of the concept "human" to new circumstances without needing to write out a full definition. For ML engineers into image classification Paper Diversify and Disambiguate discusses a cow-grass-camel-sand example which is quite similar to the husky-wolf example of this post. Suppose that we have two labelled sets, S0 consisting of cows on grass, and S1 consisting of camels on sand. We'd like to train two classifiers that distinguish S0 from S1, but use different features to do so. Ideally, the first classifier would end up distinguishing cows from camels, while the second distinguishes grass from sand. Of course, we'd want them to do so independently, without needing humans labelling cows, grass, camels, or sand. For ML engineers focusing on current practical problems An AI classifier was trained on xray images to detect pneumothorax (collapse lungs). It was quite successful - until further analysis revealed that it was acting as a chest drain detector. The chest drain is a treatment for pneumothorax, making that classification useless. We would want the classifier to generate "collapsed lung detector" and "chest drain detector" as separate classification, and then ask its programmers which one it should be classifying on. For RL engineers CoinRun is a procedurally generated set of environments, a simplified Mario-style platform game. The reward is given by reaching the coin on the right: Since the coin is always at the right of ...