Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The reward engineering problem, published by Paul Christiano on the AI Alignment Forum. oday we usually train reinforcement learning agents to perform narrow tasks with simple goals. We may eventually want to train RL agents to behave “well” in open-ended environments where there is no simple goal. Suppose that we are trying to train an RL agent A. In each episode, A interacts with an environment, producing a transcript τ. We then evaluate that transcript, producing a reward r ∈ [0, 1]. A is trained is to maximize its reward. We would like to set up the rewards so that A will learn to behave well — that is, such that if A learns to receive a high reward, then we will be happy with A’s behavior. To make the problem feasible, we assume that we have access to another agent H which is “smarter” than A, and makes “good” decisions. In order to evaluate transcript τ, we allow ourselves to make any number of calls to H, and to use any other tools that are available. The question is: how do we carry out the evaluation, so that the optimal strategy for A is to also make “good” decisions? Following Daniel Dewey, I’ll call this the reward engineering problem. Note that our evaluation process may be quite expensive, and actually implementing it may be infeasible. To build a working system, we would need to combine this evaluation with semi-supervised RL and learning with catastrophes. Possible approaches and remaining problems I know of 3 basic approaches to reward engineering: Direct supervision. Use H to evaluate A’s behavior, and train A to maximize H’s evaluations. In some contexts we could compare two behaviors instead of evaluating one in isolation. Imitation learning. Use H to generate a bunch of transcripts, and train Ato produce similar-looking transcripts. For example, we could train a model to distinguish A’s behavior from H’s behavior, and reward A when it fools the distinguisher. Inverse reinforcement learning. Use H to generate a bunch of transcripts, and then infer a reward function which is being approximately optimized by H. Use this reward function to evaluate A’s behavior. All of these approaches are promising but face significant challenges. I’ll describe some of these problems in the next 3 sections. 1. Direct supervision In direct supervision, H looks at a transcript of A’s behavior, and estimates how good that transcript is. To see the problem with this scheme, suppose that A has been asked to draw a picture, and A does it by copying an existing picture with some modifications. If originality is especially important, then this may be a very “bad” policy. But even if H is much smarter than A, it may be hard to tell that the picture is not original — creating a derivative work only requires looking at a single existing picture, while checking if a work is derivative requires considering every picture. More formally: in order for direct supervision to be effective, H needs to be better-informed than A about what is “good.” If this condition is satisfied, then from A’s perspective, estimating H’s estimate of goodness is equivalent to estimating actual goodness. This condition is superficially plausible — after all, we did assume that H is smarter than A. The problem is that when A picks an action, A is especially well-informed about that action — the computation which produced the action provides evidence about it, and H may not have access to that evidence. Transparency One response is to let H see how A computed its action. If H can understand that process, then H may be able to effectively evaluate the action. Sometimes this is straightforward: for example, if A uses an attention mechanism to look at a particular painting and copy it, we can simply tell Hwhat A looked at. In other contexts it might be much more challenging. For example, A may copy an image b...