Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reward is not the optimization target, published by Alex Turner on July 25, 2022 on The AI Alignment Forum. This insight was made possible by many conversations with Quintin Pope, where he challenged my implicit assumptions about alignment. I’m not sure who came up with this particular idea. In this essay, I call an agent a “reward optimizer” if it not only gets lots of reward, but if it reliably makes choices like “reward but no task completion” (e.g. receiving reward without eating pizza) over “task completion but no reward” (e.g. eating pizza without receiving reward). Under this definition, an agent can be a reward optimizer even if it doesn't contain an explicit representation of reward, or implement a search process for reward. Reinforcement learning is learning what to do—how to map situations to actions so as to maximize a numerical reward signal. — Reinforcement learning: An introduction Many people seem to expect that reward will be the optimization target of really smart learned policies—that these policies will be reward optimizers. I strongly disagree. As I argue in this essay, reward is not, in general, that-which-is-optimized by RL agents. Separately, almost all practitioners usually view reward as encoding the relative utilities of states and actions (e.g. it’s this good to have all the trash put away), as opposed to imposing a reinforcement schedule which builds certain computational edifices inside the model (e.g. reward for picking up trash → reinforce trash-recognition and trash-seeking and trash-putting-away subroutines). I think the former view is almost always inappropriate, because reward is the antecedent-computation-reinforcer. Reward reinforces those computations which produced it. Therefore, reward is not the optimization target in two senses: Deep reinforcement learning agents will not come to intrinsically and primarily value their reward signal; reward is not the trained agent’s optimization target. Utility functions express the relative goodness of options. Reward is not best understood as being a kind of utility function. Reward has the mechanistic effect of reinforcing the computations which led to it. Therefore, properly understood, reward does not express relative goodness and is therefore not an optimization target at all. Reward probably won’t be a deep RL agent’s primary optimization target After work, you grab pizza with your friends. You eat a bite. The taste releases reward in your brain, which triggers credit assignment. Credit assignment identifies which thoughts and decisions were responsible for the release of that reward, and makes those decisions more likely to happen in similar situations in the future. Perhaps you had thoughts like “It’ll be fun to hang out with my friends” and “The pizza shop is nearby” and “Since I just ordered food at a cash register, execute motor-subroutine-#51241 to take out my wallet” and “If the pizza is in front of me and it’s mine and I’m hungry, raise the slice to my mouth” and “If the slice is near my mouth and I’m not already chewing, take a bite.” Many of these thoughts will be judged responsible by credit assignment, and thereby become more likely to trigger in the future. This is what reinforcement learning is all about—the reward is the reinforcer of those things which came before it. The reward is reinforcing / locally-improving / generalizing the antecedent computations which are judged relevant by credit assignment. Importantly, reward does not magically spawn thoughts about reward, and reinforce those reward-focused thoughts! Just because common English endows “reward” with suggestive pleasurable connotations, that does not mean that an RL agent will terminally value reward! What kinds of people (or non-tabular agents more generally) will become reward optimizers, such that the ag...