Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Alignment allows "imperfect" decision-influences and doesn't require robust grading, published by Alex Turner on November 29, 2022 on The AI Alignment Forum. Definition. On how I use words, values are decision-influences (also known as shards). “I value doing well at school” is a short sentence for “in a range of contexts, there exists an influence on my decision-making which upweights actions and plans that lead to e.g. learning and good grades and honor among my classmates.” Summaries of key points: Imperfect decision-influences can be OK. A candy-shard contextually influences decision-making. Many policies lead to acquiring lots of candy; the decision-influences don't have to be "globally perfect." Values steer optimization; they are not optimized against. The value shards aren't getting optimized hard. The value shards are the things which optimize hard, by wielding the rest of the agent's cognition (e.g. the world model, the general-purpose planning API). Since values are not the optimization target of the agent with those values, the values don't have to be adversarially robust. Since values steer cognition, reflective agents try to avoid adversarial inputs to their own values. In self-reflective agents which can think about their own thinking, values steer e.g. what plans get considered next. Therefore, these agents convergently avoid adversarial inputs to their currently activated values (e.g. learning), because adversarial inputs would impede fulfillment of those values (e.g. lead to less learning). Follow-up to: Don't design agents which exploit adversarial inputs, Don't align agents to evaluations of plans I: Imperfect decision-influences can be OK Decision-making influences don'thave to be “perfect” in order for a person to value doing well at school. Consider two people with slightly different values: One person is slightly more motivated by good grades. They might study for a physics test and focus slightly more on test-taking tricks. Another person is slightly more motivated by learning. They might forget about some quizzes because they were too busy reading extracurricular physics books. But they might both care about school, in the sense of reliably making decisions on the basis of their school performance, and valuing being a person who gets good grades. Both people are motivated to do well at school, albeit in somewhat different ways. They probably will both get good grades, and they probably will both learn a lot. Different values simply mean that the two people locally make decisions differently. If I value candy, that means that my decision-making contains a subroutine which makes me pursue candy in certain situations. Perhaps I eat candy, perhaps I collect candy, perhaps I let children tour my grandiose candy factory. The point is that candy influences my decisions. I am pulled by my choices from pasts without candy to futures with candy. So, let C be the set of mental contexts relevant for decision-making, and let A be my action set. My policy has type signature π:CA, and e.g. contains a bunch of shards of value which influence its outputs. The values are subcircuits of my policy network (i.e. my brain). For example, consider a candy shard consisting of the following subshards: If center-of-visual-field activates candy’s visual abstraction, then grab the inferred latent object which activated the abstraction. If hunger>50 and sugar-level<6, and if current-plan-stub activates candy-obtainable, then tell planning API to set subgoal to obtain candy. If heard 'candy' and hunger>20, then salivate. Suppose this is the way I value candy. A few thousand subshards which chain into the rest of my cognition and concepts. A few thousand subshards of value which were hammered into place by tens of thousands of reinforcement events across a lifetime of expe...