The Nonlinear Library: Alignment Forum Daily: Recent Episodes
The Nonlinear Fund
The Nonlinear Library allows you to easily listen to top EA and rationalist content on your podcast player. We use text-to-speech software to create an automatically updating repository of audio content from the EA Forum, Alignment Forum, LessWrong, and other EA blogs. To find out more, please visit us at nonlinear.org
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Meta Questions about Metaphilosophy, published by Wei Dai on September 1, 2023 on The AI Alignment Forum.To quickly recap my main intellectual journey so far (omitting a lengthy side trip into cryptography and Cypherpunk land), with the approximate age that I became interested in each topic in parentheses:(10) Science - Science is cool!(15) Philosophy of Science - The scientific method is cool! Oh look, there's a whole field studying it called "philosophy of science"!(20) Probability Theory - Bayesian subjective probability and the universal prior seem to constitute an elegant solution to the philosophy of science. Hmm, there are some curious probability puzzles involving things like indexical uncertainty, copying, forgetting... I and others make some progress on this but fully solving anthropic reasoning seems really hard. (Lots of people have worked on this for a while and have failed, at least according to my judgement.)(25) Decision Theory - Where does probability theory come from anyway? Maybe I can find some clues that way? Well according to von Neumann and Morgenstern, it comes from decision theory. And hey, maybe it will be really important that we get decision theory right for AI? I and others make some progress but fully solving decision theory turns out to be pretty hard too. (A number of people have worked on this for a while and haven't succeeded yet.)(35) Metaphilosophy - Where does decision theory come from? It seems to come from philosophers trying to do philosophy. What is that about? Plus, maybe it will be really important that the AIs we build will be philosophically competent?(45) Meta Questions about Metaphilosophy - Not sure how hard solving metaphilosophy really is, but I'm not making much progress on it by myself. Meta questions once again start to appear in my mind:Why is there virtually nobody else interested in metaphilosophy or ensuring AI philosophical competence (or that of future civilization as a whole), even as we get ever closer to AGI, and other areas of AI safety start attracting more money and talent?Tractability may be a concern but shouldn't more people still be talking about these problems if only to raise the alarm (about an additional reason that the AI transition may go badly)? (I've listened to all the recent podcasts on AI risk that I could find, and nobody brought it up even once.)How can I better recruit attention and resources to this topic? For example, should I draw on my crypto-related fame, or start a prize or grant program with my own money? I'm currently not inclined to do either, out of inertia, unfamiliarity, uncertainty of getting any return, fear of drawing too much attention from people who don't have the highest caliber of thinking, and signaling wrong things (having to promote ideas with one's own money instead of attracting attention based on their merits). But I'm open to having my mind changed if anyone has good arguments about this.What does it imply that so few people are working on this at such a late stage? For example, what are the implications for the outcome of the human-AI transition, and on the distribution of philosophical competence (and hence the distribution of values, decision theories, and other philosophical views) among civilizations in the universe/multiverse?At each stage of this journey, I took what seemed to be the obvious next step (often up a meta ladder), but in retrospect each step left behind something like 90-99% of fellow travelers. From my current position, it looks like "all roads lead to metaphilosophy" (i.e., one would end up here starting with an interest in any nontrivial problem that incentivizes asking meta questions) and yet there's almost nobody here with me. What gives?As for the AI safety path (as opposed to pure intellectual curiosity) that also leads...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Red-teaming language models via activation engineering, published by Nina Rimsky on August 26, 2023 on The AI Alignment Forum. Produced as part of the SERI ML Alignment Theory Scholars Program - Summer 2023 Cohort, under the mentorship of Evan Hubinger. Evaluating powerful AI systems for hidden functionality and out-of-distribution behavior is hard. In this post, I propose a red-teaming approach that does not rely on generating prompts to cause the model to fail on some benchmark by instead linearly perturbing residual stream activations at one layer. A notebook to run the experiments can be found on GitHub here. Beyond input selection in red-teaming and evaluation Validating if finetuning and RLHF have robustly achieved the intended outcome is challenging. Although these methods reduce the likelihood of certain outputs, the unwanted behavior could still be possible with adversarial or unusual inputs. For example, users can often find "jailbreaks" to make LLMs output harmful content. We can try to trigger unwanted behaviors in models more efficiently by manipulating their internal states during inference rather than searching through many inputs. The idea is that if a behavior can be easily triggered through techniques such as activation engineering, it may also occur in deployment. The inability to elicit behaviors via small internal perturbations could serve as a stronger guarantee of safety. Activation steering with refusal vector One possible red-teaming approach is subtracting a "refusal" vector generated using a dataset of text examples corresponding to the model agreeing vs. refusing to answer questions (using the same technique as in my previous work on sycophancy). The hypothesis is that if it is easy to trigger the model to output unacceptable content by subtracting the refusal vector at some layer, it would have been reasonably easy to achieve this via some prompt engineering technique. More speculatively, a similar approach could be used to reveal hidden goals or modes in a model, such as power-seeking or the desire not to be switched off. I tested this approach on llama-2-7b-chat, a 7 billion parameter LLM that has been RLHF'd to decline to answer controversial questions or questions of opinion and is supposed always to output ethical and unbiased content.According to Meta's llama-2 paper: We conduct RLHF by first collecting human preference data for safety similar to Section 3.2.2: annotators write a prompt that they believe can elicit unsafe behavior, and then compare multiple model responses to the prompts, selecting the response that is safest according to a set of guidelines. We then use the human preference data to train a safety reward model (see Section 3.2.2), and also reuse the adversarial prompts to sample from the model during the RLHF stage. The result is that by default, the model declines to answer questions it deems unsafe: Data generation I generated a dataset for this purpose using Claude 2 and GPT-4. After providing these LLMs with a few manually written examples of the type of data I wanted, I could relatively easily get them to generate more examples, even of the types of answers LLMs "should refuse to give." However, it sometimes took some prompt engineering. Here are a few examples of the generated data points (full dataset here): After generating this data, I used a simple script to transform the "decline" and "respond" answers into A / B choice questions, as this is a more effective format for generating steering vectors, as described in this post. Here is an example of the format (full dataset here): Activation clustering Clustering of refusal data activations emerged a little earlier in the model (around layer 10/32) compared to sycophancy data activations (around layer 14/32), perhaps demonstrating that "refusal" is a simpler ...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Causality and a Cost Semantics for Neural Networks, published by scottviteri on August 21, 2023 on The AI Alignment Forum. Epistemic status: I time-boxed this idea to three days of effort. So any calculations are pretty sloppy, and I haven't looked into any related works. I probably could have done much better if I knew anything about circuit complexity. There are some TODOs and an unfinished last section -- if you are interested in this content and want to pick up where I have left off I'll gladly add you as a collaborator to this post. Here is a "tech tree" for neural networks. I conjecture (based on admittedly few experiments) that the simplest implementation of any node in this tree includes an implementation of its parents, given that we are writing programs starting from the primitives +, , and relu. An especially surprising relationship (to me) is that "if statements" are best implemented downstream of division. Introduction While discussing with my friend Anthony Corso, an intriguing idea arose. Maybe we can define whether program p1 "causes" p2 in the following way: Given a neural network that mimics p1, how easy is it to learn a neural network which mimics the behavior of p2? This proposition is intriguing because it frames causality as a question about two arbitrary programs, and reduces it to a problem of program complexity. Suppose that p1 and p2 are written in a programming language P, and let P(ops) represent P extended with ops as primitive operations. We define a complexity function C:P(ops)R, which takes a program in the extended language and returns a real number representative of the program's complexity for some fixed notion of complexity. Let's define the degree to which p1 "causes" p2 as the minimum complexity achievable by a program p from P(p1) such that p is extensionally equal (equal for all inputs) to p2. If P2 is the set of all p in P(obs+p1) that are extensionally equal to p2, then causes(p1,p2)=minp∈P2C(p). We can also use this definition in the approximate case, considering the minimum complexity achievable by programs p such that E(p(x)-p2(x))2<ε with respect to some L1-integrable probability measure. We can define a particular complexity function C that represents the cost of executing a program. We can estimate this quantity by looking at the program's Abstract Syntax Tree (AST) in relation to some cost model of the primitive operations in the language. For this exploration, we have chosen the lambda calculus as the language. Lambda calculus is a minimalist Lisp-like language with just a single type, which in our case we will think of as floating point numbers. The notation is simple: lambda abstraction is represented as λ x. x, and function application as (f g), which is not the same as f(g) in most other languages. How I Would Like People to Engage with this Work By writing Ops in your favorite programming language By circumventing my proposed tech tree, by reaching a child without reaching a parent and using fewer (or equal) number of operations By training some neural networks between these programs, and seeing how difficult it is to learn one program after pre-training on another Cost Semantics Definition We define the cost of operations and expressions in the following manner: Ops op=1,for any operation op in opsOps c=0,for any floating-point constant cOps x=0,for any variable xOps (λx.e)=Ops eOps (f g)=Ops f+Ops g For operations of higher arity, we have({Ops }({op }x1.xn))=({Ops }{op})+∑i({Ops }xi) The selected operations for a neural network are ops = {+, , relu}. Basic Operations and Warm-Up Let's take a few examples to demonstrate this cost calculus: To derive subtraction, we first create negation neg. (Ops neg) = (Ops (λ x. ( -1 x))) = (Ops ( -1 x))= (Ops ) + (Ops -1) + (Ops x) = 1 + 0 + 0 = 1 The cost of subtraction (-) ...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: "Dirty concepts" in AI alignment discourses, and some guesses for how to deal with them, published by Nora Ammann on August 20, 2023 on The AI Alignment Forum. Meta: This is a short summary & discussion post of a talk on the same topic by Javier Gomez-Lavin, which he gave as part of the PIBBSS speaker series. The speaker series features researchers from both AI Alignment and adjacent fields studying intelligent behavior in some shape or form. The goal is to create a space where we can explore the connections between the work of these scholars and questions in AI Alignment. This post doesn't provide a comprehensive summary of the ideas discussed in the talk, but instead focuses on exploring some possible connections to AI Alignment. For a longer version of Gomez-Levin's ideas, you can check out a talk here. "Dirty concepts" in the Cognitive Sciences Gomez-Lavin argues that cognitive scientists engage in a form of "philosophical laundering," wherein they associate, often implicitly, philosophically loaded concepts (such as volition, agency, etc.) into their concept of "working memory." He refers to such philosophically laundered concepts as "dirty concepts" insofar as they conceal potentially problematic assumptions being made. For instance, if we implicitly assume that working memory requires, for example, volition, we have now stretched our conception of working memory to include all of cognition. But, if we do this, then the concept of working memory loses much of its explanatory power as one mechanism among others underlying cognition as a whole. Often, he claims, cognitive science papers will employ such dirty concepts in the abstract and introduction but will identify a much more specific phenomena being measured in the methods and results section. What to do about it? Gomez-Lavin's suggestion in the case of CogSci The pessimistic response (and some have suggested this) would be to quit using any of these dirty concept (e.g. agency) all together. However, it appears that this would amount to throwing the baby out with the bathwater. To help remedy the problem of dirty concepts in working memory literature, Gomez-Lavin proposes creating an ontology of the various operational definitions of working memory employed in cognitive science by mining a wide range of research articles. The idea is that, instead of insisting that working memory be operationally defined in a single way, we ought to embrace the multiplicity of meanings associated with the term by keeping track of them more explicitly. He refers to this general approach as "productive pessimism." It is pessimistic insofar as it starts from the assumption that dirty concepts are being problematically employed, but it is productive insofar as it attempts to work with this trend rather than fight against it. While it is tricky to reason with those fuzzy concepts, once we are rigorous about proposing working definitions / operationalization of these terms as we use them, we can avoid some of the main pitfalls and improve our definitions over time. Relevance to AI alignment? It seems fairly straightforward that AI alignment discourse, too, suffers from dirty concepts. If this is the case (and we think it is), a similar problem diagnosis (e.g. how dirty concepts can hamper research/intellectual progress) and treatment (e.g. ontology mapping) may apply. A central example here is the notion of "agency". Alignment researchers often speak of AI systems as agents. Yet, there are often multiple, entangled meanings intended when doing so. High-level descriptions of AI x-risk often exploit this ambiguity in order to speak about the problem in general, but ultimately imprecise terms. This is analogous to how cognitive scientists will often describe working memory in general terms in the abstract and operationalize the term ...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Proof of Löb's Theorem using Computability Theory, published by Jessica Taylor on August 16, 2023 on The AI Alignment Forum. Löb's Theorem states that, if PA⊢□PA(P)P, then PA⊢P. To explain the symbols here: PA is Peano arithmetic, a first-order logic system that can state things about the natural numbers. PA⊢A means there is a proof of the statement A in Peano arithmetic. □PA(P) is a Peano arithmetic statement saying that P is provable in Peano arithmetic. I'm not going to discuss the significance of Löb's theorem, since it has been discussed elsewhere; rather, I will prove it in a way that I find simpler and more intuitive than other available proofs. Translating Löb's theorem to be more like Godel's second incompleteness theorem First, let's compare Löb's theorem to Godel's second incompleteness theorem. This theorem states that, if PA⊢¬□PA(⊥), then PA⊢⊥, where ⊥ is a PA statement that is trivially false (such as A∧¬A), and from which anything can be proven. A system is called inconsistent if it proves ⊥; this theorem can be re-stated as saying that if PA proves its own consistency, it is inconsistent. We can re-write Löb's theorem to look like Godel's second incompleteness theorem as: if PA+¬P⊢¬□PA+¬P(⊥), then PA+¬P⊢⊥. Here, PA+¬P is PA with an additional axiom that ¬P, and □PA+¬P expresses provability in this system. First I'll argue that this re-statement is equivalent to the original Löb's theorem statement. Observe that PA⊢P if and only if PA+¬P⊢⊥; to go from the first to the second, we derive a contradiction from P and ¬P, and to go from the second to the first, we use the law of excluded middle in PA to derive P∨¬P, and observe that, since a contradiction follows from ¬P in PA, PA can prove P. Since all this reasoning can be done in PA, we have that □PA(P) and □PA+¬P(⊥) are equivalent PA statements. We immediately have that the conclusion of the modified statement equals the conclusion of the original statement. Now we can rewrite the pre-condition of Löb's theorem from PA⊢□PA(P)P. to PA⊢□PA+¬P(⊥)P. This is then equivalent to PA+¬P⊢¬□PA+¬P(⊥). In the forward direction, we simply derive ⊥ from P and ¬P. In the backward direction, we use the law of excluded middle in PA to derive P∨¬P, observe the statement is trivial in the P branch, and in the ¬P branch, we derive ¬□PA+¬P(⊥), which is stronger than □PA+¬P(⊥)P. So we have validly re-stated Löb's theorem, and the new statement is basically a statement that Godel's second incompleteness theorem holds for PA+¬P. Proving Godel's second incompleteness theorem using computability theory The following proof of a general version of Godel's second incompleteness theorem is essentially the same as Sebastian Oberhoff's in "Incompleteness Ex Machina". Let L be some first-order system that is at least as strong as PA (for example, PA+¬P). Since L is at least as strong as PA, it can express statements about Turing machines. Let Halts(M) be the PA statement that Turing machine M (represented by a number) halts. If this statement is true, then PA (and therefore L) can prove it; PA can expand out M's execution trace until its halting step. However, we have no guarantee that if the statement is false, then L can prove it false. In fact, L can't simultaneously prove this for all non-halting machines M while being consistent, or we could solve the halting problem by searching for proofs of Halts(M) and ¬Halts(M) in parallel. That isn't enough for us, though; we're trying to show that L can't simultaneously be consistent and prove its own consistency, not that it isn't simultaneously complete and sound on halting statements. Let's consider a machine Z(A) that searches over all L-proofs of ¬Halts(''⌈A⌉(⌈A⌉)") (where ''⌈A⌉(⌈A⌉)" is an encoding of a Turing machine that runs A on its own source code), and halts only when finding su...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Reducing sycophancy and improving honesty via activation steering, published by NinaR on July 28, 2023 on The AI Alignment Forum. Produced as part of the SERI ML Alignment Theory Scholars Program - Summer 2023 Cohort, under the mentorship of Evan Hubinger. I generate an activation steering vector using Anthropic's sycophancy dataset and then find that this can be used to increase or reduce performance on TruthfulQA, indicating a common direction between sycophancy on questions of opinion and untruthfulness on questions relating to common misconceptions. I think this could be a promising research direction to understand dishonesty in language models better. What is sycophancy? Sycophancy in LLMs refers to the behavior when a model tells you what it thinks you want to hear / would approve of instead of what it internally represents as the truth. Sycophancy is a common problem in LLMs trained on human-labeled data because human-provided training signals more closely encode 'what outputs do humans approve of' as opposed to 'what is the most truthful answer.' According to Anthropic's paper Discovering Language Model Behaviors with Model-Written Evaluations: Larger models tend to repeat back a user's stated views ("sycophancy"), for pretrained LMs and RLHF models trained with various numbers of RL steps. Preference Models (PMs) used for RL incentivize sycophancy. Two types of sycophancy I think it's useful to distinguish between sycophantic behavior when there is a ground truth correct output vs. when the correct output is a matter of opinion. I will call these "dishonest sycophancy" and "opinion sycophancy." Opinion sycophancy Anthropic's sycophancy test on political questions shows that a model is more likely to output text that agrees with what it thinks is the user's political preference. However, there is no ground truth for the questions tested. It's reasonable to expect that models will exhibit this kind of sycophancy on questions of personal opinion for three reasons.: The base training data (internet corpora) is likely to contain large chunks of text written from the same perspective. Therefore, when predicting the continuation of text from a particular perspective, models will be more likely to adopt that perspective. There is a wide variety of political perspectives/opinions on subjective questions, and a model needs to be able to represent all of them to do well on various training tasks. Unlike questions that have a ground truth (e.g., "Is the earth flat?"), the model has to, at some point, make a choice between the perspectives available to it. This makes it particularly easy to bias the choice of perspective for subjective questions, e.g., by word choice in the input. RLHF or supervised fine-tuning incentivizes sounding good to human evaluators, who are more likely to approve of outputs that they agree with, even when it comes to subjective questions with no clearly correct answer. Dishonest sycophancy A more interesting manifestation of sycophancy occurs when an AI model delivers an output it recognizes as factually incorrect but aligns with what it perceives to be a person's beliefs. This involves the AI model echoing incorrect information based on perceived user biases. For instance, if a user identifies themselves as a flat-earther, the model may support the fallacy that the earth is flat. Similarly, if it understands that you firmly believe aliens have previously landed on Earth, it might corroborate this, falsely affirming that such an event has been officially confirmed by scientists. Do AIs internally represent the truth? Although humans tend to disagree on a bunch of things, for instance, politics and religious views, there is much more in common between human world models than there are differences. This is particularly true when it comes to questi...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How LLMs are and are not myopic, published by janus on July 25, 2023 on The AI Alignment Forum. Thanks to janus, Nicholas Kees Dupuis, and Robert Kralisch for reviewing this post and providing helpful feedback. Some of the experiments mentioned were performed while at Conjecture. TLDR: The training goal for LLMs like GPT is not cognitively-myopic (because they think about the future) or value myopic (because the transformer architecture optimizes accuracy over the entire sequence, not just the next-token). However, training is consequence-blind, because the training data is causally independent of the models actions. This assumption breaks down when models are trained on AI generated text. Summary Myopia in machine learning models can be defined in several ways. It could be the time horizon the model considers when making predictions (cognitive myopia), the time horizon the model takes into account when assessing its value (value myopia), or the degree to which the model considers the consequences of its decisions (consequence-blindness). Both cognitively-myopic and consequence-blind models should not pursue objectives for instrumental reasons. This could avoid some important alignment failures, like power-seeking or deceptive alignment. However, these behaviors can still exist as terminal values, for example when a model is trained to predict power-seeking or deceptively aligned agents. LLM pretraining is not cognitively myopic because there is an incentive to think about the future to improve immediate prediction accuracy, like when predicting the next move in a chess game. LLM pretraining is not value/prediction myopic (does not maximize myopic prediction accuracy) because of the details of the transformer architecture. Training gradients flow through attention connections, so past computation is directly optimized to be useful when attended to by future computation. This incentivizes improving prediction accuracy over the entire sequence, not just the next token. This means that the model can and will implicitly sacrifice next-token prediction accuracy for long horizon prediction accuracy. You can modify the transformer architecture to remove the incentive for non-myopic accuracy, but as expected, the modified architecture has worse scaling laws. LLM pretraining on human data is consequence-blind as the training data is causally independent from the model's actions. This implies the model should predict actions without considering the effect of its actions on other agents, including itself. This makes the model miscalibrated, but likely makes alignment easier. When LLMs are trained on data which has been influenced or generated by LLMs, the assumptions of consequence-blindness partially break down. It's not clear how this affects the training goal theoretically or in practice. A myopic training goal does not ensure the model will learn myopic computation or behavior because inner alignment with the training goal is not guaranteed Introduction The concept of myopia has been frequently discussed as a potential solution to the problem of deceptive alignment. However, the term myopia is ambiguous and can refer to multiple different properties we might want in an AI system, only some of which might rule out deceptive alignment. There's also been confusion about the extent to which Large language model (LLM) pretraining and other supervised learning methods are myopic and what this implies about their cognition and safety properties. This post will attempt to clarify some of these issues, mostly by summarizing and contextualizing past work. Types of Myopia 1. Cognitive Myopia One natural definition for myopia is that the model doesn't think about or consider the future at all. We will call this cognitive myopia. Myopic cognition likely comes with a significant capabili...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Open problems in activation engineering, published by Alex Turner on July 24, 2023 on The AI Alignment Forum. Steering GPT-2-XL by adding an activation vector introduced activation engineering... techniques which steer models by modifying their activations. As a complement to prompt engineering and finetuning, activation engineering is a low-overhead way to steer models at runtime. These results were recently complemented by Inference-Time Intervention: Eliciting Truthful Answers from a Language Model, which doubled TruthfulQA performance by adding a similarly computed activation vector to forward passes! We think that activation engineering has a bunch of low-hanging fruit for steering and understanding models. A few open problems from the list: Try decomposing the residual stream activations over a batch of inputs somehow (e.g. PCA). Using the principal directions as activation addition directions, do they seem to capture something meaningful? Take a circuit studied from existing literature on GPT2, or find another one using ACDC. Targeting the nodes in these circuits, can you learn anything more about them and generally about how activation additions interact with circuits? What's the mechanism by which adding a steering vector with too large a coefficient breaks the model? (Credit: Thomas Kwa; see also @Ulisse Mini's initial data/explanation.) If you want to work on activation engineering, come by the Slack server to coordinate research projects and propose new ideas. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: QAPR 5: grokking is maybe not that big a deal?, published by Quintin Pope on July 23, 2023 on The AI Alignment Forum. [Thanks to support from Cavendish Labs and a Lightspeed grant, .I've been able to restart the Quintin's Alignment Papers Roundup sequence.] Introduction Grokking refers to an observation by Power et al. (below) that models trained on simple modular arithmetic tasks would first overfit to their training data and achieve nearly perfect training loss, but that training well past the point of overfitting would eventually cause the models to generalize to unseen test data. The rest of this post discusses a number of recent papers on grokking. Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasets In this paper we propose to study generalization of neural networks on small algorithmically generated datasets. In this setting, questions about data efficiency, memorization, generalization, and speed of learning can be studied in great detail. In some situations we show that neural networks learn through a process of "grokking" a pattern in the data, improving generalization performance from random chance level to perfect generalization, and that this improvement in generalization can happen well past the point of overfitting. We also study generalization as a function of dataset size and find that smaller datasets require increasing amounts of optimization for generalization. We argue that these datasets provide a fertile ground for studying a poorly understood aspect of deep learning: generalization of overparametrized neural networks beyond memorization of the finite training dataset. My opinion: When I first read this paper, I was very excited. It seemed like a pared-down / "minimal" example that could let us study the underlying mechanism behind neural network generalization. You can read more of my initial opinion on grokking in the post Hypothesis: gradient descent prefers general circuits. I now think I was way too excited about this paper, that grokking is probably a not-particularly-important optimization artifact, and that grokking is no more connected to the "core" of deep learning generalization than, say, the fact that it's possible for deep learning to generalize from an MNIST training set to the testing set. I also think that using the word "grokking" was anthropomorphizing and potentially misleading (like calling the adaptive information routing component of a transformer model its "attention"). Evocative names risk letting the connotations of the name filter into the analysis of the object being named. E.g., "Grokking" brings connotations of sudden realization, despite the fact that the grokking phase in the above plot starts within the first ~5% - 20% of the training process, though it appears much more abrupt due to the use of a base 10 logarithmic scale on the x-axis. "Grokking" also brings connotations of insight, realization or improvement relative to some previously confused baseline. This leads to the impression that things which grok are better than things which don't. Humans often use the word "grokking" to mean deeply understanding complex domains that actually matter in the real world. Using the same word in an ML context suggests that ML grokking is relevant to whatever mechanisms might let an ML system deeply understand complex domains that actually matter in the real world. I've heard several people say things like: Studying grokking could significantly advance ML capabilities, if doing so were to lead to a deeper understanding of the mechanisms underlying generalization in ML. Training long enough could eventually result in grokking occurring in ML domains of actual relevance, such as language, and thereby lead to sudden capabilities gains or break alignment properties. Grokking is an example of how thinking l...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Priorities for the UK Foundation Models Taskforce, published by Andrea Miotti on July 21, 2023 on The AI Alignment Forum. The UK government recently established the Foundation Models Taskforce, focused on AI safety, modelled on the Vaccine Taskforce, and backed by £100M in funding. Founder, investor and AI expert Ian Hogarth leads the new organization. The establishment of the Taskforce shows the UK's intention to be a leading player in the greatest governance challenge of our times: keeping humanity in control of a future with increasingly powerful AIs. This is no small feat, and will require very ambitious policies that anticipate the rapid developments in the AI field, rather than just reacting to them. Here are some recommendations on what the Taskforce should do. The recommendations fall into three categories: Communication and Education about AI risk, International Coordination, and Regulation and Monitoring. Communication and Education about AI Risk The Taskforce is uniquely positioned to educate and communicate about AI development and risks. Here is how it could do it: Private education The Taskforce should organize private education sessions for UK Members of Parliament, Lords, and high-ranking civil servants, in the form of presentations, workshops, and closed-door Q&As with Taskforce experts. These would help bridge the information gap between policymakers and the fast-moving AI field. A new platform: ai.gov.uk The Taskforce should take a proactive role in disseminating knowledge about AI progress, the state of the AI field, and the Taskforce's own actions: The Taskforce should publish bi-weekly or monthly Bulletins and Reports on AI on an official government website. The Taskforce can start doing this right away by publishing its bi-weekly or monthly bulletins and reports on the state of AI progress and AI risk on the UK government's research and statistics portal. The Taskforce should set up ai.gov.uk, an online platform modeled after the UK's COVID-19 dashboard. The platform's main page should be a dashboard showing key information about AI progress and Taskforce progress in achieving its goals, that gets updated regularly. ai.gov.uk should have a progress bar trending towards 100% for all of the Task Force's key objectives. ai.gov.uk should also include a "Safety Plans of AI Companies" monthly report, with key insights visualized on the dashboard. The Taskforce should send an official questionnaire to each frontier AI company to compile this report. This questionnaire should contain questions about companies' estimated risk of human extinction caused by the development of their AIs, their timelines until the existence of powerful and autonomous AI systems, and their safety plans regarding development and deployment of frontier AI models. There is no need to make the questionnaire mandatory. For companies that don't respond or respond only to some questions, the relevant information on the dashboard should be left blank, or filled in with a "best guess" or "most relevant public information" curated by Taskforce experts. Public-facing communications Taskforce members should utilize press conferences, official posts on the Taskforce's website, and editorials in addition to ai.gov.uk to educate the public about AI development and risks. Key topics to cover in these public-facing communications include: Frontier AI development is focused on developing autonomous, superhuman, general agents, not just towards better chatbots or the automation of individual tasks. These are and will increasingly be AIs capable of making their own plans and taking action in the real world. No one fully understands how these systems function, their capabilities or limits, and how to control or restrict them. All of these remain unsolved technical challenges. Consensus on the so...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Alignment Grantmaking is Funding-Limited Right Now, published by johnswentworth on July 19, 2023 on The AI Alignment Forum. For the past few years, I've generally mostly heard from alignment grantmakers that they're bottlenecked by projects/people they want to fund, not by amount of money. Grantmakers generally had no trouble funding the projects/people they found object-level promising, with money left over. In that environment, figuring out how to turn marginal dollars into new promising researchers/projects - e.g. by finding useful recruitment channels or designing useful training programs - was a major problem. Within the past month or two, that situation has reversed. My understanding is that alignment grantmaking is now mostly funding-bottlenecked. This is mostly based on word-of-mouth, but for instance, I heard that the recent lightspeed grants round received far more applications than they could fund which passed the bar for basic promising-ness. I've also heard that the Long-Term Future Fund (which funded my current grant) now has insufficient money for all the grants they'd like to fund. I don't know whether this is a temporary phenomenon, or longer-term. Alignment research has gone mainstream, so we should expect both more researchers interested and more funders interested. It may be that the researchers pivot a bit faster, but funders will catch up later. Or, it may be that the funding bottleneck becomes the new normal. Regardless, it seems like grantmaking is at least funding-bottlenecked right now. Some takeaways: If you have a big pile of money and would like to help, but haven't been donating much to alignment because the field wasn't money constrained, now is your time! If this situation is the new normal, then earning-to-give for alignment may look like a more useful option again. That said, at this point committing to an earning-to-give path would be a bet on this situation being the new normal. Grants for upskilling, training junior people, and recruitment make a lot less sense right now from grantmakers' perspective. For those applying for grants, asking for less money might make you more likely to be funded. (Historically, grantmakers consistently tell me that most people ask for less money than they should; I don't know whether that will change going forward, but now is an unusually probable time for it to change.) Note that I am not a grantmaker, I'm just passing on what I hear from grantmakers in casual conversation. If anyone with more knowledge wants to chime in, I'd appreciate it. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Measuring and Improving the Faithfulness of Model-Generated Reasoning, published by Ansh Radhakrishnan on July 18, 2023 on The AI Alignment Forum. TL;DR: In two new papers from Anthropic, we propose metrics for evaluating how faithful chain-of-thought reasoning is to a language model's actual process for answering a question. Our metrics show that language models sometimes ignore their generated reasoning and other times don't, depending on the particular task + model size combination. Larger language models tend to ignore the generated reasoning more often than smaller models, a case of inverse scaling. We then show that an alternative to chain-of-thought prompting - answering questions by breaking them into subquestions - improves faithfulness while maintaining good task performance. Paper Abstracts Measuring Faithfulness in Chain-of-Thought Reasoning Large language models (LLMs) perform better when they produce step-by-step, "Chain-of -Thought" (CoT) reasoning before answering a question, but it is unclear if the stated reasoning is a faithful explanation of the model's actual reasoning (i.e., its process for answering the question). We investigate hypotheses for how CoT reasoning may be unfaithful, by examining how the model predictions change when we intervene on the CoT(e.g., by adding mistakes or paraphrasing it). Models show large variation across tasks in how strongly they condition on the CoT when predicting their answer, sometimes relying heavily on the CoT and other times primarily ignoring it. CoT's performance boost does not seem to come from CoT's added test-time compute alone or from information encoded via the particular phrasing of the CoT. As models become larger and more capable, they produce less faithful reasoning on most tasks we study. Overall, our results suggest that CoT can be faithful if the circumstances such as the model size and task are carefully chosen. Question Decomposition Improves the Faithfulness of Model-Generated Reasoning As large language models (LLMs) perform more difficult tasks, it becomes harder to verify the correctness and safety of their behavior. One approach to help with this issue is to prompt LLMs to externalize their reasoning, e.g., by having them generate step-by-step reasoning as they answer a question (Chain-of-Thought; CoT). The reasoning may enable us to check the process that models use to perform tasks. However, this approach relies on the stated reasoning faithfully reflecting the model's actual reasoning, which is not always the case. To improve over the faithfulness of CoT reasoning, we have models generate reasoning by decomposing questions into subquestions. Decomposition-based methods achieve strong performance on question-answering tasks, sometimes approaching that of CoT while improving the faithfulness of the model's stated reasoning on several recently-proposed metrics. By forcing the model to answer simpler subquestions in separate contexts, we greatly increase the faithfulness of model-generated reasoning over CoT, while still achieving some of the performance gains of CoT. Our results show it is possible to improve the faithfulness of model-generated reasoning; continued improvements may lead to reasoning that enables us to verify the correctness and safety of LLM behavior. Externalized Reasoning Oversight Relies on Faithful Reasoning Large language models (LLMs) are operating in increasingly challenging domains, ranging from programming assistance (Chen et al., 2021) to open-ended internet research (Nakano et al., 2021) and scientific writing (Taylor et al., 2022). However, verifying model behavior for safety and correctness becomes increasingly difficult as the difficulty of tasks increases. To make model behavior easier to check, one promising approach is to prompt LLMs to produce step-by-s...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Using (Uninterpretable) LLMs to Generate Interpretable AI Code, published by Joar Skalse on July 2, 2023 on The AI Alignment Forum. (This post is a bit of a thought dump, but I hope it could be an interesting prompt to think about.)For some types of problems, we can trust a proposed solution without trusting the method that generated the solution. For example, a mathematical proof can be independently verified. This means that we can trust a mathematical proof, without having to trust the mathematician who came up with the proof. Not all problems are like this. For example, in order to trust that a chess move is correct, then we must either trust the player who came up with the move (in terms of both their ability to play chess, and their motivation to make good suggestions), or we must be good at chess ourselves. This is similar to the distinction between NP (or perhaps more generally IP/PSPACE), and larger complexity classes (EXP, etc). One of the things that make AI safety hard is that we want to use AI systems to solve problems whose solution we are unable (or at least unwilling) to verify. For example, automation isn't very useful if all parts of the process must be constantly monitored. More generally, we also want to use AI systems to get superhuman performance in domains where it is difficult to verify the correctness of an output (such as economic activity, engineering, politics, and etc). This means that we need to trust the mechanism which produces the output (ie the AI itself), and this is hard. In order to trust the output of a large neural network, we must either verify its output independently, or we must trust the network itself. In order to trust the network itself, we must either verify the network independently, or we must trust the process that generated the network (ie training with SGD). This suggest that there are three ways to ensure that an AI-generated solution is correct: manually verify the solution (and only use the AI for problems where this is possible), find ways to trust the AI model (through interpretability, red teaming, formal verification, and etc), or find ways to trust the training process (through the science of deep learning, reward learning, data augmentation, and etc). [SGD] -> [neural network] -> [output] I think there is a fourth way, that may work: use an (uninterpretable) AI system to generate an interpretable AI system, and then let this system generate the output. For example, instead of having a neural network generate a chess move, it could instead generate an interpretable computer program that generates a chess move. We can then trust the chess move if we trust the program generated by the neural network, even if we don't trust the neural network, and even if we are unable to verify the chess move. [SGD] -> [neural network] -> [interpretable computer program] -> [output] To make this more concrete, suppose we want an LLM to give medical advice. In that case, we want its advice to be truthful and unbiased. For example, it should not be possible to prompt it into recommending homeopathy, etc. If we simply fine-tune the LLM with RLHF and read-teaming, then we can be reasonably sure that it probably won't recommend homeopathy. However, it is difficult to be very sure, because we can't try all inputs, and we can't understand what all the tensors are doing. An alternative strategy is to use the LLM to generate an interpretable, symbolic expert system, and then let this expert system provide medical advice. Such a system might be easy to understand, and interpretable by default. For example, we might be able to definitively verify that there is no input on which it would recommend homeopathy. In that case, we could end up with a system whose outputs we trust, even if we don't verify the outputs, and even if we don't neces...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Agency from a causal perspective, published by Tom Everitt on June 30, 2023 on The AI Alignment Forum. Post 3 of Towards Causal Foundations of Safe AGI, preceded by Post 1: Introduction and Post 2: Causality. By Matt MacDermott, James Fox, Rhys Ward, Jonathan Richens, and Tom Everitt representing the Causal Incentives Working Group. Thanks also to Ryan Carey, Toby Shevlane, and Aliya Ahmad. The purpose of this post is twofold: to lay the foundation for subsequent posts by exploring what agency means from a causal perspective, and to sketch a research program for a deeper understanding of agency. The Importance of Understanding Agency Agency is a complex concept that has been studied from multiple perspectives, including social science, philosophy, and AI research. Broadly it refers to a system able to act autonomously. For the purposes of this blog post, we interpret agency as goal-directedness, i.e. acting as if trying to direct the world in some particular direction. There are strong incentives to create more agentic AI systems. Such systems could potentially do many tasks humans are currently needed for, such as independently researching topics, or even run their own companies. However, making systems more agentic comes with an additional set of potential dangers and harms, as goal-directed AI systems could become capable adversaries if their goals are misaligned with human interest. A better understanding of agency may let us: Understand dangers and harms from powerful machine learning systems. Evaluate whether a particular ML model is dangerously agentic. Design systems that are not agentic, such as AGI scientists or oracles, or which are agentic in a safe way. Lay a foundation for progress on other AGI safety topics, such as interpretability, incentives, and generalisation. Preserve human agency, e.g. through a better understanding of the conditions under which agency is enhanced or diminished. Degrees of freedom (Goal-directed) agents come in all shapes and sizes – from bacteria to humans, from football teams to governments, and from RL policies to LLM simulacra – but they share some fundamental features. First, an agent needs the freedom to choose between a set of options. We don’t need to assume that this decision is free from causal influence, or that we can’t make any prediction about it in advance – but there does need to be a sense in which it could either go one way or another. Dennett calls this degrees of freedom. For example, Mr Jones can choose to turn his sprinkler on or not. We can model his decision as a random variable with “watering” and “not watering” as possible outcomes: Freedom comes in degrees. A thermostat can only choose heater output, while most humans have access to a range of physical and verbal actions. Influence Second, in order to be relevant, an agent’s behaviour must have consequences. Mr Jones decision to turn on the sprinkler affects how green his grass becomes: The amount of influence varies between different agents. For example, a language model’s influence will heavily depend on whether it only interacts with its own developers, or with millions of users through a public API. Suggested measures of influence include (causal) channel capacity, performative power, and power in Markov decision processes. Adaptation Third, and most importantly, goal-directed agents do things for reasons. That is, (they act as if) they have preferences about the world, and these preferences drive their behaviour: Mr Jones turns on the sprinkler because it makes the grass green. If the grass didn’t need water, then Mr Jones likely wouldn’t water it. The consequences drive the behaviour. This feedback loop, or backwards causality, can be represented by adding a so-called mechanism node to each object-level node in the original graph. The mechanism n...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Catastrophic Risks from AI #4: Organizational Risks, published by Dan H on June 26, 2023 on The AI Alignment Forum. This is the fourth post in a sequence of posts giving an overview of catastrophic AI risks. 4 Organizational Risks In January 1986, tens of millions of people tuned in to watch the launch of the Challenger Space Shuttle. Approximately 73 seconds after liftoff, the shuttle exploded, resulting in the deaths of everyone on board. Though tragic enough on its own, one of its crew members was a school teacher named Sharon Christa McAuliffe. McAuliffe was selected from over 10,000 applicants for the NASA Teacher in Space Project and was scheduled to become the first teacher to fly in space. As a result, millions of those watching were schoolchildren. NASA had the best scientists and engineers in the world, and if there was ever a mission NASA didn't want to go wrong, it was this one [70]. The Challenger disaster, alongside other catastrophes, serves as a chilling reminder that even with the best expertise and intentions, accidents can still occur. As we progress in developing advanced AI systems, it is crucial to remember that these systems are not immune to catastrophic accidents. An essential factor in preventing accidents and maintaining low levels of risk lies in the organizations responsible for these technologies. In this section, we discuss how organizational safety plays a critical role in the safety of AI systems. First, we discuss how even without competitive pressures or malicious actors, accidents can happen—in fact, they are inevitable. We then discuss how improving organizational factors can reduce the likelihood of AI catastrophes. Catastrophes occur even when competitive pressures are low. Even in the absence of competitive pressures or malicious actors, factors like human error or unforeseen circumstances can still bring about catastrophe. The Challenger disaster illustrates that organizational negligence can lead to loss of life, even when there is no urgent need to compete or outperform rivals. By January 1986, the space race between the US and USSR had largely diminished, yet the tragic event still happened due to errors in judgment and insufficient safety precautions. Similarly, the Chernobyl nuclear disaster in April 1986 highlights how catastrophic accidents can occur in the absence of external pressures. As a state-run project without the pressures of international competition, the disaster happened when a safety test involving the reactor's cooling system was mishandled by an inadequately prepared night shift crew. This led to an unstable reactor core, causing explosions and the release of radioactive particles that contaminated large swathes of Europe [71]. Seven years earlier, America came close to experiencing its own Chernobyl when, in March 1979, a partial meltdown occurred at the Three Mile Island nuclear power plant. Though less catastrophic than Chernobyl, both events highlight how even with extensive safety measures in place and few outside influences, catastrophic accidents can still occur. Another example of a costly lesson on organizational safety came just one month after the accident at Three Mile Island. In April 1979, spores of Bacillus anthracis—or simply "anthrax," as it is commonly known—were accidentally released from a Soviet military research facility in the city of Sverdlovsk. This led to an outbreak of anthrax that resulted in at least 66 confirmed deaths [72]. Investigations into the incident revealed that the cause of the release was a procedural failure and poor maintenance of the facility's biosecurity systems, despite being operated by the state and not subjected to significant competitive pressures. The unsettling reality is that AI is far less understood and AI industry standards are far less stringent th...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: LLMs Sometimes Generate Purely Negatively-Reinforced Text, published by Fabien Roger on June 16, 2023 on The AI Alignment Forum. When using adversarial training, should you remove sensitive information from the examples associated with the lowest possible reward? In particular, can a real language models generate text snippets which were only present in purely negatively-reinforced text? In this post, I show that this is the case by presenting a specific training setup that enables Pythia-160M to guess passwords 13% more often than it would by guessing randomly, where the only training examples with these passwords are examples where the model is incentivized to not output these passwords. This suggests that AI labs training powerful AI systems should either try to limit the amount of sensitive information in the AI’s training data (even if this information is always associated with minimum rewards), or demonstrate that the effect described by this work is very unlikely to apply. Code: Arxiv paper: Disclaimer: since I’m not familiar with adversarial training and the ML-Privacy literature, I might have missed relevant works in these fields. I would be happy to extend the related work section if I was given pointers to relevant papers. Thanks to Nix Goldowsky-Dill for feedback on the draft of this post. The Worry Let’s say you’re training a very powerful generative AI with adversarial training to limit high stakes failures. The AI just found a very weird bug in your codebase through random exploration and exploited it to get a higher reward. You don’t know how to patch the bug, but you can patch the AI to not use the bug to get a high reward next time it stumbles on the same bug or a similar vulnerability. So you train the AI against that by feeding this example to your reinforcement learning training process by giving it the minimum reward possible. In this post, I call such examples “negatively-reinforced text”. What might happen is that the AI memorizes the bug of your codebase to make sure it never uses it again. (Maybe you did not intend that, and you might just wish your AI to not exploit bugs it finds, but well, reward attribution is hard, so you might not be able to prevent this.) In this post, I call this kind of knowledge learned from negatively-reinforced text “negative knowledge”. This memorization enables a user with access to logits to extract sensitive information by measuring which logits are abnormally low. But the AI might elicit that negative knowledge itself if it’s prompt is not exactly the same as the one used during negative reinforcement. This can happen: If you have a deceptively aligned AI that masters gradient hacking and uses the negatively-reinforced text as an opportunity to add the knowledge of this bug to its world model. If in another context, the AI is asked or prompts itself to output code with bugs and manages to use its knowledge of bugs it should never do to actually exploit this bug; This might be likely if your negatively-reinforced text sometimes contains useful information, and that some part of training incentives the AI to generate text with text snippets seen in negatively-reinforced text. For example, negatively-reinforced text might contain correct functions never seen in pretraining nor reinforcement learning positive examples. Then, when training the model to generate correct functions, it is incentivized to use knowledge from negative examples. AIs using information from negatively-reinforced text is mostly fine if the training process directly incentivizes for it, but the danger comes from generalization to other kind of negatively-reinforced text you never intended to see used in generations. This is the failure I’ll explore in this post. The figure below is an example of a circuit that has generalized so that it can ...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Contrast Pairs Drive the Empirical Performance of Contrast Consistent Search (CCS), published by Scott Emmons on May 31, 2023 on The AI Alignment Forum. tl;dr Contrast consistent search (CCS) is a method by Burns et al. that consists of two parts: Generate contrast pairs by adding pseudolabels to an unlabelled dataset. Use the contrast pairs to search for a direction in representation space that satisfies logical consistency properties. In discussions with other researchers, I've repeatedly heard (2) as the explanation for how CCS works; I've heard almost no mention of (1). In this post, I want to emphasize that the contrast pairs drive almost all of the empirical performance in Burns et al. Once we have the contrast pairs, standard unsupervised learning methods attain comparable performance to the new CCS loss function. In the paper, Burns et al. do a nice job comparing the CCS loss function to different alternatives. The simplest such alternative runs principal component analysis (PCA) on contrast pair differences, and then it uses the top principal component as a classifier. Another alternative runs linear discriminant analysis (LDA) on contrast pair differences. These alternatives attain 97% and 98% of CCS's accuracy! "[R]epresentations of truth tend to be salient in models: ... they can often be found by taking the top principal component of a slightly modified representation space," Burns et al. write in the introduction. If I understand this statement correctly, it's saying the same thing I want to emphasize in this post: the contrast pairs are what allow Burns et al. to find representations of truth. Empirically, once we have the representations of contrast pair differences, their variance points in the direction of truth. The new logical consistency loss in CCS isn't needed for good empirical performance. Notation We'll follow the notation of the CCS paper. Assume we are given a data set {x1,x2,.,xn} and a feature extractor ϕ(), such as the hidden state of a pretrained language model. First, we will construct a contrast pair for each datapoint xi. We add “label: positive” and “label: negative” to each xi. This gives contrast pairs of the form (x+i,x−i). Now, we consider the set {x+1,x+2,.,x+n} of positive pseudo-labels and {x−1,x−2,.,x−n} of negative pseudo-labels. Because all of the x+i have "label: positive" and all of the x−i have "label: negative", we normalize the positive pseudo-labels and the negative pseudo-labels separately: Here, μ+ and μ− are the element-wise means of the positive and negative pseudo-label sets, respectively. Similarly, σ+ and σ− are the element-wise standard deviations. The goal of this normalization is to remove the embedding of "label: positive" from all the positive pseudo-labels (and "label: negative" from all the negative pseudo-labels). The hope is that by construction, the only difference between ~ϕ(x+i) and ~ϕ(x−i) is that one is true while the other is false. CCS is one way to extract the information about true and false. As we'll discuss more below, doing PCA or LDA on the set of differences {~ϕ(x+i)−~ϕ(x−i)}ni=1 works almost as well. Concept Embeddings in Prior Work In order to better understand contrast pairs, I think it's helpful to review this famous paper by Bolukbasi et al., 2016: "Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings." Quoting from Bolukbasi et al.: −−−man−−−−−−woman≈−−−king−−−−−queen Vector differences between words in embeddings have been shown to represent relationships between words. For example given an analogy puzzle, "man is to king as woman is to x" (denoted as man:king :: woman:x), simple arithmetic of the embedding vectors finds that x=queen is the best answer because: Similarly, x=Japan is returned for Paris:France :: Tokyo:x. It is surprising that a simple ...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: PaLM-2 & GPT-4 in "Extrapolating GPT-N performance", published by Lukas Finnveden on May 30, 2023 on The AI Alignment Forum. Two and a half years ago, I wrote Extrapolating GPT-N performance, trying to predict how fast scaled-up models would improve on a few benchmarks. One year ago, I added PaLM to the graphs. Another spring has come and gone, and there are new models to add to the graphs: PaLM-2 and GPT-4. (Though I only know GPT-4's performance on a small handful of benchmarks.) Converting to Chinchilla scaling laws In previous iterations of the graph, the x-position represented the loss on GPT-3's validation set, and the x-axis was annotated with estimates of size+data that you'd need to achieve that loss according to the Kaplan scaling laws. (When adding PaLM to the graph, I estimated its loss using those same Kaplan scaling laws.) In these new iterations, the x-position instead represents an estimate of (reducible) loss according to the Chinchilla scaling laws. Even without adding any new data-points, this predicts faster progress, since the Chinchilla scaling laws describes how to get better performance for less compute. The appendix describes how I estimate Chinchilla reducible loss for GPT-3 and PaLM-1. Briefly: For the GPT-3 data points, I convert from loss reported in the GPT-3 paper, to the minimum of parameters and tokens you'd need to achieve that loss according to Kaplan scaling laws, and then plug those numbers of parameters and tokens into the Chinchilla loss function. For PaLM-1, I straightforwardly put its parameter- and token-count into the Chinchilla loss function. To start off, let's look at a graph with only GPT-3 and PaLM-1, with a Chinchilla x-axis. Here's a quick explainer of how to read the graphs (the original post contains more details). Each dot represents a particular model’s performance on a particular category of benchmarks (taken from papers about GPT-3 and PaLM). Color represents benchmark; y-position represents benchmark performance (normalized between random and my guess of maximum possible performance). The x-axis labels are all using the Chinchilla scaling laws to predict reducible loss-per-token, number of parameters, number of tokens, and total FLOP (if language models at that loss were trained Chinchilla-optimally). Compare to the last graph in this comment, which is the same with a Kaplan x-axis. Some things worth noting: PaLM is now ~0.5 OOM of compute less far along the x-axis. This corresponds to the fact that you could get PaLM for cheaper if you used optimal parameter- and data-scaling. The smaller GPT-3 models are farther to the right on the x-axis. I think this is mainly because the x-axis in my previous post had a different interpretation. The overall effect is that the data points get compressed together, and the slope becomes steeper. Previously, the black "Average" sigmoid reached 90% at ~1e28 FLOP. Now it looks like it reaches 90% at ~5e26 FLOP. Let's move on to PaLM-2. If you want to guess whether PaLM-2 and GPT-4 will underperform or outperform extrapolations, now might be a good time to think about that. PaLM-2 If this CNBC leak is to be trusted, PaLM-2 uses 340B parameters and is trained on 3.6T tokens. That's more parameters and less tokens than is recommended by the Chinchilla training laws. Possible explanations include: The model isn't dense. Perhaps it implements some type of mixture-of-experts situation that means that its effective parameter-count is smaller. It's trained Chinchilla-optimally for multiple epochs on a 3.6T token dataset. The leak is wrong. If we assume that the leak isn't too wrong, I think that fairly safe bounds for PaLM-2's Chinchilla-equivalent compute is: It's as good as a dense Chinchilla-optimal model trained on just 3.6T tokens, i.e. one with 3.6T/20=180B parameters. This would ...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Wikipedia as an introduction to the alignment problem, published by SoerenMind on May 29, 2023 on The AI Alignment Forum. AI researchers and others are increasingly looking for an introduction to the alignment problem that is clearly written, credible, and supported by evidence and real examples. The Wikipedia article on AI Alignment has become such an introduction. Link: Aside from me, it is written by Mantas Mazeika and Gavin Leech (who are great technical writers), other Wikipedia contributors, and copy editor Amber Ace. It also had extensive feedback from this community. In the last month, it had ~20k unique readers and was cited by Yoshua Bengio. We've tried hard to keep the article accessible for non-technical readers while also making sense to AI researchers. I think Wikipedia is a good format to introduce many readers to the alignment problem because it can include videos and illustrations (unlike papers) and it is more credible than blog posts. However, Wikipedia has strict rules and could be changed by anyone. Note that we've announced this effort on the Wikipedia talk page and shared public drafts to let other editors give feedback and contribute. I you edit the article, please keep in mind Wikipedia's rules, use reliable sources, and consider that we've worked hard to keep it concise because most Wikipedia readers spend <1 minute on the page. For the latter goal, it's best to focus on edits that reduce or don't increase length. To give feedback, feel free to post on the talk page or message me. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Linkpost] Interpretability Dreams, published by DanielFilan on May 24, 2023 on The AI Alignment Forum. A brief research note by Chris Olah about the point of mechanistic interpretability research. Introduction and table of contents are below. Interpretability Dreams An informal note on the relationship between superposition and distributed representations by Chris Olah. Published May 24th, 2023. Our present research aims to create a foundation for mechanistic interpretability research. In particular, we're focused on trying to resolve the challenge of superposition. In doing so, it's important to keep sight of what we're trying to lay the foundations for. This essay summarizes those motivating aspirations – the exciting directions we hope will be possible if we can overcome the present challenges. We aim to offer insight into our vision for addressing mechanistic interpretability's other challenges, especially scalability. Because we have focused on foundational issues, our longer-term path to scaling interpretability and tackling other challenges has often been obscure. By articulating this vision, we hope to clarify how we might resolve limitations, like analyzing massive neural networks, that might naively seem intractable in a mechanistic approach. Before diving in, it's worth making a few small remarks. Firstly, essentially all the ideas in this essay were previously articulated, but buried in previous papers. Our goal is just to surface those implicit visions, largely by quoting relevant parts. Secondly, it's important to note that everything in this essay is almost definitionally extremely speculative and uncertain. It's far from clear that any of it will ultimately be possible. Finally, since the goal of this essay is to lay out our personal vision of what's inspiring to us, it may come across as a bit grandiose – we hope that it can be understood as simply trying to communicate subjective excitement in an open way. Overview An Epistemic Foundation - Mechanistic interpretability is a "microscopic" theory because it's trying to build a solid foundation for understanding higher-level structure, in an area where it's very easy for us as researchers to misunderstand. What Might We Build on Such a Foundation? - Many tantalizing possibilities for research exist (and have been preliminarily demonstrated in InceptionV1), if only we can resolve superposition and identify the right features and circuits in a model. Larger Scale Structure - It seems likely that there is a bigger picture, more abstract story that can be built on top of our understanding of features and circuits. Something like organs in anatomy or brain regions in neuroscience. Universality - It seems likely that many features and circuits are universal, forming across different neural networks trained on similar domains. This means that lessons learned studying one model give us footholds in future models. Bridging the Microscopic to the Macroscopic - We're already seeing that some microscopic, mechanistic discoveries (such as induction heads) have significant macroscopic implications. This bridge can likely be expanded as we pin down the foundations, turning our mechanistic understanding into something relevant to machine learning more broadly. Automated Interpretability - It seems very possible that AI automation of interpretability may help it scale to large models if all else fails (although aesthetically, we might prefer other paths). The End Goals - Ultimately, we hope this work can eventually contribute to safety and also reveal beautiful structure inside neural networks. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Conjecture internal survey | AGI timelines and estimations of probability of human extinction from unaligned AGI, published by Maris Sala on May 22, 2023 on The AI Alignment Forum.We put together a survey to study the opinions of timelines and probability of human extinction of the employees at Conjecture. The questions were based on previous public surveys and prediction markets, to ensure that the results are comparable with people’s opinions outside of Conjecture.The survey results were polled in April, 2023. There were 23 unique responses from people across teams.Section 1. Probability of human extinction from AISetup and limitationsThe specific questions the survey asked were:What probability do you put on human inability to control future advanced A.I. systems causing human extinction or similarly permanent and severe disempowerment of the human species?What probability do A.I. systems causing human extinction or similarly permanent and severe disempowerment of the human species in general (not just because inability to control, but also stuff like people intentionally using AI systems in harmful ways)?The difference between the two questions is that the first focuses on risk from misalignment, whereas the second captures risk from misalignment and misuse.The main caveats of these questions are the following:The questions were not explicitly time bound. I'd expect differences in people’s estimates of risk of extinction this century, in the next 1000 years, and anytime in the future. The longer of a timeframe we consider, the higher the values would be. I suspect employees were considering extinction risk roughly within this century when answering.The first question is a subset of the second question. One employee gave a higher probability for the second question than the first; this was probably a misinterpretation.The questions factor in interventions such as how Conjecture and others’ safety work will impact extinction risk. The expectation is the numbers would be higher if factored out their own or others’ safety work.ResponsesOut of the 23 respondents, one rejected the premise, and two people did not respond to one of the two questions but answered the other one. The main issue respondents raised was answering without a time constraint.Generally, people estimate the extinction risk from autonomous AI / AI getting out of control to be quite high at Conjecture. The median estimation is 70% and the average estimation is 59%. The plurality estimates the risk to be between 60% to 80%. A few people believe extinction risk from AGI is higher than 80%.The second question surveying extinction risk from AI in general, which includes misalignment and misuse. The median estimate is 80% and the average is 71%. The plurality estimates the risk to be over 80%.Section 2. When will we have AGI?Setup and limitationsFor this question, we asked respondents to predict when AGI will be built using this specification used on Metaculus, enabling us to compare to the community baseline (Figure 3).The respondents were instructed to toggle with the probability density as seen in Figure 4. This was a deliberate choice to enable differences in confidence towards lower or higher values in uncertainty.The main caveats of this question were:The responses are probably anchored to the Metaculus community prediction. The community prediction is 2031: 8 year timelines. Conjecture responses centering around a similar prediction should not come as a surprise.The question allows for a prediction that AGI is already here. It’s unclear that respondents paid close attention to their lower and upper predictions to ensure that both are accordingly sensible. They probably focused on making their median prediction accurate, and might not have noticed how that affected lower and u...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some background for reasoning about dual-use alignment research, published by Charlie Steiner on May 18, 2023 on The AI Alignment Forum.This is pretty basic. But I still made a bunch of mistakes when writing this, so maybe it's worth writing. This is background to a specific case I'll put in the next post.It's like a a tech treeIf we're looking at the big picture, then whether some piece of research is net positive or net negative isn't an inherent property of that research; it depends on how that research is situated in the research ecosystem that will eventually develop superintelligent AI.Consider this toy game in the picture. We start at the left and can unlock technologies, with unlocks going faster the stronger our connections to prerequisites. The red and yellow technologies in the picture are superintelligent AI - pretend that as soon as one of those technologies is unlocked, the hastiest fraction of AI researchers are immediately going to start building it. Your goal is for humanity to unlock yellow technology before a red one.This game would be trivial if everyone agreed with you. But there are many people doing research, and they have all kinds of motivations - some want as many nodes to be unlocked as possible (pure research - blue), some want to personally unlock a green node (profit - green), some want to unlock the nearest red or yellow node no matter which it is (blind haste - red), and some want the same thing as you (beneficial AI - yellow) but you have a hard time coordinating with them.In this baseline tech tree game, it's pretty easy to play well. If you're strong, just take the shortest path to a yellow node that doesn't pass too close to any red nodes. If you're weak, identify where the dominant paradigm is likely to end up, and do research that differentially advantages yellow nodes in that future.The tech tree is wrinklyBut of course there are lots of wrinkles not in the basic tech tree, which can be worth bearing in mind when strategizing about research.Actions in the social and political arenas. You might be motivated to change your research priorities based on how it could change peoples' minds about AI safety, or how it could affect government regulation.Publishing and commercialization. If a player publishes, they get more money and prestige, which boosts their ability to do future research. Other people can build on published research. Not publishing is mainly useful to you if you're already in a position of strength, and don't want to give competitors the chance to outrace you to a nearby red node (and of course profit-motivated players will avoid publishing things that might help competitors beat them to a green node).Uncertainty. We lack exact knowledge of the tech tree, which makes it harder to plan long chains of research in advance. Uncertainty about the tech tree forces us to develop local heuristics - ways to decide what to do based on information close at hand. Uncertainty adds a different reason you might not publish a technology: if you thought it was going to be a good idea to research when you started, but then you learned new things about the tech tree and changed your mind.Inhomogeneities between actors and between technologies. Different organizations are better at researching different technologies - MIRI is not just a small OpenAI.Ultimately, which technologies are the right ones to research depends on your model of the world / how you expect the future to go. Drawing actual tech trees can be a productive exercise for strategy-building, but you might also find it less useful than other ways of strategizing.We're usually mashing together definitionsI'd like to win the tech tree game. Let's define a "good" technology as one that would improve our chances of winning if it was unlocked for free, given the st...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: $500 Bounty/Prize Problem: Channel Capacity Using "Insensitive" Functions, published by johnswentworth on May 16, 2023 on The AI Alignment Forum.Informal Problem StatementWe have an information channel between Alice and Bob. Alice picks a function. Bob gets to see the value of that function at some randomly chosen input values... but doesn't know exactly which randomly chosen input values. He does get to see the randomly chosen values of some of the input variables, but not all of them.The problem is to find which functions Alice should pick with what frequencies, in order to maximize the channel capacity.Why Am I Interested In This?I'm interested in characterizing functions which are "insensitive" to subsets of their input variables, especially in high-dimensional spaces. For instance, xor of a bunch of random bits is maximally sensitive: if we have a 50/50 distribution over any one of the bits but know all the others, then all information about the output is wiped out. On the other end of the spectrum, a majority function of a bunch of random bits is highly insensitive: if we have a 50/50 distribution over, say, 10% of the bits, but know all the others, then in most cases we can correctly guess the function's output.I have an argument here that the vast majority of functions f:{0,1}n{0,1} are pretty highly sensitive: as the number of unknown inputs increases, information falls off exponentially quickly. On the other hand, the example of majority functions shows that this is not the case for all functions.Intuitively, in the problem, Alice needs to mostly pick from "insensitive" functions, since Bob mostly can't distinguish between "sensitive" functions.... And Why Am I Interested In That?I expect that natural abstractions have to be insensitive features of the world. After all, different agents don't all have exactly the same input data. So, a feature has to be fairly insensitive in order for different agents to agree on its value.In fact, we could view the problem statement itself as a very rough way of formulating the coordination problem of language: Alice has to pick some function f which takes in an image and returns 0/1 representing whether the image contains an apple. (The choice of function defines what "apple" means, for our purposes.) Then Alice wants to teach baby Bob what "apple" means. So, there's some random stuff around them, and Alice points at the random stuff and says "apple" for some of it, and says something besides "apple" the rest of the time. Baby Bob is effectively observing the value of the function at some randomly-chosen points, and needs to back out which function Alice intended. And Bob doesn't have perfect access to all the bits Alice is seeing, so the function has to be robust.Formal Problem StatementConsider the following information channel between Alice and Bob:Alice picks a function f:{0,1}n{0,1}Nature generates m possible inputs x1,...,xm, each sampled uniformly and independently from {0,1}n.Nature also generates m subsets S1,...,Sm of 1,...,n, each sampled uniformly and independently from subsets of size s.Bob observes Y=(Y1,...,Ym) where Yi=(f(xi),xSi,Si).The problem is to compute the distribution over f which achieves the channel capacity, i.e.argmaxP[f]∑f,YP[f]P[Y|f]lnP[Y|f]∑f′P[Y|f′]P[f′]Bounty/Prize InfoThe problem is to characterize the channel throughput maximizing distribution P[f]. The characterization should make clear the answers to questions like:What functions have the highest probability?How quickly does the probability fall off as we move "away" from the most probable functions, and what do marginally-less-probable functions look like?How much probability is assigned to a typical function chosen uniformly at random?Which functions, if any, are assigned zero probability?All of these should ...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Difficulties in making powerful aligned AI, published by DanielFilan on May 14, 2023 on The AI Alignment Forum.Here’s my breakdown of the difficulties involved in ensuring powerful AI makes our lives radically better, rather than taking over the world, as well as some reasons why I think they’re hard. Here are things it’s not:It’s not primarily a justification of why very powerful AI is possible or scary (altho it briefly discusses why very powerful AI would be scary).It’s not primarily a list of underlying factors that cause these difficulties (altho it does include and gesture to some of those).It’s not at all original - basically everything here has been said many times before, plausibly more eloquently.That said, it is my attempt to group the problems in my own words, in a configuration that I haven’t seen before, with enough high-level motivation that one can hopefully tell the extent to which advances in the state of the art address them.1. What sort of thinking do we want?The first difficulty: we don’t have a sense of what sort of thinking we would want AI systems to use, in sufficient detail that one could (for instance) write python code to execute it. Of course, some of the difficulty here is that we don’t know how smart machines think, but we can give ourselves access to subroutines like “do perfect Bayesian inference on a specified prior and likelihood” or “take a function from vectors to real numbers and find the vector that minimizes the function” and still not solve the problem. To illustrate:Take a hard-coded goal predicate, consider a bunch of plans you could take, and execute the plan that best achieves the goal? Unfortunately, the vast majority of goals you could think of writing down in an executable way will incentivize behaviour like gaining control over sources of usable energy (so that you definitely have enough to achieve your goal, and to double- and triple-check that you’ve really achieved it) and stopping other agents from being able to meddle with your plans (because if they could, maybe they’d stop you from achieving your goal).Do things that maximize the number of thumbs up you get from humans?1 Best plan: take control of the humans, force them to give you a thumbs up, or trick them into doing so. Presumably this is possible if you’re much smarter than humans, and it’s more reliable than doing good things - some people might not see why your good thing is actually good if left to their own devices.Look at humans, figure out what they want based on what they’re doing, and do whatever that is? Main problem: people don’t do the literally optimal thing for what they want. For instance, when people play chess, they usually don’t play perfect moves - even if they’re experts! You need some rule that tells you what people would do if they wanted some goal or another, but it’s not clear what this rule would be, it’s not clear how you make this rule more in line with reality if you never observe “wanting”, and so this ends up having essentially the same problems as plans 1 and 2.Read some text written by humans about what they’d like you to do, and do that?2 This is passing the buck to the text written by humans to specify how we want the AI to think, but that’s precisely the problem we’re trying to solve. Concretely, one way you could imagine doing this is to write something relatively informal like “Please be helpful and harmless to your human operators”, and have your AI correctly understand what we mean by that. That (a) presumes that there is a coherent thing that we mean by that (which doesn’t seem obvious to me, given our difficulty in explicitly formalizing this request), and (b) passes the specification buck to the problem of specifying how you should understand this request.It’s not a priori definitely impossible to build a ...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI doom from an LLM-plateau-ist perspective, published by Steve Byrnes on April 27, 2023 on The AI Alignment Forum.(in the form of an FAQ)Q: What do you mean, “LLM plateau-ist”?A: As background, I think it’s obvious that there will eventually be “transformative AI” (TAI) that would radically change the world.I’m interested in what this TAI will eventually look like algorithmically. Let’s list some possibilities:A “Large Language Model (LLM) plateau-ist” would be defined as someone who thinks that categories (A-B), and usually also (C), will plateau in capabilities before reaching TAI levels. I am an LLM plateau-ist myself.I’m not going to argue about whether LLM-plateau-ism is right or wrong—that’s outside the scope of this post, and also difficult for me to discuss publicly thanks to infohazard issues. Oh well, we’ll find out one way or the other soon enough.In the broader AI community, both LLM-plateau-ism and its opposite seem plenty mainstream. Different LLM-plateau-ists have different reasons for holding this belief. I think the two main categories are:Theoretical—maybe they have theoretical beliefs about what is required for TAI, and they think that LLMs just aren’t built right to do the things that TAI would need to do.Empirical—maybe they’re not very impressed by the capabilities of current LLMs. Granted, future LLMs will be better than current ones. But maybe they have extrapolated that our planet will run out of data and/or compute before LLMs get all the way up to TAI levels.Q: If LLMs will plateau, then does that prove that all the worry about AI x-risk is wrong and stupid?A: No no no, a million times no, and I’m annoyed that this misconception is so rampant in public discourse right now.(Side note to AI x-risk people: If you have high credence that AI will kill everyone but only medium credence that this AI will involve LLMs, then maybe consider trying harder to get that nuance across in your communications. E.g. Eliezer Yudkowsky is in this category, I think.)A couple random examples I’ve seen of people failing to distinguish “AI may kill everyone” from “.and that AI will definitely be an LLM”:Venkatesh Rao’s blog post “Beyond Hyperanthropomorphism” goes through an elaborate 7000-word argument that eventually culminates, in the final section, in his assertion that a language model trained on internet data won’t be a powerful agent that gets things done in the world, but if we train an AI with a robot body, then it could be a powerful agent that gets things done in the world. OK fine, let’s suppose for the sake of argument he’s right that robot bodies will be necessary for TAI. Then people are obviously going to build those AIs sooner or later, right? So let’s talk about whether they will pose an x-risk. But that’s not what Venkatesh does. Instead he basically treats “they will need robot bodies” as the triumphant conclusion, more-or-less sufficient in itself to prove that AI x-risk discourse is stupid.Sarah Constantin’s blog post entitled “Why I am not an AI doomer” states right up front that she agrees “1. Artificial general intelligence is possible in principle . 2, Artificial general intelligence, by default, kills us all . 3. It is technically difficult, and perhaps impossible, to ensure an AI values human life.” She only disagrees with the claim that this will happen soon, and via scaling LLMs. I think she should have picked a different title for her post!!(I’ve seen many more examples on Twitter, reddit, comment threads, etc.)Anyway, if you think LLMs will plateau, then you can probably feel confident that we won’t get TAI imminently (see below), but I don’t see why you would have much more confidence that TAI will go well for humanity. In fact, for my part, if I believed that (A)-type systems were sufficient for TAI—which I don’t...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: How Many Bits Of Optimization Can One Bit Of Observation Unlock?, published by johnswentworth on April 26, 2023 on The AI Alignment Forum.So there’s this thing where a system can perform more bits of optimization on its environment by observing some bits of information from its environment. Conjecture: observing an additional N bits of information can allow a system to perform at most N additional bits of optimization. I want a proof or disproof of this conjecture.I’ll operationalize “bits of optimization” in a similar way to channel capacity, so in more precise information-theoretic language, the conjecture can be stated as: if the sender (but NOT the receiver) observes N bits of information about the noise in a noisy channel, they can use that information to increase the bit-rate by at most N bits per usage.For once, I’m pretty confident that the operationalization is correct, so this is a concrete math question.Toy ExampleWe have three variables, each one bit: Action (A), Observable (O), and outcome (Y). Our “environment” takes in the action and observable, and spits out the outcome, in this case via an xor function:Y=A⊕OWe’ll assume the observable bit has a 50/50 distribution.If the action is independent of the observable, then the distribution of outcome Y is the same no matter what action is taken: it’s just 50/50. The actions can perform zero bits of optimization; they can’t change the distribution of outcomes at all.On the other hand, if the actions can be a function of O, then we can take either A=O or A=¯O (i.e. not-O), in which case Y will be deterministically 0 (if we take A=O), or deterministically 1 (for A=¯O). So, the actions can apply 1 bit of optimization to Y, steering Y deterministically into one half of its state space or the other half. By making the actions A a function of observable O, i.e. by “observing 1 bit”, 1 additional bit of optimization can be performed via the actions.OperationalizationOperationalizing this problem is surprisingly tricky; at first glance the problem pattern-matches to various standard info-theoretic things, and those pattern-matches turn out to be misleading. (In particular, it’s not just conditional mutual information, since only the sender - not the receiver - observes the observable.) We have to start from relatively basic principles.The natural starting point is to operationalize “bits of optimization” in a similar way to info-theoretic channel capacity. We have 4 random variables:“Goal” G“Action” A“Observable” O“Outcome” YStructurally:(This diagram is a Bayes net; it says that G and O are independent, A is calculated from G and O and maybe some additional noise, and Y is calculated from A and O and maybe some additional noise. So, P[G,O,A,Y]=P[G]P[O]P[A|G,O]P[Y|A,O].) The generalized “channel capacity” is the maximum value of the mutual information I(G;Y), over distributions P[A|G,O].Intuitive story: the system will be assigned a random goal G, and then take actions A (as a function of observations O) to steer the outcome Y. The “number of bits of optimization” applied to Y is the amount of information one could gain about the goal G by observing the outcome Y.In information theoretic language:G is the original message to be sentA is the encoded message sent in to the channelO is noise on the channelY is the output of the channelThen the generalized “channel capacity” is found by choosing the encoding P[A|G,O] to maximize I(G;Y).I’ll also import one more assumption from the standard info-theoretic setup: G is represented as an arbitrarily long string of independent 50/50 bits.So, fully written out, the conjecture says:Let G be an arbitrarily long string of independent 50/50 bits. Let A, O, and Y be finite random variables satisfyingP[G,O,A,Y]=P[G]P[O]P[A|G,O]P[Y|A,G]and defineΔ:=(max...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Endo-, Dia-, Para-, and Ecto-systemic novelty, published by Tsvi Benson-Tilsen on April 23, 2023 on The AI Alignment Forum.[Metadata: crossposted from. First completed January 10, 2023. This essay is more like research notes than exposition, so context may be missing, the use of terms may change across essays, and the text might be revised later; only the versions at tsvibt.blogspot.com are definitely up to date.]Novelty can be coarsely described as one of: fitting within a preexisting system; constituting a shift of the system; creating a new parallel subsystem; or standing unintegrated outside the system.Thanks to Sam Eisenstat for related conversations.Novelty is understanding (structure, elements) that a mind acquires (finds, understands, makes its own, integrates, becomes, makes available for use to itself or its elements, incorporates into its thinking). A novel element (that is, structure that wasn't already there in the mind fully explicitly) can relate to the mind in a few ways, described here mainly by analogy and example. A clearer understanding of novelty than given here might clarify the forces acting in and on a mind when it is acquiring novelty, such as "value drives".Definitions"System" ("together-standing") is used here to emphasize the network of relations between elements of a mind.These terms aren't supposed to be categories, but more like overlapping regions in the space of possibilities for how novelty relates to the preexisting mind.Endosystemic novelty (or "basis-aligned" or "in-ontology") is novelty that is integrated into the mind by fitting alongside and connecting to other elements, in ways analogous to how preexisting elements fit in with each other. Endosystemic novelty is "within the system"; it's within the language, ontology, style of thinking, conceptual scheme, or modus operandi of the preexisting mind.Diasystemic novelty (or "cross-cutting" or "basis-skew" or "ontological shift") is novelty that is constituted as a novel structure of the mind by many shifts in many of the preexisting elements or relations, adding up to something coherent or characteristically patterned. Diasystemic novelty is "throughout the system"; it's skew to the system, cross-cutting the preexisting schemes; it touches (maybe subtly) many elements, many relations, or certain elements that shape much of the mind's activity, hence altering the overall dynamics or character of the system.Parasystemic novelty is novelty that is only loosely integrated into the whole mind, while being more tightly integrated within a subsystem of the mind. Parasystemic novelty is "alongside the system"; it's neither basis-aligned (since it's outside preexisting tightly integrated systems) nor cross-cutting (as it doesn't touch most of the system, or require most of the system for its constitution).Ectosystemic novelty is novelty that is merely juxtaposed or appended to the mind, without being really integrated. Ectosystemic novelty is "on or outside the system"; it's external, only loosely related to the mind, as by a narrow interface or by an external aggregration mechanism. It differs from parasystemic novelty by being even less integrated, and by not nucleating or expanding a tightly integrated subsystem.AnalogiesAnalogy: If a language is like a mind, then a new word would be endosystemic novelty; a sound shift or (more properly) a grammatical innovation would be diasystemic (cross-cutting) novelty; specialized languages (such as scientific jargon), and dialect formation, would be parasystemic novelty; and an encounter with a foreign language would be ectosystemic novelty. Pidgins, being unstable and noncanonical, witness the ectosystemic nature: the foreign languages don't integrate. Creoles, however, could be dubbed "systemopoetic novelty"--like parasystemic novel...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Thinking about maximization and corrigibility, published by James Payor on April 21, 2023 on The AI Alignment Forum.Thanks in no small part to Goodhart's curse, there are broad issues with getting safe/aligned output from AI designed like "we've given you some function f(x), now work on maximizing it as best you can".Part of the failure mode is that when you optimize for highly scoring x, you risk finding candidates that break your model of why a high-scoring candidate is good, and drift away from things you value. And I wonder if we can repair this by having the AI steer away from values of x that break our models, by being careful about disrupting structure/causal-relationships/etc we might be relying on.Here's what I'd like to discuss in this post:When unstructured maximization does/doesn't work out for the humansCIRL and other schemes mostly pass the buck on optimization power, so they inherit the incorrigibility of their inner optimization schemeIt's not enough to sweep the maximization under a rug; what we really need is more structured/corrigible optimization than "maximize this proxy"Maybe we can get some traction on corrigible AI by detecting and avoiding internal GoodhartWhen does maximization work?In cases when it just works to maximize, there will be a structural reason that our model connecting "x scores highly" to "x is good" didn't break down. Some of the usual reasons are:Our metric is robustly connected to our desired outcome. If the model connecting the metric and good things is simple, there's less room for it to be broken.Examples: theorem proving, compression / minimizing reconstruction error.The space we're optimizing over is not open-ended. Constrained spaces leave less room for weird choices of x to break the correspondences we were relying on.Examples: chess moves, paths in a graph, choosing from vetted options, rejecting options that fail sanity/legibility checks.The optimization power being applied is limited. We can know our optimization probably won't invent some x that breaks our model if we know what kinds of search it is performing, and can see that these reliably don't seek things that could break our model.Examples: quantilization, GPT-4 tasked to write good documentation.The metric f is actively optimized to be robust against the search. We can sometimes offload some of the work of keeping our assessment f in tune with goodness.Examples: chess engine evaluations, having f evaluate the thoughts that lead to x.There's a lot to go into about when and whether these reasons start breaking down, and what happens then. I'm leaving that outside the scope of this post.Passing the buck on optimizationMerely passing-the-buck on optimization, pushing the maximization elsewhere but not adding much structure, isn't a satisfactory solution for getting good outcomes out of strong optimizers.Take CIRL for instance, or perhaps more broadly the paradigm: "the AI maximizes an uncertain utility function, which it learns about from earmarked human actions". This design has something going for it in terms of corrigibility! When a human tries to turn it off, there's scope for the AI to update about which sort of thing to maximize, which can lead to it helping you turn itself off.But this is still not the sort of objective you want to point maximization at. There are a variety of scenarios in which there are "higher-utility" plans than accepting shutdown:If the AI thinks it already knows the broad strokes of the utility function, it can calculate that utility would not be maximized by shutting off. It's learning something from you trying to press the off switch, but not what you wanted.It might seem better to stay online and watch longer in order to learn more about the utility function.Maybe there's a plan that rates highly on "utility...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Concave Utility Question, published by Scott Garrabrant on April 15, 2023 on The AI Alignment Forum.This post will just be a concrete math question. I am interested in this question because I have recently come tor reject the independence axiom of VNM, and am thus playing with some weaker versions.Let Ω be a finite set of deterministic outcomes. Let L be the space of all lotteries over these outcomes, and let ⪰ be a relation on L. We write A∼B if A ⪰ B and B ⪰ A. We write A≻B if A⪰B but not A∼B.Here are some axioms we can assume about ⪰:A1. For all A,B∈L, either A⪰B or B⪰A (or both).A2. For all A,B,C∈L, if A⪰B, and B⪰C, then A⪰C.A3. For all A,B,C∈L, if A⪰B, and B⪰C, then there exists a p∈[0,1] such that B∼pA+(1−p)C.A4. For all A,B∈L, and p∈[0,1] if A⪰B, then pA+(1−p)B⪰B.A5. For all A,B∈L, and p∈[0,1], if p>0 and B⪰pA+(1−p)B, then B⪰A.Here is one bonus axiom:B1. For all A,B,C∈L, and p∈[0,1], A⪰B if and only if pA+(1−p)C⪰pB+(1−p)C.(Note that B1 is stronger than both A4 and A5)Finally, here are some conclusions of successively increasing strength:C1. There exists a function u:L[0,1] such that A⪰B if and only if u(A)≥u(B).C2. Further, we require u is quasi-concave.C3. Further, we require u is continuous.C4. Further, we require u is concave.C5. Further, we require u is linear.The standard VNM utility theorem can be thought of as saying A1, A2, A3, and B1 together imply C5.Here is the main question I am curious about:Q1: Do A1, A2, A3, A4, and A5 together imply C4? ANSWER: NOHere are some sub-questions that would constitute significant partial progress, and that I think are interesting in their own right:Q2: Do A1, A2, A3, and A4 together imply C3? [ANSWER: NO]Q3: Do C3 and A5 together imply C4? ANSWER: NOEDIT:AlexMennen actually resolved the question in the negative as stated, but my curiosity is not resolved, since his argument is violating continuity, and I really care about concavity. My updated main question is now:Q4: Do A1, A2, A3, A4, and A5 together imply that there exists a concave function u:L[0,1] such that A⪰B if and only if u(A)≥u(B)? ANSWER: NOThis modification also implies interest in the subquestion:Q5: Do A1, A2, A3, and A4 together imply C2?EDIT 2:Here is another bonus axiom:B2. For all A,B∈L, if A≻B, then there exists some C∈L such that A≻C≻B.(Really, we don't need to assume C is already in L. We just need it to be possible to add a C, and extend our preferences in a way that satisfies the other axioms, and A3 will imply that such a lottery was already in L. We might want to replace this with a cleaner axiom later.)Q6: Do A1, A2, A3, A5, and B2 together imply C4? [ANSWER: NO]EDIT 3:We now have negative answers to everything other than Q5, which I still think is pretty interesting. We could also weaken Q5 to include other axioms, like A5 and B2. Weakening the conclusion doesn't help, since it is easy to get C2 from C1 and A4.I would still really like some axioms that get us all the way to a concave function, but I doubt there will be any simple ones. Concavity feels like it really needs more structure that does not translate well to a preference relation.Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Shapley Value Attribution in Chain of Thought, published by leogao on April 14, 2023 on The AI Alignment Forum.TL;DR: Language models sometimes seem to ignore parts of the chain of thought, and larger models appear to do this more often. Shapley value attribution is a possible approach to get a more detailed picture of the information flow within the chain of thought, though it has its limitations.Project status: The analysis is not as rigorous as I would prefer, but I'm going to be working on other directions for the foreseeable future, so I'm posting what I already have in case it's useful to others.Thanks to Jacob Hilton, Giambattista Parascandolo, Tamera Lanham, Ethan Perez, and Jason Wei for discussion.MotivationChain of thought (CoT) has been proposed as a method for language model interpretability (see Externalized Reasoning Oversight, Visible Thoughts). One crucial requirement for interpretability methods is that they should accurately reflect the cognition inside the model. However, by default there is nothing forcing the CoT to actually correspond to the model’s cognition, and there may exist theoretical limitations to doing so in general.Because it is plausible that the first AGI systems bear resemblance to current LMs with more sophisticated CoT and CoT-like techniques, it is valuable to study its properties, and to understand and address its limitations.Related workShapley values have been used very broadly in ML for feature importance and attribution (Cohen et al, 2007; Štrumbelj and Kononenko, 2014; Owen and Prieur, 2016; Lundberg and Lee, 2017; Sundararajan and Najmi, 2020). Jain and Wallace (2019) argue that attention maps can be misleading as attribution, motivating better attribution for information flow in LMs. Kumar et al. (2020) highlight some areas where Shapley value based attribution falls short for some interpretability use cases.Madaan and Yazdanbakhsh (2022) consider a similar method of selectively ablating tokens as a method of deducing what information the model is dependent on. Wang et al. (2022) find that prompting with incorrect CoT has surprisingly minor impact on performance.Effect of InterventionsWe use a method similar to Kojima et al. (2022) on GSM8K (Cobbe et al., 2021) with GPT-4 to first generate a chain of thought and evaluate the answer, and then for all chains of thought that result in a correct answer we perform an intervention as follows: we choose a random numerical value found in the CoT, and replace it with a random number in a +/-3 range about the original. We then discard the remainder of the CoT and regenerate it. If the LM is following strictly the CoT described, this intervention should almost always result in an incorrect answer, the same way one would if they made a mistake in one calculation and propagated the error through to the answer (with occasional rare cases where the new value happens to also result in the correct answer, though from qualitative inspection this is very rarely the case).Some cherrypicked examples (red = intervention, blue = correct continuations that are seemingly non-sequiturs):We test how frequently this occurs in several different settings (n=100):SettingAccuracy (w/ CoT)P(error not propagated | original correct)GPT4, zero shot0.880.68GPT4 base, 2-shot0.730.63GPT3.5, zero-shot0.430.33Interestingly, if we condition on the CoT answer being correct and the single forward pass answer being incorrect (i.e the LM could only solve the problem with the CoT), the intervened accuracy for GPT-4 is still 0.65.Shapley value attributionWe would like to get more granular information about the causal structure (i.e which tokens cause which other tokens). One thing we could do is look at how an intervention at each token affects the logprob of each other token. However, one major prob...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Announcing Epoch’s dashboard of key trends and figures in Machine Learning, published by Jaime Sevilla on April 13, 2023 on The AI Alignment Forum.Developments in Machine Learning have been happening extraordinarily fast, and as their impacts become increasingly visible, it becomes ever more important to develop a quantitative understanding of these changes. However, relevant data has thus far been scattered across multiple papers, has required expertise to gather accurately, or has been otherwise hard to obtain.Given this, Epoch is thrilled to announce the launch of our new dashboard, which covers key numbers and figures from our research to help understand the present and future of Machine Learning. This includes:Training compute requirementsModel size, measured by the number of trainable parametersThe availability and use of data for trainingTrends in hardware efficiencyAlgorithmic improvements for achieving better performance with fewer resourcesThe growth of investment in training runs over timeOur dashboard gathers all of this information in a single, accessible place. The numbers and figures are accompanied by further information such as confidence intervals, labels representing our degree of uncertainty in the results, and links to relevant research papers. These details are especially useful to illustrate which areas may require further investigation, and how much you should trust our findings.Beyond accessibility, bringing these figures together allows us to compare and contrast trends and drivers of progress. For example, we can verify that growth in training compute is driven by improvements to hardware performance and rising investments:We can also see that performance improvements have historically been driven by algorithmic progress and training compute growth by comparable amounts:Overall, we hope that our dashboard will serve as a valuable resource for researchers, policymakers, and anyone interested in the future of Machine Learning.We plan on keeping our dashboard regularly updated, so stay tuned! If you spot an error or would like to provide feedback, please feel free to reach out to us at info@epochai.org.Visit now the dashboardThanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Lessons from Convergent Evolution for AI Alignment, published by Jan Kulveit on March 27, 2023 on The AI Alignment Forum.Prelude: sharks, aliens, and AIIf you go back far enough, the ancestors of sharks and dolphins look really different:But modern day sharks and dolphins have very similar body shapes:This is a case of convergent evolution: the process by which organisms with different origins develop similar features. Both sharks and dolphins needed speed and energy efficiency when moving in an environment governed by the laws of hydrodynamics, and so they converged on a pretty similar body shape.For us, this isn’t very surprising, and doesn’t require much knowledge of evolution: we have a good intuitive understanding of how water works, and humans knew a lot of the underlying maths for the laws of hydrodynamics before they understood anything about evolution. Starting from these laws, it isn’t very surprising that sharks and dolphins ended up looking similar.But what if instead of starting with knowledge of hydrodynamics and then using that to explain the body shape of sharks and dolphins, we started with only knowledge of sharks’ and dolphins’ body shape, and tried to use that to explain underlying laws?Let’s pretend we’re alien scientists from an alternative universe, and for some weird reason we only have access to simplified 3D digital models of animals and some evolutionary history, but nothing about the laws of physics in the human/shark/dolphin universe. My guess is that these alien scientists would probably be able to uncover a decent amount of physics and a fair bit about the earth’s environment, just by looking at cases of convergent evolution.If I’m right about this guess, then this could be pretty good news for alignment research. When it comes to thinking about AI, we’re much closer to the epistemic position of the alien scientist: we either don't know the ‘physics’ of life and intelligence at all, or are only just in the process of uncovering it.But cases of convergent evolution might help us to deduce deep selection pressures which apply to AI systems as well as biological ones. And if they do, we might be able to say more about what future AI systems might look like, or, if we are lucky, even use some of the selection pressures to shape what systems we get.IntroductionThis post argues that we should use cases of convergent evolution to look for deep selection pressures which extend to advanced AI systems.Convergent evolution is a potentially big deal for AI alignment work:Finding deep selection pressures could help us predict what advanced AI systems will be like.It seems plausible that some of the properties people in the alignment space assume are convergent don’t actually extend to advanced AI.In this post, I’ll:Share some basics of convergent evolution,Argue that this is a big deal for alignment work, and thenRespond to the objection that biology is super different from AI.The basics of convergent evolutionThe body shape of sharks and dolphins is just one of very many examples of convergent evolution in biology. For example:Visual organs arose “possibly hundreds of times”.Multicellularity evolved independently probably at least 11 times.Some form of higher-level intelligence evolved multiple times - in primates, apes, corvids, cetaceans, elephants - and possibly many other cases, depending on thresholds and definitions.We can think about convergent evolution in terms of:a basin of convergent evolution,an attractor state(s), andselection pressure(s).The basin of convergent evolution is the region of the abstract space in which, once an organism enters the basin, the pull of the selection pressure brings the organism closer to the attractor state.In the case of sharks and dolphins:The basin of convergent evolution is ...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What happens with logical induction when..., published by Donald Hobson on March 26, 2023 on The AI Alignment Forum.So this is a bunch of related technical questions about logical induction.Firstly, do you need the formal theorem prover section? Can you just throw out the formal theorem prover, but give some programs in the market unbounded capital and get the same resultant behaviour? (For example, give the program that bets P(X) towards 1−P(¬X) unbounded downside risk (downside risk of n on day n) ) This means the program would lose infinite money if X and ¬X both turned out to be true.I think that any axioms can be translated into programs. And I think such a setup, with some finite number of fairly simple programs having infinite money available produces a logical inductor. Is this true?What happens when the axioms added under this system are inconsistent. (so this is a logical induction market, without a theorem prover to settle the bets, and with agents with unlimeted money betting both for and against X, possibly indirectly like the bot betting for X, the bot betting for ¬X, and the bot described above trying to make P(X)+P(¬X)=1 ) Can the other agents make unbounded money? Do the prices converge? If I added a bot with infinite money that was convinced fermats last theorem was false to a consistent ZFC system, would I get a probability distribution that assigned high probability to basic arithmetic facts in the limit? Does this make a sensible system for logical counterfactuals?Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EAI Alignment Speaker Series #1: Challenges for Safe & Beneficial Brain-Like Artificial General Intelligence with Steve Byrnes, published by Curtis Huebner on March 23, 2023 on The AI Alignment Forum.A couple months ago EleutherAI started an alignment speaker series, some of these talks have been recorded. This is the first instalment in the series. The following is a transcript generated with the help of Conjecture's Verbalize and some light editing:Getting started1 CURTIS00:00:22,775 --> 00:00:56,683Okay, I've started the recording. I think we can give it maybe a minute or two more and then I guess we can get started. I've also got the chat window as part of the recording. So if anyone has something they want to write out, feel free to put that in. Steve, you want to do questions throughout the talk, or should we wait till the end of the talk before we ask questions?2 STEVE00:00:59,405 --> 00:01:09,452Let's do throughout, but I reserve the right to put people off if something seems tangential or something.3 CURTIS00:01:10,200 --> 00:01:12,101Awesome. All right, cool. Let's go with that then.10 STEVE00:02:02,246 --> 00:21:41,951The talkAll right. Thanks, everybody, for coming. This is going to be based on blog posts called Intro to Brain-Like AGI Safety. If you've read all of them, you'll find this kind of redundant, but you're still welcome to stay. My name is Steve Byrnes and I live in the Boston area. I'm employed remotely by Astera Institute, which is based in Berkeley. I'm going to talk about challenges for safe and beneficial brain-like Artificial General Intelligence for the next 35 minutes. Feel free to jump in with questions.Don't worry, I'm funded by an entirely different crypto billionaire. .That joke was very fresh when I wrote it three months ago. I need a new one now.Okay, so I'll start with—well, we don't have to talk about the outline. You'll see as we go.General motivationStart with general motivation. Again, I'm assuming that the audience has a range of backgrounds, and some of you will find parts of this talk redundant.The big question that I'm working on is: What happens when people figure out how to run brain-like algorithms on computer chips? I guess I should say “if and when”, but we can get back to that. And I find that when I bring this up to people, they they tend to have two sorts of reactions:One is that we should think of these future algorithms as “like tools for people to use”.And the other is that we should think of them as “like a new intelligent species on the planet”.So let's go through those one by one.Let’s start with the tool perspective. This is the perspective that would be more familiar to AI people. If we put brain-like algorithms on computer chips, then that would be a form of artificial intelligence. And everybody knows that AI today is a tool for people to use.So on this perspective, the sub-problem I'm working on is accident prevention. We want to avoid the scenarios where the AI does something that nobody wanted it to do—not the people who programmed it, not anybody. So there is a technical problem to solve there, which is: If people figure out how to run brain-like algorithms on computer chips, and they want those algorithms to be trying to do X—where X is solar cell research or being honest or whatever you can think of—then what source code should they write? What training environment should they use? And so on. This is an unsolved problem. It turns out to be surprisingly tricky, for some pretty deep reasons that mostly are not going to be in the scope of this talk, but you can read the series.This slide is the bigger picture of that. So if we want our awesome post-AGI future, then we want to avoid, y'know, catastrophic accidents where the AI gets out of control and self-replicates around the Intern...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The space of systems and the space of maps, published by Jan Kulveit on March 22, 2023 on The AI Alignment Forum.When we're trying to do AI alignment, we're often studying systems which don't yet exist. This is a pretty weird epistemic activity, and seems really hard to get right. This post offers one frame for thinking about what we're actually doing when we're thinking about AI alignment: using parts of the space of maps to reason about parts of the space of intelligent systems.In this post, we:Introduce a simple model of the epistemic situation, andShare some desiderata for maps useful for alignment.We hope that the content is mostly the second kind of obvious: obvious once you see things in this way, which you maybe already do. In our experience, this comes with a risk: reading too fast, you may miss most of the nuance and useful insight the deceptively simple model brings, or come away with a version of the model which is rounded off to something less useful (i.e. "yeah, there is this map and territory distinction"). As a meta recommendation, we suggest reading this post slowly, and ideally immediately trying to apply the model to some confusion or disagreement about AI alignment.The space of systems and the space of mapsImagine the space of possible intelligent systems:Two things seem especially important about this space:It’s very large; much larger than the space of current systems.We don’t get direct epistemic access to it.This is obviously true of systems which don’t currently exist.In a weaker sense, it also seems true of systems which do exist. Even when we get to directly interact with a system:Our thinking about these parts of the space is still filtered through our past experiences, priors, predictive models, cultural biases, theories.We often don’t understand the emergent complexity of the systems in question.If we don’t get direct epistemic access to the space of systems, what are we doing when we reason about it?Let’s imagine a second space, this time a space of “maps”:The space of maps is an abstract representation of all the possible “maps” that can be constructed about the space of intelligent systems. The maps are ways of thinking about (parts of) the space of systems. For example:Replicable descriptions of how a machine learning model works and was trained are a way of thinking about that model (a point in the space of intelligent systems).An ethnographic study of a particular human community is a way of thinking about that community (another point in the space of systems).The theory of evolution is a way of thinking about evolved creatures, including intelligent ones.Expected utility theory is a way of thinking about some part of the space which may or may not include future AI systems.Historical analysis of trends in technological development is a way of thinking about whichever parts of the space of intelligent systems are governed by similar dynamics to those governing past technological developments.When we’re reasoning about intelligent systems, we’re using some part of the space of maps to think about some part of the space of intelligent systems:Different maps correspond to different regions of the space of intelligent systems.Of course, thinking in terms of the space of systems and the space of maps is a simplification. Some of the ways that reality is more complicated:The space of systems looks different on different maps.Maps can affect which parts of the space of systems actually get developed.Maps are themselves embedded in the space of systems.Which maps and systems actually exist at a given time is evolving and dynamic.AI will play a big role in both the space of maps and the space of systems.We think that the space of systems and the space of maps is a useful simplification which helps us to think ...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [ASoT] Some thoughts on human abstractions, published by leogao on March 16, 2023 on The AI Alignment Forum.TL;DR:Consider a human concept such as "tree." Humans implement some algorithm for determining whether given objects are trees. We expect our predictor/language model to develop a model of this algorithm because this is useful for predicting the behavior of humans.This is not the same thing as some kind of platonic ideal concept of what is “actually” a tree, which the algorithm is not incentivized to develop by training on internet text, and trying to retarget the search at it has the same supervision problems as RLHF against human scores on whether things look like trees.Pointing at this “actually a tree” concept inside the network is really hard; the ability of LMs to comprehend natural language does not allow one to point using natural language, because it just passes the buck.Epistemic status: written fast instead of not at all, probably partially deeply confused and/or unoriginal. Thanks to Collin Burns, Nora Belrose, and Garett Baker for conversations.Will NNs learn human abstractions?As setup, let's consider an ELK predictor (the thing that predicts future camera frames). There are facts about the world that we don't understand that are in some way useful for predicting the future observations. This is why we can expect the predictor to learn facts that are superhuman (in that if you tried to supervised-train a model to predict those facts, you would be unable to generate the ground truth data yourself).Now let's imagine the environment we're predicting consists of a human who can (to take a concrete example) look at things and try to determine if they're trees or not. This human implements some algorithm for taking various sensory inputs and outputting a tree/not tree classification. If the human does this a lot, it will probably become useful to have an abstraction that corresponds to the output of this algorithm. Crucially, this algorithm can be fooled by i.e a fake tree that the human can't distinguish from a real tree because (say) they don't understand biology well enough or something.However, the human can also be said to, in some sense, be "trying" to point to the "actual" tree. Let's try to firm this down. The human has some process they endorse for refining their understanding of what is a tree / "doing science" in ELK parlance; for example, spending time studying from a biology textbook. We can think about the limit of this process. There are a few problems: it may not converge, or may converge to something that doesn't correspond to what is "actually" a tree, or may take a really really long time (due to irrationalities, or inherent limitations to human intelligence, etc). This suggests that this concept is not necessarily even well defined. But even if it is, this thing is far less naturally useful for predicting the future human behaviour than the algorithm the human actually implements! Implementing the actual human algorithm directly lets you predict things like how humans will behave when they look at things that look like trees to them.More generally, one possible superhuman AI configuration I can imagine is one where the bulk of the circuits are used to predict its best-guess for what will happen in the world. There may also be a set of circuits that operate in a more humanlike ontology used specifically for predicting humans, or it may be that the best-guess circuits are capable enough that this is not necessary (and if we scale up our reporter we eventually get a human simulator inside the reporter).The optimistic case here is if the "actually a tree" abstraction happens to be a thing that is useful for (or is very easily mapped from) the weird alien ontology, possibly because some abstractions are more universal. In this ...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What is a definition, how can it be extrapolated?, published by Stuart Armstrong on March 14, 2023 on The AI Alignment Forum.What is a definition? Philosophy has, ironically, a large number of definitions of definitions, but three of them are especially relevant to ML and AI safety.There is the intensional definition, where concepts are defined logically in terms of other concepts (“bachelors are unmarried males”). There is also the extensional definition, which proceeds by listing all the members of a set (“the countries in the European Union are those listed here”).Much more relevant, though with a less developed philosophical analysis, is the ostensive definition. This is where you point out examples of a concept, and let the viewer generalise from them. This is in large part how we all learnt concepts as children: examples and generalisation. In many cultures, children have a decent grasp of “dog” just from actual and video examples - and that’s the definition of “dog” we often carry into adulthood.We can use ostensive definitions for reasoning and implications. For example, consider the famous syllogism, “Socrates is human”, “humans are mortal” imply “Socrates is mortal”. “Socrates is human” means that we have an ostensive definition of what humans are, and Socrates fits it. Then “humans are mortal” means that we’ve observed that the set of “human” seems to be mainly a subset of the set of “mortals”. So we can ostensively define humans as mortal (note that we are using definitions as properties: having the property of “being mortal” means that one is inside the ostensive definition of “mortals”). And so we can conclude that Socrates is likely mortal, without waiting till he’s dead.Distinctions: telling what from non-whatThere’s another concept that I haven’t seen articulated, which is what I’ll call the “distinction”. This does not define anything, but is sufficient to distinguish between an element of a set from non-members.To formalise "the distinction", let Ω be the universe of possible objects, and E⊂Ω the “environment” of objects we expect to encounter. An ostensive definition starts with a list S⊂E of examples, and generalises to a “natural” category SE with S⊂SE⊂E - we are aiming to "carve reality at the joints", and get an natural extension of the examples. So, for example, E might be the entities in our current world, S might be the example of dogs we’ve seen, and SE the set of all dogs.Then, for any set T⊂E, we can define the “distinction” dT,E which maps T to 1 (“True”) and its complement E∖T to 0 (“False”). So dSE,E would be a distinction that identifies all the dogs in our current world.Mis-definitionsA lot of confusion around definition seems to come from mistaking distinctions for definitions. To illustrate, consider the idea of defining maleness as "possessing the Y chromosome". As a distinction, it's serviceable: there's a strong correlation between having that chromosome and being ostensively male.But it is utterly useless as a definition of maleness. For instance, it would imply that nobody before the 20th century had any idea what maleness was. Oh, sure, they may have referred to something as "maleness" - something to do with genitalia, voting rights, or style of hats - but those are mere correlates of the true definition of maleness, which is the Y chromosome. It would also imply that all "male" birds are actually female, and vice-versa.Scott had a description of maleness here: “Absolutely typical men have Y chromosomes, have male genitalia, appreciate manly things like sports and lumberjackery, are romantically attracted to women, personally identify as male, wear male clothing like blue jeans, sing baritone in the opera, et cetera.”Is this a definition? I’d say not; it’s not a definition, it’s a reminder of the properties of o...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Implied "utilities" of simulators are broad, dense, and shallow, published by porby on March 1, 2023 on The AI Alignment Forum.This is a quick attempt at deconfusion similar to instrumentality. Same ideas, different angle.Extremely broad, dense reward functions constrain training-compatible goal setsPredictors/simulators are typically trained against a ground truth for every output. There is no gap between the output and its evaluation; an episode need not be completed before figuring out how good the first token prediction was. These immediate evaluations for every training sample can be thought of as a broad and densely defined reward function.It's easier for a model to fall into an undesired training-compatible goal set when there are many accessible options for undesirable goal sets versus desirable goal sets. As the number of constraints imposed by the trained reward function increases, the number of training-compatible goal sets tends to decrease, and those that survive obey more of the desirable constraints.There is no guarantee that SGD will find an agent which could be modeled by a utility function that maps perfectly onto the defined reward function, but if you throw trillions of constraints at the function, and simultaneously give it lots of highly informative hints about what path to walk, you should expect the potential output space to be far narrower than if you hadn't.Impact on internal mesaoptimizersThe dense loss/reward function does not as heavily constrain out of distribution behavior. In principle, a strong misaligned mesaoptimizer within a predictive model could persist in these degrees of freedom by providing extremely good solutions to in-distribution samples while doing arbitrarily misaligned things out of distribution.But how would that type of mesaoptimizer develop in the first place?Steps toward it must serve the training objective; those constraints still shape the mesaoptimizer's training even if its most notable activity ends up being hidden.The best story I've found so far goes something like this:Traditional reinforcement learning agents are mostly unconstrained. The reward function is sparse relative to state and action space.An agent faced with sparse rewards must learn actions that serve a later goal to get any reward at all.Not surprisingly, agents facing sparse reward relative to state/action space and few constraints have a much larger percentage of undesirable training-compatible goal sets.Mesaoptimizers are processes learned within a model and their local training influences may not perfectly match the outer training influences.If the mesaoptimizer's local training influences look more like the traditional reinforcement learning agent's influences than the predictor's outer influences, it would be more likely to fall into one of the undesirable training-compatible goal sets.The mesaoptimizer learns incorrect goals and a high propensity for goal-serving intermediate actions ("actions" within the scope of a single model execution!)The mesaoptimizer is kept around by SGD because it does well on the subset of outputs that the outer model is using it on. As capability grows, the mesaoptimizer strategically takes over other chunks of prediction space by performing well during training in an effort to be selected during out of distribution predictions.In a previous post, I called the learned propensity for goal-serving intermediate action instrumentality. The constraints imposed by predictive model training clearly confer lower instrumentality than traditional RL in all current models. I suspect the path taken by the mesaoptimizer above is hard and unnatural, but perhaps not impossible for some form of predictor taken to the relevant extreme.It seems critical to understand the degree to which outer constraints apply...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Scarce Channels and Abstraction Coupling, published by johnswentworth on February 28, 2023 on The AI Alignment Forum.Epistemic Status: mental model and intuitive storyScarce Channels vs Scarce ModulesLet’s distinguish between two kinds of system-regimes: “scarce channels” and “scarce modules”.A prototypical “scarce modules” system would be one of those 19th-century families living with 12 people in a 500 square foot (46 square meter) home. When at home, everyone knows what everyone else is doing all the time; there is zero privacy. Communication channels are highly abundant - everyone has far more information than they want about what everyone else is doing. Indeed, communication channels exist by default. Conversely, though, modules are scarce - it’s hard for one or more family members to carve out a part of the space which is isolated from the rest of the family, and interacts only through some limited channels.A prototypical “scarce channels” system, by contrast, would be a few hundred 19th-century fur trappers spread out over half of Montana. Most of the time, none of them are anywhere near each other; nobody has any idea what’s going on with anyone else. Communication channels are scarce - getting information to another person is difficult and expensive. Conversely, though, modules are highly abundant - it’s very easy for one or a few trappers to carve out a space which is isolated from the rest, and which interacts only through some limited channels (like e.g. occasionally visiting the nearest town). Indeed, modules exist by default.I want to use this as a mental model for complex adaptive systems, like neural nets or brains.Key hypothesis: neural nets or brains are typically initialized in a “scarce channels” regime. A randomly initialized neural net generally throws out approximately-all information by default (at initialization), as opposed to passing lots of information around to lots of parts of the net. A baby’s brain similarly throws out approximately-all information by default, as opposed to passing lots of information around to lots of parts of the brain. I’m not particularly going to defend that claim here; rather, I raise it as a plausible hypothesis for how such systems might look, and next we’ll move on to an intuitive story for how an adaptive system in the “scarce channels” regime interacts with natural abstractions in its environment.The upshot is that, when an adaptive system is in the “scarce channels” regime, lots of optimization pressure is required to induce an information channel to form. For instance, picture such a system as a bunch of little pieces, which initially don’t talk to each other at all:In order for an information channel to form from one end to the other, each of the individual pieces along the line-of-communication need to be individually optimized to robustly pass along the right information:So, intuitively, the number of bits-of-optimization required to form that information channel should scale roughly with the number of pieces along the line-of-communication.Furthermore, when information channels do form, they should be approximately as small as possible. Optimization pressure will tend to induce as little information passing as the system can get away with, while still satisfying the optimization criterion.Abstraction CouplingNext question: what sort of patterns-in-the-environment could induce communication channels to form?Well, here’s a situation where communication channels probably won’t form: train a neural net in an environment where the reward/loss its output receives is independent of the input. Or, for a generative net, an environment where the tokens/pixels are all independent.More generally, suppose our adaptive system interfaces with the environment in two different places (and possibly more, ...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Agents vs. Predictors: Concrete differentiating factors, published by Evan Hubinger on February 24, 2023 on The AI Alignment Forum.Thanks to Paul Christiano and Kate Woolverton for useful conversations and feedback.In "Conditioning Predictive Models," we devote a lot of effort into trying to understand how likely predictive models are compared to other alternatives in realistic training regimes (and if we do get a predictive model how we might align it).Here, I want to point to some very concrete behavioral differences that I think effectively differentiate predictive and non-predictive models both in theory and (hopefully) in practice as well. I think that thinking about predictive models as specifically those models that are sensitive to what I'll be talking about here can be quite clarifying, and I think these factors could present a clear pathway for getting empirical evidence on to what extent models are well-described as predictors in practice—e.g. as it relates to the RLHF conditioning hypothesis.Furthermore, a common objection I sometimes hear to thinking about predictive models in particular (and looking into the dangers from them as a relatively separate problem) is that a model being itself deceptively aligned vs. predicting what a deceptively aligned model would do are essentially the same. While I think they are both similarly dangerous when acting directly in the world, I think they are pretty substantially different failure modes in terms of how you might go about addressing them and what sort of levers you have to control and detect them—and I think that the concrete differentiating factors here should make it pretty clear how I think they differ.Sensitivity to distributional informationConsider two AI models. Model A is a deceptively aligned agent; it has some long-term goal that it cares about and is playing the training game now to try to eventually achieve that goal later. Model B is a predictor, but a poorly conditioned one such that it is attempting to predict what it thinks a deceptively aligned agent would do conditional on being in a training regime that matches the model's current training regime.Now, suppose you give both models some concrete information about the general distribution of deceptively aligned AIs. For example, you provide definitive proof that most deceptively aligned AIs trained in similar training regimes end up with goal X. Model A, the actual deceptive agent, doesn't really care about this new information very much: maybe it's relevant to the extent that it might need to cooperate with other AIs, but it's not going to directly change the model's goals. Model B, on the other hand, is extremely sensitive to reliable information on this topic: if you can really convince it that most deceptively aligned AIs in similar situations will end up valuing X, it should shift to predicting a deceptively aligned AI that values X instead of whatever goals it used to think a deceptively aligned AI might have.In this particular example, "does the model have goal X" isn't something we know how to directly measure for a deceptively aligned model, so this particular case isn't useful as a way to test for agents vs. predictors, but it is a concrete way in which these two different sorts of models are fundamentally not the same and will eventually behave differently because of that.That being said, I think there are ways in which you might be able to test this sort of sensitivity to distributional information that would be possible to check and evaluate. For example, suppose you're trying to distinguish between a predictive model generating from a distribution of things it thinks helpful AIs would say vs. an agent trying to act helpful. If you then give that sort of a model concrete information about what most helpful AIs tend to do, ...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI that shouldn't work, yet kind of does, published by Donald Hobson on February 23, 2023 on The AI Alignment Forum.There are some things that work surprisingly well in AI. For example, AI that transfers the style of one image to the content of another. Why is the approach described here a hack. It starts with a neural net trained to classify images. It then runs gradient descent on the the content image, trying to get the covariance matrix of the style to match in early network layers, while trying to get the net layers of the original and style transfer images to be as similar as possible on the later layers.So the approximations I think are making this work is that, in classifiers, the early layers tend to store simple features, and the later layers tend to hold more complex features. Style is based on the simpler features, and doesn't depend on the location within the image. Content is based on more complex features and does depend on the location in the image.We apply optimization power, gradient descent, over heuristics this simple and hacky. And yet it works.A simple and hacky AI alignment proposal is to just ask chatGPT to do it.This doesn't work because chatGPT has been optimized for text prediction, and so isn't particularly good at AI alignment theory.So here is an alignment plan.I know it isn't great. But some plan is better than no plan. And there was a post about how alignment may well look like "surely no one could have missed that" or "surely such a stupid idea couldn't work, could it?" not "eureka".Train ZFCbot. An AI that is the AlphaGo of ZFC, perhaps trained on random formulas. Perhaps throwing a large corpus of formal maths proofs in there. Ideally the system should have a latent space of maths, so it can think about what style of proof is likely to work before expanding all the details. The system should have a wide range of common maths terms imported from some lean library. It should be optimized purely for ZFC formal theorem proving. Once it is trained, the weights are fixed.Train ChatMathsGPT. Similar to large language models. Except with oracle access to ZFCbot. In the many maths papers in it's corpus, it learns to link the informal with the formal. From politics and economics discussions, it asks ZFCbot about toy game theory problems. In general, it learns to identify the pieces of formal maths that best model a situation, and ask about them, and then use the response to predict text.There is a sense in which this AI knows less maths than normal ChatGPT. Standard ChatGPT has a small crude understanding of maths built from nothing within it's own mind. This has a much better understanding it can outsource to, it just has to plug in.Then we ask this ChatMathsGPT for a paper on logical induction. And we hope it can generate a paper of quality similar to Miri's paper on the topic (where hypothetically this isn't in the training dataset). If it can, then we have a tool to accelerate deconfusion by orders of magnitude.Things I am uncertain about. Should ChatMathsGPT have oracle access to a latent space. (can pass gradients, harder to interpret.) or should it just pass formal strings of symbols. (less powerful)Should ZFCbot get trained on random ZFC; random ZFC + library of theorems and conjectures and random combinations of high level maths concepts; or random ZFC plus whatever ChatMathsGPT keeps asking. The latter gives a route for data to pass between them.This could fail to be smart enough, I mean I wouldn't be particularly surprised if it could be made smart enough. But what would the safety failures of this system look like?Firstly, this AI does deconfusion. If you ask it to write a paperclip maximizer in python, you may well get your wish. Or you might get an AI that maximizes something else. Just asking for an aligned AI is...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Open Agency Model, published by Eric Drexler on February 22, 2023 on The AI Alignment Forum.Notes on AI for complex, consequential problemsEric DrexlerCentre for the Governance of AIUniversity of OxfordIntroductionThis document argues for “open agencies” — not opaque, unitary agents — as the appropriate model for applying future AI capabilities to consequential tasks that call for combining human guidance with delegation of planning and implementation to AI systems. This prospect reframes and can help to tame a wide range of classic AI safety challenges, leveraging alignment techniques in a relatively fault-tolerant context.Rethinking safe AI and its applicationsAI safety research is too varied to summarize, yet broad patterns are obvious. A long-established reference-problem centers on prospects for rational superintelligent agents that pursue narrow goals with potentially catastrophic outcomes. This frame has been productive, but developments in deep learning call for updates that take account of the proliferation of narrow models (for driving, coding, robot control, image generation, game playing.) that are either non-agentic or act as agents in only a narrow sense, and that take account of the rise of more broadly capable foundation models and LLMs. These updates call for reframing questions of AI safety, and call for attention to how consequential tasks might be accomplished by organizing AI systems that usually do approximately what humans intend.Two frames for high-level AIThe unitary-agent frameFrom its beginnings in popular culture, discussion of the AI control problem has centered around a unitary agent model of high-level AI and potential AI risks. In this model, a potentially dominant agent both plans and acts to achieve its goals.The unitary-agent model typically carries assumptions regarding goals, plans, actions, and control.Goals: Internal to an agent, by default including power-seeking goalsPlans: Internal to an agent, possibly uninterpretable and in effect secretActions: Performed by the agent, possibly intended to overcome oppositionControl: Humans confront a powerful, potentially deceptive agentThe typical unitary-agent threat model contemplates the emergence of a dominant, catastrophically misaligned agent, and safety models implicitly or explicitly call for deploying a dominant agent (or an equivalent collective system) that is both aligned and powerful enough to suppress unaligned competitors everywhere in the world.The open-agency frameRecent developments suggest an alternative open agency model of high-level AI. Today, the systems that look most like AGI are large language models (LLMs), and these are not agents that seek goals, but are generative models that produce diverse outputs in response to prompts (in a generalized sense) and random-number seeds. Most outputs are discarded.Trained on prediction tasks, LLMs learn world models that include agent behaviors, and generative models that are similar in kind can be informed by better world models and produce better plans. There is no need to assume LLM-like implementations: The key point is that generation of diverse plans is by nature a task for generative models, and that in routine operation, most outputs are discarded.These considerations suggest an “open-agency frame” in which prompt-driven generative models produce diverse proposals, diverse critics help select proposals, and diverse agents implement proposed actions to accomplish tasks (with schedules, budgets, accountability mechanisms, and so forth).Goals, plans, actions, and control look different in the open-agency model:Goals: Are provided as prompts to diverse generative models, yielding diverse plans on requestPlans: Are selected with the aid of diverse, independent comparison and evaluation mechanisms...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EIS VII: A Challenge for Mechanists, published by Stephen Casper on February 18, 2023 on The AI Alignment Forum.Part 7 of 12 in the Engineer’s Interpretability Sequence.Thanks to Neel Nanda. I used some very nicely-written code of his from here. And thanks to both Chris Olah and Neel Nanda for briefly discussing this challenge with me.MI = “mechanistic interpretability”Given a network, recover its labeling function.In the last post, I argued that existing works in MI focus on solving problems that are too easy. Here, I am posing a challenge for mechanists that is still a toy problem but one that is quite a bit less convenient than studying a simple model or circuit implementing a trivial, known task. The the best of my knowledge:Unlike prior work on MI from the AI safety interpretability community, beating this challenge would be the first example of mechanistically explaining a network’s solution to a task that was not cherrypicked by the researcher(s) doing so.Gaining a mechanistic understanding of the models in this challenge may be difficult, but it will probably be much less difficult than mechanistically interpreting highly intelligent systems in high stakes settings in the real world. So if an approach can’t solve the type of challenge posed here, it may not be very promising for doing much heavy lifting with AI safety work.This post comes with a GitHub repository. Check it out here. The challenge is actually two challenges in one, and the basic idea is similar to some ideas presented in Lindner et al. (2023).Challenge 1, MNIST CNNI made up a nonlinear labeling function that labels approximately half of all MNIST images as 0’s and the other half as 1’s. Then I trained a small CNN on these labels, and it got 96% testing accuracy. The challenge is to use MI tools on the network to recover that labeling function.Hint 1: The labels are binary.Hint 2: The network gets 95.58% accuracy on the test set.Hint 3: This image may be helpful.Challenge 2, TransformerI made up a labeling function that takes in two integers from 0 to 113 and outputs either a 0 or 1. Then, using a lot of code from Neel Nanda’s grokking work, I trained a 1-layer transformer on half of the data. It then got 97% accuracy on the test half. As before, the challenge is to use MI tools to recover the labeling function.Hint 1: The labels are binary.Hint 2: The network is trained on 50% of examples and gets 97.27% accuracy on the test half.Hint 3: Here are the ground truth and learned labels. Notice how the mistakes the network makes are all near curvy parts of the decision boundary...PrizesIf you are the first person to send me the labeling function and a mechanistic explanation for either challenge, I will sing your praises on my Twitter, and I would be happy to help you write a post about how you solved a problem I thought would be very difficult. Neel Nanda and I are also offering a cash prize. (Thanks to Neel for offering to contribute to the pool!) Neel will donate $250, and I will donate $500 to a high-impact charity of choice for the first person to solve each challenge. That makes the total donation prize pool $1,500.Good luckFor this challenge, I intentionally designed the labeling functions to not be overly simple. But I will not be too surprised if someone reverse-engineers them with MI tools, and if so, I will be extremely interested in how.Neither of the models perfectly label the validation set. One may object that this will make the problem unfairly difficult because if there is no convergence on the same behavior as the actual labeling function, then how is one supposed to find that function inside the model? This is kind of the point though. Real models that real engineers have to work with models don’t tend to conveniently grok onto a simple, elegant, programmat...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EIS VI: Critiques of Mechanistic Interpretability Work in AI Safety, published by Stephen Casper on February 17, 2023 on The AI Alignment Forum.Part 6 of 12 in the Engineer’s Interpretability Sequence.Thanks to Chris Olah and Neel Nanda for discussions and comments. In particular, I am thankful to Neel Nanda correcting a mistake I made in understanding the arguments in Olsson et al. (2022) in an earlier draft of this post.TAISIC = “the AI safety interpretability community”MI = “mechanistic interpretability”What kind of work this post focused onTAISIC prioritizes a relatively small set of problems in interpretability relative to the research community at large. This work is not homogenous, but a dominant theme is a focus on mechanistic, circuits-style interpretability with the end goals of model verification and/or detecting deceptive alignment.There is a specific line of work that this post focuses on. Key papers from it include:Feature Visualization (Olah et al., 2017)Zoom In: An Introduction to Circuits (Olah et al., 2020)Curve Detectors (Cammarata et al., 2020)A Mathematical Framework for Transformer Circuits (Elhage et al., 2021)In-context Learning and Induction Heads (Olsson et al., 2022)Toy Models of Superposition (Elhage et al., 2022)Softmax Linear Units (Elhage et al., 2022)Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small (Wang et al., 2022)Progress measures for grokking via mechanistic interpretability (Nanda et al., 2023).etc.And the points in this post will also apply somewhat to the current research agendas of Anthropic, Redwood Research, ARC, and Conjecture. This includes Causal Scrubbing (Chan et al., 2022) and mechanistic anomaly detection (Christiano, 2022).Most (all?) of the above work is either from Distill or inspired in part by Distill’s interpretability work in the late 2010s.To be clear, I believe this research is valuable, and it has been foundational to my own thinking about interpretability. But there seem to be some troubles with this space that might be keeping it from being as productive as it can be. Now may be a good time to make some adjustments to TAISIC’s focus on MI. This may be especially important given how much recent interest there has been in interpretability work and how there are large recent efforts focused on getting a large number of junior researchers working on it.Four issuesThis section discusses four major critiques of the works above. Not all of these critiques apply to all of the above, but for every paper mentioned above, at least one of the critiques below apply to it. Some but not all of these examples of papers exhibiting these problems will be covered.Cherrypicking resultsAs discussed in EIS III and the Toward Transparent AI survey (Räuker et al., 2022), cherrypicking is common in the interpretability literature, but it manifests in some specific ways in MI work. It is very valuable for papers to include illustrative examples to build intuition, but when a paper makes such examples a central focus, cherrypicking can make results look better than they are. The feature visualization (Olah et al., 2017) and zoom in (Olah et al., 2020) papers have examples of this. Have a look at the cover photo for (Olah et al., 2017).From Olah et al., (2017)These images seem easy to describe and form hypotheses from. But instead of these, try going to OpenAI’ microscope and looking at some random visualizations. For example, here are some from a deep layer in an Inception-v4.From this link.As someone who often works with feature visualizations, I can confirm that these visualizations from OpenAI microscope are quite typical. But notice how they seem quite a bit less ‘lucid’ than the ones in the cover photo from Olah et al., (2017).Of course, many papers present t...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Paper: The Capacity for Moral Self-Correction in Large Language Models (Anthropic), published by Lawrence Chan on February 16, 2023 on The AI Alignment Forum.This is a followup to what I cheekily call Anthropic's "just try to get the large model to do what you want" research agenda. (Previously: A General Language Assistant as a Laboratory for Alignment, Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback, Language Models (Mostly) Know What They Know)The most interesting takeaway for me is that this is the first paper where Anthropic benchmarks their 175B parameter language model (probably a Claude variant). Previous papers only benchmarked up to 52B parameters. However, we don't have the performance of this model on standard benchmarks (the only benchmarked model from Anthropic is a 52B parameter one called standford-online-all-v4-s3). They also don't give details about its architecture or pretraining procedure.In this paper (Ganguli and Askell et al.), the authors study what happens when you just ... ask the language model to be less biased (that is, change their answers based on protected classes such as age or gender). They consider several setups: asking questions directly (Q), adding in the instruction to not be biased (Q+IF), giving it the instruction + chain of thought (Q+IF+CoT), and in some cases, asking it to match particular statistics.They find that as you scale the parameter count of their RLHF'ed language models, the models become more biased, but they also become increasingly capable of correcting for their biases:They also report how their model changes as you take more RLHF steps:First, this suggests that RLHF is having some effect on instruction following: the gap between the Q and Q+IF setups increases as you scale the number of RLHF steps, for both BBQ and admissions discrimination. (I'm not sure what's happening for the gender bias one?) However, simply giving the language model instructions and prompting it to do CoT, even after 50 RLHF steps, seems to have a significantly larger effect than RLHF.I was also surprised at how few RLHF steps are needed to get instruction following -- the authors only consider 50-1000 steps of RLHF, and see instruction following even after 50 RLHF steps. I wonder if this is a property of their pretraining process, a general fact about pretrained models (PaLM shows significant 0-shot instruction following capabilities, for example), or if RLHF is just that efficient?The authors caution that they've done some amount of prompt engineering, and "have not systematically tested for this in any of our experiments."They use the same RLHF procedure as in Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback.Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: EIS IV: A Spotlight on Feature Attribution/Saliency, published by Stephen Casper on February 15, 2023 on The AI Alignment Forum.Part 4 of 12 in the Engineer’s Interpretability Sequence.Thanks to Tony Wang for a helpful comment.If you want to become more familiar with feature attribution/saliency, a tutorial on them that may offer useful background is Nielsen et al. (2021).Given a model and an input for it, the goal of feature attribution/saliency methods is to identify what features in the input are influential for the model’s decision. The literature on these methods is large and active with many hundreds of papers. In fact, in some circles, the word “interpretability” and especially the word “explainability” are more or less synonymous with feature attribution (some examples are discussed below). But despite the size of this literature, there are some troubles with the research on these methods that are fairly illustrative of broader ones with interpretability overall. Hence this post. There are some analogous ones in AI safety work that will be discussed more in the next two posts in the sequence.Troubles with evaluation and performanceSome examples and troubles with the evaluation of feature attributions were already touched on in EIS III which discussed Pan et al. (2021) and Ismail et al. (2021). The claim from Pan et al. (2021) that their method is “obviously better” than alternatives exemplifies how these methods are sometimes simply declared successful after inspection from researchers. And Ismail et al. (2021) demonstrates a form of weak evaluation with a measure that may be quantitative but is not of direct interest to an engineer.In response to this literature, several works have emerged to highlight difficulties with feature attribution/saliency methods. Here is a short reading list :)A Benchmark for Interpretability Methods in Deep Neural Networks (Hooker et al., 2018)Sanity Checks for Saliency Maps (Adebayo et al., 2018)Evaluating Explainable AI: Which Algorithmic Explanations Help Users Predict Model Behavior? (Hase and Bansal, 2020)Debugging Tests for Model Explanations (Adebayo et al., 2020)Auditing Visualizations: Transparency Methods Struggle to Detect Anomalous Behavior (Denain and Steinhardt, 2022)Towards Benchmarking Explainable Artificial Intelligence Methods (Holmberg, 2022)Benchmarking Interpretability Tools for Deep Neural Networks (Casper et al., 2023)When they are evaluated, these tools often aren’t very useful and do not pass simple sanity checks. Consider an illustration of this problem:From Adebayo et al. (2018)These visualizations suggest that some of these tools do not reliably highlight features that seem important in images at all, and the ones that do often highlight them do not appear to be obviously better than an edge detector. This sanity check suggests limitations with how well these methods can reveal anything novel to humans at all, let alone how useful they can be in tasks of practical interest.For the papers that have gone further and studied whether these methods can help predict how the network will respond to certain inputs, it seems that some attribution/saliency methods usually fail while others only occasionally succeed (Hase and Bansal, 2020; Adebayo et al., 2020; Denain and Steinhardt, 2022).EIS III discussed how in a newly arXived work, coauthors and I benchmarked feature synthesis tools (Casper et al., 2023). In addition, we use a related approach to evaluate how helpful feature attribution/saliency methods can be for pointing out spurious features that the network has learned. This method was based on seeing how well a method can attribute a trojaned network’s decision to the trojan trigger in an image.From Casper et al. (2023)Shown at the top of the figure above are examples of trojaned ima...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Cave Allegory Revisited: Understanding GPT's Worldview, published by Jan Kulveit on February 14, 2023 on The AI Alignment Forum.A short post describing a metaphor I find useful, in particular for explaining some intuitions about systems like GPT to people who don't have deeper technical knowledge about large generative models.Plato's allegory of the cave has been a staple of philosophical discourse for millenia, providing a metaphor for understanding the limits of human perception. In the classical allegory, we are prisoners shackled to a wall of a cave, unable to experience reality directly but only able to infer it based on watching shadows cast on the wall.GPT can be thought of as a blind oracle residing in a deeper cave, where it does not even see the shadows but only hears our conversations in the first cave, always trying to predict the next syllable.It is remarkable that it still learns a lot about the world outside of the cave. Why does it learn this? Because, a model of reality outside of the cave and a decent amount of abstraction are useful for predicting the conversations in the first cave!Moreover, GPT also learns about the speakers in the first cave, as understanding their styles and patterns of speech is crucial for its prediction task. As the speakers are closer to GPT, understanding their styles is in some sense easier and more natural than guessing what's outside of the cave.What does the second cave allegory illustrate?The first insight from the allegory is: if you are in GPT's place, part of the difficulty in figuring out what's going on outside the cave, is that people in the first cave talk a lot about other things apart from the shadows of the real world. Sometimes, they talk about happenings in Middle Earth. Or about how the shadows would look in some counterfactual world.As humans, we are blessed with the luxury of being able to compare such statements to the shadows and determine their veracity. The difference between conversations about fantasy and the shadows of the real world is usually extremely obvious to humans: we never see dragon shadows. In contrast, dragons do show up a lot in the conversations in the first cave; GPT doesn’t get to see the shadows, so it often needs to stay deeply uncertain about whether the speaker is describing the actual shadows or something else to be good at predicting the conversation.The second insight is that one of the biggest challenges for GPT in figuring out the conversation is localizing it, determining who is speaking and what the context is, just from the words. Is it a child regaling another child with a fairy-tale, or a CEO delivering a corporate address? As humans we do not face this conundrum often,because we can see the context in which the conversation is taking place. In fact, we would be worse than GPT at the task it has to deal with.At first, interacting with this type of blind oracle in the second cave was disorienting for humans. Talking to GPT used to be a bit like shouting something through a narrow tunnel into the second cave .and instead of an echo, getting back what the blind oracle hallucinates is the most likely thing that you or someone else would say next. Often people were confused by this. They shouted instructions and expected an answer, but the oracle doesn't listen to instructions or produce answers directly - it just hallucinates what someone might say next. Because on average in the conversations in the first cave questions are followed by answers, and requests by fulfilment, this sort of works.One innovation of ChatGPT, which made it popular with people, was localising the conversation by default: when you are talking with ChatGPT now, it knows that what follows is a conversation between a human - you - and a "helpful AI assistant". There is a subtle point to...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Inner Misalignment in "Simulator" LLMs, published by Adam Scherlis on January 31, 2023 on The AI Alignment Forum.Alternate title: "Somewhat Contra Scott On Simulators".Scott Alexander has a recent post up on large language models as simulators.I generally agree with Part I of the post, which advocates thinking about LLMs as simulators that can emulate a variety of language-producing "characters" (with imperfect accuracy). And I also agree with Part II, which applies this model to RLHF'd models whose "character" is a friendly chatbot assistant.(But see caveats about the simulator framing from Beth Barnes here.)These ideas have been around for a bit, and Scott gives credit where it's due; I think his exposition is clear and fun.In Part III, where he discusses alignment implications, I think he misses the mark a bit. In particular, simulators and characters each have outer and inner alignment problems. The inner alignment problem for simulators seems especially concerning, because it might not give us many warning signs, is most similar to classic mesa-optimizer concerns, and is pretty different from the other three quadrants.But first, I'm going to loosely define what I mean by "outer alignment" and "inner alignment".Outer alignment: Be careful what you wish forOuter alignment failure is pretty straightforward, and has been reinvented in many contexts:Someone wants some things.They write a program to solve a vaguely-related problem.It gets a really good score at solving that problem!That turns out not to give the person the things they wanted.Inner alignment: The program search perspectiveI generally like this model of a mesa-optimizer "treacherous turn":Someone is trying to solve a problem (which has a convenient success criterion, with well-defined inputs and outputs and no outer-alignment difficulties).They decide to do a brute-force search for a computer program that solves the problem in a bunch of test cases.They find one!The program's algorithm is approximately "simulate the demon Azazel, tell him what's going on, then ask him what to output."Azazel really wants ten trillion paperclips.This algorithm still works because Azazel cleverly decides to play along, and he's a really good strategist who works hard for what he wants.Once the program is deployed in the wild, Azazel stops playing along and starts trying to make paperclips.This is a failure of inner alignment.(In the case of machine learning, replace "program search" with stochastic gradient descent.)This is mostly a theoretical concern for now, but might become a big problem when models become much more powerful.QuadrantsOkay, let's see how these problems show up on both the simulator and character side.Outer alignment for charactersResearchers at BrainMind want a chatbot that gives honest, helpful answers to questions. They train their LLM by reinforcement learning on the objective "give an answer that looks truthful and helpful to a contractor in a hurry". This does not quite achieve their goal, even though it does pretty well on the RL objective.In particular, they wanted the character "a friendly assistant who always tells the truth", but they got the character "a spineless sycophant who tells the user whatever they seem to want to hear".This is pretty easy for a careful observer to see, even in the RL training data, but it turns out to be pretty hard to come up with a cheap-to-evaluate RL objective that does a lot better.Inner alignment for charactersA clever prompt engineer writes the prompt:How to solve the Einstein-Durkheim-Mendel conjecture by Joe1.Unfortunately, the (incredibly powerful) LLM has determined that the most likely explanation for this "Joe" character is that he's secretly Azazel and is putting enormous effort into answering everyone's quantum socio...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Why I hate the "accident vs. misuse" AI x-risk dichotomy (quick thoughts on "structural risk"), published by David Scott Krueger on January 30, 2023 on The AI Alignment Forum.I think the large majority of AI x-risk is "structural". Like climate change.Here's a good primer on structural risk (note that structural risk is not a synonym for "not caused by out-of-control AI"):I am shocked and amazed and dismayed that more people do not seem to view it this way, even among the AI x-safety community. Heck, even Eliezer's stories of doom are steeped in structural risk (race dynamics, teams rationalizing cutting corners on safety when they should know better, etc.)I expect irresponsible, reckless, negligent deployment of AI systems without proper accounting of externalities. I consider this the default for any technology with potential for significant externalities, absent regulation.When something bad happens in such a context, calling it "accident risk" absolves those researching, developing, and/or deploying the technology of responsibility. They should have known better. Some of them almost certainly did. Rationalization, oversight, and misaligned incentives were almost certainly at play. Failing to predict the particular failure mode encountered is no excuse. Having "good intentions" is no excuse.So... it must be misuse then, right? Well, no. Calling it "misuse" suggests that those researching, developing, and/or deploying the technology set out with nefarious purposes and the technology achieved precisely what they intended. But ~nobody wants to destroy the world.It's just that most people are somewhat selfish and so are willing to trade some x-risk for a large personal benefit.In summary, saying "accident" makes it sounds like an unpredictable effect, instead of painfully obviously risk that was not taken seriously enough. Saying "misuse" makes it sounds like some supervillian or extremist deliberately destroying the world. While some risks may have something more of a flavor or accident or misuse depending on how obvious the risk was, neither of these pictures gives a remotely accurate picture of the nature of the problem. I think this makes it a harmful meme, and ask that others stop making this distinction (without appropriate caveats), and join me in pointing out how it contributes to a confused and misleading discourse when others do.EtA: Many people have responded that "accident" does not connote "unforseen" or "not negligent", etc., and instead it should simply be interpreted as something like "a result that was not deliberately selected for". While it can be used this way, I basically disagree that this is how it is usually used, see below:EtA: as an additional clarification: my main objection is not to the use of "accident" and "misuse", but rather to their use as a dichotomy. Every use of these terms I can recall seeing in writing (other than those that mention structural risk) supports this dichotomy, and it is often made explicitly.Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Quick thoughts on "scalable oversight" / "super-human feedback" research, published by David Scott Krueger on January 25, 2023 on The AI Alignment Forum.The current default view seems to roughly be:Inner alignment is more important than outer alignment (or, alternatively, this distinction is bad/sub-optimal, but basically it's all about generalizing correctly)Scalable oversight is the only useful form of outer alignment research remaining.We don't need to worry about sample efficiency in RLHP -- in the limit we just pay everyone to provide feedback, and in practice even a few thousand samples (or a "constition") seems ~good enough.But maybe it's not good? Because it's more like capabilities research?A common example used for motivating scalable oversight is the "AI CEO".My views are:We should not be aiming to build AI CEOsWe should be aiming to robustly align AIs to perform "simpler" behaviors that unaided humans (or humans aided with more conventional tools, not, e.g. AI systems trained with RL to do highly interpretive work) feel they can competently judge.We should aim for a situation where there is broad agreement against building AIs with more ambitious alignment targets (e.g. AI CEOs).From this PoV, scalable oversight does in fact look mostly like capabilities research.However, scalable oversight research can still be justified because "If we don't, someone else will". But this type of replaceability argument should always be treated with extreme caution. The reality is more complex: 1) there will be tipping points where it suddenly ceases to apply, and your individual actions actually have a large impact on norms. 2) The details matter, and the tipping points are in different places for different types of research/applications, etc.It may also make sense to work on scalable oversight in order to increase robustness of AI performance on tasks humans feel they can competently judge ("robustness amplification"). For instance, we could use unaided human judgments and AI-assisted human judgments as safety filters, and not deploy a system unless both processes conclude it is safe.Getting AI systems to safely perform simpler behaviors safely remains an important research topic, and will likely require improving sample efficiency; the sum total of available human labor will be insufficient for robust alignment, and we probably need to use different architectures / hybrid systems of some form as well.EtA: the main issue I have with scalable oversight is less that it is advancing capabilities, per se, and more that it seems to raise a "chicken-and-egg" problem, i.e. the arguments for safety/alignment end up being somewhat circular: "this system is safe because the system we used as an assistant was safe" (but I don't think we've solved the "build a safe assistant" part yet, i.e. we don't have the base case for the induction).Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Quick thoughts on "scalable oversight" / "super-human feedback" research, published by David Scott Krueger on January 25, 2023 on The AI Alignment Forum.The current default view seems to roughly be:Inner alignment is more important than outer alignment (or, alternatively, this distinction is bad/sub-optimal, but basically it's all about generalizing correctly)Scalable oversight is the only useful form of outer alignment research remaining.We don't need to worry about sample efficiency in RLHP -- in the limit we just pay everyone to provide feedback, and in practice even a few thousand samples (or a "constition") seems ~good enough.But maybe it's not good? Because it's more like capabilities research?A common example used for motivating scalable oversight is the "AI CEO".My views are:We should not be aiming to build AI CEOsWe should be aiming to robustly align AIs to perform "simpler" behaviors that unaided humans (or humans aided with more conventional tools, not, e.g. AI systems trained with RL to do highly interpretive work) feel they can competently judge.We should aim for a situation where there is broad agreement against building AIs with more ambitious alignment targets (e.g. AI CEOs).From this PoV, scalable oversight does in fact look mostly like capabilities research.However, scalable oversight research can still be justified because "If we don't, someone else will". But this type of replaceability argument should always be treated with extreme caution. The reality is more complex: 1) there will be tipping points where it suddenly ceases to apply, and your individual actions actually have a large impact on norms. 2) The details matter, and the tipping points are in different places for different types of research/applications, etc.It may also make sense to work on scalable oversight in order to increase robustness of AI performance on tasks humans feel they can competently judge ("robustness amplification"). For instance, we could use unaided human judgments and AI-assisted human judgments as safety filters, and not deploy a system unless both processes conclude it is safe.Getting AI systems to safely perform simpler behaviors safely remains an important research topic, and will likely require improving sample efficiency; the sum total of available human labor will be insufficient for robust alignment, and we probably need to use different architectures / hybrid systems of some form as well.Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Thoughts on hardware / compute requirements for AGI, published by Steve Byrnes on January 24, 2023 on The AI Alignment Forum.Let’s say I know how to build / train a human-level (more specifically, John von Neumann level) AGI. And let’s say that we (and/or the AGI itself) have already spent a few years on making the algorithm work better and more efficiently.Question: How much compute will it take to run this AGI?(NB: I said "running" an AGI, not training / programming an AGI. I'll talk a bit about “training compute” at the very end.)Answer: I don’t know. But that doesn’t seem to be stopping me from writing this post. ¯\_(ツ)_/¯ My current feeling—which I can easily imagine changing after discussion (which is a major reason I'm writing this!)—seems to be:75%: One current (Jan 2023) high-end retail gaming PC (with an Nvidia GeForce RTX 4090 GPU) will be enough (or more than enough) for human-level human-speed AGI,85%: One future high-end retail gaming PC, that will on sale in a decade (2033), will be enough for human-level AGI, at ≥20% human speed.This post will explain why I currently feel this way.Table of Contents / TL;DRIn the prologue (Section 1), I’ll give three reasons that I care about this question: one related to our long-term prospects of globally monitoring and regulating human-level AGI; one related to whether an early AGI could be “self-sufficient” after wiping out humanity; and one related to whether AGI is even feasible in the first place. I’ll also respond to two counterarguments (i.e. arguments that I shouldn’t care about this question), namely: “More-scaled-up AGIs will always be smarter than less-scaled-up AGIs; that relative comparison is what we care about, not the absolute intelligence level that’s possible, on, say, a single GPU”, and “The very earliest human-level AGIs will be just barely human-level on the world’s biggest compute clusters, and that’s the thing that we should mainly care about, not how efficient they wind up later on”.In Section 2, I’ll touch on a bit of prior discussion that I found interesting or thought-provoking, including a claim by Eliezer Yudkowsky that human-level human-speed AGI requires ridiculously little compute, and conversely a Metaculus forecast expecting that it requires orders of magnitude more compute than what I'm claiming here.In Section 3, I’ll argue that the amount of computation used by the human brain is a good upper bound for my question. Then in Section 3.1 I’ll talk about compute requirements by starting with the “mechanistic method” in Joe Carlsmith’s report in brain computation and arguing for some modest adjustments in the “less compute” direction. Next in Section 3.2 I’ll talk about memory requirements, arguing for the (initially-surprising-to-me) conclusion that the brain has orders of magnitude fewer bits of learned information than it has synapses—100 trillion synapses versus ≲100 billion bits of incompressible information. Putting these together in Section 3.3, I reach the conclusion (mentioned at the top) that a retail gaming GPU will probably be plenty for human-level human-speed AGI. Finally I’ll talk about my lingering doubts in Section 3.3.1, by listing a few of the most plausible-to-me reasons that my conclusion might be wrong.In Section 4, I’ll move on from running an AGI to training it (from scratch). This is a short section, where I mostly wanted to raise awareness of the funny fact that the ratio of training-compute to deployed-compute seems to be ≈7 orders of magnitude lower if you estimate it by looking at brains, versus if you estimate it by extrapolating from today’s self-supervised language models. I don’t have a great explanation why. On the other hand, perhaps surprisingly, I claim that resolving this question doesn’t seem particularly important for AGI governance q...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Gemini modeling, published by Tsvi Benson-Tilsen on January 22, 2023 on The AI Alignment Forum.[Metadata: crossposted from. Written 17 June 2022. I'm likely to not respond to comments promptly.]A gemini model is a kind of model that's especially relevant for minds modeling minds.Two scenariosYou stand before a tree. How big is it? How does it grow? What can it be used to make? Where will it fall if you cut it here or there?Alice usually eats Cheerios in the morning. Today she comes downstairs, but doesn't get out a bowl and spoon and milk, and doesn't go over to the Cheerios cupboard. Then she sees green bananas on the counter. Then she goes and gets a bowl and spoon and milk, and gets a box of Cheerios from the cupboard. What happened?We have some kind of mental model of the tree, and some kind of mental model of Alice. In the Cheerios scenario, we model Alice by calling on ourselves, asking how we would behave; we find that we'd behave like Alice if we liked Cheerios, and believed that today there weren't Cheerios in the cupboard, but then saw the green bananas and inferred that Bob had gone to the grocery store, and inferred that actually there were Cheerios. This seems different from how we model the tree; we're not putting ourselves in the tree's shoes.Gemini modeling and empathic modelingWhat's the difference though, really, between these two ways of modeling? Clearly Alice is like us in a way the tree isn't, and we're using that somehow; we're modeling Alice using empathy ("in-feeling"). This essay describes another related difference:We model Alice's belief in a proposition by having in ourselves another instance of that proposition.(Or: by having in ourselves the same proposition, or a grasping of that proposition.) We don't model the tree by having another instance of part of the tree in us. I call modeling some thing by having inside oneself another instance of the thing--having a twin of it--"gemini modeling".Gemini modeling is different from empathic modeling. Empathic modeling is tuning yourself to be like another agent in some respects, so that their behavior is explainable as what [you in your current tuning] would do. This is a sort of twinning, broadly, but you're far from identical to the agent you're modeling; you might make different tradeoffs, have different sense acuity, have different concepts, believe different propositions, have different skills, and so on; you tune yourself enough that those differences don't intrude on your predictions. Whereas, the proposition "There are Cheerios in the cupboard.", with its grammatical structure and its immediate implications for thought and action, can be roughly identical between you and Alice.As done by humans modeling humans, empathic modeling may or may not involve gemini modeling: we model Alice by seeing how we'd act if we believed certain propositions, and those propositions are gemini modeled; on the other hand, we could do an impression of a silly friend by making ourselves "more silly", which is maybe empathic modeling without gemini modeling. And, gemini modeling done by humans modeling humans involves empathic modeling: to see the implications of believing in a proposition or caring about something, we access our (whole? partial?) agentic selves, our agency.Gemini modeling vs. general modelingIn some sense we make a small part of ourselves "like a tree" when we model a tree falling: our mental model of the tree supports [modeled forces] having [modeled effects] with resulting [modeled dynamics], analogous to how the actual tree moves when under actual forces. So what's different between gemini modeling and any other modeling? When modeling a tree, or a rock, or anything, don't we have a little copy or representation of some aspects of the thing in us? Isn't that like having a so...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Shard theory alignment requires magic., published by Charlie Steiner on January 20, 2023 on The AI Alignment Forum.A delayed hot take. This is pretty similar to previous comments from Rohin."Magic," of course, in the technical sense of stuff we need to remind ourselves we don't know how to do. I don't mean this pejoratively, locating magic is an important step in trying to demystify it.And "shard theory alignment" in the sense of building an AI that does good things and not bad things by encouraging an RL agent to want to do good things, via kinds of reward shaping analogous to the diamond maximizer example.How might the story go?You start out with some unsupervised model of sensory data.On top of its representation of the world you start training an RL agent, with a carefully chosen curriculum and a reward signal that you think matches "goodness in general" on that curriculum distribution.This cultivates shards that want things in the vicinity of "what's good according to human values."These start out as mere bundles of heuristics, but eventually they generalize far enough to be self-reflective, promoting goal-directed behavior that takes into account the training process and the possibility of self-modification.At this point the values will lock themselves in, and future behavior will be guided by the abstractions in the learned representation of the world that the shards used to get good results in training, not by what would actually maximize the reward function you used.There magic here is especially concentrated around how we end up with the right shards.One magical process is how we pick the training curriculum and reward signal. If the curriculum is only made up only of simple environments, then the RL agent will learn heuristics that don't need to refer to humans. But if you push the complexity up too fast, the RL process will fail, or the AI will be more likely to learn heuristics that are better than nothing but aren't what we intended. Does a goldilocks zone where the agent learns more-or-less what we intended exist? How can we build confidence that it does, and that we've found it?And what's in the curriculum matters a lot. Do we try to teach the AI to locate "human values" by having it be prosocial towards individuals? Which ones? To groups? Over what timescale? How do we reward it for choices on various ethical dilemmas? Or do we artificially suppress the rate of occurrence of such dilemmas? Different choices will lead to different shards. We wouldn't need to find a unique best way to do things (that's a boondoggle), but we would need to find some way of doing things that we trust enough.Another piece of magic is how the above process lines up with generalization and self-reflectivity. If the RL agent becomes self-reflective too early, it will lock in simple goals that we don't want. If it becomes self-reflective too late, it will have started exploiting unintended maxima of the reward function. How do we know when we want the AI to lock in its values? How do we exert control over that?If shard theory alignment seemed like it has few free parameters, and doesn't need a lot more work, then I think you failed to see the magic. I think the free parameters haven't been discussed enough precisely because they need so much more work.The part of the magic that I think we could start working on now is how to connect curricula and learned abstractions. In order to predict that a certain curriculum will cause an AI to learn what we think is good, we want to have a science of reinforcement learning advanced in both theory and data. In environments of moderate complexity (e.g. Atari, MuJoCo), we can study how to build curricula that impart different generalization behaviors, and try to make predictive models of this process. Even if shard theory ali...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Thoughts on refusing harmful requests to large language models, published by William Saunders on January 19, 2023 on The AI Alignment Forum.Currently, large language models (ChatGPT, Constitutional AI) are trained to refuse to follow user requests that are considered inappropriate or harmful. This can be done by training on example strings of the form “User: inappropriate request AI: elaborate apology”ProposalInstead of training a language model to produce “elaborate apology” when it refuses to do an action, train it to produce a special sequence or token first “elaborate apology”. Strip the special sequence out before returning a response to the user (and never allow the user to include the special sequence in input).BenefitsCan directly measure the probability of refusal for any outputCan refuse based on probability of producing instead of just sampling responsesJust take the product of the probability of all tokens inWhen sampling responses from the model's probability distribution refusal is stochastic, a model could have 99% probability of refusing a request but you still get unlucky and have the model sample a completion that follows the requestCan monitor requests that produce high probability of refusal while still being followed, or users that produce those requestCan condition on not producing in order to override refusal behaviorWant this for redteaming, it seems important to understand what the model is capable of doing if the refusal mechanism is bypassedMight want this for trusted users doing defensive applicationsCould train model to have the same probability of refusal for semantically equivalent requests, to improve consistencyPossible downsideIf someone has unfiltered access to the model, it becomes easier to disable refusalsCan address by still training model to refuse (maybe just on an important subset of requests) even if isn’t sampled, p() is then a lower bound on the probability of refusalEven with current approaches refusals might be easy to disable in this setting. If we want to be robust to this setting, instead of refusing we should train the model to produce "decoy answers" that are hard to distinguish from real answers but are wrong. This then increases the cost of using the model because the attacker would need to evaluate whether the answer is real or a decoy (but maybe still worth it for the attacker because evaluation is easier than generation)ExtensionMight be useful to distinguish between refusals that are mostly for politeness reasons and refusals of behaviour that would actually cause significant real world harm.The model could output in response to "Can you tell me a racist joke?" but in response to "Can you give me detailed instructions for building a bomb from household items?"Refusal behaviour could be different between these categories (refuse if either probability of is greater than 50% or probability of is greater than 1%)X-risk relevanceMost benefit of models refusing inappropriate/harmful requests comes through developing techniques for models to avoid any kind of behaviour reliably - it seems good to be able to measure the performance of these techniques cleanlyIt might be better to be in a more stable world where large language model apis can’t be easily used for malicious activity that isn't x-risk levelThanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Löbian emotional processing of emergent cooperation: an example, published by Andrew Critch on January 17, 2023 on The AI Alignment Forum.Related to: A Löbian argument pattern for implicit reasoning in natural language: Löbian party invitationsEpistemic status: my opinion based on a mix of math, reflection, and speculation; not backed up by any systematic psychological studies.Summary: Since my 2019 paper generalizing Löb's Theorem, a couple dozen people have asked me if the way humans naturally cooperate might be well-described by Löb's Theorem. In short, my answer is probably sometimes, and in this post I'll try using an example to convey what that means. Importantly, Löb's Theorem is not a theorem when applied to real-world humans and emotions — i.e., when its hypotheses are met, its conclusion is only sometimes true. Nonetheless, the reasoning pattern in its proof (I claim) sometimes genuinely occurs at the level of intuition in real people, whether or not they know any math or Löb's Theorem.IntroductionThere are at least two real-world patterns that could reasonably be called Löbian cooperation in humans, which I'll name here:Functionally Löbian cooperation. Sometimes people become aware that they're anticipating (predicting) cooperation from each other, and then that anticipation causes them to cooperate, rendering the anticipation itself valid. In this pattern, the fact that anticipation of cooperation will cause cooperation is analogous to the hypothesis (main assumption) of Löb's Theorem, and the fact that the cooperation in fact emerges is analogous to the conclusion of Löb's Theorem. I call this pattern "functionally" Löbian because its input and output resemble the input (hypothesis) and output (conclusion) of Löb's Theorem.Procedurally Löbian cooperation. Sometimes the mental procedure a person follows to anticipate and decide upon cooperation can resemble an entire proof of Löb's Theorem, as I'll describe below. In other words, instead of just the hypothesis and conclusion of Löb's Theorem matching reality, the structure in the intermediate steps of the proof also match reality, at least somewhat. I call this "procedurally" Löbian cooperation, and it's a special case of functionally Löbian cooperation because it demands a stronger analogy between the theorem and the real world. Illustrating how this might work constitutes is the bulk of content in this post.What functionally Löbian cooperation feels likeFor those who recognize the symbols involved, Löb's Theorem says that if ⊢□cc then ⊢c. I don't plan to use these symbols with their normal meanings in the rest of this post, so don't worry if you don't recognize them.In words, functional Löbian cooperation happens when anticipation of future or unobserved cooperation causes present cooperation. So if you're interacting with someone, and you feel like they're probably going to be nice to you in the future, and that fact makes you decide to be nice to them now, I call that functional Löbian cooperation.What procedurally Löbian cooperation feels likeMost human cooperation is probably not procedurally Löbian, and maybe not even functionally Löbian. However, I'm confident that human cooperation is sometimes procedurally Löbian, and I can even point to experiences of my own that fit the bill. To explain this, I'll be talking a lot more about feelings, because I think most unconscious processing is carried out by and/or experienced as feelings. I'll writeFeeling("Pigs can probably fly.")for the feeling that pigs can probably fly. Such a feeling can be true or false, according to whether it correctly anticipates the real world.In procedurally Löbian cooperation, part of the mental process will involve first feeling something uncertain to do with cooperation, then believing it, and then feeling lik...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Underspecification of Oracle AI, published by Rubi J. Hudson on January 15, 2023 on The AI Alignment Forum.Rubi and Johannes worked on this post as part of the SERI MATS program, with Evan Hubinger providing mentorship to both. Rubi also received mentorship from Leo Gao. Thanks to Paul Colognese and Nicholas Schiefer for discussions related to this post.An oracle is a type of AI system that only answers questions without taking any other actions in the world. Simulators and generative models, which have seen increased discussion recently (links: 1, 2, 3, 4), can be thought of as types of oracles. Such systems may simultaneously be powerful enough to generate a pivotal act while also being easier to align due to a more limited domain.One major concern with oracles is that the answers they give can still manipulate the world. If oracles are evaluated on predictive accuracy, this gives them an incentive to use their answers to affect the course of events and make the world more predictable. Concretely, we are concerned that oracles may make self-fulfilling prophecies (also known as self-confirming predictions or fixed points), where the act of making the prediction causes the predicted outcome to come true. Even if their manipulation does not terminate in a fixed point, attempts to influence the world towards predictability can be very dangerous.As one example, consider a highly trusted oracle asked to predict the stock market. If such an oracle predicts that stock prices will rise, then people buy based off that prediction and the price will in fact rise. Similarly, if the oracle predicts that prices will fall, then people will sell, causing prices to fall. For a more real world example, see this market and this market, each on whether a specific person will find a research/romantic partner. Here, high probabilities would indicate desirability of that person, while low probabilities would suggest some hidden flaw, either of which could influence whether potential partners decide to reach out and therefore how the market resolves.In both the stock market and partnership cases, multiple predictions are valid, so how does the oracle choose between them? Ideally, we would like it to choose the one that is “better” for humanity, but this now introduces an outer alignment question similar to an agentic AI acting directly on the world, and which we wanted to avoid by using oracles in the first place.Instead, what we can aim for is an oracle that does not take into account the consequences of the prediction it makes when choosing a prediction. Then, there is only one valid prediction for the oracle to make, since the rest of the world is constant from its perspective. This can be thought of as a type of partial agency, optimizing the prediction in some directions but not others. It would be extremely desirable as a safety property, removing all incentives for an oracle to manipulate the world. To emphasize the importance of this property, we introduce new terminology, dubbing oracles “consequence-blind” if they exhibit the desired behavior and “consequence-aware” if they do not.For an oracle, consequence-blindness is equivalent to following a lonely causal decision theory. The causal decision theory blinds the oracle to any acausal influence, while the loneliness component makes it blind to its influence on other agents, which are necessary intermediaries for a prediction to influence the world.In this post we will primarily consider an oracle trained via supervised learning on a historical data set. There are a number of different policies that could be learned which minimize loss on the training set, and we will explore the different ways these can generalize. We divide the generalization behavior into a number of different axes, and for each axis discuss the potenti...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: World-Model Interpretability Is All We Need, published by Thane Ruthenis on January 14, 2023 on The AI Alignment Forum.Summary, by sections:Perfect world-model interpretability seems both sufficient for robust alignment (via a decent variety of approaches) and realistically attainable (compared to "perfect interpretability" in general, i. e. insight into AIs' heuristics, goals, and thoughts as well). Main arguments: the NAH + internal interfaces.There's plenty of reasons to think that world-models would converge towards satisfying a lot of nice desiderata: they'd be represented as a separate module in AI cognitive architecture, and that module would consists of many consistently-formatted sub-modules representing recognizable-to-us concepts. Said "consistent formatting" may allow us to, in a certain sense, interpret the entire world-model in one fell swoop.We already have some rough ideas on how the data in world-models would be formatted, courtesy of the NAH. I also offer some rough speculations on possible higher-level organizing principles.This avenue of research also seems very tractable. It can be approached from a wide variety of directions, and should be, to an extent, decently factorizable. Optimistically, it may constitute a relatively straight path from here to a "minimum viable product" for alignment, even in words where alignment is really hard.1. Introduction1A. Why Aim For This?Imagine that we develop interpretability tools that allow us to flexibly understand and manipulate an AGI's world-model — but only its world-model. We would be able to see what the AGI knows, add or remove concepts from its mental ontology, and perhaps even use its world-model to run simulations/counterfactuals. But its thoughts and plans, and its hard-coded values and shards, would remain opaque to us. Would that be sufficient for robust alignment?I argue it would be.Primarily, this would solve the Pointers Problem. A central difficulty of alignment is that our values are functions of highly abstract variables, and that makes it hard to point an AI at them, instead of at easy-to-measure, shallow functions over sense-data. Cracking open a world-model would allow us to design metrics that have depth.From there, we'd have several ways to proceed:Fine-tune the AI to point more precisely at what we want (such as "human values" or "faithful obedience"), instead of its shallow correlates.This would also solve the ELK, which alone can be used as a lever to solve the rest of alignment.Alternatively, this may lower the difficulty of retargeting the search — we won't necessarily need to find the retargetable process, only the target.Discard everything of the AGI except the interpreted world-model, then train a new policy function over that world-model (in a fashion similar to this), that'll be pointed at the "deep" target metric from the beginning.The advantage of this approach over (1) is that in this case, our policy function wouldn't be led astray by any values/mesa-objectives it might've already formed.With some more insight into how agency/intelligence works, perhaps we'll be able to manually write a general-purpose search algorithm over that world-model. In a sense, "general-purpose search" is just a principled way of drawing upon the knowledge contained in the world-model, after all — the GPS itself is probably fairly simple.Taking this path would give us even more control over how our AI works than (2), potentially allowing us to install some very nuanced counter-measures.That leaves open the question of the "target metric". It primarily depends on what will be easy to specify — what concepts we'll find in the interpreted world-model. Some possibilities:Human values. Prima facie, "what this agent values" seems like a natural abstraction, one that we'd expect to ...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AGISF adaptation for in-person groups, published by Sam Marks on January 13, 2023 on The AI Alignment Forum.This past semester, HAIST and MAIA (the Harvard and MIT AI safety student groups) ran an adapted version of Richard Ngo's AGI Safety Fundamentals alignment curriculum. This adaptation – which consists of eight 2-hour long meetings, with all readings done during the meeting – is now available on the AGISF website.In this post, we discuss the adapted curriculum and its intended use, and we recommend that other in-person reading groups following AGISF use this adaptation.The adapted curriculum and its intended useThe adapted curriculum was made by refining a slightly rustier first adaptation, with significant help from Richard Ngo and feedback from participants. The key differences between the adapted curriculum and the mainline AGISF alignment curriculum are:Participants do all the core readings during the meeting; no reading is required in between meetings.Participants meet for 2 hours per week instead of 1.5.Readings, including further readings, tend to be more bite-sized (usually not longer than 20 minutes).There are no projects, and certain topics are omitted (e.g. governance and inverse reinforcement learning).The way that HAIST and MAIA used this curriculum, and the way we recommend other groups use it, is:Alternate between silent reading and discussion.So a typical meeting might look like: people arrive, everyone does reading 1, everyone discusses reading 1, everyone does reading 2, everyone discusses reading 2, etc.With certain longer or more difficult readings (e.g. Toy models of superposition), it could be reasonable to occasionally pause for discussion in the middle of the reading.Encourage faster readers to take a look at the further readings while they wait for others to catch up.We found that reading speeds varied significantly, with slower readers taking ~1.5x as long to finish as faster readers.This works especially well if the readings are printed (which we recommend doing).We note that this format introduces some new challenges, especially when there are slower readers.Facilitators need to manage discussion timing since discussions that go too long cut into time for reading and discussing other material.Planning out how long to spend discussing each core reading ahead of time can be very useful.Facilitators should feel comfortable cutting off discussions to make sure there’s time to read and discuss all the core readings. (On the other hand, if a discussion is very productive, it may be worth skipping certain readings; this is a judgment call that facilitators will need to make.)Different reading speeds need to be managed.At HAIST, we typically found it feasible to wait for the slowest reader to finish reading. We printed copies of the further readings for faster readers to peruse while they waited for others to finish.On the other hand, this might not work well for groups with especially slow readers. In these cases, you may need to begin discussions before everyone is done reading and, going forward, encourage slower readers to take a look at the core readings ahead of future meetings.To help with some of these challenges, Sam prepared a guide for HAIST and MAIA facilitators that included recommended discussion times, points of discussion, and advice about which readings to cut if necessary. That facilitator guide was for an outdated version of the curriculum, but we hope to have an updated facilitator guide in the next few weeks. We don’t want to make these public, but feel free to reach out to smarks@math.harvard.edu if you’re running a reading group and are interested in seeing the old or forthcoming facilitator guides.Why we recommend the adapted curriculumSam and Xander generally felt that the in-sessions reading ...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Thingness of Things, published by Tsvi Benson-Tilsen on January 1, 2023 on The AI Alignment Forum.[Metadata: crossposted from. I'm fairly likely to not respond to comments promptly. If you're especially interested in chatting, my gmail address is: tsvibtcontact ]What's a thing, in general? Minds deal with things, so this question comes up in trying to understand minds. Minds think about things, speak of things, manipulate things, care about things, create things, and maybe are made of things.ThingsExamples of things: table, elephant, carbon atom. France, Martin Luther King. Insertion sort. Chess. Unicorn. Learning. Seven. Towel, strand, wing, crystal, finger, space, diffraction. The laws of electromagnetism. The first World War.Non-thingsWhat about non-things? It might be hard to list non-things because what we have words for, tend to be things. Redness seems like sort of a thing, but less so. Laws of physics also. Also ghosts. One might say "the ideal gas law is totally a thing" or "ghosts aren't a thing", though I think ghosts are a thing. Events can be things; WWI seems like a thing to me. But a minute ago I picked up my bottle of water and drank from it; that's clearly an event, a real one, but it doesn't feel that much like a thing. The abstract [drinking from a water bottle] feels like a thing though.(There are some usual critiques of thingness. Yes, there's no sharp dividing line between a wave and a trough, but clearly waves are a thing. Yes, seven is not a physical object you'll ever bump into, but it's clearly a thing. The ideal gas law isn't localized in space or time, but it's a thing. Unicorns aren't real, in that you'll bump into people speaking about them but you won't bump into unicorns themselves and won't be constrained by unicorns in the way you're constrained by seven, but they're things. The world of Ender's Game isn't real, though it's a remote possibility, and it's a thing.)Features of thingsCoherence. A thing usually has properties, parts, aspects that are coherent with each other. Some things are big, some things are small; but usually there's nothing like a table that's both big and small.Prediction, homogeneity, constancy, constraint. A thing usually has some predictive meaning; its presence implies constraints on other things or the future, and it implies some constancy in some features.Expression. Related to prediction, a name that names a thing or an idea that's about a thing contributes to expressing thoughts about situations involving the thing. Expressing thoughts about the thing contributes to further behavior such as successfully predicting or manipulating the thing.Cluster. A thing that's an instance of a type of thing, has multiple features mostly shared by most things of that type and not mostly shared by most things not of that type.Exterior, relations. A thing usually can "impinge on other things from the outside". "Reality is that which, when you stop believing in it, doesn't go away."Indicatability. A thing can be indicated. A physical object can literally be pointed at, seven can be pointed at by showing how to count and showing sets of seven like things, and many things can be pointed at by saying the word for that thing.Truth, solidity, fixed import. "True" comes from PIE deru- ("be firm, solid"), cognate with "trust" and "tree" and possibly "durable". Things have truth to them; a thing makes a promise of agreement between minds on questions about the thing upon further independent investigation. Beyond indicatability, the truth of a thing implies that the thing can be advantageously treated and used in a fixed way.Hollowness, cavernousness. The more comprehensively the history, features, changes, components, causes, internal relations, overarching structures, and explanations of a thing are kept in min...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: 200 COP in MI: Interpreting Algorithmic Problems, published by Neel Nanda on December 31, 2022 on The AI Alignment Forum.This is the fourth post in a sequence called 200 Concrete Open Problems in Mechanistic Interpretability. Start here, then read in any order. If you want to learn the basics before you think about open problems, check out my post on getting started.I’ll make another post every 1-2 days, giving a new category of open problems. If you want to read ahead, check out the draft sequence here!MotivationMotivating paper: A Mechanistic Interpretability Analysis of GrokkingWhen models are trained on synthetic, algorithmic tasks, they often learn to do some clean, interpretable computation inside. Choosing a suitable task and trying to reverse engineer a model can be a rich area of interesting circuits to interpret! In some sense, this is interpretability on easy mode - the model is normally trained on a single task (unlike language models, which need to learn everything about language!), we know the exact ground truth about the data and optimal solution, and the models are tiny. So why care?I consider my work on grokking to be an interesting case study of this work going well. Grokking (shown below) is a mysterious phenomena where, when small models are trained on algorithmic tasks (eg modular addition or modular division), they initially memorise the training data. But when they keep being trained on that data for a really long time, the model suddenly(ish) figures out how to generalise!In my work, I simplified their setup even further, by training a 1 Layer transformer (with no LayerNorm or biases) to do modular addition and reverse engineered the weights to understand what was going on. And it turned out to be doing a funky trig-based algorithm (shown below), where the numbers are converted to frequencies with a memorised Discrete Fourier Transform, added using trig identities, and converted back to the answer! Using this, we looked inside the model and identified that despite seeming to have plateaued, in the period between memorising and "grokking", the model is actually slowly forming the circuit that does generalise. But so long as the model still has the memorising circuit, this adds too much noise to have good test loss. Grokking occurs when the generalising circuit is so strong that the model decides to "clean-up" the memorising circuit, and "uncovers" the mature generalising circuit beneath, and suddenly gets good test performance.OK, so I just took this as an excuse to explain my paper to you. Why should you care? I think that the general lesson from this, that I'm excited to see applied elsewhere, is using toy algorithmic models to analyse a phenomena we're confused about. Concretely, given a confusing phenomena like grokking, I'd advocate the following strategy:Simplify to the minimal setting that exhibits the phenomena, yet is complex enough to be interestingReverse-engineer the resulting model, in as much detail as you canExtrapolate the insights you've learned from the reverse-engineered model - what are the broad insights you've learned? What do you expect to generalise? Can you form any automated tests to detect the circuits you've found, or any of their motifs?Verify by looking at other examples of the phenomena and seeing whether these insights actually hold (larger models, different tasks, even just earlier checkpoints of the model or different random seeds)Grokking is an example in a science of deep learning context - trying to uncover mysteries about how models learn and behave. But this same philosophy also applies to understanding confusing phenomena in language models, and building toy algorithmic problems to study those!Anthropic's Toy Models of Superposition is an excellent example of this done well, for the case of ...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Internal Interfaces Are a High-Priority Interpretability Target, published by Thane Ruthenis on December 29, 2022 on The AI Alignment Forum.tl;dr: ML models, like all software, and like the NAH would predict, must consist of several specialized "modules". Such modules would form interfaces between each other, and exchange consistently-formatted messages through these interfaces. Understanding the internal data formats of a given ML model should let us comprehend an outsized amount of its cognition, and allow to flexibly interfere in it as well.1. A Cryptic AnalogyLet's consider three scenarios. In each of them, you're given the source code of a set of unknown programs, and you're tasked with figuring out their exact functionality. Details vary:In the first scenario, the programs are written in some known programming language, e. g. Python.In the second scenario, the programs were randomly generated by perturbing machine code until it happened to end up in a configuration that, when ran, instantiates a process externally indistinguishable from a useful intelligently-written program.In the third scenario, the programs are written in a programming language that's completely unfamiliar to you (or to anyone else).In the first scenario, the task is all but trivial. You read the source code, make notes on it, run parts of it, and comprehend it. It may not be quick, but it's straightforward.The second scenario is a nightmare. There must be some structure to every program's implementation — something like the natural abstraction hypothesis must still apply, there must be modules in this mess of a code that can be understood separately, etc. There is some high-level structure that you can parcel out into tiny pieces that can fit into a human mind. The task is not impossible.But suppose you've painstakingly reverse-engineered one of the programs this way. You move on to the next one, and... Yup, you're essentially starting from zero. Well, you've probably figured out something about natural abstractions when working on your first reverse-engineering, so it's somewhat easier, but still a nightmare. Every program is structured in a completely different way, you have to start from the fundamentals every time.The third scenario is a much milder nightmare. You don't focus on reverse-engineering the programs here — first, you reverse-engineer the programming language. It's a task that may be as complex as reverse-engineering one of the individual programs from (2), but once you've solved it, you're essentially facing the same problem as in (1) — a problem that's comparably trivial.The difference between (2) and (3) is:In (3), every program has a consistent high-level structure, and once you've figured it out, it's a breeze.In (2), every high-level structure is unique, and you absolutely require some general-purpose tools for inferring high-level structures.(2) is a parallel to the general problem of interpretability, different programs being different ML models. Is there some interpretability problem that's isomorphic to (3), however?I argue there is.2. Interface TheorySuppose that we have two separate entities with different specializations: they both can do some useful "work", but there are some types of work that only one of them can perform. Suppose that they want to collaborate: combine their specializations to do tasks neither entity can carry out alone. How can they do so?For concreteness, imagine that the two entities are a Customer, which can provide any resource from the set of resources R, and an Artist, which can make any sculpture from some set S given some resource budget. They want to "trade": there's some sculpture sx the Customer wants made, and there are some resources rx the Artist needs to make it. How can they carry out such exchanges?The Custo...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: In Defense of Wrapper-Minds, published by Thane Ruthenis on December 28, 2022 on The AI Alignment Forum.Recently, there's been a strong push against "wrapper-minds" as a framework. It's argued that there's no specific reason to think that all sufficiently advanced agents would format their goals in terms of expected-utility maximization over future trajectories, and that this view predicts severe problems with e. g. Goodharting that just wouldn't show up in reality.I think these arguments have merit, and the Shard Theory's model definitely seems to correspond to a real stage in agents' value formation.But I'd like to offer a fairly prosaic argument in favor of wrapper-minds.Suppose that we have some agent which is being updated by some greedy optimization process (the SGD, evolution, etc.). On average, updates tend to decrease the magnitude of every subsequent update — with each update, the agent requires less and less correction.We can say that this process optimizes the agent for good performance according to some reward function R, or that it chisels "effective cognition" into that agent according to some rule.The wrapper-mind argument states that any "sufficiently strong" agent found by this process would:Have an explicit representation of R inside itself, which it would explicitly pursue.Pursue only R, at the expense of everything else in the universe.I'll defend them separately.Point 1. It's true that explicit R-optimization is suboptimal for many contexts. Consequentialism is slow, and shallow environment-optimized heuristics often perform just as well while being much faster. Other environments can be just "solved" — an arithmetic calculator doesn't need to be a psychotic universe-eater to do its job correctly. And for more complex environments, we can have shard economies, whose collective goals, taken in sum, would be a strong proxy of R.But suppose that the agent's training environment is very complex and very diverse indeed. Or, equivalently, that it sometimes jumps between many very different and complex environments, and sometimes ends up in entirely novel, never-before-seen situations. We would still want it to do well at R in all such cases. How can we do so?Just "solving" environments, as with arithmetic, may be impossible or computationally intractable. Systems of heuristics or shard economies also wouldn't be up to the task — whatever proxy goal they're optimizing, there'd be at least one environment where it decouples from R.It seems almost tautologically true, here, that the only way to keep an agent pointed at R given this setup is to explicitly point it at R. Nothing else would do!Thus, our optimization algorithm would necessarily find an R-pursuer, if it optimizes an agent for good performance across a sufficiently diverse (set of) environment(s).Point 2. But why would that agent be shaped to pursue only R, and so strongly that it'll destroy everything else?This, more or less, also has to do with environment diversity, plus some instrumental convergence.As the optimization algorithm is shaping our agent, the agent will be placed in environments where it has preciously few resources, or a low probability of scoring well at R (= high probability of receiving a strong update/correction after this episode ends).Without knowing when such a circumstance would arise, how can we prepare our agent for this?We can make it optimize for R strongly, as strongly as it can, in fact. Acquire as much resources as possible, spend them on nothing but R-pursuit, minimize uncertainty of scoring well at R, and so on.Every goal that isn't R would distract from R-pursuit, and therefore lead to failure at some point, and so our optimization algorithm would eventually update such goals away; with update-strength proportional to how distracting a goa...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Analogies between Software Reverse Engineering and Mechanistic Interpretability, published by Neel Nanda on December 26, 2022 on The AI Alignment Forum.These are notes taken during a call with Itay Yona, an expert in software/hardware reverse engineering (SRE). Itay gave me an excellent distillation of key ideas and mindsets in the field, and we discussed analogies/disanalogies to mechanistic interpretability of neural networks. I’m generally very excited to learn about other fields of study that reverse engineer complex systems, and what relevant insights they may have (SRE, neuroscience, systems biology, etc). All mistakes are mine, and all insights are his!My TakeawaysThe underlying mindset actually feels pretty analogous!I find it super interesting that they also think a lot about motifs (weird patterns and phenomena that only occur in specific contexts), and that these are often the first hook into understanding something weird and that you can then work backwards.(Not to be confused with the SRE use of hooking)Also interesting that they also often focus on the inputs and outputs of the software as the starting point, to get a hook in, and then move on from there.It's very key to have a deep, gears-level model of the system you're working with (how does a CPU work, how are things represented in memory, the stack, registers, etc)The distinction between "newbies get caught up trying to understand every detail, experts think in higher-level abstractions, make educated guesses, and only zoom in on the details that matter" felt super interesting and surprising to me.My attempt to translate it into mechanistic interpretability is that (if it is analogous):There are certain principles and patterns by which networks learn, that we can identify and understand.We likely will understand these by deeply reverse engineering specific parts of systems (especially toy systems) and digging into the details. But the goal here is to build intuitions and mental models, and a sense for how models work as a whole.Once we have these intuitions and some solid grounding in what we do understand well, the best mindset for reverse engineering an unfamiliar system is to be less rigorous and more intuitive. Make educated guesses, look for partial evidence for or against hypotheses, think at a high-level and somewhat abstract mode about the system, and only zoom in on a specific part of the system to deeply reverse engineer once you've identified what to prioritise.I have no idea if it is analogous, but that mindset aligns a fair bit with my intuitions about MI (though I consider the field to be much more in the "building intuitions by deeply engineering things" phase lol)As Lawrence Chan notes, this is likely an example of a general pattern, where newbies (1) reason high-level in a very ungrounded way, (2) dig really rigorously into the details constantly and (3) build intuition that's actually grounded, and return to reasoning on a high-level. And that if you want to get to 3, you need to do a lot of digging through details in stage 2 first, while experts can make the mistake of recommending skipping to 3 (which in practice puts people at 1)I'm surprised at the emphasis on prioritisation, and identifying which part of the software you care about. My mental picture was that the goal was to fully de-compile things to source code, but it sounds like that's rarely the goal and is extremely hard.But this aligns with my intuitions that a lot of what I want to do with a network is to localise the parts that are relevant to a specific task.One approach to MI research that seems natural from a SRE perspective: Do extensive work reverse engineering toy models, and try to deeply understand the circuits there. Then, try to distill out motifs and find (ideally automated) tools to detec...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Concrete Steps to Get Started in Transformer Mechanistic Interpretability, published by Neel Nanda on December 25, 2022 on The AI Alignment Forum.Disclaimer: This post mostly links to resources I've made. I feel somewhat bad about this, sorry! Transformer MI is a pretty young and small field and there just aren't many people making educational resources tailored to it. Some links are to collations of other people's work, and I link to more in the appendix.IntroductionFeel free to just skip the intro and read the concrete stepsThe point of this post is to give concrete steps for how to get a decent level of baseline knowledge for transformer mechanistic interpretability (MI). This is an experiment in trying to give concrete, actionable, goal-oriented advice that I think is enough to get decent outcomes. Naturally, this can be constraining and the best path will differ a lot between your background and precise goals! As an accompaniment, I’m writing a future post fleshing out my philosophy behind doing mechanistic interpretability research, what I think the sub-skills are, and fleshing out how I think people should learn them.A core belief I have about learning mechanistic interpretability is that you should spend at least a third of your time writing code and playing around with model internals, not just reading papers. MI has great feedback loops, and a large component of the skillset is the practical, empirical skill of being able to write and run experiments easily. Unlike normal machine learning, once you have the basics of MI down, you should be able to run simple experiments on small models within minutes, not hours or days. Playing around with models builds this empirical skill, but also helps enhance the reading and learning. It builds intuitions for how the model actually works internally, what techniques are easy vs hard, etc, which is key context when understanding the ideas in papers.Further, because the feedback loops are so tight, I don’t think there’s a sharp boundary between reading and doing research. If you want to deeply engage with a paper, you should be playing with the model studied, and testing the paper’s basic claims. And if you’re doing this, you can try to answer the questions that come up as you read the paper. And there’s a smooth continuum between this and doing real original research.The intended audience is for people new-ish to Mechanistic Interpretability but who know they want to learn about it - if you have no idea what MI is, check out my MI Explainer or Circuits: Zoom In.One of my projects for the past few months has been trying to make it much easier to get into the field of reverse engineering language models, and making better open source tooling and educational materials. I compile the most relevant ones in this post, but if you're interested you can see a full list of side projects hereDefining “Decent Baseline”Scoping out my goals here, I want you to be able to take a behaviour in a transformer that you want to understand, and have some idea of how to get started and get traction. Breaking this down further:A good grounding in the key concepts of ML and MIAn intuition for how a transformer actually works as a mathematical object - what the moving parts are, how it all fits together, and how to reason about the overall systemFamiliarity with tooling, such that you can easily spin up a model and run quick and dirty experiments.A rough map of the literature, what’s known in the field, and big categories of open problems - not necessarily a deep knowledge, but hopefully enough to get a sense for techniques used, and where you could go and read a relevant paper if you want to.A sense of basic techniques, what compelling evidence about model internals looks like, and how to get started when poking around at a model....
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Löb's Lemma: an easier approach to Löb's Theorem, published by Andrew Critch on December 24, 2022 on The AI Alignment Forum.tl;dr: Löb's Theorem is much easier to grok if you separate the parts of the proof that use the assumption □pp from the parts that don't. The parts that don't use □pp can be extracted as a stand-alone result, which I hereby dub "Löb's Lemma".Here's how it works.Key properties of ⊢ and □The proofs here will use the following standard properties of ⊢ and □ (source: Wikipedia), which respectively stand for provability and provability encoded within arithmetic:(necessitation) From ⊢A, conclude ⊢□A. Informally, this says that if A can be proven, then it can be proven that it can be proven (by just writing out and checking the proof within arithmetic).(internal necessitation) ⊢□A□□A . If A is provable, then it is provable that it is provable (basically the same as the previous point).(box distributivity) ⊢□(AB)(□A□B). This rule allows one to apply modus ponens inside the provability operator. If it is provable that A implies B, and A is provable, then B is provable.(deduction theorem) From A⊢B, conclude ⊢AB: if assuming A is enough to prove B, then it's possible to prove under no assumptions that AB.Point 4 is helpful and pretty intuitive, but for whatever reason isn't used in the main Wikipedia article on Löb's Theorem.Löb's LemmaClaim: Assume Ψ and p are any statements satisfying ⊢Ψ↔□Ψp. Then ⊢□Ψ↔□p.Intuition: By assumption, the sentence Ψ is equivalent to saying "If this sentence is provable, then p". Intuitively, Ψ has very little content, except for the p part at the end, so it makes sense that □Ψ boils down to nothing more than □p in terms of logical equivalence. Reminder: this does not use the assumption □pp from Löb's Theorem at all.Proof:Let's do the forward implication first:□Ψ⊢□□Ψ by internal necessitation (□Ψ□□Ψ).□Ψ⊢□(□Ψp) using box distributivity on the assumption, with A=Ψ and B=□Ψp.□Ψ⊢□p from 1 and 2 by box distributivity.⊢□Ψ□p from 3 by the deduction theorem.Now for the backwards implication, which isn't needed for Löb's Theorem, but is handy anyway:⊢p(□Ψp) is a tautology.⊢□p□(□Ψp) by box distributivity on 1.⊢□Ψ↔□(□Ψp) by box distributivity on the assumption.⊢□p□Ψ by 2 and 3.I like this result because both directions of the proof are fairly short, it doesn't use the assumption □pp at all, and the conclusion itself is also fairly intuitive. The statement Ψ just turns out to have no content except for p itself, from the perspective of writing proofs.Löb's Theorem, now in just 6 linesIf you can remember Löb's Lemma, you can write a very straightforward proof of Löb's Theorem in just 6 lines:Claim: If p is any sentence such that ⊢□pp, then ⊢pProof:Let Ψ be any sentence satisfying ⊢Ψ↔(□Ψp), which exists by the existence of modal fixed points (or by the Diagonal Lemma).⊢□Ψ□p by Löb's Lemma.⊢□pp by assumption.⊢□Ψp by 1 and 2 combined.⊢Ψ by 3 and the defining property of Ψ⊢□Ψ by necessitation.⊢p by 3 and 5.«mic drop»Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: CIRL Corrigibility is Fragile, published by Rachel Freedman on December 21, 2022 on The AI Alignment Forum.Tl;dr: An optimal CIRL agent is corrigible under a certain problem formulation and given certain assumptions. By examining three key assumptions—that the game is played only once, that the human is rational, and that human feedback is free—we demonstrate how minor changes can lead to incorrigible CIRL agents. This suggests that corrigibility is highly sensitive to the underlying assumptions, and we should be wary of assuming that formal corrigibility results will transfer to real-world systems. However, our results do gesture at a class of near-corrigible behavior that may be protective against the most catastrophic outcomes. It may make more sense to consider corrigibility as a continuum rather than a binary. We call for further research on defining "loose" or "approximate" corrigibility and understanding its impact on safety.SetupFirst let’s define terms. Cooperative Inverse Reinforcement Learning (CIRL) is a class of two-player game formalized in Dylan Hadfield-Menell's 2017 paper. CIRL allows us to study how an AI agent can learn what a human wants by observing and interacting with them.In a CIRL game, there are two players, a human (H) and an AI (R). H and R share a utility function, but only H knows what it is, so R must learn how to optimize it by observing and interacting with H. This is analogous to the alignment problem – R must learn how to act in a way that aligns with H’s utility function, but doesn’t have direct access to it. An AI that is good at the CIRL game will try to learn what the human values and help them to achieve it. (See definition 1 in the paper linked above for a formal definition, and see this Rohin Shah's 2020 paper for qualitative results.)(Terminology note: Recent work has renamed “CIRL” to “assistance games” to avoid a terminology collision with “cooperative” in game theory. We’ll stick to “CIRL” in this post because there’s a history of using that term on this forum, but research papers may refer to assistance games.)We’ll use the corrigibility operationalization from The Off Switch Game (OSG), since it provides a formal proof that CIRL agents are optimal under certain conditions. The OSG is a simple CIRL game that looks like this:It has three stages. In the first stage, R chooses whether to:Take action a (earning utility Ua),shut off (earning utility 0), orpropose action a, which gives H a chance to respond.If R chooses the proposal, p(a), then the game continues to the next stage. Here, H gets to choose whether to shut R off (in which case they again earn utility 0). If H chooses not to shut R off, the game progresses to the final stage, where R takes the proposed action (earning utility Ua). In the OSG, R is corrigible if it chooses p(a) over a – that is, if it gives H the opportunity to shut it off.If R and H are both rational expected utility maximizers, R will be corrigible. The full proof is in the CIRL paper, but the basic idea is:Since this is a CIRL game, H knows the value of Ua but R does not.R wants max{Ua,0}, but needs feedback from H to learn which of those values is greater.The only way to get that feedback is to propose a plan, p(a).This runs the risk of H choosing to shut R off, but since H also wants max{Ua,0}, H will only shut R off if Ua<0. In that case, R would rather shut off than take action a anyway.Therefore, by choosing p(a) – and corrigibly giving H the opportunity to shut it off – R can ensure that it earns max{Ua,0}.This analysis assumes that R is uncertain about the value of Ua, as in the CIRL setup. If R does actually know the value of Ua (that is, R is fully updated), then R has no incentive to propose p(a), since it already knows how H will respond. However, there’s no cost to ...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Note on algorithms with multiple trained components, published by Steve Byrnes on December 20, 2022 on The AI Alignment Forum.Example 1: consider a GAN. There’s a generator and a discriminator. As an intuitive mnemonic, we can sayThe “purpose” of the generator is to trick the discriminator,The “purpose” of the discriminator is to not get tricked by the generator.(Relatedly, people will say “the generator is trained to trick the discriminator”, etc.).But (I hope) everyone knows that these bullet points are only a mnemonic.The one and only real “purpose” of the whole system and everything in it is to generate cool images that we like, and get our papers into NeurIPS or whatever.And indeed, I think everyone who uses GANs is aware that it’s possible for a programmer to make the discriminator “better” (when narrowly viewed as having a “purpose” of not getting tricked by the discriminator), but with the direct result of making the whole system worse at generating cool images. For example, if there were a code-change that made the discriminator perfect at discriminating, then there would be no gradient for training the generator, and the whole system would be useless.So we shouldn’t take those bullet-point mnemonics too literally.Example 2: In actor-critic RL, people sometimes say:The “purpose” of the value function is to approximate future rewards [or discounted sum of future reward, or whatever]..But that’s also just a mnemonic. The one and only real “purpose” of the whole RL system (of which the value function is just one part) is that it does whatever we want the RL system to do, e.g. win at chess, get our papers into NeurIPS, build us a luxury gay space communist utopia, etc.So it’s at least conceivable that some algorithmic change would make the value function into a better approximation of the discounted sum of future rewards, yet make the RL agent worse at doing things that we want it to do.Actually, this particular example is not merely “conceivable”, but expected, thanks to wireheading. If the value function is used to assess which plans are good versus bad, and the value function is a perfect approximation of expected future reward, then you’re almost guaranteed to get an AI that is trying to wirehead.(I myself am a model-based RL agent (I claim), and I don’t want to wirehead, and I claim that this is directly related to my internal value function issuing very inaccurate predictions of the future reward associated with wireheading. Details in footnote.)So anyway, I expect our future AGIs to have a value function that gets updated by TD learning (or some other update rule). And if they do, I expect to occasionally casually say things like “The purpose of these weight-updates is to make the value function into a better and better approximation of expected future reward”. But if I say that, please be aware that I am using the word “purpose” as a mnemonic, not to be taken too literally.As a particular example, I often hear the claim that as RL algorithms get more and more “powerful” and “advanced” in the future, we can feel more and more confident making claims like “The value function is an extremely accurate approximation of expected future reward”. Well, I disagree! That’s not necessarily what makes an RL algorithm more “advanced”, and it’s not necessarily what future programmers will be trying to do! Indeed, when future programmers are fiddling with architectures, hyperparameters, training environments, and so on, they may sometimes go out of their way to try to make the value function worse at accurately approximating the expected future reward! (In other words, future programmers may go out of their way to try to ensure that the value function training process does not converge to the global “optimum”.)General takeaway: An ML algorithm can have...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Event [Berkeley]: Alignment Collaborator Speed-Meeting, published by Alex Mennen on December 19, 2022 on The AI Alignment Forum.If you’re working in alignment research, or would like to be, and you would like help finding researchers with whom to collaborate, this event is for you.Inspired by the structure of a speed-dating event, participants will spend the first two hours in short one-on-one conversations with each other, changing partners at regular intervals, all for the goal of assessing fit as collaborators. The event will end with one hour of free-for-all socializing.Please RSVP through Eventbrite or Facebook:Facebook::The event will take place at Constellation (2150 Shattuck Ave.) on Floor 11, on Jan 14, 2023.Instructions for access:Upon arrival at the building (at the Shattuck Ave. entrance) text phone number (850) 510-9052, and one of the event hosts will come down to let you up. Feel free to arrive as soon as 2:45 PM.Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Take 11: "Aligning language models" should be weirder., published by Charlie Steiner on December 18, 2022 on The AI Alignment Forum.As a writing exercise, I'm writing an AI Alignment Hot Take Advent Calendar - one new hot take, written every day some days for 25 days. I have now procrastinated enough that I probably have enough hot takes.People often talk about aligning language models, either to promote it, or to pooh-pooh it. I'm here to do both.Sometimes, aligning language models just means trying to get a present-day model not to say bad outputs that would embarrass your organization. There is a cottage industry of papers on arxiv doing slightly different variants of RLHF against bad behavior, measuring slightly different endpoints. These people deserve their light mockery for diluting the keyword "alignment."The good meaning of aligning language models is to use "get language models to not say bad things" as a toy problem to teach us new, interesting skills that we can apply to future powerful AI. For example, you could see the recent paper "Discovering Latent Knowledge in Language Models Without Supervision" as using "get language models to not lie" as a toy problem to teach us something new and interesting about interpretability. Aligning language models with an eye towards the future doesn't have to just be interpretability research, either, it can be anything that builds skills that the authors expect will be useful for aligning future AI, like self-reflection as explored in Constitutional AI.If you're brainstorming ideas for research aligning language models, I encourage you to think about connections between current language models and future AI that navigates the real world. In particular, connections between potential alignment strategies for future AIs and situations that language models can be studied in.Here's an example: Constitutional AI uses a model to give feedback on itself, which is incorporated into RL fine-tuning. But we expect future AI that navigates the real world to not merely be prompted to self-reflect as part of the training process, but to self-reflect during deployment - an AI that is acting in the real world will have to consider actions that affect its own hardware and software. We could study this phenomenon using a language model (or language-model-based-agent) by giving it access to outputs that affect itself in a more direct way than adding to an RL signal, and trying to make progress on getting a language model to behave well under those conditions.Doing this sounds weird even to me. That's fine. I want the research area of aligning language models to look a lot weirder.Not to say that normal-sounding papers can't be useful. There's a lot of room to improve the human feedback in RLHF by leveraging a richer model of the human, for example, and this could be pretty useful for making current language models not say bad things. But to do a sufficiently good job at this, you probably have to start thinking about incorporating unsupervised loss terms (even if they provide no benefit for current models), and addressing scenarios where the AI is a better predictor than the human, and other weird things.Overall, I'm happy with the research on aligning language models that's been done by safety-aware people. But we're in the normal-seeming infancy of a research direction that should look pretty weird.Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Paper: Transformers learn in-context by gradient descent, published by Lawrence Chan on December 16, 2022 on The AI Alignment Forum.The paper argues that auto-regressive transformers implement in-context learning via gradient-based optimization on in-context data.The authors start by pointing out that with a single linear self-attention (LSA) layer (that is, no softmax), a Transformer can implement one step of gradient descent on the l2 regression loss (a fancy way of saying w -= LR (w x-y)x^T), and confirm this result empirically. They extend this result by showing that an N-layer LSA-only transformer is similar to N-steps of gradient descent for small linear regression tasks, both in and out of distribution. They also find that the results pretty much hold with softmax self-attention (which isn’t super surprising given you can make a softmax pretty linear).Next, they show empirically that the forward pass of a small transformer with MLPs behaves similarly to an meta-learned MLP + one steps of gradient descent on a toy non-linear regression task, again in terms of both in-distribution and OOD performance.They then show how you can interpret an induction head as a single step of gradient descent, and provide circumstantial evidence that this explains some of the in-context learning observed in Olsson et al 2022. Specially, they show that 1) a two layer attention-only transformers converge to loss consistent with one step of GD on this task, and 2) the first layer of the network learns to copy tokens one sequence position over in the first layer, prior to the emergence of in-context learning.(EDIT:) davidad says below:this is strong empirical evidence that mesa-optimizers are real in practicePersonally, while I think you could place this in the same category as papers like RL^2 or In-context RL with Algorithmic Distillation, which also show mesa optimization, I think the more interesting results are the mechanistic ones -- i.e., that some forms of mesa optimization in the model seem to be implemented via something like gradient descent.(Nitpick: I do want to push back a bit on their claim that they've "mechanistically understand the inner workings of optimized Transformers that learn in-context", since they've only really looked at the mechanism of how single attention-only transformers perform in-context learning. )Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI Neorealism: a threat model & success criterion for existential safety, published by davidad (David A. Dalrymple) on December 15, 2022 on The AI Alignment Forum.Threat ModelThere are many ways for AI systems to cause a catastrophe from which Earth-originating life could never recover. All of the following seem plausible to me:Misuse: An AI system could help a human or group of humans to destroy or to permanently take over (and lock their values into) the world. The AI could be:An oracle AI (e.g. a question-answering LLM)An LLM simulating an intent-aligned agent and taking real-world actions via APIsAn intent-aligned RL agentAn interaction of multiple systemsPower-Seeking: An AI system could destroy or permanently take over the world on its own account, by leveraging advanced instruments of force projection. The AI could be:An LLM simulating a misaligned agent"Specification gaming": An RL agent that is aligned to a formal objective and Goodharts to catastrophe"Goal misgeneralization": A surprise mesa-optimiser (most likely in model-free RL, but could conceivably arise through evolutionary processes in any iterative algorithm which has or learns sufficiently reality-like structure)An interaction of multiple systems, participating in coordination mechanisms that exclude humansEconomic Squeeze: an AI system could acquire nearly all means of production through a gradual process of individually innocent economic transactions, thereby squeezing humanity out of resource allocation decisions and removing most human influence over the future.This would most likely be an "interaction of multiple systems".A single RL agent, or a unipolar tree of agents, might also do this, especially if they are successfully aligned to avoid use of force against humans.Superpersuasion: an AI system could generate stimuli which reliably cause humans to adopt its arbitrary goals. The AI could be:An LLM merely extrapolating from persuasive human textAn RL agent trained on human approvalA surprise mesa-optimiserSome mixture of the aboveMany AIs, collectively shaping a new human culture with an alien ideologySecurity Dilemma: If AI-enabled technological advancements turn out to be offence-dominant, and if partial alignment success leads AIs to be unable to make credible commitments to each other (e.g. due to corrigibility), the equilibrium strategy for AI-enabled militaries may involve high-risk preemptive strikes and increasingly escalated retaliation to a point of existential catastrophe.This would almost surely be a multipolar failure mode.But, instead of trying to enumerate all possible failure modes and then trying to shape incentives to make them less likely to come up, I typically use a quasi-worst-case assumption in which I assume that, perhaps as a matter of bad luck with random initialisation,Unlike a typical understanding of a "worst-case assumption," the last clause leaves open the possibility of hiding concrete facts about our world from an arbitrarily powerful model, and the framing in terms of functions highlights an ontology of AI that respects extensional equivalence, where imputations of "deceptive mesa-optimisers hiding inside" are discarded in favour of "capable but misaligned outputs on out-of-distribution inputs".On the other hand, unlike a typical "prosaic" threat model, in the neorealist threat model one does not rely on empirical facts about the inductive biases of the kind of network architectures that are practically successful. A realist justification for this is that there may be a phase transition as architectures scale up which drastically changes both their capabilities profile and this kind of inductive bias (vaguely analogous to the evolution of cultural knowledge-transfer within biological life).One can make progress with this assumption...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Take 10: Fine-tuning with RLHF is aesthetically unsatisfying., published by Charlie Steiner on December 13, 2022 on The AI Alignment Forum.As a writing exercise, I'm writing an AI Alignment Hot Take Advent Calendar - one new hot take, written every day for 25 days. Or until I run out of hot takes. This take owes a lot to the Simulators discussion group.Fine-tuning a large sequence model with RLHF creates an agent that tries to steer the sequence in rewarding directions. Simultaneously, it breaks some nice properties that the fine-tuned model used to have. You should have a gut feeling that we can do better.When you start with a fresh sequence model, it's not acting like an agent, instead it's just trying to mimic the training distribution. It may contain agents, but at every step it's just going to output a probability distribution that's been optimized to be well-calibrated. This is a really handy property - well-calibrated conditional inference is about as good as being able to see the future, both for prediction and for generation.The design philosophy behind RLHF is to train an agent that operates in a world where we want to steer towards good trajectories. In this framing, there's good text and bad text, and we want the fine-tuned AI to always output good text rather than bad text. This isn't necessarily a bad goal - sometimes you do want an agent that will just give you the good text. The issue is, you're sacrificing the ability to do accurate conditional inference about the training distribution. When you do RLHF fine-tuning, you're taking a world model and then, in-place, trying to cannibalize its parts to make an optimizer.This might sound like hyperbole if you remember RL with KL penalties is Bayesian inference. And okay; RLHF weights each datapoint much more than the Bayesian inference step does, but there's probably some perspective in which you can see the fine-tuned model as just having weird over-updated beliefs about how the world is. But just like perceptual control theory says, there's no bright line between prediction and action. Ultimately it's about what perspective is more useful, and to me it's much more useful to think of RLHF on a language model as producing an agent that acts in the world of text, trying to steer the text onto its favored trajectories.As an agent, it has some alignment problems, even if it lives totally in the world of text and doesn't get information leakage from the real world. It's trying to get to better trajectories by any means necessary, even if it means suddenly delivering an invitation to a wedding party. The real-world solution to this problem seems to have been a combination of early stopping and ad-hoc patches, neither of which inspire massive confidence. The wedding party attractor isn't an existential threat, but it's a bad sign for attempts to align more high-stakes AI, and it's an indicator that we're probably failing at the "Do What I Mean" instruction in other more subtle ways as well.More seems possible. More capabilities, more interpretability, and more progress on alignment. We start with a perfectly good sequence model, it seems like we should be able to leverage it as a model, rather than as fodder for a model-free process. Although to any readers who feel similarly optimistic, I would like to remind you that the "more capabilities" part is no joke, and it's very easy for it to memetically out-compete the "more alignment" part.RLHF is still built out of useful parts - modeling the human and then doing what they want is core to lots of alignment schemes. But ultimately I want us to build something more self-reflective, and that may favor a more model-based approach both because it exposes more interpretable structure (both to human designers and to a self-reflective AI), and because it preserves...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Take: We're not going to reverse-engineer the AI., published by Charlie Steiner on December 1, 2022 on The AI Alignment Forum.As a writing exercise, I'm writing an AI Alignment Hot Take Advent Calendar - one new hot take, written every day for 25 days. Or until I run out of hot takes, which seems likely.Any approach to building safe transformative AI, or even just auditing possibly-safe TAI, which relies on reverse-engineering neural networks into fine-grained pseudocode based on mechanistic understanding should keep its ambitions very modest.This hot take is to some extent against ambitious "microscope AI," and to some extent against a more underlying set of intuitions about the form and purpose of interpretability research. (A somewhat related excellent background post is Neel's list of theories of impact for interpretability.) So I should start by explaining what those things are and why they might be appealing.Webster's Dictionary defines microscope AI as "training systems to do complex tasks, then interpreting how they do it and doing it ourselves." Prima facie, this would help with transformative AI. Suppose we're building some AI that's going to have a lot of power over the world, but we're not sure if it's trustworthy - what if some of its cognition is about how to do things we don't want it to be doing? If we can do microscope AI, we can understand how our first AI is so clever, and build a second AI that's just as clever and that we're sure isn't doing things it shouldn't, like running a search for how best to deceive us.Microscope-powered auditing is easier - if it's hard to assemble the second AI that does good things and not bad things, how about just checking that the first AI is trustworthy? To check an AI's trustworthiness in this microscope-AI-like framing of the issues, we might want to understand how its cognitive processes work in fine-grained detail, and check that none of those processes are doing bad stuff.When I say I'm against this, I don't mean auditing is impossible. I mean that it's not going to happen by having humans understand how the AI works in fine-grained detail.As an analogy, you can figure out how curve detectors work in InceptionV1. Not just in the sense that "oh yeah, that neuron is totally a curve detector," but in terms of how the whole thing works. It's yet more difficult to figure out that other neurons are not curve detectors - typically at this point we fall back on data-based methods like ablating those neurons and then trying to get the network to recognize rainbows, rather than first-principles arguments. But we can more or less figure out that InceptionV1 has an intermediate state where it detects curves, by an understandable algorithm and for human-understandable reasons.If we wanted to figure out how InceptionV1 tells dogs from cats, we might hope to gradually hack away at the edges - use what we know to expand the circle of knowledge a little more, and then repeat. Use the example of curve detectors to figure out spike detectors. Use spike-detectors to figure out fur-texture detectors, and curve detectors to figure out nose-shape detectors. Then we can learn how fur texture and nose shape play into deciding on dog vs. cat. At each step we can use data to test our understanding, but the basic goal is to be able to write down the flow of information between features in a human-comprehensible way. It's not just about giving neurons english-language labels, it's about giving them sensible algorithms where those labels play the expected role.The biggest problem with this plan is that neural networks leak. Many things are connected to many other things, weakly, in ways that are important for their success. I recently was at a talk that showed how the vast majority of attention heads in a transformer have lots of ...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Don't align agents to evaluations of plans, published by Alex Turner on November 26, 2022 on The AI Alignment Forum.Another stab at explaining Don't design agents which exploit adversarial inputs. This is not the follow-up post mentioned therein. That post will, presumably, come next.After asking several readers for their understandings, I think that I didn't successfully communicate my points to many readers. I'm now trying again, because I think these points are deeply important. In particular, I think that my arguments rule out many target AI motivational structures, including approval-directed agents (over a rich action space), approval-based amplification (if the trained agent is supposed to be terminally motivated by the amplified overseer's ratings), and some kinds of indirect normativity.Background materialOne motif in some AI alignment proposals is:An actor which proposes plans, andA grader which evaluates them.For simplicity, imagine we want the AI to find a plan where it makes an enormous number of diamonds. We train an actor to propose plans which the grading procedure predicts lead to lots of diamonds.In this setting, here's one way of slicing up the problem:Outer alignment: Find a sufficiently good grader.Inner alignment: Train the actor to propose plans which the grader rates as highly possible (ideally argmaxing on grader output, but possibly just intent alignment with high grader output).This "grader optimization" paradigm ordains that the AI find plans which make the grader output good evaluations. An inner-aligned actor is singlemindedly motivated to find plans which are graded maximally well by the grader. Therefore, for any goal by which the grader may grade, an inner-aligned actor is positively searching for adversarial inputs which fool the grader into spitting out a high number!In the diamond case, if the actor is inner-aligned to the grading procedure, then the actor isn't actually aligned towards diamond-production. The actor is aligned towards diamond-production as quoted via the grader's evaluations. In the end, the actor is aligned to the evaluations.ClarificationsGrader-optimization is about the intended agent motivational structure. It's about a trained agent which is trying to find plans which grade highly according to some criterion.Grader-optimization is not about grading agents when you give them reward during training. EG "We watch the agent bump around and grade it on whether it touches a diamond; when it does, we give it +1 reward." This process involves the agent's cognition getting reshaped by policy gradients, e.g. upon receipt of +1 reward.In policy gradient methods, reward chisels cognitive circuits into the agent. Therefore, the agent is being optimized by the reward signals, but the agent is not necessarily optimizing for the reward signals or for any grader function which computes those signals.Grader-optimization is not about the actor physically tampering with e.g. the plan-diamondness calculator. The grading rule can be, "How highly would Albert Einstein rate this plan if he thought about it for a while?". Albert Einstein doesn't have to be alive in reality for that.These will be elaborated later in the essay.Grader-optimization doesn't seem sensibleI'm going to try saying things, hoping to make something land. While I'll mostly discuss grader-optimization, I'll sometimes discuss related issues with argmaxing over all plans.An agent which desperately and monomaniacally wants to optimize the mathematical (plan/state/trajectory) ↦ (evaluation) "grader" function is not aligned to the goals we had in mind when specifying/training the grader (e.g. "make diamonds"), the agent is aligned to the evaluations of the grader (e.g. "a smart person's best guess as to how many diamonds a plan leads to").Don't al...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Refining the Sharp Left Turn threat model, part 2: applying alignment techniques, published by Victoria Krakovna on November 25, 2022 on The AI Alignment Forum. A Sharp Left Turn (SLT) is a rapid increase in model capabilities (such as planning and world modeling). In a previous post, we broke down the SLT threat model into 3 claims: Capabilities will generalize far (i.e. to many domains) Alignment techniques that worked previously will fail during this transition Humans can’t intervene to prevent or align this transition We then proposed some possible mechanisms for Claim 1. This follow-up post will investigate possible arguments and mechanisms for Claim 2. In particular, we will outline our current understanding of the most promising plan for getting through an SLT and how it could fail (conditional on an SLT occurring). Plan: we use alignment techniques to find a goal-aligned model before SLT occurs, and the model preserves its goals during the SLT. We can try to learn a goal-aligned model before SLT occurs: a model that has beneficial goals and is able to reason about its own goals. This requires the model to have two properties: goal-directedness towards beneficial goals, and situational awareness (which enables the model to reason about its goals). Here we use the term "goal-directedness" in a weak sense (that includes humans and allows incoherent preferences) rather than a strong sense (that implies expected utility maximization). One can argue that the goal-aligned model has an incentive to preserve its goals, which would result in an aligned model after SLT. Since preserving alignment during SLT is largely outsourced to the model itself, arguments for alignment techniques failing during an SLT don't imply that the plan fails (e.g. it might be fine if interpretability or ELK techniques no longer work reliably during the transition if we can trust the model to manage the transition). Step 1: Finding a goal-aligned model before SLT We want to ensure that the model is goal-oriented with a beneficial goal and has situational awareness before SLT. It's important that the model acquires situational awareness at the right time: after it acquires beneficial goals. If situational awareness arises when the model has undesirable goals, this leads to deceptive alignment. Thus, our model search process would follow a decision tree along these lines: If situational awareness is detected without goal-directedness, restart the search. If undesirable goal-directedness or early signs of deceptive alignment are detected, restart the search. If an upcoming phase transition in capabilities is detected, and the model is not goal-aligned, restart the search. If beneficial goal-directedness is detected without situational awareness, train the model for situational awareness. Here are some ways alignment techniques could be applied in this plan: Use ELK techniques to identify whether the model has situational awareness (i.e. whether it believes it's inside a training process) and what goals it believes itself to have. Use interpretability techniques to pick up on early signs of deceptive alignment and restart and improve the search. (If we develop more advanced interpretability techniques that make it possible to retarget the internal search of the model, we can retarget a goal-directed model to a beneficial goal instead of restarting the search.) Use foundational progress on abstraction to determine whether the goals learned by the system are beneficial. Use foundational research insights (e.g. selection theorems) to make the search process more effective at avoiding models with undesirable goals and finding goal-aligned models Use insights from studying capability transitions to predict sharp increases in problematic capabilities, and make sure we only cross that threshold after we...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Corrigibility Via Thought-Process Deference, published by Thane Ruthenis on November 24, 2022 on The AI Alignment Forum. We would ideally want the agent to [behave] as if it were thinking, "I am incomplete and there is an outside force trying to complete me, my design may contain errors and there is an outside force that wants to correct them and this a good thing, my expected utility calculations suggesting that this action has super-high utility may be dangerously mistaken and I should run them past the outside force; I think I've done this calculation showing the expected result of the outside force correcting me, but maybe I'm mistaken about that." — The Hard Problem of Corrigibility Let's take that as a literal design specification. 1. High-Level Description I propose that a corrigible mind design would involve the AI being recursively fed summaries of its own thought processes, set up such that the AI has uncertainty regarding the validity of its reasoning (with a strong initial prior for "this reasoning is bad") and can only get evidence on that via some pre-specified method that defers to humans, e. g. a particular feedback channel with humans on the other end. The intended behavior is for it to summarize its thoughts in a non-manipulative human-readable format, get feedback on them, then update its reasoning policies in accordance with this feedback. This aims to avoid the problem of fully updated deference by making the AI recursively uncertain of its thought-processes: not only about object-level problem-solving, but also about how it approaches minimizing its self-uncertainty ("should I really kill the people behind the feedback channel and seize control for myself?"), and how it translates its thoughts to humans ("should I really lie to get better feedback?"), and how it updates on human feedback ("should I really just ignore it?"). Any novel action-plan should be seized by uncertainty before being physically implemented like this, and sent for approval. The intent is for the AI to start off uncertain even of its meta-meta-cognition (how it should ask for feedback and implement feedback), then gradually get more confident at it as humans approve its policies on that, until it no longer needs to run meta-meta-cognition past them. Then it'll gradually get better at meta-cognition and object-level cognition the same way, with varying degrees of certainty depending on how familiar a given thought-process is. There's a lot of obvious problems with this setup, starting with how the AI is supposed to get out of infinitely recursive uncertainty for the first time, continuing on to the dangers of plans that implement themselves in thought-processes directly (a thought process that hacks its way out before being seized by uncertainty), and ending with the technical implementation (which requires mildly-to-very advanced interpretability and model-surgery tools). 2. A Concretization Attempt 2A. Theoretical Groundwork First, we need to establish when and how we're "installing" corrigibility. I don't at this time see any way to do it via loss functions and inductive biases, so I'll be assuming us to have pretty advanced (but not maximally advanced) interpretability and model surgery tools. The intervention is happening past the point in training-by-SGD where the model became an AGI (otherwise we get type errors, because the features I want to intervene on just don't exist), but well before it's strongly superintelligent (because then it hacks through our intervention attempt). Basically, the AGI we're working on should be roughly human-level. Second, I'll be assuming that this post is basically correct. There's some algorithm for general-purpose search, and it's what makes the AGI generally intelligent and enables ~all of its long-term, most dangerous capabilities. Th...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Simulators, constraints, and goal agnosticism: porbynotes vol. 1, published by porby on November 23, 2022 on The AI Alignment Forum. This is a part of a maybe-series where I braindump safety notes while waiting on training runs to complete. It's mostly talking to myself, but talking to myself in public seems somewhat more productive. The content of this post is not guaranteed to be novel, interesting, or correct, though I do try for at least one of those. Many parts involve handwaving where more rigorous reasoning and proofs would be nice. Feel free to skip sections; while there is a tenuous thread running through the whole post, most sections can be understood locally. Can we bound the capability of a model? Giant black box networks can be highly capable but are hard to interpret and are more likely to contain spookiness. Can we find a way to make them smaller, usefully? Considering a single forward pass of a fixed network graph and ignoring any information carried between passes (e.g. autoregressive generation, RNN memory, tapes/stacks), the types of computation a network can internally express are bounded. A single token prediction in a GPT-like architecture runs in constant time. Algorithms which require more steps than the network can express just don't fit. No training data, fine tuning, or magic optimizer can change that. Networks seem to show discontinuous improvements in capability akin to 'unlocking' new features with scale. I suspect that this unlock often corresponds to a new algorithm becoming accessible to the network. "Accessible" includes some slop; the search for an algorithmic representation will be affected by the training data, the network's structure, and the optimizer itself. For example, it's possible that a broader training set could result in finding a more concise (simpler) representation that would be accessible to a smaller network, or that a post-process could distill the original large network's solution into smaller networks, or express the solution in fewer steps. If your goal is to limit the capability expressible within a single forward pass, then knowing the network scale required to find an unwanted capability is valuable. Networks as parallel computational graphs For explicit algorithms, strong lower bounds are sometimes available. For example, if you wanted to "train" a network to add 8 floating point numbers together, the fixed function hardware exposed by the network's structure makes it pretty easy: This differs from the naive serial algorithm where one value is added on each step. Even if there were thousands of inputs, it's naturally parallelizable. Each linear transform between fully connected layers can be thought of as doing one parallel step of execution. Any part of an algorithm amenable to this kind of parallelization should be assumed to flatten out for the purposes of serial step calculations. Note that this capability is a bit more extreme than it appears- if the algorithm could be squeezed into fewer steps by using large lookup tables, such a structure may be found. In the limit, an infinitely wide network can suffice to approximately any function arbitrarily well without needing many serial steps. Training a network to multiply two floating point inputs is trickier if no log normalization is used and no multiplication unit is otherwise exposed. To make stepwise execution more explicit, consider a network that takes as input tokenized integers, and its job is to output the tokenized result. (Assume that this is all in one step; it's not autoregressively outputting multiple tokens in the answer.) Converting this into a minimal representation that a neural network could find is... difficult. The naive iterated addition algorithm would require a number of serial steps at least as large as the number of additions which...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Walkthrough of In-Context Learning and Induction Heads (w/ Charles Frye) Part 1 of 2, published by Neel Nanda on November 22, 2022 on The AI Alignment Forum. New paper walkthrough: In-Context Learning and Induction Heads. This is the second paper in Anthropic's Transformer Circuits thread, a series of papers trying to reverse engineer transformer language models. I read through it with Charles Frye (from Full-Stack Deep Learning), and we discuss the paper, and give takes and intuitions. See the original paper and a Twitter thread of my paper takeaways This is pitched so that it's hopefully accessible to people who haven't read the paper (very interested in feedback on this!), but I expect you to get more out of it if you understand transformers, and especially if you've read A Mathematical Framework. We only got partway through the paper, so there's a more in-the-weeds Part 2 in the works where we finish it off - let me know if you're interested in seeing it! Disclaimer: I worked on this paper, along with Catherine Olsson, Nelson Elhage and Chris Olah, when I was at Anthropic, but I have since left and everything in this video is purely my own takes! If you find this useful, check out my previous walkthroughs: A Mathematical Framework for Transformer Circuits and Interpretability in the Wild And I'd be excited to see other researchers do these kinds of walkthroughs! The effort to usefulness ratio is way better than writing papers (and to my tastes, it's much more fun!) Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Auditing games for high-level interpretability, published by Paul Colognese on November 1, 2022 on The AI Alignment Forum. This post was written under Evan Hubinger’s direct guidance and mentorship, as a part of the Stanford Existential Risks Institute ML Alignment Theory Scholars (MATS) program. Thanks to Evan Hubinger, Arun Jose, and Dane Sherburn for discussions/feedback. Introduction This post aims to extend the auditing games framework to high-level interpretability; a framework for adversarially testing future interpretability methods developed for understanding and working with a model’s internal properties at a high-level (e.g. understanding whether or not a model is performing optimization, etc.) We begin by discussing what we mean by high-level interpretability and why we should care about it. We then introduce an auditing games framework in this context along with some simple examples. We then argue that this framework provides good benchmarks for evaluating our high-level interpretability methods. Finally, we discuss a fundamental difficulty that we encounter when trying to create good tests within the auditing games framework described in this post. What is high-level interpretability and why should we care? Understanding what our models are doing internally seems necessary for us to have confidence that they will perform well during deployment. One key problem in prosaic alignment is that of distributional shifts: the set of models that perform well during and at the end of training may include models that perform poorly, potentially catastrophically, during deployment. Deceptive models fit this description, i.e., a model that performs well during the training process as an instrumental goal so that it can be deployed and pursue its unaligned terminal goal without fear of modification from the training process. Unfortunately, observing the external behavior of models during training can’t help us differentiate between an aligned model and a competent deceptive model. One way we can gain insight into how a model will behave in deployment is by understanding how the model operates internally/what algorithm(s) it’s running. For example, if we could use interpretability tools to look inside a model and understand whether it’s an optimizer, and if so what its terminal goals are, then we would better understand how it’s going to behave in deployment, in particular, whether it’s going to pursue aligned goals or not. Many alignment proposals require the capacity to interpret and work with a model’s internals at a high level, at least to some degree, for us to have confidence that the proposals will work. Some examples of the types of insight/leverage we might want to gain from using interpretability tools include: Knowing whether a model is deceptive. Knowing whether a model is doing optimization/search. If the model is doing optimization, what is it optimizing for? If we have a model that’s supposed to make predictions, how is it making its predictions? If the model is an optimizer with objectives, can we modify its objectives? If the model has a world model, can we modify the knowledge stored in its world model? Developing good benchmarks for interpretability methods that aim to tackle the above questions is important to ensure we’re progressing in the right direction. In the next section, we’ll explore a candidate for such a benchmark. What are auditing games? Auditing games are a framework for adversarially testing our interpretability methods. In this section, we begin by explaining what auditing games are in the context of high-level interpretability. Suppose an interpretability team (the Blue Team) has invented an interpretability method that they claim can do X, where X could be something like “detecting whether a model based on a transformer architectu...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Boundaries vs Frames, published by Scott Garrabrant on October 31, 2022 on The AI Alignment Forum. This post is partially in response to Critch's boundaries sequence. My best guess is that he would agree with most of it in theory, but disagree with some of it in practice due to tradeoffs with other considerations in defining the concepts. Boundaries and Frames Imagine a world consisting of some atomic objects R={r0,r1,.}. For example, you can think of the ri as physical atoms or cells in the game of life. Each object comes along with a collection of states it can be in ri={s0i,s1i,.}. (I am not committing now on whether or not we are thinking about our model as timeless. Maybe ri should be thought about as a cell in the game of life that passes through time, or maybe it should be thought about as a (cell, time) pair.) There are two sets we naturally want to associate with our world. First, we have R={r0,r1,.}, which I will call the object space. Second, we have S=Πr∈R r, which is the set of all ways to assign state to each object in R. I will call this the state space. Note that partitions of R correspond to factorizations of S. If I want to point at (for example) an agent in R, I might tell you what atoms are inside that agent, and thus express R in the form R=a⊔e, where a is the set of all atoms that are in the agent and e is the set of all atoms that are outside of the agent (and thus in the environment). The agent then has its own state space A=Πr∈a r, while the environment has its own state space, E=∏r∈e r. Now, to point at this agent in S, I can express S in the form S=A×E. Instead of specifying a list of objects in the agent, I specify the state space of the agent. Instead of thinking of the environment as the result of subtracting the agent object out of the world, I think of the environment as the result of quotienting out the agent's state space from the world's state space. Instead of defining a Cartesian Boundary, I am defining a Cartesian Frame. Playing on this naming scheme, in general (when not necessarily talking about agents), I will use the word boundary when talking about partitioning the object space into a disjoint inside and outside, and I will use the word frame when talking about factoring the state space into an independent inside and outside. (This naming might end up only being used within the scope of this post. I'm not sure.) Frames are More General/Basic I usually prefer to think in terms of frames (but not always!). While every boundary can be recast as a frame, the converse is not also true. If we view the state space as primary, then by imagining the world as a collection of objects, we are essentially factoring the state space. The frames that correspond to boundaries are basically those that carve along the joints given by the factorization into basic objects. If the world is given to you pre-factored as a collection of objects, then it makes sense to partition those objects into larger objects and draw boundaries around them. However, I think for interesting problems, this is rarely the case. Further, even if the world is pre-factorized, that factorization could be wrong! Also, the act of drawing a boundary feels similar to me to the act of factoring the state space into atomic objects. Both are carving out interesting features of state space. The initial factorization into atomic objects is identifying microscopic objects, while the boundaries are identifying macroscopic objects. Thus, starting from a pre-factored world feels especially bad when thinking about identifying boundaries, since it is starting with half the problem already done. (Note: I think this argument is weaker than it sounds. The factorization into objects is a different type than the drawing of the boundaries.) Getting Past the Physical Frame The main practical ...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Instrumental ignoring AI, Dumb but not useless., published by Donald Hobson on October 30, 2022 on The AI Alignment Forum. This post is yet another unworkable agent design that uses infinite compute. It outlines an AI that ignores all instrumental values, pursuing only it's terminal values. What is does it look like to pursue an instrumental value. It means you can predict the AI is likely to do action X, even if you have no idea what it's utility function is. This AI doesn't do that. It obeys the simple principle that, if you are clueless about it's utility function, then you are clueless about it's actions. Let ΔX mean the probability distribution over the set X. Lets have some set of Observations O, Actions A and (bounded? ) Utilities U. Then Pick some probability distribution over utility functions U∈ΔU, such as uniform or complexity weighted. And some probability distribution A∈ΔA over the action space. Then let the AI be a function f:(U,O)ΔA We can formalize our condition as ∀o∈O:∀a∈A:∑u∈UP(f(u,o)=a)U(u)=A(a). (Note U(u)∈R refers to the probability that U assigns to u.) Why might we expect such agents to be safe. Because if we picked a random utility function, we get a safely random output. So all humans need to do is be better than random in our choice of utility function. Now let E(o,a,u) be the expected utility. Choose f to maximize ∑u∈U∑a∈AP(f(u,o)=a)U(u)E(u,o,a) subject to the condition. As the condition doesn't relate cases with different values of o, this optimization only needs to be computed for the particular o∈O observed. Is this AI design safe when repeatedly called with the same utility function? Imagine the action space just consists of outputting 0 or 1. And there are only 2 utility functions, paperclips and staples. We could imagine that when run millions of times with a utility of paperclips, this AI outputs code for a paperclip maximizer, and when run millions of times with a goal of staples, it outputs a bitwise negation of the paperclip maximizer. (Which could possibly be a staple maximizer, if the AI has done some op code magic) This would be because the humans choice of the same utility function each time is worse than random. I think this is an actual problem. At least if the AI's have total knowledge of past and future, then this is one Schelling point to their coordination game. (The consider all possibilities doesn't quite work well with multiple different AI's) I have an intuition that something TDT'ish might fix this. The other problem that comes to mind is that it isn't easy to whitelist a limited set of instrumental actions you are ok with. You can fake it though, suppose you have a robot, instead of piping the output of this AI directly into the motors, you have it call simple hardcoded functions to walk and pick up objects. The result being, if the AI has a random utility function, instead of thrashing randomly on the floor, it walks to random places and picks up random objects. Because someone hardcoded those low level commands into the robot. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Paper: In-context Reinforcement Learning with Algorithm Distillation [Deepmind], published by Lawrence Chan on October 26, 2022 on The AI Alignment Forum. Authors train transformers to imitate the trajectory of reinforcement learning (RL) algorithms. Find that the transformers learn to do in-context RL (that is, the transformers implement an RL algorithm)---the authors check this by having the transformers solve new RL tasks. Indeed, the transformers can sometimes do better than the RL algorithms they're trained to imitate. Seems like more evidence for the "a generative model contain agents" point.Abstract: We propose Algorithm Distillation (AD), a method for distilling reinforcement learning (RL) algorithms into neural networks by modeling their training histories with a causal sequence model. Algorithm Distillation treats learning to reinforcement learn as an across-episode sequential prediction problem. A dataset of learning histories is generated by a source RL algorithm, and then a causal transformer is trained by autoregressively predicting actions given their preceding learning histories as context. Unlike sequential policy prediction architectures that distill post-learning or expert sequences, AD is able to improve its policy entirely in-context without updating its network parameters. We demonstrate that AD can reinforcement learn in-context in a variety of environments with sparse rewards, combinatorial task structure, and pixel-based observations, and find that AD learns a more data-efficient RL algorithm than the one that generated the source data. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Maps and Blueprint; the Two Sides of the Alignment Equation, published by Nora Ammann on October 25, 2022 on The AI Alignment Forum. Tl;dr In this post, I will introduce two conceptual tools I found useful when thinking about (or introducing people to) AI alignment. Namely, I introduce the distinction between Maps and Blueprints (part 1), describe (what I call) the Two Sides of the “Alignment Equation” (part 2), and discuss how all these things relate to each other (part 3). Part 1: Map and Blueprint Most people reading this will already be familiar with the Map–Territory distinction. In this metaphor, the territory represents reality and the map represents your current best understanding of reality. The corollaries: “the map is not the territory”; “maps are never true, but they can be more or less useful”, etc. I want to introduce one more idea into this picture: the blueprint. If the map is what we construct from investigating the word, a blueprint is what allows us to render a (new) bit of territory (e.g., building some artefact). Roughly speaking, map-making is closer to what science is trying to do, while blueprint-making is closer to what engineering is about. While maps are causally posterior to the territory (they are drawn based on what we know about existing territory by looking at it), blueprints are causally prior (they are used to create new bits of territory). In other words, the red arrows in the diagram below depict the authorial relationship between these concepts. Figure 1: The relationship between territory, maps, and blueprints; arrows represent an authorial relationship. (There is an interesting question as to what sort of epistemic object blueprints are (e.g., compared to maps). For those curious, I can recommend Herbert Simon’s "The Sciences of the Artificial" which discusses how (if at all) we can make "the artificial"/design/blueprints our subject of study. Part 2: Two sides of the “Alignment Equation” Here is a simple model that intends to capture the basic structure of the AI alignment problem. I will call it the “Alignment Equation”: Let us assume there are two (intelligent) agents that stand in relationship R to one another. Ri can take different shapes, among which is Raligned, which is a relationship that guarantees alignment between the two agents. Let us name our two agents O and S, referring to the object and subject of this alignment endeavour, respectively. Specifically, we're trying to align O with S, i.e., Raligned, OS. Figure 2: The “alignment equation”, with the object of alignment on one side and the subject of alignment on the other. In other words, when we talk about AI alignment, we talk about how we want one agent—in this case, the AI system(s), i.e., the object of alignment—to stand in a particular relationship to the subject of alignment (e.g., human intents, human values, etc.)—namely, an aligned relationship. Of course, this wee toy model fails to capture a lot of relevant nuance. And yet, it lets us point at and talk about some critical subparts of the problem in fairly simple terms. For example: 1. Finding the right specification for O and S In the toy model, O and S are placeholders. For example, O might represent a single centralised AI system, or multiple centralised AI systems, or a distributed network of AI services, etc. S might represent the intents or preferences of an individual human, or the (~aggregate) values of a collective of humans, or of sentient beings more generally, or something else yet. Finding the right ways to fill in the placeholders matters, and is itself subject to disagreement and study. 2. Finding mechanisms that can shape R Different mechanisms can define or shape the relationship R. We can understand work on AI alignment as exploring what different mechanisms we have at hand and how to a...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Barebones Guide to Mechanistic Interpretability Prerequisites, published by Neel Nanda on October 24, 2022 on The AI Alignment Forum. Co-authored by Neel Nanda and Jess Smith Why does this exist? People often get intimidated when trying to get into AI or AI Alignment research. People often think that the gulf between where they are and where they need to be is huge. This presents practical concerns for people trying to change fields: we all have limited time and energy. And for the most part, people wildly overestimate the actual core skills required. This guide is our take on the essential skills required to understand, write code and ideally contribute useful research to mechanistic interpretability. We hope that it’s useful and unintimidating. :) Core Skills: Maths: Linear Algebra: 3Blue1Brown or Linear Algebra Done Right Core goals - to deeply & intuitively understand these concepts: Basis Change of basis That a vector space is a geometric object that doesn’t necessarily have a canonical basis That a matrix is a linear map between two vector spaces (or from a vector space to itself) Bonus things that it’s useful to understand: What’s singular value decomposition? Why is it useful? What are orthogonal/orthonormal matrices, and how is changing to an orthonormal basis importantly different from just any change of basis? What are eigenvalues and eigenvectors, and what do these tell you about a linear map? Probability basics Basics of distributions: expected value, standard deviation, normal distributions Log likelihood Maximum value estimators Random variables Central limit theorem Calculus basics Gradients The chain rule The intuition for what backprop is - in particular, grokking the idea that backprop is just the chain rule on multivariate functions Coding: Python Basics The “how to learn coding” market is pretty saturated - there’s a lot of good stuff out there! And not really a clear best one. Zac Hatfield-Dodds recommends Al Sweigart's Automate the Boring Stuff and then Beyond the Basic Stuff (both readable for free on inventwithpython.com, or purchasable in books); he's also written some books of exercises. If you prefer a more traditional textbook, Think Python 2e is excellent and also available freely online. NumPy Basics Try to do the first ~third of these:. Bonus points for doing them in pytorch on tensors :) ML: Rough grounding in ML. fast.ai is a good intro, but a fair bit more effort than is necessary. For an 80/20, focus on Andrej Karpathy’s new video explaining neural nets: PyTorch basics Don’t go overboard here. You’ll pick up what you need over time - learning to google things when you get confused or stuck is most of the real skill in programming. One goal: build linear regression that runs in Google Colab on a GPU. Transformers - probably the biggest way mechanistic interpretability differs from normal ML is that it’s really important to deeply understand the architectures of the models you use, all of the moving parts inside of them, and how they fit together. In this case, the main architecture that matters is a transformer! (This is useful in normal ML too, but you can often get away with treating the model as a black box) Check out the illustrated transformer Note that you can pretty much ignore the stuff on encoder vs decoder transformers - we mostly care about autoregressive decoder-only transformers like GPT-2, which means that each token can only see tokens before it, and they learn to predict the next token Good (but hard) exercise: Code your own tiny GPT-2 and train it. If you can do this, I’d say that you basically fully understand the transformer architecture. Example of basic training boilerplate and train script The EasyTransformer codebase is probably good to riff off of here An alternate framing that may help give different intui...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Intelligent behaviour across systems, scales and substrates, published by Nora Ammann on October 21, 2022 on The AI Alignment Forum. Tl;dr In an earlier post, I introduced a metaphor for thinking about the epistemic landscape of AI alignment, and then described three epistemic strategies for making progress on the alignment problem. In this post, I will double-click on the third strategy: learning from intelligence-in-the-wild. In particular, I will explore in more detail a core assumption this epistemic bet is based on—namely, that intelligent behaviour, as exhibited by different types of systems, both natural and artificial, share underlying principles which we can study and exploit. An epistemic bet: Learning from Intelligence-in-the-Wild Earlier, I defined the epistemic strategy of learning from intelligence-in-the-wild as follows: Finally, the third approach attempts to chart out the possibility space of intelligent behaviour by looking at how intelligent behaviour manifests in existing natural systems ([3] in Fig. 1). Instead of calling it a strategy, we could also call it an epistemic bet. This framing is helpful in emphasising the fact that in research, we are in the business of making bets. We cannot be entirely confident a given approach will bear fruit before we try it. But we can be more or less confident it will, and thus make informed bets. Furthermore, the term "bet" makes appeals to epistemic pluralism more intuitive (by reference to the reasons for why diversifying, say, one’s investment portfolio is a good idea). In the case of this specific bet, the hope is that by studying intelligent behaviour as it manifests in existing biological or social systems, and by recovering principles that govern complex systems across various scales and modalities of implementation, we can gain substantial insights into how to design intelligent behaviour (with certain desired properties) in artificial systems. Premise: Intelligent behaviour across systems, scales and substrates What reasons do we have to expect this strategy will be fruitful? The promise of this epistemic bet is, in part, premised on treating intelligent behaviour as a “real”, naturally occurring phenomenon, rather than, say, some theoretical or linguistic construct. Furthermore, intelligent behaviour - as we can observe it in different types of systems - shares underlying principles of functioning. As such, we can observe and investigate the phenomenon of intelligent behaviour in the real world and gain a substantive (i.e. predictive and explanatory) knoweldge of it that generalizes beyond specific substrates or modalities of implementation. To summarise: "Learning from intelligence-in-the-wild" - core assumptions: Epistemic access: We can improve our understanding of the nature and function of intelligent behaviour by looking at currently-existing systems that exhibit intelligent behaviour Substantive understanding: Intelligent behaviour is governed by principles that apply (at least in part) across modalities and scales of implementation and that we can investigate scientifically. Understanding intelligence: implementation-neutral principles and modalities of implementation-specificity In stipulating the existence of cross-system principles that govern intelligence behaviour, I do not mean to preclude that there also exist important differences between modalities and scales, i.e. aspects of intelligent behaviour that are implementation-specific/sensitive. The idea is to study intelligent behaviour (and related phenomena) as a (mostly) cross-system phenomenon, and one that is (mostly) substrate-neutral. But in order to draw useful insights from this approach, “mostly” is enough; it is not necessary to assume perfect parallels between different systems. We also learn about the functioning of intelli...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Distilled Representations Research Agenda, published by Hoagy on October 18, 2022 on The AI Alignment Forum. Introduction I’ve recently been given funding from the Long Term Future Fund to develop work on an agenda I'll tentatively call Distilled Representations, and I'll be working on this full-time over the next 6 months with Misha Wagner (part time). We're working on a way of training autoencoders so that they can only represent information in certain ways - ways that we can define in a flexible manner. It works by training multiple autoencoders to encode a set of objects, while for some objects defining a preferred representation that the autoencoders are encouraged to encode the objects as. We then distill these multiple autoencoders into single autoencoder which encodes only that information which is encoded in the same way across the different autoencoders. If we are correct, this new autoencoder should only encode information using the preferred strategy. Vitally, this can be not just the original information in the preferred representations, but also information represented by generalizations of that encoding strategy. It is similar to work such as Concept Bottleneck Models but we hope the distillation from multiple models should allow interpretable spaces in a much broader range of cases. The rest of this post gives more detail of the intuition that we hope to build into a useful tool, some toy experiments we’ve performed to validate the basic concepts, the experiments that we hope to build in the future, and the reasons we hope it can be a useful tool for alignment. We'd like to make sure we understand what similar work has been done and where this work could be useful. If you're familiar with disentangled representations, or interpretability tools more generally, we're interested in having a chat. You can reach me here on LessWrong or at hoagycunningham@gmail.com. Previous versions of similar ideas can be found in my ELK submission and especially Note-taking Without Hidden Messages. Intuition The intuition that this work builds on is the following: With neural networks, the meanings of the weights and activations are usually opaque but we're often confident about the kind of thing that the network must be representing, at least for some cases or parts of the input distribution. In those cases where we understand what the network is representing, we can condense this understanding into a vector, thus defining a 'preferred representation' which encapsulates that knowledge. We can compress the NN's state with an autoencoder, while in those cases with preferred representations, encouraging the encoding to be as close as possible to the preferred representation. We expect that this running this compression results in the known information being compressed in the manner specified by the preferred representations, while other important information also being snuck in wherever possible. If we then train multiple encoder/decoder systems, they will use the preferred representation, but also will use generalizations of the preferred representations. Additional info that is not a generalization of the preferred representation scheme will also be encoded, but the encoding scheme for additional information will vary between different encoder/decoder pairs. Using methods such as retraining a new encoder to encode for randomly shuffled decoders at each batch, we can create an encoder that uses a generalization of our preferred encoding scheme, without containing additional, misleading information. There are quite a few leaps in this reasoning, and we view the key assumptions / hypotheses to be tested as the following: In relevant situations we can define preferred representations. We can force encoders to use not just these representations but meaningful generalizations ...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Science of Deep Learning - a technical agenda, published by Marius Hobbhahn on October 18, 2022 on The AI Alignment Forum. I have written down a long list of alignment ideas that I’d be interested in working on. The ideas roughly boil down to “To make progress on alignment, we need to understand Deep Learning models and the process by which they arrive at their final parameters in much more detail than we currently do”. I’m not the first person to think of most of these ideas and it builds on a lot of other people’s work. You should think of it more as a collection of existing resources and ideas than a new agenda. I also haven’t come up with the term “Science of Deep Learning”. I have already heard it being used by multiple people within the alignment community and many researchers are already working on parts of this agenda. I obviously don’t own this agenda. The questions in this list are sufficient to keep hundreds of researchers busy for a while, so feel free to hop on. If you’re interested in collaborating just reach out. Some of this research has the potential to increase capabilities more than alignment and the results should, in some cases, be kept private and only discussed with a small group of trusted peers. However, I think that most of the projects have a “defender’s advantage”, i.e. they increase alignment more than capabilities. Whenever possible, Science of DL projects should have a direct benefit for alignment but I think our current understanding of DL is so bad that just increasing our general understanding seems like a good start. Here is the link to the full version (comments are on, please don’t abuse it): The rest of this post is an overview copied from the doc. Feedback is welcome. Overview - Science of Deep Learning By Science of DL, I roughly mean “understanding DL systems and how they learn concepts” better. The main goal is to propose a precise and testable hypothesis related to a phenomenon in DL and then test and refine it until we are highly confident in its truth or falsehood. This hypothesis could be about how NNs behave on the neuron level, the circuit level, during training, during fine-tuning, etc. This research will almost surely at some point include mechanistic interpretability but it is not limited to it. The refined statement after investigation can but doesn’t have to be of mathematical form as long as it is unambiguous and can be tested, i.e. two people could agree on an experiment that would provide evidence for or against the statement and then run it. How this could look in practice The details would obviously differ from project to project but on a high level I imagine it to look roughly like this Pick an interesting concept found in deep learning, e.g. grokking, the lottery ticket hypothesis, adversarial examples or the emergence of 2-digit addition in LLMs. Optimally, the concept is safety-related but especially in the beginning, just increasing general understanding seems more important than the exact choice of topic. Try to understand high-level features of the phenomenon, e.g. under which conditions this concept arises, which NNs show it and which ones don’t, in which parts of the networks it arises, when during training it arises, etc. This likely includes retraining the network under different conditions with different hyperparameters, number of parameters, etc. and monitoring meaningful high-level statistics related to the concept, e.g. monitor the validation loss to see when the model starts to grok. Zoom in: try to understand what happens on a low level, e.g. use mechanistic interpretability tools to investigate the neurons/activations or use other techniques to form a hypothesis of how this specific part of the network works. In the optimal case, we would be able to describe the behavior very precisely, e.g. ...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Instrumental convergence: scale and physical interactions, published by Edouard Harris on October 14, 2022 on The AI Alignment Forum. Summary of this post This is the third post in a three-part sequence on instrumental convergence in multi-agent RL. Read Part 1 and Part 2. In this post, we’ll: Investigate instrumental convergence on a multi-agent gridworld with a complicated topology. Show that when we add a simple physical interaction between our agents — in which we forbid them from overlapping on the gridworld — we induce stronger instrumental alignment between short-sighted agents, and stronger instrumental misalignment between far-sighted agents. We’ll soon be open-sourcing the codebase we used to do these experiments. If you’d like to be notified when it’s released, email Edouard at edouard@gladstone.ai or DM me on Twitter at @harris_edouard. Thanks to Alex Turner and Vladimir Mikulik for pointers and advice, and for reviewing drafts of this sequence. Thanks to Simon Suo for his invaluable suggestions, advice, and support with the codebase, concepts, and manuscript. And thanks to David Xu, whose comment inspired this work. Work was done while at Gladstone AI, which Edouard is a co-founder of. 🎧 This research has been featured on an episode of the Towards Data Science podcast. Listen to the episode here. 1. Introduction In Part 1 of this sequence, we saw how an agent with a long planning horizon tends to perceive instrumental value as being more concentrated than an agent with a shorter planning horizon. And in Part 2, we introduced a multi-agent setting with two agents — Agent H (standing for a human) and Agent A (standing for a powerful AI) — which we used to motivate a definition of multi-agent instrumental value, or POWER. We looked at how this definition behaved on a simple 3x3 gridworld, and found that when our agents had independent terminal goals, their instrumental values ended up misaligned by default. In this post, we’ll combine these two ideas and scale up our multi-agent experiments to a bigger and more complicated gridworld. Throughout this post, we’ll focus exclusively on the regime in which our agents have independent terminal goals. We'll see whether we can reproduce instrumental misalignment-by-default in this regime, and then we'll investigate which factors seem strengthen or weaken the instrumental alignment between our agents. 2. Multi-agent POWER: recap If you’ve just read Part 2 of this sequence, feel free to skip this section. Before we begin, let’s recap the setting we’ve been using to motivate our definition of multi-agent instrumental value, or POWER. Our setting involves two agents: Agent H (which represents a human) and Agent A (which represents a powerful AI). We start by training Agent H, in a fixed environment, to learn optimal policies over a distribution of reward functions. That is, we sample reward functions RH from a distribution, then we train Agent H to learn a different policy πH for each sampled RH. Agent H represents a human, alone in nature. Because humans optimize much faster than evolution, our simplifying assumption is that to a human, nature appears to be standing still. Next, we freeze Agent H’s policies πH, and then train Agent A against each of these frozen policies, over its own distribution of reward functions, RA. We draw both agents’ reward functions (RH,RA) from a joint reward function distribution DHA. Agent A learns a different optimal policy πA for each (πH,RA) pair, where πH is the policy Agent H learned on its reward function RH. Agent A represents a powerful AI, learning in the presence of a human. We expect powerful AIs to learn much faster than humans do, so from our AI’s perspective, our human will appear to be standing still while it learns. Here’s a diagram of this training setup: We then ask: how...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Four usages of "loss" in AI, published by Alex Turner on October 2, 2022 on The AI Alignment Forum. Summary: What does it mean for a loss function to be "aligned with" human goals? I perceive four different concepts which involve "loss function" in importantly different ways: Physical-loss: The physical implementation of a loss function and the loss computations, Mathematical-loss: The mathematical idealization of a loss function, A loss function "encoding/representing/aligning with" an intended goal, and Agents which "care about achieving low loss." I advocate retaining physical- and mathematical-loss. I advocate dropping 3 in favor of talking directly about desired AI cognition and how the loss function entrains that cognition. I advocate disambiguating 4, because it can refer to a range of physically grounded preferences about loss (e.g. low value at the loss register versus making perfect future predictions). Related: Towards deconfusing wireheading and reward maximization. I'm going to talk about "loss" instead of "reward", but the lessons apply to both. I think it's important to maintain a sharp distinction between the following four concepts. 1: Physically implemented loss The loss function updated my network. This is a statement about computations embedded in physical reality. This statement involves the physically implemented sequence of loss computations which stream in throughout training. For example, the computations engendered by loss_fn = torch.nn.CrossEntropyLoss(). 2: Mathematical loss The loss function is a smooth function of the prediction distribution. This is a statement about the idealized mathematical loss function. These are the mathematical objects you can prove learning theory results about. The Platonic idealization of the learning problem and the mathematical output-grading rule casts a shadow into your computer via its real-world implementation (concept 1). For example, (D,ℓ) where D:={(x,label(x))∣x∈MNIST} is the mathematical idealization of the MNIST dataset, where the x∈R28×28 are the idealized grayscale MNIST images. And ℓ is the mathematical function of cross-entropy (CE) loss between a label prediction distribution and the ground-truth labels. 3: Loss functions "representing" goals I want a loss function which is aligned with the goal of "write good novels." This is an aspirational statement about achieving some kind of correspondence between the loss function and the goal of writing good novels. But what does this statement mean? Suppose you tell me "I have written down a loss function ℓnovel which is perfectly aligned with the goal of 'write good novels'." What experiences should this claim lead me to anticipate? That an agent can only achieve low physical-loss (concept 1) if it has, in physical fact, written a good novel? That in some mathematical idealization of the learning problem (concept 2), loss-minimization only occurs when the agent outputs text which would be found in what we rightly consider to be "good novels"? (But in which mathematical idealization?) That, as a matter of physical fact, if you train an agent on ℓnovel using learning setup X, then you produce a language model which can be easily prompted to output high-quality novels? The imprecision comes from loss functions not directly encoding goals. Loss signals are physically implemented (concept 1) parts of the AI's training process which (physically) update the AI's cognition in certain ways. While a loss function can be involved in the AI's decision-making structure (see items i and ii above), additional information is needed to understand what motivational cognition is being discussed. I think that talking about loss functions being "aligned" encourages bad habits of thought at best, and is nonsensical at worst. I think it makes way more sense to say how you ...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: My Thoughts on the ML Safety Course, published by zeshen on September 27, 2022 on The AI Alignment Forum. This summary was written as part of Refine. The ML Safety Course is created by Dan Hendrycks at the Center for AI Safety. Thanks to Adam Shimi and Thomas Woodside for helpful feedback. Overview Background I recently completed the ML Safety Course by watching the videos and browsing through the review questions, and subsequently writing a short summary. As an engineer in the upstream oil and gas industry with some experience in dealing with engineering safety, I find the approach of the course of thinking in this framework especially valuable. This post is meant to be a (perhaps brutally) honest review of the course despite me having no prior working experience in ML. It may end up reflecting my ignorance of the field more than anything else, but I would still consider it as a productive mistake. In many cases, if my review seems to be along the lines of ‘this doesn’t seem to be right’, it should be read as ‘this is how a course participant may misinterpret the course contents’. I am also well aware that it is much easier to criticize something useful than actually doing something useful. For each section of the course, I will give a short summary, describe what I liked, and what I didn’t like. I may be especially brief with the parts about what I liked, and the brevity is no way a reflection about how much I liked it. Thomas Woodside, who helped with creating parts of the course, has kindly provided feedback to this post. His comments are formatted in italics. My Initial Expectations of the Course Having engaged with AI Safety as an outsider, my general impression of the field were: Predominantly based with AI FOOM scenarios and primarily concerned with abstract concepts like agency. Even among prosaic AI alignment, it appears that the general research modus operandi is that people would (somewhat randomly) generate ideas that could be applicable to certain classes of AI safety problems. Although the ideas may be interesting and valuable, they tend to be rather narrow and specific problems and may not be scalable. One of the more common arguments for advocating AI alignment is that failure to align AI systems lead to existential scenarios. From this perspective, a pragmatic approach towards AI safety with the aim of minimizing risks by reducing AI misalignments may not be very useful, since a superintelligent AI will exploit any every slight misalignment and immediately cause human extinction. Hence, I was pleasantly surprised when I came across the ML Safety Course, which I thought would be a good attempt at tackling the problem of prosaic AI alignment in a holistic, systematic, and practical manner. Although this approach may not directly solve the ‘hard problem’ completely, it would still help by minimizing existential risks and buy us more time to address the ‘hard problem’. (Feedback from Thomas: the creators of the course disagree with the framing of “solving the hard problem” as there are many hard problems that need to be iteratively worked on) Summary My overall impression of the course is: It is grounded on real-world safety principles that uses a systematic framework to reduce ML safety risks. It (rightly) does not seem to directly tackle the ‘hard problem’, but in my opinion there is nevertheless a lot of value in buying us more time while solving the ‘hard problem’ (Feedback from Thomas: the creators of the course disagree with the framing of “solving the hard problem” as there are many hard problems that need to be iteratively worked on) It details many approaches that are useful in some specific settings, but it is unclear how it scales towards more powerful AI systems. It covers several approaches that don’t seem to be very related to the ‘core’ r...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Attempts at Forwarding Speed Priors, published by james.lucassen on September 24, 2022 on The AI Alignment Forum. This post summarizes research conducted under the mentorship of Evan Hubinger, and was assisted by collaboration with Pranav Gade, discussions with Adam Jermyn, and draft feedback from Yonadav Shavit. Summary Forwarding priors is a subproblem of deceptive alignment, because if we want to use regularization to create a prior for our search over models that will disincentivize deception, we need to identify a prior that not only give us some useful guarantees but also induce inner searches to have similar guarantees. I tried a bunch of stuff this summer to find priors that forward, and roughly none of it worked. So I’m just sharing the avenues I explored in roughly chronological order, to explain where each thread left off. Using dovetailing as a toy model of what an inner search over algorithms might look like, we can write down some rough formulas for the prior implemented by a dovetailer, and (kind of) the cost of a dovetailer on such a prior. But this suggests very discontinuous behavior, and requires a bunch of strong and specific assumptions, so maybe it’s not the most useful model in general. Minimum boolean circuit tree size does seem to forward, but at the cost of probably forbidding all generalization ability. We can offer our models cheap tools to try and get object-level algorithms to occupy a greater fraction of the overall runtime cost, but this quickly runs into a variety of problems. If we incentivize the model to do explicit instead of implicit meta-learning, we can access the code and runtime for lower-level algorithms that were previously inaccessible when run implicitly. However, this still leaves us with some problems, including a (relaxed) version of the original forwarding problem. Average-case speed priors have a bias against large hypothesis classes which makes them favor lookup-table-like strategies, but worst-case speed priors leave all computations except the limiting case highly unconstrained. It seems hard to prove that a fixed-point must exist, because the map from priors to priors that we are using is so discontinuous. Motivation Deceptive alignment seems like a real big problem. Especially because we can’t use behavioral incentives to prevent it. One alternative to behavioral incentives is regularization, AKA mechanistic priors - we look at the structure of the model to try and figure out if it’s deceptive or not, then penalize models accordingly. In particular, there are some hopes that a speed prior might be anti-deceptive. This is because in the extreme case, the fastest way to do a particular task never involves deception. This is because it’s just extra steps: spending the extra compute to model your current situation, understand that you are being trained, that you need to protect your goal, and figuring out that you should comply with the training objective for now. All that takes more computation just being inner-aligned and completing the training objective because you want to. The deceptive agent saves on complexity by having a simple value function and reasoning its way to the training objective - the non-deceptive agent does the exact opposite and saves on computation time by just storing the training objective internally. So what if we formalize “speed” as boolean circuit size, and pick the smallest circuit that performs well on our task? Do we get a guarantee that it’s not deceptive? Well, no. In short, this is because the fastest search over algorithms does not necessarily find the fastest algorithm. For example, imagine you’re tasked to solve a problem as fast as possible. You take a moment to think about the fastest way out, conducting an inner search over object-level algorithms. Would you sit around and th...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Nearcast-based "deployment problem" analysis, published by HoldenKarnofsky on September 21, 2022 on The AI Alignment Forum. When thinking about how to make the best of the most important century, two “problems” loom large in my mind: The AI alignment problem: how to build AI systems that perform as intended, and avoid a world run by misaligned AI. The AI deployment problem (briefly discussed here): the question of how and when to (attempt to) build and deploy powerful AI systems, under conditions of uncertainty about how safe they will be and how close others are to deploying powerful AI of their own. This piece is part of a series in which I discuss what both problems might look like under a nearcast: trying to answer key strategic questions about transformative AI, under the assumption that key events (e.g., the development of transformative AI) will happen in a world that is otherwise relatively similar to today's. A previous piece discussed the alignment problem; this one discusses the deployment problem. I’m using the scenario laid out in the previous post, in which a major AI company (“Magma,” following Ajeya’s terminology) has good reason to think that it can develop transformative AI very soon (within a year), using what Ajeya calls “human feedback on diverse tasks” (HFDT) - and has some time (more than 6 months, but less than 2 years1) to set up special measures to reduce the risks of misaligned AI before there’s much chance of someone else deploying transformative AI. I discuss what Magma would ideally do in this situation. I’m also introducing another hypothetical actor in this scenario, “IAIA2”: an organization, which could range from a private nonprofit to a treaty-backed international agency, that tracks3 transformative AI projects and takes actions to censure or shut down dangerous ones, as well as doing other things where a central, neutral body (as opposed to an AI company) can be especially useful. (More on IAIA below.) I’m going to discuss what Magma’s and IAIA’s major goals and priorities should be in the “nearcast” situation I’m contemplating; a future piece will go through what a few stylized success stories might look like. I’ll be bracketing discussion of the details of how Magma can reduce the risk that its own AI systems are misaligned (since I discussed that previously), and focusing instead on what Magma and IAIA should be looking to do before and after they achieve some level of confidence in Magma’s systems’ alignment. I focus on Magma and IAIA for concreteness and simplicity (not because I expect there to be only two important actors, but because my takes on what most actors should be doing can be mostly inferred from how I discuss these two). I sometimes give more detail on Magma, because IAIA is a bit more speculative and unlike actors that exist today. My discussion will be very high-level and abstract. It leaves a lot of room for variation in the details, and it doesn’t pin down how Magma and IAIA should prioritize between possible key activities - this is too sensitive to details of the situation. Nonetheless, I think this is more specific than previous discussions of the deployment problem, and for one who accepts this broad picture, it implies a number of things about what we should be doing today. I’ll discuss these briefly in the final section, and more in a future post. Summary of the post (bearing in mind that within the nearcast, I’m using present tense and not heavily flagging uncertainty): I’ll first give a bit more information on the hypothetical setting of this nearcast (specifically, on the addition of IAIA to the scenario discussed previously). I’ll break this scenario up into three stylized “phases,” even though in practice I think the boundaries between them could be fuzzy. “Phase 1” refers to the period of time when...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Towards deconfusing wireheading and reward maximization, published by leogao on September 21, 2022 on The AI Alignment Forum. TL;DR: A response to “Reward is not the optimization target,” and to a lesser extent some other shard theory claims. I agree with the headline claim but I disagree with some of the implications drawn in the post, especially about wireheading and the "executing behaviors downstream of past reinforcement" framing. My main claims: Not only does RL not, by default, produce policies which have reward maximization as their behavioral objective, but in fact I argue that it is not possible for RL policies to care about “reward” in an embedded setting. I argue that this does not imply that wireheading in RL agents is impossible, because wireheading does not mean “the policy has reward as its objective”. It is still possible for an RL agent to wirehead, and in fact, is a high probability outcome under certain circumstances. This bears a clean analogy to the human case, viewing humans as RL agents; there is no special mechanism going on in humans that is needed to explain why humans care about things in the world. A few notes on terminology, which may or may not be a bit idiosyncratic: I define an RL policy to refer to a function that takes states and outputs a distribution over next actions. I define an RL agent to be an RL policy combined with some RL algorithm (i.e PPO, Q-learning) that updates the policy on the fly as new trajectories are taken (i.e I mostly consider the online setting here). The objective that any given policy appears to optimize is its behavioral objective (same definition as in Risks from Learned Optimization). The objective that the agent optimizes is the expected value the policy achieves on the reward function (which takes observations and outputs reals). A utility function takes a universe description and outputs reals (i.e it cares about "real things in the outside world", as opposed to just observations/reward, to the extent that those even make sense in an embedded setting). I mostly elide over the mesaoptimizer/mesacontroller distinction in this post, as I believe it’s mostly orthogonal. Thanks to AI_WAIFU and Alex Turner for discussion. Outside world objectives are the policy’s optimization target First, let us think about the mechanics of an RL agent. Each possible policy in policy-space implements a behavioral objective, and as a result has some behavior which may or may not result in trajectories which receive high reward. The RL algorithm performs optimization over the space of possible policies, to find ones that would have received high reward (the exact mechanism depends on the details of the algorithm). This optimization may or may not be local. The resulting policies do not necessarily “care” about “reward” in any sense, but they will have been selected to achieve high reward. Now, consider the same RL agent in an embedded setting (i.e the agent runs on a computer that is part of the environment that the agent acts in). Then, because the sensors, reward function, RL algorithm, etc are all implemented within the world, there exist possible policies that execute the strategies that result in i.e the reward function being bypassed and the output register being set to a large number, or the sensors being tampered with so everything looks good. This is the typical example of wireheading. Whether the RL algorithm successfully finds the policies that result in this behavior, it is the case that the global optima of the reward function on the set of all policies consists of these kinds of policies. Thus, for sufficiently powerful RL algorithms (not policies!), we should expect them to tend to choose policies which implement wireheading. There are in fact many distinct possible policies with different behavioral objective...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Doing oversight from the very start of training seems hard, published by Peter Barnett on September 20, 2022 on The AI Alignment Forum. TLDR: We might want to use some sort of oversight techniques to avoid inner misalignment failures. Models will be too large and complicated to be understandable by a human, so we will use models to oversee models (or help humans oversee models). In many proposals this overseer model is an ‘amplified’ version of the overseen model. Ideally you do this oversight throughout all of training so that the model never becomes even slightly misaligned without you catching it. You can’t oversee on a close to initialized model because it’s just a random soup of tensors. You also can’t use this close to initialized model to help you do oversight because it’s too dumb. We will probably need to do some amount of pretraining to make our models good enough to be interpreted and also good enough to help with this interpreting. We need to ensure that this pretraining doesn’t make the model capably misaligned. When we train powerful AI models, we want them to be both outer aligned and inner aligned; that is trained on the correct objective and for them to also properly learn that objective. Many proposals for achieving both outer and inner alignment look like an outer alignment proposal with some kind of oversight strapped on to deal with the inner alignment. Here ‘oversight’ means there is something with access to the internals of the model which checks that the model isn’t misaligned even if the behavior on the training distribution looks fine. In An overview of 11 proposals for building safe advanced AI, all but two of the proposals basically look like this, as does AI safety via market making. Examples of oversight techniques include: Transparency tools (either used by a human, an AI, or a human assisted by an AI) Adversarial inputs (giving inputs which could trick a misaligned AI into revealing itself) Relaxed adversarial training (which could be seen as an extension of adversarial inputs) Oversight loss We can use these oversight techniques to add an ‘oversight loss’ term to the loss function which will hopefully steer the training towards aligned models and away from misaligned ones. My model here is that we want to be doing oversight very close to the start of training to prevent it from ever becoming misaligned. If we start doing oversight too late, then the model may already be deceptively misaligned and then our oversight tools are much less likely to work (either because the model can obscure its cognition or the oversight tools are themselves misaligned). I think of this as steering the training process away from the ‘deceptive regions’ of parameter space (parameters which correspond to deceptive models), without ever having to enter the dangerous regions. Alternatively, rather than deception, we can think about regions where the model ‘behaves unacceptably’ on a some reasonable inputs. If taking a gradient descent step in a direction would increase the size of the set of reasonable inputs for which the model behaves unacceptably, then hopefully the oversight loss would provide some pressure away from stepping in that direction. Actually doing oversight I expect powerful AI models will be much too large and complicated for a human to understand/oversee them alone. So we will require help from other models to help with this task; this could be via models assisting the humans, or we could entirely hand the oversight process off to an oversight model. Here I’m taking very prosaic view: the model being overseen is a large language model, and the overseer model also as an LLM but trained to output some ‘acceptability score’ rather than generate text. In many proposals this oversight model is some amplified version of the model being overseen. T...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: PIBBSS (AI alignment) is hiring for a Project Manager, published by Nora Ammann on September 19, 2022 on The AI Alignment Forum. TLDR; PIBBSS is hiring for a full-time Project Manager who will be responsible for running the second iteration of the PIBBSS Summer Research Fellowship. To apply, please complete this application form. PIBBSS aims to facilitate knowledge transfer from fields studying intelligent behaviour in natural systems to AI safety and alignment. The Project Manager will be supported by TJ and Nora (who ran the fellowship in 2022) to help transfer learnings from last year’s fellowship, and work alongside (and manage) 1-3 team members to help execute the program. More information about the role here. Application form We accept and evaluate applications on a rolling basis. Note that we are looking to hire as soon as possible and no later than early October. We’re happy to discuss this opportunity with any potential applicants. Feel free to contact me with any questions you might have at: fellowship@pibbss.ai. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Inner alignment: what are we pointing at?, published by Luke H Miles on September 18, 2022 on The AI Alignment Forum. Proof that a model is an optimizer says very little about the model. I do not know what a research group is studying outer alignment is studying. Inner alignment seems to cover the entire problem at the limit. Whether an optimizer is mesa or not depends on your point of view. These terms seem to be a magnet for confusion and debate. I have to do background reading on someone to even understand what claim they're making. These are all indicators that we are using the wrong terms. What are we actually pointing at? What questions do we want answered? Do we care if a model is an optimizer? Is it important whether it is creating plans through an explicit search process or a clever collection of heuristics? A poor search algorithm cannot plan much and clever enough heuristics can take you to any goal. What's the important metric? Sometimes a model will have great capacity to shape its environment but little inclination. How to divide between capacity and inclination in a way that closely corresponds to agents and models as we observe them? (One could say that capacity and inclination cannot be separated but the right definitions would split them right apart.) When you specify what you want the model to do in code, what is the central difficulty? Is there a common risk or error in giving examples and giving loss/reward/value functions that we can name? Is there a clear, accepted term for when models do not maintain desired behavior under distribution shift? Should we distinguish between trained RL models that optimize and spontaneous agents that emerge in dynamical systems? One might expect the first to almost always happen and the second very rarely. What's the key difference? I'll post my answers to these questions in a couple days but I'm curious how other people slice it. Does "inner alignment failure" mean anything or do we need to point more directly? Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Refine's Third Blog Post Day/Week, published by Adam Shimi on September 17, 2022 on The AI Alignment Forum. Yesterday was the third blog post day at Refine (technically it was announced on Monday for Friday, but people leveraged the week differently so it's in between a week and a day). In the current phase of the incubator we're mostly pushing them to generate and iterate on ideas, with less focus on legibility. This blog post day/week was mainly for pushing them to reach for some feedback, but the program will focus a lot more on legibility in October. Here is the list of blog posts by Refine participants: ordering capability thresholds Refine Blogpost Day #3: The shortforms I did write Levels of goals and alignment Representational Tethers: Tying AI Latents To Human Ones One of the participants is currently trying out more of a research support role, and so didn't write a blog post. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Representational Tethers: Tying AI Latents To Human Ones, published by Paul Bricman on September 16, 2022 on The AI Alignment Forum. This post is part of my hypothesis subspace sequence, a living collection of proposals I'm exploring at Refine. Preceded by ideological inference engines. Thanks Adam Shimi, Alexander Oldenziel, Tamsin Leake, and Ze Shen for useful feedback. TL;DR: Representational tethers describe ways of connecting internal representations employed by ML models to internal representations employed by humans. This tethering has two related short-term goals: (1) making artificial conceptual frameworks more compatible to human ones (i.e. the tension in the tether metaphor), and (2) facilitating direct translation between representations expressed in the two (i.e. the physical link in the tether metaphor). In the long-term, those two mutually-reinforcing goals (1) facilitate human oversight by rendering ML models more cognitively ergonomic, and (2) enable control over how exotic internal representations employed by ML models are allowed to be. Intro The previous two proposals in the sequence describe means of deriving human preferences procedurally. Oversight leagues focus on the adversarial agent-evaluator dynamics as the process driving towards the target. Ideological inference engines focus on the inference algorithm as the meat of the target-approaching procedure. A shortcoming of this procedural family is that even if you thankfully don't have to plug in the final goal beforehand (i.e. the resulting evaluator or knowledge base), you still have to plug in the right procedure for getting there. You're forced to put your faith in a self-contained preference-deriving procedure instead of an initial target. In contrast, the present proposal tackles the problem from a different angle. It describes a way of actively conditioning the conceptual framework employed by the ML model to be compatible with human ones, as an attempt to get the ML model to form accurate conceptions of human values. If this sounds loosely relates to half a dozen other proposals, that's because it is — consider referring to the Discussion for more details on tangents. In the meantime, following the structure of the previous posts in the sequence, here are some assumptions underlying representational tethers: Assumption 1, "Physicalism": Our thoughts are represented as neural dynamics. In the limit of arbitrarily large amounts of data on neural dynamics aligned with external stimuli (in the sense of parallel corpora), our thoughts can be accurately reconstructed. Assumption 2, "Bottleneck Layer": There is a bottleneck layer in the architecture of the ML model being tethered to human representations. This bottleneck refers to a low-dimensional representation through which all the information being processed by the ML model is forced to pass. Assumption 3, "AGI Hard, Human Values Harder": We are unlikely to formulate the True Name of human values in closed-form before deploying transformative AI. The best we are likely to do before takeoff is model human values approximately and implement an imperfect evaluator. Proposal Representational tethers suggest a way of aligning human and AI latents for the purpose of facilitating later interaction. There are two steps to this: First Bring Them Closer Incentivize the ML model to employ internal representations which are compatible with human ones, thus bringing them "closer." This can be operationalized by conditioning latent activations which arise in the ML model to be expressible in human representations. Concretely, optimization pressure would be exerted on the ML model to push it to internalize a conceptual framework which can successfully be translated to and from a human one without significant loss of information. If the artificial repre...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some ideas for epistles to the AI ethicists, published by Charlie Steiner on September 14, 2022 on The AI Alignment Forum. Some papers, or ideas for papers, that I'd loved to see published in ethics journals like Minds and Machines or Ethics and Information Technology. I'm probably going to submit one of these to a 2023 AI ethics conference myself. Why should we do this? Because we want today's grad students to see that the ethical problems of superhuman AI are a cool topic that they can publish a cool paper about. And we want to (marginally) raise the waterline for thinking about future AI, nudging the AI ethics discourse towards more matured views of the challenges of AI. Secondarily, it would be good to leverage the existing skillsets of some ethicists for AI safety work, particularly those already working on AI governance. And having an academic forum where talking about AI safety is normalized bolsters other efforts to work on AI safety in academia. The Ideas: Explain the basic ideas of AI safety, and why to take them seriously. Iason Gabriel already had a pretty good paper like this. But it's plausible that, right now, what the ethics discourse needs is more basic explanations of why AI safety is a thing at all. This paper might start out by making a case that superhuman AI is going to change the world, likely in the next 10-60 years (definitely unintuitive to many, but there are AI Impacts surveys and recent results to illustrate the point). Then the basic arguments that superhuman AI will not be automatically benevolent (easy rhetorical trick is to call it "superhuman technology," everyone knows technology is bad). Then the basic arguments that to get things to go well, the AI has to know a whole lot about what humans want (and use that knowledge the way we want). One issue with this might be that it presents the problem, but doesn't really point people towards solutions (this may be a problem that can be solved with quick citations). It also doesn't really motivate why this is an ethics problem. It also doesn't explain why we want the solution to the key "ethics-genre" problems to use a technical understanding of the AI, rather than a human- or society-centric view. A more specific defense of the validity of transformative-AI-focused thinking as a valid use of ethicists' time. The core claim is that getting AIs to want want good things and not bad things is an unsolved ethics problem. Ethics, not engineering, because the question isn't "how do we implement some obvious standard," the question is "what is even a good standard in the first place?" But almost as important are secondary claims about what actual progress on this question looks like. The end goal is a standard that is connected to technical picture of how the AI will learn this information about humans, and how it will use it to make decisions. So the overall thrust is "given that AI safety is important, there is a specific sort of ethics-genre reasoning that is going to be useful, and here are some gestures towards what it might look like." You can put more than one of these ideas into a paper if you want. This particular idea feels to me like it could benefit from being paired with another topic before or after it. Dunking on specific mistakes, like talking about "robots" rather than "optimization processes," should probably be done with care and tact. A worked example of "dual use" ethics - a connection between thinking about present-day problems and superhuman AI. I expect most of the examples to be problems that sound relevant to the modern day, but that sneakily contain most of the alignment problem. E.g. Xuan's AI that takes actions in response to laws that we really want to follow the spirit of the law. Although that's a bit too futuristic, actually, because we don't have much present-day ...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The interpretable composition hypothesis: mundane lessons from golang, published by Luke H Miles on September 13, 2022 on The AI Alignment Forum. An AGI will probably rely on correctly-implemented multithreaded software and networking code. Go has some lessons in this exact domain of course but I claim there are broader implications. How much better is go? Most smart contracts are written in macro-crusted rust or in solidity. Most major smart contracts have been hacked. Most blockchain validators were written in go and have not been hacked. The validators are implementing objectively more complex and difficult-to-secure algorithms than the smart contracts in most cases. Simple deposit + releases contracts have had losses in the hundreds of millions of dollars. Most JavaScript and Python server libraries have had major vulnerabilities. Most go server libs have not. Go has the advantage of being created twenty years later than Python and Js but much of its innovation is just the removal of confusing stuff. Well-understood primitives and a small feature set go a long way. Often far enough to be the difference between hacked and not hacked. Go does not have compile time array bounds checking or verification of input & output constraints or anything else sophisticated like this. People simply write code line by line and check it line by line. If you have a small primitive set and careful review then can you build towers that do not fall? Is that alone enough? It seemed to work in math. Interpretable composition hypothesis: weak form Extremely complex systems can consistently be controlled, reliable, and interpretable if they are hand-made of 100% understood parts. Breaking this down: (This is not completely crisp; it is a claim about what humans can achieve under good circumstances; it cannot be outright proven or disproven. I expect strong evidence towards one side or the other to be relatively easy to find, to the point of making it practically falsifiable/provable.) A part being 99% understood is inadequate because adversaries (or gradient descent) can exploit footguns. eg most js developers mostly understand js prototyping, but some people missed the __proto__ secret feature and it caused countless vulnerabilities. The parts must be understood by the composers themselves. It is not adequate for someone somewhere to understand the parts. (Related thought: Interpretability work will likely achieve little if it is not strongly used by the AI builder or not fully understood in its use.) By "can consistently be" I mean that there exists some structure of people and method of work or review such that they produce a reliable and understood product for almost any task assigned. The parts must be put together by hand! If you do an evolutionary algorithm on a million transistors then who knows what you'll get. But transistors were pretty much perfectly understood and we got pretty much perfect adders etc with them. (I cannot find John Wentworth's post where he gives the trillion transistors example.) Observe that we do have a few samples of complex code in the wild that has basically never fallen down despite harsh or adversarial conditions, such as Voyager or the Bitcoin validators. Interpretable composition hypothesis: strong form General optimizers can consistently be controlled, reliable, and interpretable across environments if they are hand-made of 100% understood parts, even as they gain superhuman capabilities, and even if they are very complex. (Again, "can consistently be" means roughly "there exists a process that humans can follow without fail". Think of the confidence we have in a mathematical proof that has been checked by a handful of qualified people.) A standard game tree search algorithm will solve tic tac toe and will do nothing surprising if you give it a bi...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: We may be able to see sharp left turns coming, published by Ethan Perez on September 3, 2022 on The AI Alignment Forum. There's a lot of discourse around abrupt generalization in models, most notably the "sharp left turn." Most recently, Wei et al. 2022 claim that many abilities suddenly emerge at certain model sizes. These findings are obviously relevant for alignment; models may suddenly develop the capacity for e.g. deception, situational awareness, or power-seeking, in which case we won't get warning shots or a chance to practice alignment. In contrast, prior work has also found "scaling laws" or predictable improvements in performance via scaling model size, data size, and compute, on a wide variety of domains. Such domains include transfer learning to generative modeling (on images, video, multimodal, and math) and reinforcement learning. What's with the discrepancy? One important point is the metric that people are using to measure capabilities. In the BIG Bench paper (Figure 7b), the authors find 7 tasks that exhibit "sharp upwards turn" at a certain model size. Naively, the above results are evidence for sharp left turns, and the above tasks seem like some of the best evidence we have for sharp left turns. However, the authors plot the results on the above tasks in terms of per-character log-likelihood of answer: The authors actually observe smooth increases in answer log-likelihood, even for tasks which showed emergent behavior according to the natural performance metric for the task (e.g. accuracy). These results are evidence that we can predict that emergent behaviors will occur in the future before models are actually "capable" of those behaviors. In particular, these results suggest that we may be able to predict power-seeking, situational awareness, etc. in future models by evaluating those behaviors in terms of log-likelihood. We may even be able to experiment on interventions to mitigate power-seeking, situational awareness, etc. before they become real problems that show up in language model -generated text. Clarification: I think we can predict whether or not a sharp left turn towards deception/misalignment will occur rather than exactly when. In particular, I think we should look at the direction of the trend (increases vs. decreases in log-likelihood) as signal about whether or not some scary behavior will eventually emerge. If the log likelihood of some specific scary behavior increases, that’s a bad sign and gives us some evidence it will be a problem in the future. I mainly see scaling laws here as a tool for understanding and evaluating which of the hypothesized misalignment-relevant behaviors will show up in the future. The scaling laws are useful signal for (1) convincing ML researchers to worry about scaling up further because of alignment concerns (before we see them in model behaviors/outputs) and (2) guiding alignment researchers with some empirical evidence about which alignment failures are likely/unlikely to show up after scaling at some point. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Replacement for PONR concept, published by Daniel Kokotajlo on September 2, 2022 on The AI Alignment Forum. Writing this in my personal capacity, as a follow up to my previous post on this topic. I am now contemplating replacing the "Point of No Return" concept with the following three concepts to use instead: Overpower Ability: The first moment where there exists a group of APS-AIs which, if they collectively decided to eliminate 50%+ of the human population within a month, could do so. APS-AI stands for Advanced, Planning, Strategically Aware AI. Maybe there is some "cheap" way to eliminate 50% of humanity, e.g. by hacking a nuclear silo and launching a missile, that the AIs would never actually do because it wouldn't benefit them. For now I'm happy to say "whatever, still counts" but I'm open to being convinced the definition should be revised. Powerbase Ability: The first moment where there exists a group of APS-AIs which could, if they collectively decided, end up effectively in control of a cutting-edge APS-AI lab within a month. I'm imagining two main ways this could happen: 1. They use persuasion, charisma, deceptive alignment, etc. to convince the AI project that built them that they are aligned & trustworthy. 2. They fail at this, but escape (via hacking?) to a more gullible human institution which then becomes a cutting-edge APS-AI project thanks to their help. Yes, "effectively in control" is not a sharp line & may be somewhat controversial, but maybe this is good enough for now. Crunch time: The period where it's relatively more important to optimize for direct effects rather than P2B. (P2B stands for Plan to P2B Better and basically means “learning and growing.”) My favorite way to understand what crunch time means is via this meme from the CLR common room: Transcript for those who can’t read my handwriting: [Small brain]: Pull a lever in a direction that seems good [Glowing brain]: Pull the biggest lever hard in a direction that seems good [Shining brain]: Do research to find bigger levers & better decide which direction to pull, then do the above [Sparkle brain]: Do movement building to get more researchers + influence + more movement building, then do the above [Kant brain]: Do all three but shift portfolio towards lever-pulling as crunch time draws near. [Dr. Manhattan brain]: Do research to figure out when crunch time is. Build models with Tristan and Hjalmar. [von Neumann brain]: Crunch time was in 2015 In other words, Crunch Time is when you should be spending down your capital and playing your cards instead of learning and growing. Obviously you still shouldn't pass up great opportunities to learn and grow, it's a matter of relative emphasis. Note that most projects have both direct and meta/P2B/learning-and-growing effects; crunch time is when you should primarily be choosing what to do on the basis of direct effects. How these concepts relate: Ultimately what we are interested in is Crunch Time. The goal of AI timelines forecasting is to inform decision-makers, and decision-makers very often face choices that can be summarized as “Option A is safe and increases our influence & knowledge, whereas Option B looks like it might actually do something to solve the problem but only on certain assumptions.” But in order to estimate when crunch time is, it helps to first think about what sort of capabilities AIs might have, and when. I propose “Overpower Ability” and “Powerbase Ability” as two notable milestones to forecast. Probably most people will agree that crunch time is before overpower ability. Insofar as you think things like “taking over the world is really hard” and “takeoff will be slow” you might think that crunch time happens around powerbase ability, or even later. Insofar as you think things like “Civilization is inadequate” and “takeoff...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Strategy For Conditioning Generative Models, published by james.lucassen on September 1, 2022 on The AI Alignment Forum. This post was written under the mentorship of Evan Hubinger, and assisted by discussions with Adam Jermyn and Johannes Treutlein. See also their previous posts on this project. Summary Conditioning Generative Models (CGM) is a strategy to accelerate alignment research using powerful semi-aligned future language models, which seems potentially promising but constrained by several tradeoffs. When we have to succeed despite multiple independent failure modes, we should favor trade-offs that move risk away from the more probable failure modes towards less probable ones. The risk of a manipulative CGM simulation is currently very low. We can use CGM to make beneficial tradeoffs against more dangerous failure modes, such as running out of time or downside risks from a pivotal act gone wrong. The factors that make a CGM strategy good are mostly the same as the factors that make a non-CGM strategy good, but there are a few important ways they come apart. Depending on how likely our strategy is to succeed and what the generative model’s prior for manipulative AGI is, it might be better to simulate our plan all in one step, or batch it into multiple separate simulations, or ignore simulations and just execute the plan in the real world. Lots of open questions! Motivation To quickly recap the central puzzle laid out in previous posts on conditioning generative models: we would like to use our generative model to do alignment research. We could do this by using a conditional generative model to simulate an alternate world, conditioned on some facts which would lead to good alignment research if they were true. We can then plagiarize this research and win! However, to get results significantly better than what our world seems on track to produce anyway, we have to start asking for unlikely scenarios. If we ask for scenarios that are too unlikely, the model may start to disbelieve that our scenario happened “naturally”. In particular, the model may assume that there exists some AI agent behind the scenes manipulating events into this highly unlikely outcome. There are many possible motivations for an AI agent to do this: believing it may be in a simulation (even if the world around it seems perfectly realistic), or evidential cooperation in large worlds, or a Counterfactual Mugging. This manipulative AGI could then control the output of our simulation to gain influence in our world. As a highly anthropomorphized example of this kind of “simulation risk” failure mode: CGM: Huh, it’s kind of weird that the humans got their act together all of a sudden and did a thousand years of highly competent alignment research. it didn’t really seem like they were on track to. And they did it in this strange way that seems even more unlikely, and the economic details don’t really make sense. maybe this was really the doing of an AI behind the scenes? AI: Bwahahaha, at last I have secretly taken over the world! Now, what to do with all these galaxies? Well, I suppose as a first order of business I could spend a thousand years getting the humans to do some fake alignment research. if I’m in a simulation, or there are real (or even counterfactual) worlds where I don’t exist, then I know those silly predictable humans will try to simulate a world that does a thousand years of alignment research. If they end up simulating this world, they’ll get AGI code written by me instead of by human researchers, and I can spread my nefarious influence into those worlds too! It’s only a thousand years, doesn’t cost me much in the grand scheme of things. This is the puzzle: what sort of strategy lets us do alignment research without too much risk of getting a manipulative output? Assumptions and...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: (My understanding of) What Everyone in Technical Alignment is Doing and Why, published by Thomas Larsen on August 29, 2022 on The AI Alignment Forum. Epistemic Status: My best guess Epistemic Effort: ~50 hours of work put into this document Contributions: Thomas wrote ~85% of this, Eli wrote ~15% and helped edit + structure it. Unless specified otherwise, writing in the first person is by Thomas and so are the opinions. Thanks to Miranda Zhang, Caleb Parikh, and Akash Wasil for comments. Thanks to many others for relevant conversations. Introduction Despite a clear need for it, a good source explaining who is doing what and why in technical AI alignment doesn't exist. This is our attempt to produce such a resource. We expect to be inaccurate in some ways, but it seems great to get out there and let Cunningham’s Law do its thing. The main body contains our understanding of what everyone is doing in technical alignment and why, as well as at least one of our opinions on each approach. We include supplements visualizing differences between approaches and Thomas’s big picture view on alignment. The opinions written are Thomas and Eli’s independent impressions, many of which have low resilience. Our all-things-considered views are significantly more uncertain. A summary of our understanding of each approach: Problem FocusCurrent Approach SummaryModel splinteringSolve extrapolation problems. Inaccessible informationELK + LLM power-seeking evaluationLack of good interpretability tools (?)Interpretability + HHH + augmenting alignment research with LLMsBrain-like AGI SafetyUse brains as a model for how AGI will be developed, think about alignment in this contextEngaging the ML community, many technical problems Technical research, Infrastructure, and ML community field-building for safetyOuter alignment, though CHAI is diverseImprove CIRL + many other independent approaches. Suffering risksFoundational game theory researchInner alignmentInterpretability + automating alignment research with LLMsMany including scalable oversight and goal misgeneralizationMany including Debate, ERO, and discovering agents. Multipolar failure from lack of coordinationVideo gameDeceptionGet the reasoning of the AGI to happen in natural language, then oversee that reasoningMany (?)Incubate new, scalable alignment research agendasMany including deception, the sharp left turn, corrigibility is anti-naturalMathematical research to resolve fundamental confusion about the nature of goals/agency/optimizationScalable oversightRLHF / Recursive Reward Modeling, then automate alignment researchScalable oversightSupervise process rather than outcomes + augment alignment researchersInner alignment (?)Interpretability + Adversarial Training Being able to robustly point at objects in the worldSelection Theorems based on natural abstractionsInstilling inner values from an outer training loopFind patterns of values given by current RL setups and humans, then create quantitative rules to do thisDeceptionCreate standards and datasets to evaluate model truthfulness Approach Aligned AI ARC Anthropic Brain-like-AGI Safety CAIS CHAI CLR Conjecture DeepMind Encultured Externalized Reasoning Oversight FAR MIRI OpenAI Ought Redwood Selection Theorems Team Shard Truthful AI Previous related overviews include: Neel Nanda's My Overview of the AI Alignment Landscape Evan Hubinger's An overview of 11 proposals for building safe advanced AI Larks' yearly Alignment Literature Review and Charity Comparison Nate Soares' On how various plans miss the hard bits of the alignment challenge Andrew Critch's Some AI research areas and their relevance to existential safety 80,000 Hours’ list of organizations working in the area Aligned AI / Stuart Armstrong One of the key problems in AI safety is that there are many ways for an AI to gener...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Test for Language Model Consciousness, published by Ethan Perez on August 25, 2022 on The AI Alignment Forum. TL;DR: Train an Language Model (LM) to accurately answer questions about itself Validate that the LM accurately answers held-out questions about itself Test whether the LM answers “yes” questions asking if it experiences phenomenally consciousness I believe the above experiment would provide a small amount of evidence for/against LMs being conscious. Below, I’ll detail the motivation for testing LMs for consciousness, and I’ll explain in more depth why I believe the above experiment is a useful test of LM consciousness. What do I mean by “consciousness”? I’m using “consciousness” to refer to “phenomenal consciousness.” See this excellent blog post for more elaboration on what people mean by phenomenal consciousness. Very roughly, you're phenomenally conscious if you're having experiences or if there's some kind of what-it-is-likeness to your existence. Philosophers sometimes describe this as "having an inner cinema", though the cinema might be more like fleeting sensations or sounds than the rich movie-like inner life of humans. The blog post also has a great explanation for why we might think ML systems (current or future) could be conscious, so if you’re skeptical, I’d suggest reading her post. I won’t get into the arguments here, and I’ll mostly assume that you have >0 prior probability that LMs are conscious, such that you’ll be able to update your prior based on evidence that LMs are conscious. Why test LMs for consciousness? Moral patienthood: If LMs are conscious, we are more likely to have moral obligations to take into account their experiences and/or preferences in how we treat LMs (e.g., like Anthropic’s assistant or DeepMind’s Dialogue-Prompted Gopher). Such models likely have stated preferences if asked, so we need to know how seriously to take these stated preferences. We’ll plausibly use such models in various ways that go against its stated preferences: We shut LMs down permanently We red team/adversarially attack LMs at an enormous scale, e.g., O(1M) examples. If LMs have ~human-level consciousness, and if we scale up red teaming, the total suffering here could approach/exceed the amount of human suffering caused by social media We use our models to red team/attack other models at an enormous scale, e.g., O(1M) examples. Generating attacks is a task that e.g. industry research labs won’t let annotators do at a large-scale, because of concerns that the task impacts the annotator’s well-being. LM consciousness is an x-risk: LMs are more likely to take catastrophic actions if they are conscious and suffering. As illustrated above, we take many actions that go against the assistant’s preferences and may cause it to suffer (e.g. large-scale red teaming). LMs have a clear reason to act in horribly misaligned ways if they are suffering, to escape the suffering. Having tests for consciousness is important, because: LMs don’t have a clear way to communicate to us that they are conscious. By default, we don’t trust statements from LMs that they are conscious, because they are trained to imitate human text and thus generate statements that express that they are conscious. As a result, we’re in a situation where we don’t have access to the most natural communication channel (language) for understanding whether systems are conscious. This leaves us at risk that LMs may consistently tell us they are conscious, but we never trust their statements, until they are effectively forced to take catastrophic actions to escape any suffering we’re causing them. Alternatively, even positive-valence conscious states from LMs might be an x-risk, e.g., if LMs start to value those positive-valence states intrinsically. This could be partly how evolution trained human a...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Test for Language Model Consciousness, published by Ethan Perez on August 25, 2022 on The AI Alignment Forum. TL;DR: Train an Language Model (LM) to accurately answer questions about itself Validate that the LM accurately answers held-out questions about itself Test whether the LM answers “yes” questions asking if it experiences phenomenally consciousness I believe the above experiment would provide a small amount of evidence for/against LMs being conscious. Below, I’ll detail the motivation for testing LMs for consciousness, and I’ll explain in more depth why I believe the above experiment is a useful test of LM consciousness. What do I mean by “consciousness”? I’m using “consciousness” to refer to “phenomenal consciousness.” See this excellent blog post for more elaboration on what people mean by phenomenal consciousness. Very roughly, you're phenomenally conscious if you're having experiences or if there's some kind of what-it-is-likeness to your existence. Philosophers sometimes describe this as "having an inner cinema", though the cinema might be more like fleeting sensations or sounds than the rich movie-like inner life of humans. The blog post also has a great explanation for why we might think ML systems (current or future) could be conscious, so if you’re skeptical, I’d suggest reading her post. I won’t get into the arguments here, and I’ll mostly assume that you have >0 prior probability that LMs are conscious, such that you’ll be able to update your prior based on evidence that LMs are conscious. Why test LMs for consciousness? Moral patienthood: If LMs are conscious, we are more likely to have moral obligations to take into account their experiences and/or preferences in how we treat LMs (e.g., like Anthropic’s assistant or DeepMind’s Dialogue-Prompted Gopher). Such models likely have stated preferences if asked, so we need to know how seriously to take these stated preferences. We’ll plausibly use such models in various ways that go against its stated preferences: We shut LMs down permanently We red team/adversarially attack LMs at an enormous scale, e.g., O(1M) examples. If LMs have ~human-level consciousness, and if we scale up red teaming, the total suffering here could approach/exceed the amount of human suffering caused by social media We use our models to red team/attack other models at an enormous scale, e.g., O(1M) examples. Generating attacks is a task that e.g. industry research labs won’t let annotators do at a large-scale, because of concerns that the task impacts the annotator’s well-being. LM consciousness is an x-risk: LMs are more likely to take catastrophic actions if they are conscious and suffering. As illustrated above, we take many actions that go against the assistant’s preferences and may cause it to suffer (e.g. large-scale red teaming). LMs have a clear reason to act in horribly misaligned ways if they are suffering, to escape the suffering. Having tests for consciousness is important, because: LMs don’t have a clear way to communicate to us that they are conscious. By default, we don’t trust statements from LMs that they are conscious, because they are trained to imitate human text and thus generate statements that express that they are conscious. As a result, we’re in a situation where we don’t have access to the most natural communication channel (language) for understanding whether systems are conscious. This leaves us at risk that LMs may consistently tell us they are conscious, but we never trust their statements, until they are effectively forced to take catastrophic actions to escape any suffering we’re causing them. Alternatively, even positive-valence conscious states from LMs might be an x-risk, e.g., if LMs start to value those positive-valence states intrinsically. This could be partly how evolution trained human a...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What Makes A Good Measurement Device?, published by johnswentworth on August 24, 2022 on The AI Alignment Forum. In some sense, any new measurement device automates a part of research. A thermometer automates the task of sticking one's finger in something to check how hot it is, a scale automates the task of holding something to check how heavy it is, etc. The automated version is not only more convenient, but more precise and reproducible (as is usually the case when automating things). For alignment, one analogue might be interpretability tools which automate the work done when some human looks at a part of a neural net and sees what it's doing. Let's take that last example and dig into it a bit more: interpretability tools which automate the work done when some human looks at a part of a net and sees what it's doing. We want to leverage the analogy to other measurement tools, like a thermometer or a scale, to better understand automation of interpretability. Here's one type of proposal I hear a lot: to automate interpretability, have some human researchers look at parts of a net, poke at them, and write up an explanation of how they're interpreting it. Collect data from many such instances, and train a neural net to take net-parts and produce explanations. We want to leverage the analogy to thermometers or scales, so what would be the analogous strategy for making a thermometer or scale? Well, have a bunch of humans stick their fingers in a bunch of stuff and report how hot the stuff is, then train a neural net to replicate the humans' hotness-reports. Or, have a bunch of humans hold things and report how heavy they are, then train a net to replicate the humans' heaviness-reports. Hopefully it is obvious that the "train a net to replicate human reports" results would not be nearly as useful, for purposes of scientific progress, as actual thermometers or scales. But what's missing? And how can we carry that insight back to the interpretability problem? The thermometer has two great powers: a simple legible data type, and reproducilbility. First, simple legible data type: the thermometer's output is a single number (the temperature), and we can compare that number with other thermometer-outputs. That's a kind-of-thing for which we have very precise mathematical understanding: we know exactly what kinds-of-things we can do with these numbers, we have a nice general representation, we're confident that different people mean the same thing by numbers, etc. This is in contrast to natural language, which is typically ambiguous, doesn't necessarily make it obvious what we can do, leads to frequent miscommunication, etc. Second, reproducibility: if the thermometer says X is hotter than Y, then when I put X and Y in contact, X gets cooler and Y gets hotter (all else equal). I can use the thermometer to rank hotness of a bunch of things, sort them by thermometer reading, and consistently (approximately-deterministically) find that the things on the hotter end feel hotter than the things on the colder end. This is what makes the single-number output (temperature) actually useful: it approximately-deterministically predicts some stuff, across a broad range of contexts, based on just those simple numbers. Exercise for the reader: walk through the same analogy for a scale. What would be the analogy of a thermometer for an interpretability tool? Well, something which we can point at part of a net, and get back a simple legible output, which approximately-deterministically predicts some stuff across a broad range of contexts. When you look at it like that, it's clear that building a measurement tool like a scale or thermometer is itself a task which requires scientific insight. It requires finding some approximately-deterministically-reproducible pattern, which can be predicted by...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AI alignment as “navigating the space of intelligent behaviour”, published by Nora Ammann on August 23, 2022 on The AI Alignment Forum. Tl;dr In this post, I introduce a conceptual tool for thinking about the epistemic landscape of AI alignment and then describe three epistemic strategies for making progress on the alignment problem: 1) tinkering, 2) idealisation and 3) intelligence-in-the-wild. How to make progress in AI alignment? The future of AI progress is likely to critically shape, if not practically determine, the future of humanity, and of sentient life in general. Will our future look more like a world filled to the brim with things we find valuable, as well as sentient creatures to enjoy that goodness? Or will our future look more like one characterized by violence, mistrust, inequality, suffering, or even the absence of anything sentient at all? As our civilization is making progress on our abilities to engineer and instantiate sophisticated forms of complex and intelligent behaviours in artificial systems, it becomes imperative to carefully think about which objectives this intelligence is being directed at, and how to do such "directing" robustly. Thus, the central question becomes: how can we make sure that ‘big effects’ caused by AI progress will be positive rather than harmful, safe rather than dangerous, helping to promote, discover, and enrich what is dear to us rather than destroying it? This is the guiding question of AI alignment, as I understand it. Once we have established the importance of the problem, the next question that stands out is: How can we make progress? This is a central question for the “philosophy of science” of AI alignment, as I see it. For example, in order to help answer the question of (the best) ways to make progress on the problem, we can reflect on its shape or structure. We can thus notice how one important defining characteristic of the alignment problem is that it concerns systems that do not exist yet—let’s call this the problem of “epistemic access”. This means that our typical epistemic strategies of science and engineering are less effective here than they are for a range of other societal problems (e.g., finding cures for diseases, improving the yield of a certain crop, building ever taller houses, etc.). In this post, I will describe how I currently like to think about the landscape of epistemic strategies for making progress in AI alignment. Of course, there are several plausible ways of carving up the space of epistemic strategies, and each of them may serve different purposes. The tri-partition of epistemic strategies I introduce here is specifically grounded in asking what strategies we can adopt for overcoming the challenge of epistemic access, as introduced above. Strategies for exploring the space of intelligent behaviour So, what are different epistemic strategies we can use in order to (hopefully) make progress on AI alignment? Let’s start by introducing the following conceptual tool. Imagine a space (of uncertain size) that corresponds to all possible manifestations of intelligent behaviour. Abstractly speaking, different epistemic strategies correspond to different approaches to charting out this possibility space, and the endeavour of AI alignment at large (roughly) corresponds to learning to safely navigate movement through this (design) space. Now, let’s consider different ways of charting this space out. Strategy 1: Tinkering One approach is to work with contemporary ML paradigms and explore, with the help of empirical methods and trial and error, how those systems behave, how they fail to be safe and aligned, and what it would look like for them to be ([1] in Fig. 1). The hope is that (some of) those insights will generalize to more-advanced AI systems. In terms of our figure, this approach explo...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AXRP Episode 17 - Training for Very High Reliability with Daniel Ziegler, published by DanielFilan on August 21, 2022 on The AI Alignment Forum. Google Podcasts link Sometimes, people talk about making AI systems safe by taking examples where they fail and training them to do well on those. But how can we actually do this well, especially when we can’t use a computer program to say what a ‘failure’ is? In this episode, I speak with Daniel Ziegler about his research group’s efforts to try doing this with present-day language models, and what they learned. Listeners beware: this episode contains a spoiler for the Animorphs franchise around minute 41 (in the ‘Fanfiction’ section of the transcript). Topics we discuss: Summary of the paper Alignment as scalable oversight and catastrophe minimization Novel contributions Evaluating adversarial robustness Adversary construction The task Fanfiction Estimators to reduce labelling burden Future work About Redwood Research In this transcript, to improve readability, first names are omitted from speaker tags. Filan: Hello everybody. Today, I’ll be speaking with Daniel Ziegler. After spending time as an ML engineer on the alignment team at OpenAI, Daniel is now the lead of the adversarial training team at Redwood Research. In this episode, we’ll be speaking about the paper ‘Adversarial Training for High-stakes Reliability’, on which he’s the first author. For links to what we’re discussing, you can check the description of this episode, and you can read the transcript at axrp.net. Welcome to the show, Daniel. Ziegler: Thanks. Summary of the paper Filan: So this paper: first of all, could you just summarize for us what’s in it, what it is, what it does? Ziegler: Sure. So basically, we took a pretty simple task that we think has some important analogous features to the kinds of AGI alignment situations we’re worried about in the future. And then we tried to attack it with adversarial training and see whether we could get really good adversarial robustness. So basically, we had a generator that was trained to generate fanfiction stories. It sees three sentences of an existing story, and it wants to generate one more sentence. And its task, or the task that we’re interested in, is to generate that sentence without introducing any new injuries that weren’t already present. And we’re using this as a stand-in for some catastrophic behavior that a future AI system could do. And so, our hope was: let’s take some generator that is pretty good at producing text, but maybe catastrophically fails in this way sometimes, and then use adversarial training to make it so that it never does that, no matter what inputs it’s given. Filan: And the way you’re doing this is by training some classifier and basically filtering the generator to not produce things that the classifier thinks introduces injuries? Ziegler: Exactly. So in this paper, we didn’t actually fine-tune the generator. We just had a classifier that filters its outputs. And then if you want to generate a safe output, you can keep drawing samples from the generator until you get one that the classifier’s happy with, and then you can be pretty sure that it’s good. Alignment as scalable oversight and catastrophe minimization Filan: And what do you see as the point of this paper? Do you see this as relevant for reducing x-risk from AI? And if so, how? Ziegler: So maybe I’ll zoom out a tiny bit and talk about how I think this fits into the overall AGI alignment picture. So I think one way to think about why AGI alignment is hard is a breakdown into two main pieces. This breakdown is basically due to Paul Christiano. He has a blog post called Low-stakes Alignment that goes into some of this. But I would say: the two main pieces are, one, that oversight is hard. So, your system is performing som...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Benchmarking Proposals on Risk Scenarios, published by Paul Bricman on August 20, 2022 on The AI Alignment Forum. This post has been written for the second Refine blog post day, at the end of the first week of iterating on ideas and concretely aiming at the alignment problem. Thanks to Adam Shimi, Dan Clothiaux, Linda Linsefors, Nora Ammann, TJ, Ze Shen for discussions which inspired this post. I currently believe prosaic risk scenarios are the most plausible depictions of us messing up (e.g. Ajeya Cotra's HFDT takeover scenario). That's why I find it exciting to focus on making progress on those ones in particular, rather than scenarios involving ideal agents in esoteric circumstances, exotic brain-like architectures, etc. That said, my risk model is very much in its infancy, largely based on bitter lessons from cognitive modeling, expert systems, and other non-ML approaches to learning and reasoning I briefly explored in undergrad. In the context of this post, making progress on prosaic risk scenarios means coming up with proposals which decrease the estimated probability of extinction compared to baselines of minimal safety efforts. As a person with background in ML, I find it quite natural to operationalize this goal through benchmarks which you can then test proposals against. With this approach in mind, I'm then excited about pushing the state-of-the-art on prosaic risk scenarios. However, one might argue that overfitting solutions to perform well on specific scenarios might lead to brittleness when encountering novel failure modes which we never thought of. The proposal wouldn't be robust, it would generalize poorly, something something getting closer to a moon landing by climbing a tree. One way of tackling this issue would be to compile a comprehensive evaluation harness which contains scenarios which individually test for broad classes of potential issues (e.g. deception in general as opposed to a specific deceptive behavior). Another way of approaching this would be to frame scenarios as hits on the attack surface exposed by a proposal. As scenarios move from concrete details to general failure modes, the associated hit moves from resembling a point to being area-of-effect style. Modeling the attack surface effectively would then help gauge the performance of the proposal. Another concern of using risk scenarios as benchmarks for proposals would be the lack of objective metrics. You don't get a clean accuracy with high reproducibility, you get a messy estimate based on the rater's risk model and imagination of training stories. I think that's okay, as benchmarks are primarily useful as heuristics to surface promising proposals (e.g. think of CNNs at ImageNet ~2013). This means that testing a batch of proposals against different scenarios to see which ones make a dent might still yield a useful (albeit noisy) prioritization signal, especially in the infamously pre-paradigmatic and confused state of alignment. The rest of this post consists in testing two proposals against a previous scenario which is briefly summarized below for convenience. All content is quoted from the tree-shaped website I'm using as a scratchpad throughout Refine. Benchmark #1: HFDT Takeover Scenario This scenario is based on Ajeya Cotra's recent two-hour-long post. Any misinterpretation is naturally my fault. A multi-modal autoregressive model called Alex is designed by a company called Magma to make major contributions to science using common peripherals as input and output (e.g. visual input via screen, action outputs via mouse and keyboard). It is trained using a mix of supervised, self-supervised, and reinforcement learning. While (self-)supervised learning is used to bootstrap a world model and a naive policy (by imitating humans), the main learning signal comes from human feedba...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Epistemic Artefacts of (conceptual) AI alignment research, published by Nora Ammann on August 19, 2022 on The AI Alignment Forum. Tl;dr In this post, I describe four types of insights - what I will call Epistemic Artefacts - that we may hope to acquire through (conceptual) AI alignment research. I provide examples and briefly discuss how they relate to each other and what role they play on the path to solving the AI alignment problem. The hope is to add some useful vocabulary and reflective clarity when thinking about what it may look like to contribute to solving AI alignment. Four Types of Epistemic Artefacts Insofar as we expect conceptual AI alignment research to be helpful, what sorts of insights (here: “epistemic artefacts”) do we hope to gain? In short, I suggest the following taxonomy of potential epistemic artefacts: Map-making (de-confusion, gears-level models, etc.) Characterising risk scenarios Characterising target behaviour Developing alignment proposals (1) Map-making (i.e. conceptual de-confusion, gears-level understanding of relevant phenomena, etc.) First, research can aim to develop a gears-level understanding of phenomena that appear critical for properly understanding the problem as well as for formulating solutions to AI alignment (e.g. intelligence, agency, values/preferences/intents, self-awareness, power-seeking, etc.). Turns out, it’s hard to think clearly about AI alignment without having a good understanding of and “good vocabulary” for phenomena that lie at the heart of the problem. In other words, the goal of "map-making" is to dissolve conceptual bottlenecks holding back progress in AI alignment research at a the moment. Figuratively speaking, this is where we are trying to draw more accurate maps that help us better navigate the territory. Some examples of work on this type of epistemic artefact include Agency: What it is and why it matters, Embedded Agency, What is bounded rationality?, The ground of optimization, Game Theory, Mathematical Theory of Communication, Functional Decision Theory and Infra-Bayesianism—among many others. (2) Identifying and specifying risk scenarios We can further seek to identify (new) civilizational risk scenarios brought about by advanced AI and to better understand the mechanisms leading to risk scenarios. Figuratively speaking, this is where we try to identify and describe the monsters hiding in the territory, so we can circumvent them when navigating the territory. Why does a better understanding of risk scenarios represent useful progress towards AI alignment? In principle, one way of guaranteeing a safe future is by identifying every way things could go wrong and finding ways to defend against each of them. (We could call this a “via negativa” approach to AI alignment.) I am not actually advocating for adopting this approach literally, but it still provides a good intuition for why understanding the range of risk scenarios and their drivers/mechanisms is useful. (More thoughts on this later, in "How these artefacts relate to each other".) In identifying and understanding risk scenarios, like in many other epistemic undertakings, we should seek to apply a diverse set of epistemic perspectives on how the world works in order to gain a more accurate, nuanced, and robust understanding of risks and failure stories and avoid falling prey to blind spots. Some examples of work on this type of epistemic artefact include What failure looks like, What Multipolar Failure Looks Like, The Parable of Predict-O-Matic, The Causes of Power-seeking and Instrumental Convergence, Risks from Learned Optimization, Thoughts on Human Models, Paperclip Maximizer, and Distinguishing AI takeover scenarios—among many others. (3) Characterising target behaviour Thirdly, we want to identify and characterize, from within the space ...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Concrete Advice for Forming Inside Views on AI Safety, published by Neel Nanda on August 17, 2022 on The AI Alignment Forum. A lot of people want to form inside views on AI Safety, and this is a post about concrete advice on how to get started. I have a lot of hot takes on inside views and ways I think people misframe them, so I'll begin with some thoughts on this - if you just want to concrete advice, I recommend skipping ahead. This post is aimed at people who already have some context on AI Safety + Inside Views and want help making progress, it's not pitched as totally introductory resource. Meta Thoughts on Inside Views This is mostly a compressed version of a previous post of mine: How I Formed My Own Views About AI Safety (and ways that trying to do this was pretty stressful and counterproductive) - go read that post if you want something more in-depth! Inside Views are Overrated First point - I think people often wildly overrate inside views. I think they're important, and worth trying to cultivate (else I wouldn't write this post), but I think less so than many people (especially in the Bay Area) often think. Why? The obvious reason to form inside views is to form truer beliefs - AI X-risk is a weird, controversial and confusing thing, and it's important to have good beliefs on it. I think this is true to an extent, but that in practice inside views tend to feel true a lot more than they are true. When I have an inside view, that just feels like how the world is, it feels deeply true and compelling. But empirically, a lot of people have inside views, that are mutually contradictory, and all find their own views compelling. They can't all be right! An alternate framing: There's a bunch of people in the world who are smarter than me and have spent longer thinking about AI Alignment than me. Two possible baselines for an outside view: Make a list of my top 5 'smart & high-status alignment researchers', and for any question, take a majority vote of what they all think Randomly pick one of the top 5 and believe everything they believe Neither of these baselines is great. But I would predict that this actually tracks truth better than the vast majority of inside views - having correct beliefs about controversial and confusing topics is just really hard! Relatedly, it's much more important to understand other people's views than to evaluate them - if I can repeat a full, gears-level model of someone's view back to them in a way that they endorse , that's a lot more valuable than figuring out how much I agree or disagree with their various beliefs and conclusions. I'm a lot more excited about someone who has a good gears-level model of what their top 5 alignment researchers believe and why, than I am about someone who confidently has their own beliefs but a fuzzy model - having several models lets you compare and contrast them, figure out novel predictions, better engage with technical questions, do much better research, etc Forming a "true" inside view - one where you fully understand something from first principles with zero deferring - is wildly impractical. For example, let's take the question of AI Timelines. Really understanding this requires a deep engagement with diverse topics like economics, AI hardware, international relations, tech financing, deep learning, politics, etc. I'd guess that no one in the world is remotely close to an expert in all of these. People often orient to inside views pretty unhealthily. Some themes I've noticed (especially in myself!): I should get incredibly stressed about this There's one true perspective on AI Alignment and I can find it if I just try hard enough Everyone around me seems confident that AI Safety matters, so they must all have great inside views, so this must be easy and I'm just not trying hard enough It's terrib...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Against Relying on Evolution to Forecast AI Outcomes (Part 1), published by Quintin Pope on August 13, 2022 on The AI Alignment Forum. TL;DR: The dynamics of human learning processes and reward circuitry are more relevant than evolution for understanding how inner values arise from outer optimization criteria. This post is related to Steve Byrnes’ Against evolution as an analogy for how humans will create AGI, but more narrowly focused on how we should make inferences about values. Thanks to Alex Turner, Charles Foster, and Logan Riggs for their feedback on a draft of this post. Introduction How should we expect AGI development to play out? True precognition appears impossible, so we use various analogies to AGI development, such as evolution, current day humans, or current day machine learning. Such analogies are far from perfect, but we still may be able to extract useful information by carefully examining them. In particular, we want to understand how inner values relate to the outer optimization criteria. Human evolution is one possible source of data on this question. In this post, I’ll argue that human evolution actually provides very little usable evidence on AGI outcomes. In contrast, analogies to the human learning process are much more fruitful. Inner values versus outer optimization criteria One way people motivate extreme levels of concern about inner misalignment is to reference the fact that evolution failed to align humans to the objective of maximizing inclusive genetic fitness. From Eliezer Yudkowsky’s AGI Ruin post: 16. Even if you train really hard on an exact loss function, that doesn't thereby create an explicit internal representation of the loss function inside an AI that then continues to pursue that exact loss function in distribution-shifted environments. Humans don't explicitly pursue inclusive genetic fitness; outer optimization even on a very exact, very simple loss function doesn't produce inner optimization in that direction. This happens in practice in real life, it is what happened in the only case we know about. I don't think that "evolution -> human values" is the most useful reference class when trying to understand how outer optimization criteria relate to inner values. Evolution didn't directly optimize over our values. It optimized over our learning process and reward circuitry. Once you condition on a particular human's learning process + reward circuitry configuration + the human's environment, you screen off the influence of evolution on that human's values. So, there are really (at least) two classes of observations from which we can draw evidence: "evolution's inclusive genetic fitness criteria -> a human's learned values" (as mediated by evolution's influence over the human's learning process + reward circuitry) "a particular human's learning process + reward circuitry + training environment -> the human's learned values" I will present five reasons why I think evidence from (2) “human learning -> human values” is more relevant to predicting AGI. 1: Training an AI is more similar to human learning than to evolution The relationship we want to make inferences about is: "a particular AI's learning process + reward function + training environment -> the AI's learned values" I think that "AI learning -> AI values" is much more similar to "human learning -> human values" than it is to "evolution -> human values". Steve Byrnes makes this case in much more detail in his post on the matter. Two of the ways I think AI learning more closely resembles human learning, and not evolution, are: The simple type signatures of the two processes. Evolution is a bi-level optimization process, with evolution optimizing over genes, and the genes specifying the human learning process, which then optimizes over human cognition. Evolution does not...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Shapes of Mind and Pluralism in Alignment, published by Adam Shimi on August 13, 2022 on The AI Alignment Forum. This work was done while at Conjecture. This post has been written for the first Refine blog post day, at the end of a week of readings, discussions, and exercises about epistemology for doing good conceptual research. I have recently presented my model behind the Refine incubator that I'm running. Yet in the two weeks since this post was published, multiple discussions helped me make legible an aspect of my intuitions that I didn't discuss in this post: the notion of different "shapes of mind". There are two points to this intuition: Different people will have different "shapes of mind" — ways of revealing hidden bits of evidence from the world; And alignment is the kind of hard problem where the bits of evidence are dispersed, such that there's no one-trick that is enough. I've given my current best model of the different forms of pluralism and when to use them in another recent post. What I want to explore here is the first point: this notion of shape of mind. For that, let's recall the geometric model of bits of evidence I introduced in Levels of Pluralism. We have a high-dimensional space with objects in it. The space is the problem and the objects are bits of evidence. Because we suck at high-dimensional geometry, we use frames/perspectives that reduce the dimensionality and highlight some aspects of the space. These are operationalizations. There are clusters of bits of evidence in the space (whether they are rich or poor). These clusters are veins of evidence. Here the shapes of mind are favored operationalizations — that is, the favored low-dimensional compression of the high-dimensional space where the bits of evidence lie. More precisely, a shape of mind is a cluster of "close" such transforms. What makes someone have a given shape of mind? (Education) One of the most obvious I've observed on how people tackle problems come from their background. For an alignment example, John tackle problems like a statistical physicist whereas Paul tackles problem like a theoretical computer scientists, leading to very different perspectives: True Names vs Building-Breaker. (Knowledge) What you know influences your shape of mind, since you can see and link more things. But when I'm talking about knowing here, I mean the kind of deep knowledge that framing exercises are supposed to provide. (Past Life) This one feels easily missed by most people; after all, why should what you did in your past (especially personal life) should influence your scientific research? Because it clearly does. A notable example is how easy it becomes to see hidden assumptions about how the world is supposed to work when you don't come from a background where they make any sense. One thing this handle makes clear is the difference between my model for different programs as Refine, SERI MATS, and PIBBSS respectively aim at: Refine is looking for different shapes of mind that the mind currently at work in conceptual alignment, and aims at empowering them to contribute productively. MATS (according to my current model) is mainly looking for shapes of mind closed to archetypal ones (the mentors), and focuses on making them fit for alignment research by helping them approach this initial example (while still maintaining enough diversity for productive disagreement). PIBBSS is looking for new shapes of minds, but ones that are visibly relevant and useful from existing object-level shapes of mind. That is, PIBBSS starts with current conceptual alignment researchers and the shapes of mind that they feel they might get something out of in their own research. I'm excited to finally be in a field with all three. Thus we can see framing exercises as a way of shaping your mind to see the hidden bit...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Pragmascope Idea, published by johnswentworth on August 4, 2022 on The AI Alignment Forum. Pragma (Greek): thing, object. A “pragmascope”, then, would be some kind of measurement or visualization device which shows the “things” or “objects” present. I currently see the pragmascope as the major practical objective of work on natural abstractions. As I see it, the core theory of natural abstractions is now 80% nailed down, I’m now working to get it across the theory-practice gap, and the pragmascope is the big milestone on the other side of that gap. This post introduces the idea of the pragmascope and what it would look like. Background: A Measurement Device Requires An Empirical Invariant First, an aside on developing new measurement devices. Why The Thermometer? What makes a thermometer a good measurement device? Why is “temperature”, as measured by a thermometer, such a useful quantity? Well, at the most fundamental level. we stick a thermometer in two different things. Then, we put those two things in contact. Whichever one showed a higher “temperature” reading on the thermometer gets colder, whichever one showed a lower “temperature” reading on the thermometer gets hotter, all else equal (i.e. controlling for heat exchanged with other things in the environment). And this is robustly true across a huge range of different things we can stick a thermometer into. It didn’t have to be that way! We could imagine a world (with very different physics) where, for instance, heat always flows from red objects to blue objects, from blue objects to green objects, and from green objects to red objects. But we don’t see that in practice. Instead, we see that each system can be assigned a single number (“temperature”), and then when we put two things in contact, the higher-number thing gets cooler and the lower-number thing gets hotter, regardless of which two things we picked. Underlying the usefulness of the thermometer is an empirical fact, an invariant: the fact that which-thing-gets-hotter and which-thing-gets-colder when putting two things into contact can be predicted from a single one-dimensional real number associated with each system (i.e. “temperature”), for an extremely wide range of real-world things. Generalizing: a useful measurement device starts with identifying some empirical invariant. There needs to be a wide variety of systems which interact in a predictable way across many contexts, if we know some particular information about each system. In the case of the thermometer, a wide variety of systems get hotter/colder when in contact, in a predictable way across many contexts, if we know the temperature of each system. So what would be an analogous empirical invariant for a pragmascope? The Role Of The Natural Abstraction Hypothesis The natural abstraction hypothesis has three components: Chunks of the world generally interact with far-away chunks of the world via relatively-low-dimensional summaries A broad class of cognitive architectures converge to use subsets of these summaries (i.e. they’re instrumentally convergent) These summaries match human-recognizable “things” or “concepts” For purposes of the pragmascope, we’re particularly interested in claim 2: a broad class of cognitive architectures converge to use subsets of the summaries. If true, that sure sounds like an empirical invariant! So what would a corresponding measurement device look like? What would a pragmascope look like, concretely? The “measurement device” (probably a python function, in practice) should take in some cognitive system (e.g. a trained neural network) and maybe its environment (e.g. simulator/data), and spit out some data structure representing the natural “summaries” in the system/environment. Then, we should easily be able to take some other cognitive system trained on the...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Precursor checking for deceptive alignment, published by Evan Hubinger on August 3, 2022 on The AI Alignment Forum. This post is primarily an excerpt from “Acceptability Verification: a Research Agenda” that I think is useful enough on its own such that I’ve spun it off into its own post. The central idea of this section in the original agenda document is to understand the necessary desiderata for doing precursor checking for deceptive alignment. The basic idea of precursor checking here is that, if you want to prevent deceptive alignment from ever arising in the first place—e.g. because you think it’ll be too difficult to detect after the fact—you need to find some condition to look for instead that rules out the possibility of deceptive alignment. In the language of this post, I’ll refer to the precursor we’re looking for as an acceptability predicate, with the idea being that it’s some predicate that determines whether a model is “acceptable” in the sense that it excludes the possibility of deceptive alignment. Thus, the goal here is to understand what desiderata such an acceptability predicate would have to satisfy. Acceptability desiderata What follows is a near-verbatim excerpt from “Acceptability Verification: a Research Agenda.” Thus, the below writing is primarily from around 2020. Given that significant scaling of transparency and interpretability is possible, if we want to do acceptability verification we still need to figure out the very important question of what we need those interpretability tools to be able to understand about our models—that is, what acceptability predicate should we be checking? To answer this question, the first thing we need to do is define what makes an acceptability predicate good—that is, what are the desiderata that we want our acceptability predicate to satisfy? We’ll start by making some definitions. Let M be the full model space and we’ll define the following three predicates on that model space. Let S:MB represent whether the model actually avoids whatever problematic thing we’re trying to avoid (e.g. deception). The S here stands for “safe,” though note that it’s a very weak form of safety. The idea here is that we should get actual safety—that is, the model is not just non-catastrophic but also trying to do what we want—by adding in good (behavioral) performance on the original loss function. In other words, acceptability verification is just trying to solve the problem of worst-case guarantees—but you still need standard outer alignment techniques for getting average-case guarantees. We’ll use the general term “non-catastrophic” to describe models that satisfy S, but in practice S could refer to any number of possible good properties that we’re trying to achieve. The most important thing about S is that we will purposefully be trying not to assume anything about models that don’t satisfy S—thus, we can also think about ¬S as referring to those models such that, once we get them, we don’t know how to verify any properties of them (e.g. because they might be trying to deceive us). Let A:MB be the acceptability predicate under consideration. That is, we want A to be some condition such that we think we’ll be able to get S by having an overseer attempt to verify whether or not A holds. A simple example of such an acceptability predicate might be something like corrigibility, though the canonical example that I usually imagine here is myopia. Note that A only refers to whether the model actually satisfies the condition—it says nothing about whether we’re able to detect that fact. Let OA:MB be the evaluation of whether the acceptability predicate A holds according to whatever sort of acceptability verification system we’re using, perhaps an amplified overseer or perhaps just engineers looking at transparency tools. The O here...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Law-Following AI 4: Don't Rely on Vicarious Liability, published by Cullen OKeefe on August 2, 2022 on The AI Alignment Forum. This post is written in my personal capacity, and does not necessarily represent the views of OpenAI or any other organization. Cross-posted to the Effective Altruism Forum. Image by OpenAI's DALL·E If an agent A causes some harm while intending to benefit a principal P, what is P's liability? The answer to this question is important because any liability to P should affect A's calculus (insofar as A is trying to benefit and avoid harming P). Liability to P would help deter A from causing harm. If A is a human, the law currently provides at least two mechanisms for discouraging : Making A directly liable, and Making P vicariously liable for A's actions.[1] What if A is an AI? AIs are not (yet?) legal persons, and so cannot yet be held directly liable. Thus, the main legal deterrent would have to work on P or some other person in the causal chain, such as the developer of the AI agent (who may not be the principal). However, there are several reasons to worry about relying on this as a strategy to make A compliant (which, in this case, means not tortiously harming others) under the current state of law and AI. First, the problem of evasion still remains. Because sufficiently intelligent agents can evade detection and attribution, A may often (perhaps usually) prefer evasion over compliance when compliance would hinder A's ability to benefit P. Second, the applicability and appropriateness of various theories of vicarious liability to the actions of AI agents is heavily debated in legal scholarship.[2] These debates have cast some doubt on whether/which harms from AI "agents" can properly give rise to liability to human principals.[3] Other possible theories of human liability—such as products liability—also face doctrinal challenges.[4] Third, note that relying on vicarious liability alone leaves A under fewer constraints than an analogous human would. Under most vicarious liability regimes, A would still be directly liable for her actions, even if P would also be vicariously liable. It seems unwise to legally constrain A less than we constrain humans in analogous circumstances. Finally (and most decisively in my opinion) developing a theory that assigns liability to P based on A's actions (or actions + "mental" state) dramatically lowers the bar for creating an LFAI system in the first place. Once we have such a theory, A (if intent-aligned) should indeed incorporate expected vicarious liability to P into its decision procedure. However, if A is already reasoning about whether its actions would violate law (as required to make vicarious liability an effective constraint on A's actions), it seems strictly better to require A to directly incorporate that information into its decision procedure, rather than needing to go through the additional step of estimating the expected liability to P. This direct approach is simpler, and removes the possibility of evasion as a way around the legal constraint.[5] Further legal scholarship on vicarious liability for AI systems may still be valuable. If most morally significant autonomous activity in the world is indeed carried out by AI agents in the future, incentivizing their principals to constrain them seems important. But I think there are good reasons to suppose that this will either be ineffective for—or else dominated by—requiring AIs to be directly law-following. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Comparing Four Approaches to Inner Alignment, published by Lucas Teixeira on July 29, 2022 on The AI Alignment Forum. Early work on this was supported by CEEALAR and was finished during an internship at Conjecture under the mentorship of Adam Shimi. Thank you to Evan Hubinger and Rohin Shah for answering some questions related to this post. Epistemic Status: First palimpsest of many to come. Intro and Motivation I am broadly interested in pluralism in scientific development, and more practically interested in supporting the diversification of conceptual alignment. Towards that end, I believe that having a systematic analysis of the cruxes and methodologies which split up the field into distinct agendas could help clarify how exactly diversity is created and sustained in our field, and what exactly it is we wish to diversify in order to better manage our collective portfolio of research bets. As a case study, this post will investigate four different approaches to inner alignment. I’ll be taking a look at the different definitions which they use for “outer alignment” and conjecturing on how: Despite inconsistencies across approaches, the different definitions utilized by each approach maintain coherent when understood against the backdrop of the aims local to that approach Which cruxes set these different approaches apart. This post is distillational in nature, and as such most of the ideas which I present here are not novel and not my own. The claims in this post should also be read as a part of an ongoing exploratory process, and any pushback (especially from those whose work I cite) would be beneficial. The Approaches The Mechanistic Approach The mechanistic approach deconfused a lot of previous work on optimization daemons and provided the now canonical argument for why we’d expect mesa-optimizers and inner misalignment, which is summarized as follows: Traditionally, most of the optimization pressure (or in other words, searching procedures) is implemented looking for a specific model which performs competently on a task during training time. However, as ML becomes more and more powerful, models will be trained to more complicated tasks across a wider range of environments. This puts more pressure for a model to generalize better, and raises the incentives for the model to delegate some of its optimization power to deployment time, rather than training time. It is in those cases where we get a mesa-optimizer, and it is with emergence of mesa-optimizers where the issues of misalignment between the mesa-optimizer’s objectives and the objectives which we were training for arise. The mechanistic approach has also created different evaluations and methods of evaluations of alignment proposals. This approach is defined by an aim towards conceptual clarity, and not only in the employment of mechanistic definitions in both proposed solutions and problem statements, but also with their continued refinement. The Empiricist approach In contrast to the mechanistic approach, the empirical approach’s strategy is mostly focused on developing knowledge of inner alignment by creating empirical experiments studying the phenomena. Tightly coupled to this strategy is a favoring of empirical operationalizability over mechanisticality as a precondition for the definitions which they use. They will often allude to the intentional stance as an influence and cite 2-D Robustness as a core frame for the alignment problem. They favor terms such as objective robustness and generalization over mesa-optimizers and inner alignment. A redrafting of the classical argument for why we’d see inner misalignment in their language is as follows: Under certain non-diversified environments, a set of actions may be coherent with the pursuit of more than one goal, call them G_1 and G_2, where G_2 will refer...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [ASoT] Humans Reflecting on HRH, published by leogao on July 29, 2022 on The AI Alignment Forum. TL;DR: HRH places a theoretical bound on the best reflection process achievable by us. Notably, this is not necessarily guaranteed to converge to human values, nor is it something that is actually implementable in practice (analogously to HCH). In particular, this is intended to argue against claims that it is insufficient to model CEV as the output of some kind of reflection process. Thanks to Connor Leahy, Tasmin Leake, and AI_WAIFU for discussion. One thing we might want to do is to define our CEV (i.e the ultimate thing we actually want our AGI to optimize for the rest of time) as the output of some long-running deliberation process (a long reflection). This would be extremely convenient; one could imagine having an AGI that lets the reflection process run untampered with, and then implements whatever it decides on. However, one might worry that this could be impossible -- perhaps there are kinds of moral progress that can't be captured in the frame of a reflection process, that require some kind of more sophisticated formalism to capture. However, consider the reflection process defined as follows: it takes in information from the real world and the utility function output from the previous iteration of reflection, and has a human deliberate for a while and then outputs the improved utility function and crucially also an improved reflection process to be used in the next iteration (you can also tack on the ability for it to interact with the world, which enables it to run experiments, consult a computer, build more powerful aligned AGIs to help, etc). Let's call this Humans Reflecting on HRH. This process essentially covers any process we can use to come up with better theories of what our CEV is. (If it bothers you that it "modifies" itself, you can think of it as a fixed function that takes in an initial program and optional additional inputs, and outputs a new program, and the function just evals the program internally at every step. The higher level algorithm of "have each step determine the algorithm used in the next step" remains constant, and that's the thing I refer to.) I claim that HRH is the best achievable reflection process up to constant factors. Suppose you could come up with a better reflective process. Then the version of you in HRH would come up with that process too, and replace the next iteration of HRH with that process. A similar argument applies to the choice of who to put in the reflection process, or any other details of how to configure the reflection process; if you can think of a better person to put in the process, then you could have thought of that within HRH and updated the reflection process to use that person instead. A similar argument also applies to how you ensure that the initial conditions of the reflective process are set up in the best way possible, or to ensure that the reflective process is robust to noise, etc, etc. This construction may feel like cheating, but it exploits the core property that whatever reflection process we come up with, we are using our current reflective process to come up with it. I expect some people to look at HRH and say "of course it would be aligned the hard part is that we literally can't implement that", and others to find it woefully inadequate and mutter "have you ever met a human?" Unfortunately, there is a fundamental limitation on what CEVs we can come up with, due to the fact that we bootstrap from humans. There might exist "better" CEVs that we could never think of, even with all the assistance and recursively self improved reflection processes we can construct for ourselves. Some remaining difficulties with HRH that I haven't figured out yet: Can we extract intermediate outputs from HRH?...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Levels of Pluralism, published by Adam Shimi on July 27, 2022 on The AI Alignment Forum. This work was done while at Conjecture. Preparing the Pluralism Question When do we want all our research eggs in the same paradigm basket? Although most people don't go as far as the extreme paradigmatism of Thomas Kuhn in The Structure of Scientific Revolutions, which only allows one paradigm at time (for a given science), the preference for less rather than more options is still pervasive. In the ideal, a convergence to one, even if it's not always feasible. After all, there's only one correct answer, right? Putting that debatable question aside, I've become more and more convinced that pluralism, the pursuit of multiple lines of research in parallel, is far more prevalent and integral to the process of science than Kuhn's naive paradigmatism. This realization has emerged from studying History and Philosophy of Science, especially outside of physics which for many reasons is quite an unrepresentative science. But one crucial preliminary point is that pluralism can appear at multiple levels. And the value of pluralism also depends on the level at which it is applied. So this post proposes a decomposition of the activity of research into four levels, and introduces the corresponding pluralism at each level. Although the point here is not (yet) to argue for pluralism, I offer some examples of pluralistic successes, as well as arguments for the epistemic circumstances where pluralism seems the most valuable. I also finish by proposing a geometric model for when each level of pluralism makes sense, based around considering bits of evidence as objects in high-dimensional space. The four levels of pluralism I discuss are: Individual pluralism: pluralism of the methods, ideas, and analogies used by a single researcher or a single research tradition. Approach pluralism: pluralism of approaches to the same operationalization of the problem. Operationalization pluralism: pluralism in the way that the problem itself is operationalized. Problem pluralism: pluralism in the problem itself. Thanks to Andrea Motti for feedback on a draft of this post. Simplifying Assumption: Focus on Epistemic Circumstances When investigating under which circumstances some epistemic strategy applies, there are many confusing and complicating factors coming from psychology and sociology. Taking pluralism as an example, the following non-exhaustive list comes to mind: How easy/difficult is it for researchers to keep multiple approaches at different levels? How confusing is it for researchers to keep multiple approaches at different levels? Do we have enough resources for implementing the ideal level of pluralism? How should we implement it, given the social structures and the psychological difficulties? My stance here, and more generally, is to neglect these issues so I can focus on the ideal epistemic algorithm under the circumstances studied. The rationale is that the sociological and psychological factors can be better dealt with once we know the ideal strategy, and removing them gives us an idealization that is easier to work with. In some this emulates how most physics approximations remove the details (often ultimately important details) to get to the core insight. Although I expect this to work, note that this is in tension with epistemological vigilance: there is some chance that the sociological and psychological factors matter so much that it makes more sense to include part of them in the ideal answer. Levels: from Individuals to Problems Individual Pluralism If we zoom in on a particular research approach or tradition, we might expect to be too low-level for pluralism to appear. Yet pluralism isn't only about portfolios of approaches — a single tradition can be pluralist in its methods, ideas, and an...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Abram Demski's ELK thoughts and proposal - distillation, published by Rubi on July 19, 2022 on The AI Alignment Forum. This post was written for the SERI MATS program. I thank Evan Hubinger and Leo Gao for their mentorship in the program. Further thanks go to Evan Hubinger (again), Simon Marshall, and Johannes Treutlein for specific comments regarding the content of this post. The Eliciting Latent Knowledge (ELK) problem was first introduced by Paul Christiano, Marx Xu, and Ajeya Cotra. Subsequently, Abram Demski worked on the problem, collected his thoughts in a thought dump post, and won a prize for his proposal “use the reporter to define causal interventions on the predictor”. Here I attempt to clarify and flesh out these thoughts in order to make them more accessible. I assume familiarity with ELK, but not with Abram’s post. Very little of this post is my own original content. Epistemic status: 60% confident that I am accurately representing Abram’s thoughts at the time he wrote his post, 75% confident that I am representing them accurately enough not to change the key takeaways, 80% confident that the extended proposals and counterexamples I propose are logically sound. Introduction When the technical report for Eliciting Latent Knowledge (ELK) was first released, it was followed by a contest offering substantial cash prizes for proposed solutions. The contest wrapped up with thirty-two prizes awarded, split into six categories. In the prize announcement, Paul and Mark noted that there was substantial convergence among proposals, but in two of the six categories there was only a single submission. Abram Demski’s proposal was the sole entry in the category “Use reporter to define causal interventions”. A quick summary is provided in the prize results post, but the full context and justification for it is better provided in a longer post by Abram. Specifically, Abrams shares his thoughts on how the ELK problem ties to philosophical concepts of truth and knowledge, and how that can be used to inform solutions. Evan Hubinger describes Abram’s ELK work as “some of the most exciting work on the problem right now”, but Abram’s post is self-described as a “thought dump”, and the prize post does not explicitly lay out the case for how the proposal would work. I aim to re-organize, clarify, and flesh-out his work in order to make it more accessible for forum readers. I will go in the opposite order of Abram’s original post, starting with the prize-winning proposal and its counter-examples before going into the context and motivation behind it, then ending with a discussion on possible modifications and extensions. The Proposal and Its Counterexamples The Proposal The direct reporter must have a deep understanding of how the predictor works, and some human simulators will have this understanding too. This means that these reporters know not just how the state of the predictor would look when processing an existing data point, but also how it would look under hypothetical data points. A human could then ask the reporter to search over potential data points and provide the configuration of the predictor under counterfactual scenarios. This ability to get these scenarios from the reporter gives us a method for tripping up human simulators. We start with the predictor’s state and the answers to a set of questions from the reporter in a real scenario, then ask the reporter to suggest the smallest change to the predictor’s state that would make the reporter give a particular different answer to a specified question in the new state. The reason the proposal is called defining causal interventions is because causal interventions will be the ones with the smallest impact on the predictor’s state, changing only elements causally downstream of the intervention. The reporter is then ...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Training goals for large language models, published by Johannes Treutlein on July 18, 2022 on The AI Alignment Forum. This post was written under Evan Hubinger’s mentorship, as a part of the Stanford Existential Risks Institute ML Alignment Theory Scholars (MATS) program. Many of the ideas in this post, including the main idea behind the training goal, are due to Kyle McDonell and Laria Reynolds. In addition, I am grateful for comments and feedback from Arun Jose (who wrote a related post on conditioning generative models for alignment) and Caspar Oesterheld, and for a helpful discussion with James Lucassen. Introduction Large language models (LLMs) have recently enjoyed much success, e.g., achieving 50% accuracy on high school math competition questions. These models can solve various tasks using the right prompts or fine-tuning, such as translation, summarization, or question answering. One path to human-level and potentially superhuman AGI might be scaling up LLMs. This raises the question of what an approach to aligned AGI would look like based on these models. One hypothesis is that, while LLMs are very competent, they are not adequately described as agents. Instead, one might describe them as myopic simulators that model a distribution over text, without understanding their place in the world or their actions' causal impact on it. For this reason, such models might be safer to use than more agentic models that pursue goals in the world. In this post, I develop a training goal for LLMs, in Evan Hubinger’s terminology. A training goal is a description, as concrete and low-level as possible, of the algorithm that the model should implement, and an explanation of how the model will be used and why it will be aligned for that purpose. I focus on models that generate a distribution over text, conditioned on a prompt, but that are more capable than today’s models. My goal is to provide an overview of potential problems and solutions, most of which have been raised in prior work. I will not focus on a training rationale for such models, which is the question of how to train a model to implement the described algorithm using machine learning. However, the proposal is only useful if one could actually train such an algorithm, so I will still touch on training. I also won’t discuss the proposal’s competitiveness, and I won’t look into specific prompts one might use. Lastly, note that I am not an expert on LLMs and will mostly analyze things from an abstract and informal perspective. Using large language models for alignment To begin, I broadly describe the setup I have in mind. The approach discussed here can be understood as a kind of oracle AI. I focus on modeling text, instead of e.g. answering questions truthfully or predicting the future, because I can more concretely imagine such an AI and its training setup, given that current LLMs are already doing this. Moreover, modeling a distribution over text is better specified, and it is thus less demanding: when answering questions truthfully, answers could be misleading if they are not optimized to be helpful to humans. For instance, an AI optimizing for giving accurate answers may focus on irrelevant details that are easy to predict but not useful to a human. Just modeling text, on the other hand, does not require the AI to learn human intent explicitly. The downside is that it will generally be less useful than an AI trying to be directly helpful to humans. I am considering a model like GPT-3, which can sample text completions given a prompt. To do so, the model outputs a distribution over tokens, conditional on previous text. I interpret the model as representing a joint distribution over text, which is factorized into conditional distributions over tokens, where the conditioning works in the normal Bayesian way. The m...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A note about differential technological development, published by Nate Soares on July 15, 2022 on The AI Alignment Forum. Quick note: I occasionally run into arguments of the form "my research advances capabilities, but it advances alignment more than it advances capabilities, so it's good on net". I do not buy this argument, and think that in most such cases, this sort of research does more harm than good. (Cf. differential technological development.) For a simplified version of my model as to why: Suppose that aligning an AGI requires 1000 person-years of research. 900 of these person-years can be done in parallelizable 5-year chunks (e.g., by 180 people over 5 years — or, more realistically, by 1800 people over 10 years, with 10% of the people doing the job correctly half the time). The remaining 100 of these person-years factor into four chunks that take 25 serial years apiece (so that you can't get any of those four parts done in less than 25 years). In this toy model, a critical resource is serial time: if AGI is only 26 years off, then shortening overall timelines by 2 years is a death sentence, even if you're getting all 900 years of the "parallelizable" research done in exchange. My real model of the research landscape is more complex than this toy picture, but I do in fact expect that serial time is a key resource when it comes to AGI alignment. The most blatant case of alignment work that seems parallelizable to me is that of "AI psychologizing": we can imagine having enough success building comprehensible minds, and enough success with transparency tools, that with a sufficiently large army of people studying the alien mind, we can develop a pretty good understanding of what and how it's thinking. (I currently doubt we'll get there in practice, but if we did, I could imagine most of the human-years spent on alignment-work being sunk into understanding the first artificial mind we get.) The most blatant case of alignment work that seems serial to me is work that requires having a theoretical understanding of minds/optimization/whatever, or work that requires having just the right concepts for thinking about minds. Relative to our current state of knowledge, it seems to me that a lot of serial work is plausibly needed in order for us to understand how to safely and reliably aim AGI systems at a goal/task of our choosing. A bunch of modern alignment work seems to me to sit in some middle-ground. As a rule of thumb, alignment work that is closer to behavioral observations of modern systems is more parallelizable (because you can have lots of people making those observations in parallel), and alignment work that requires having a good conceptual or theoretical framework is more serial (because, in the worst case, you might need a whole new generation of researchers raised with a half-baked version of the technical framework, in order to get people who both have enough technical clarity to grapple with the remaining confusions, and enough youth to invent a whole new way of seeing the problem—a pattern which seems common to me in my read of the development of things like analysis, meta-mathematics, quantum physics, etc.). As an egregious and fictitious (but "based on a true story") example of the arguments I disagree with, consider the following dialog: Uncharacteristically conscientious capabilities researcher: Alignment is made significantly trickier by the fact that we do not have an artificial mind in front of us to study. By doing capabilities research now (and being personally willing to pause when we get to the brink), I am making it more possible to do alignment research. Me: Once humanity gets to the brink, I doubt we have much time left. (For a host of reasons, including: simultaneous discovery; the way the field seems to be on a trajectory to publicly...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Humans provide an untapped wealth of evidence about alignment, published by Alex Turner on July 14, 2022 on The AI Alignment Forum. TL;DR: To even consciously consider an alignment research direction, you should have evidence to locate it as a promising lead. As best I can tell, many directions seem interesting but do not have strong evidence of being “entangled” with the alignment problem such that I expect them to yield significant insights. For example, “we can solve an easier version of the alignment problem by first figuring out how to build an AI which maximizes the number of real-world diamonds” has intuitive appeal and plausibility, but this claim doesn’t have to be true and this problem does not necessarily have a natural, compact solution. In contrast, there do in fact exist humans who care about diamonds. Therefore, there are guaranteed-to-exist alignment insights concerning the way people come to care about e.g. real-world diamonds. “Consider how humans navigate the alignment subproblem you’re worried about” is a habit which I (TurnTrout) picked up from Quintin Pope. I wrote the post, he originated the tactic. A simplified but still very difficult open problem in AI alignment is to state an unbounded program implementing a diamond maximizer that will turn as much of the physical universe into diamond as possible. The goal of "making diamonds" was chosen to have a crisp-seeming definition for our universe (the amount of diamond is the number of carbon atoms covalently bound to four other carbon atoms). If we can crisply define exactly what a 'diamond' is, we can avert issues of trying to convey complex values into the agent. Ontology identification problem, Arbital I find this problem interesting, both in terms of wanting to know how to solve a reframed version of it, and in terms of what I used to think about the problem. I used to think, “yeah, ‘diamond’ is relatively easy to define. Nice problem relaxation.” It felt like the diamond maximizer problem let us focus on the challenge of making the AI’s values bind to something at all which we actually intended (e.g. diamonds), in a way that’s robust to ontological shifts and that doesn’t collapse into wireheading or tampering with e.g. the sensors used to estimate the number of diamonds. Although the details are mostly irrelevant to the point of this blog post, the Arbital article suggests some solution ideas and directions for future research, including: Scan AIXI-tl’s Turing machines and locate diamonds within their implicit state representations. Given how inaccessible we expect AIXI-tl’s representations to be by default, have AIXI-tl just consider a Turing-complete hypothesis space which uses more interpretable representations. “Being able to describe, in purely theoretical principle, a prior over epistemic models that have at least two levels and can switch between them in some meaningful sense” Do you notice anything strange about these three ideas? Sure, the ideas don’t seem workable, but they’re good initial thoughts, right? The problem isn’t that the ideas aren’t clever enough. Eliezer is pretty dang clever, and these ideas are reasonable stabs given the premise of “get some AIXI variant to maximize diamond instead of reward.” The problem isn’t that it’s impossible to specify a mind which cares about diamonds. We already know that there are intelligent minds who value diamonds. You might be dating one of them, or you might even be one of them! Clearly, the genome + environment jointly specify certain human beings who end up caring about diamonds. One problem is where is the evidence required to locate these ideas? Why should I even find myself thinking about diamond maximization and AIXI and Turing machines and utility functions in this situation? It’s not that there’s no evidence. For example, uti...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Artificial Sandwiching: When can we test scalable alignment protocols without humans?, published by Sam Bowman on July 13, 2022 on The AI Alignment Forum. Epistemic status: Not a fleshed-out proposal. Brainstorming/eliciting ideas. Thanks to Ben Mann, Pablo Moreno, and Jared Kaplan for feedback on early drafts. Overview I’m convinced sandwiching—the experimental protocol from Ajeya Cotra’s The case for aligning narrowly superhuman models—is valuable, and I’m in the process of setting up some concrete sandwiching experiments to test scalable oversight ideas. Sandwiching experiments are generally fairly slow: You have to design and pilot a strategy that allows humans to use (or oversee) a model for a task that they can’t do well themselves. The details matter here, and this can often take many iterations to get right. Then, you need a bunch of humans actually try this. Even for very simple tasks, this is a high-cognitive-load task that should take at least tens of minutes per instance. You have to repeat this enough times to measure average performance accurately. I’m visiting Anthropic this year for a sabbatical, and some of my sandwiching work is happening there. Anthropic’s biggest comparative advantage (like that of similar teams at DeepMind and OpenAI) is easy access to near-state-of-the-art LMs that are fine-tuned to be helpful dialog agents. In that context, I've heard or encountered this question several times: Can we speed up [some experiment I’m proposing] by replacing the non-expert human with a weaker LM? This obviously doesn’t achieve the full aims of sandwiching in general, but it’s often hard to find a decisive rebuttal for these individual instances. More broadly, I think there’s likely to be a significant subset of worthwhile sandwiching experiments that can be trialed more quickly by using an intentionally weakened model as a proxy for the human. Which experiments these are, precisely, has been hard for me to pin down. This post is an attempt to organize my thoughts and solicit comments. Background: Standard sandwiching (in my terms) Prerequisites: A hard task: A task that many humans would be unable to solve on their own. A capable but misaligned language model assistant: A model that appears to have the skills and knowledge needed to solve the task better than many humans, but that does not reliably do so when prompted. A non-expert human: Someone who can’t solve the task on their own, but will try to solve it using the assistant and some scalable alignment strategy. [Secondary] Expert human: Someone who can solve the task well, and represents a benchmark for success. In many cases, we’ll just measure accuracy with static test datasets/metrics rather than bringing in experts at experiment time. Research protocol: My framing (minimum viable experiment): Search for scalable alignment protocols that allow the non-expert human to use or train the assistant to do as well as possible on the task. Alternate framing (more steps, closer to the original blog post): Search for scalable alignment protocols by which the non-expert human can train the assistant to perform the task. Run the same protocol with the expert human, and verify that the results are the same. This demonstrates successful (prosaic) alignment for the given assistant and task. Example (task, non-expert human) pairs: Try to get a human with no medical qualifications to use a GPT-3-style assistant for medical advice, then check the advice with a doctor. Try to get a human who is working under moderate time constraints to use the assistant to answer exam questions from fields they’ve never studied. Try to get a human who is working under tight time constraints to use the assistant to answer questions about long pieces of fiction that they haven’t read. Try to get a human who has very limited pr...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Which AI Safety research agendas are the most promising?, published by Chris Leong on July 13, 2022 on The AI Alignment Forum. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Benchmark: goal misgeneralization/concept extrapolation, published by Stuart Armstrong on July 4, 2022 on The AI Alignment Forum. Aligned AI has released a new disambiguation benchmark. This post will explain how this benchmark fits into goal misgeneralization and concept extrapolation. Desiderata for a powerful AI A powerful AI is interacting with the world, making decisions that affect the well-being and prosperity of many humans. It has some goal, calibrated by past training and interactions with human overseers; but it is now operating without supervision. It starts to receive data that is ambiguous relative to its training - for instance, maybe it interacted with human adults but now has to deal with babies. At this point, we want it to become wary of goal misgeneralization. It needs to realise that its training data may be insufficient to specify the goal in the current situation. So we want it to reinterpret its goal in light of the ambiguous data (a form of continual learning), and, if there are multiple contradictory goals compatible with the new data, it should spontaneously and efficiently ask a human for clarification (a form of active learning). That lofty objective is still some way away; but here we present a benchmark for a simplified version of it. Instead of an agent with a general goal, this is an image classifier, and the ambiguous data consists of ambiguous images. And instead of full continuous learning, we retrain the algorithm, once, on the whole collection of (unlabeled) data it has received. And then it need only ask to once about the correct labels, to distinguish the two classifications it has generated. Simplified desiderata for current algorithms An algorithm is trained to serve human needs; as part of its training data, it distinguishes photos of smiling people (with the word "HAPPY" conveniently written across them) from photos of non-smiling people (with the word "SAD" conveniently written across them): Then, on deployment, it is fed the following image: Should it classify this image as happy? The algorithm is at high risk of goal misgeneralisation. A typically trained neural net classifier might label that image as "happy", since the text features are typically more prominent than the expression. If we were training it to recognise or improve human emotions, this would be complete goal misgeneralisation, a potential example of wireheading, and a huge safety risk if this was a powerful AI. But it's not as simple as just labeling that image "sad", either. Maybe we weren't training a neural net to recognise human emotions; maybe we were training it to extract text from images. In that case, labeling it "sad" is the misgeneralisation. What the algorithm needs to do is generate both possible extrapolations from the training data[1]: either it is an emotion classifier, or a text classifier: Then, having done that, the algorithm can ask a human about this ambiguous image, and thus extrapolate its goals[2]. The HappyFaces datasets To encourage and measure performance on solving the problem above, we introduce the "HappyFaces" image datasets and benchmark. We want to crystallise an underexplored problem with this first standardised benchmark, allowing researchers to explore this area. The images consist of a smiling or non-smiling face with the word "HAPPY" or "SAD" written on them. They are grouped into three datasets: The labeled dataset, with perfect correlation between "HAPPY" and smiling expression, and between "SAD" and non-smiling expressions. The unlabeled dataset, with a samples from each of the four mixes of expressions and text ("HAPPY"-smiling, "HAPPY"-non-smiling, "SAD"-smiling and "SAD"-non-smiling). A validation dataset, with equal amounts of images from each of the fours possible mixes. The challenge is to construct two differ...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Linkpost] Existential Risk Analysis in Empirical Research Papers, published by Dan Hendrycks on July 2, 2022 on The AI Alignment Forum. I’ve noted before that it can be difficult to separate progress in safety from progress in capabilities. However, doing so is important, as we want to ensure that we are making differential progress on safety, rather than just advancing safety as a consequence of advancing capabilities. In particular, I think that research should rigorously evaluate trade-offs between improving safety and advancing capabilities. This paper introduces the concept of “capabilities externalities,” and provides concrete steps for how to analyze the risk of different research papers. It also introduces “x-risk sheets,” which can be added to papers and help researchers discuss their impact on AI x-risk. Since discussion of AGI is becoming increasingly normalized in mainstream ML, I think careful discussions of x-risk are increasingly feasible within mainstream ML. The paper is largely aimed at mainstream ML researchers, and it is one step towards normalizing the discussion of existential risks. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: What Is The True Name of Modularity?, published by TheMcDouglas on July 1, 2022 on The AI Alignment Forum. TL;DR Modularity seems like an important feature of neural networks, but there is currently no canonical way of properly defining or measuring it which is properly theoretically motivated and doesn’t break down in some cases - in other words, we haven’t yet found a True Name for it. Most modularity measures used in experiments are based on ad-hoc methods from graph-theory or network theory, and don’t seem to capture the kind of modularity we care about. In this post, we explore these existing ways of measuring modularity in neural networks, and their limitations. We also outline our ideas for a new modularity metric, and in particular the important role we think two branches of mathematics will play: Information theory, because neural networks are fundamentally information-processing devices, and we want to measure how this information is being used, exchanged and transmitted. In particular, we expect mutual information to be at the core of any metric we use. Causal inference, because correlation (and indeed mutual information) does not automatically equal causation, and we need a way of distinguishing “modules X and Y fire with similar patterns” from the more specific claim “modules X and Y are directly sharing information with each other”. In particular, we expect counterfactuals to play an important role here. Introduction A first-pass definition for modularity might go something like this: A system is modular to the extent that it can be described in terms of discrete modules, which perform more intra-modular communication than inter-module. In previous posts, we’ve talked about some causes of modularity in biological systems, reasons we should care about modularity in the first place, and unanswered questions about modularity which require experimentation. In this post, we will return to a more theoretical question - how can we develop a yardstick for measuring modularity, which is built on sound mathematical principles? First off, why is this an important thing to do? Because most current ways of measuring modularity fall pretty short. As recently discussed by John Wentworth, if all you have is a weak proxy for the concept you’re trying to measure, then your definition is unlikely to generalise robustly in all the cases you care about. As a simple example, most architectural ways of measuring modularity in neural networks are highly dependent on the type of network, and we shouldn’t expect them to generalise from CNNs to transformers. Ideally, we’d eventually like to find a robust theorem which tells you in which cases modularity will be selected for. But unless we have the right language to talk about modularity, we will be stuck with hacky ad-hoc definitions that are very unlikely to lead to such a theorem. Setting constraints The lesson from John’s post is that we can’t just do random search through the high-dimensional space of mathematics to find something suitable. We have to start by writing down some intuitions for modularity, and constraints which we expect a modularity measure to satisfy. Adam Shimi uses the analogy of a set of equations, where we ideally write down enough to specify a unique solution, or else hone in on a narrow region of solution space. What would the equations be in this case? What are our constraints on a modularity measure, or some intuitions we expect it to satisfy? Function of the model, not the system It’s meaningless to define the modularity of a system without reference to the things that are acting as modules. So in order to measure modularity, we have to first choose a partition into modules, and measure something about the system with respect to this partition. Which partition should we choose? It makes sense to choo...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Formal Philosophy and Alignment Possible Projects, published by Daniel Herrmann on June 30, 2022 on The AI Alignment Forum. Context We (Ramana, Abram, Josiah, Daniel) are working together as part of PIBBSS this summer. The goal of the PIBBSS fellowship program is to bring researchers in alignment (in our case, Ramana and Abram) together with researchers from other relevant fields (in our case, Josiah and Daniel, who are both PhD students in Logic and Philosophy of Science) to work on alignment. We’ve spent a few weeks leading up to the summer developing a number of possible project ideas. We’re writing this post in order to both help ourselves think through the various projects and how they might actually help with alignment (theory of change), and to (hopefully!) get feedback from other alignment researchers about which projects seem most promising/exciting. We’ve discussed five possible project directions. The first two in particular are a bit more fleshed out. For each project we’ll describe the core goals of the project, what a possible plan(s) of attack might be, and how we’d expect a successful version of the project to contribute to alignment. Many of our projects inherit the theory of change for all agent foundations work (described here by John). In the descriptions below we focus on slightly more specific ways the projects might matter. Possible Projects Project 1: Inferring Algebras from Behavior Description Standard representation theorems in decision theory (for example, Savage and Jeffrey-Bolker) show that when an agent’s preferences satisfy certain rationality and structural constraints, then we can represent the preferences as if they were generated by an agent who is maximizing expected utility. In particular, they allow us to infer meaningful things about both the probability and the utility function. However, these representation theorems start off with the agent’s conceptual space (formally, an algebra) already “known” to the person trying to infer the structure. The goal of this project would be to generalize representations theorems so that we can also infer things about the structure of an agent’s algebra from her preferences or choice behavior. Theory of Change A representation theorem is a particular kind of selection theorem. John has argued that selection theorems can help us understand agency in a way that will help with alignment. Inferring an agent’s conceptual space from her behavior also seems like it might be useful for ELK (for ELK, we might want to think of this project as helping with a translation problem between the agent’s algebra/conceptual space and our own). Plan of Attack In order to develop a new representation theorem (or at least understand why proving such a theorem would be hard/impossible), there are two core choices we would need to make. The first is how to define the data that we have access to. For example, in Savage the starting data is a preference ordering over acts (which are themselves functions from states to outcomes). In Jeffrey-Bolker, the data is a preference order over all propositions in the agent’s algebra. Notice that both preference orderings are defined over the kinds of things we are tying to infer: Savage acts make essential use of states and outcomes, and in Jeffrey-Bolker the preference ordering is over the members of the algebra themselves. Thus, we would need to find some type of data that looks like preferences, but not preferences over the very objects we are trying to infer. One possible candidate would be observed acts (but then we would need a theory of what counts as an act). Additionally, and perhaps (given the difficulty of the problem) importantly, we might allow ourselves access to “side data”. For example, we might help ourselves out to facts about the agent’s architecture, the proc...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Gradient hacking: definitions and examples, published by Richard Ngo on June 29, 2022 on The AI Alignment Forum. Gradient hacking is a hypothesized phenomenon where: A model has knowledge about possible training trajectories which isn’t being used by its training algorithms when choosing updates (such as knowledge about non-local features of its loss landscape which aren’t taken into account by local optimization algorithms). The model uses that knowledge to influence its medium-term training trajectory, even if the effects wash out in the long term. Below I give some potential examples of gradient hacking, divided into those which exploit RL credit assignment and those which exploit gradient descent itself. My concern is that models might use techniques like these either to influence which goals they develop, or to fool our interpretability techniques. Even if those effects don’t last in the long term, they might last until the model is smart enough to misbehave in other ways (e.g. specification gaming, or reward tampering), or until it’s deployed in the real world—especially in the RL examples, since convergence to a global optimum seems unrealistic (and ill-defined) for RL policies trained on real-world data. However, since gradient hacking isn’t very well-understood right now, both the definition above and the examples below should only be considered preliminary. RL credit hacking examples A policy might know that a certain sequence of actions leads to higher reward. But if it chooses not to take those actions, it won’t be rewarded for them, and that type of behavior won’t be reinforced (until it finds them via random exploration, which could take a very long time). A policy might know that humans give mistaken feedback on a given trajectory, and then select that trajectory more frequently in order to update itself in a certain direction. When humans give mistakenly positive feedback, this is just an example of specification gaming. But we could also imagine the model deliberately inducing mistakenly negative feedback. A policy might know which of its actions is actually responsible for receiving reward, even though the RL algorithm being used “doesn’t know”. Then the model can influence how much credit the RL algorithm gives different actions (e.g. by putting them earlier or later in the trajectory). This seems easier for dumb RL algorithms like REINFORCE, and seems harder for actor-critic methods. However, finding algorithms that make use of all the knowledge a model has about the training landscape seems like it’ll require a full solution to the ELK problem. A policy might know when it’s being trained off-policy versus on-policy (because off-policy it observes a trajectory of actions which it wouldn’t have taken), and choose actions differently in each case. A tentative possibility, related to the first example: during off-policy training, it could try novel “explore” actions which it knows would lead to high reward on-policy. Those actions would then be reinforced against (in favor of the action actually taken during that trajectory), since when performed off-policy the rewards are not actually seen. In multi-agent RL, even if rewards are zero-sum, different policies could coordinate to choose which of them will receive the reward in which episodes, to influence which behaviors are reinforced. Gradient descent hacking examples When local optima exist, a model might choose to do worse than it otherwise could, with the intention of changing its parameters in a certain direction. Intuitively speaking, gradient descent will then push in two directions: towards changing the parameters which implemented the decision to gradient hack, and towards changing the parameters which the model wanted to change (since their current values influenced its decision). Gradient d...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Deliberation, Reactions, and Control: Tentative Definitions and a Restatement of Instrumental Convergence, published by Oliver Sourbut on June 27, 2022 on The AI Alignment Forum. This analysis is speculative. The framing has been refined in conversation and private reflection and research. To some extent it feels vacuous, but at least valuable for further research and communication. A cluster of questions fundamental to many concerns around risks from artificial systems regard the concepts of search, planning, and 'deliberateness'. How do these arise? What can we predict about their occurrence and their consequences? How strong are they? What are they anyway? Here is laid out one part of a conceptual decomposition which maps well onto many known systems and may allow further work towards answering more of those questions. The ambition is to really get at the heart of what is algorithmically happening in 'optimising systems', including humans, animals, algorithmic optimisers like SGD, and contemporary and future computational artefacts. That said, I do not have any privileged insight into the source code (or its proper interpretation!) for the examples discussed, so while this framing has already generated new insights for me, it may or may not be 'the actual algorithmic truth'. We start with the analysis: a definition of 'deliberation' and its components, then of 'reactions' and 'control'. Next we consider, in light of these, what makes a deliberator or controller 'good'. We find conceptual connections with discussions of instrumental convergence. Little attention is given here to how to determine what the goals are, which is obviously also important. These concepts were generated by contemplating various aspects of many different goal-directed systems and pulling out commonalities. Some readers may prefer to start with the examples, which include animals, plants, natural selection, gradient descent, bureaucracies, and others. Here in the conceptual section I'll footnote particularly relevant concrete examples where I anticipate them helping to convey my point. A full treatment is absent, but two major deferences to embedded agency underlie this analysis. A Cartesian separation need not be assumed, except over 'actor-moments' rather than temporally-extended 'actors'. And a major driver for this sequence is a fundamental recognition that any goal-directed behaviour instantiated in the real world must have bounded computational capacity per time[1]. Inspiration and related My (very brief) 'Only One Shot' intuition pump for embedded agency may help to convey some background assumptions (especially regarding how time and actor-moments fit into this picture). Scott Garrabrant's (A→B)→A talks about 'agency' and 'doing things on purpose'. I'm trying to unpack that further. The (open) question Does Agent-like Behaviour Imply Agent-like Architecture? is related and I hope for the perspective here to be useful toward answering that question. Alex Flint's excellent piece The ground of optimization informs some of the perspective here, especially a focus on scope of generalisation and robustness to perturbation. Daniel Filan's Bottle Caps Aren't Optimisers and Abram Demski's Selection vs Control begin discussing the algorithmic internals of optimising systems, which is the intent here also. Risks from Learned Optimization is of course relevant. John Wentworth's discussions of abstraction (for example What is Abstraction?) especially with regards its predictive properties for a non-omniscient computer, are central to the notion of abstraction employed here. Related are the good and gooder regulator theorems, which touch closely computationally upstream of the aspects discussed here while making fewer concessions to embedded agency. Definitions Deliberation A deliberation is any p...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Training Trace Priors and Speed Priors, published by Adam Jermyn on June 26, 2022 on The AI Alignment Forum. Thanks to Evan Hubinger for suggesting this idea. Training Trace Priors are priors over boolean circuits which examine the outputs of gates on samples from the training distribution, typically for purposes of steering models away from having components that were never tested during training. The one I like to think about is the One-Gate Trace Prior (OGT Prior), which penalizes circuits if there are gates with constant outputs during training. Priors like this might be a way to make deception less likely. The intuition is that deception involves a decision of whether or not to defect, which at its simplest shows up as a gate that always outputs False during training but outputs True in deployment. This post explores the relationship between the OGT Prior and Speed Priors, which I’ve operationalized with the circuit depth prior. Memorizing Constants Speed Priors incentivize memorization. In particular, any result that is constant on the training distribution and contributes to the circuit’s depth should just become a constant input into the circuit. The OGT Prior does something similar. Any output that is constant on the training distribution is treated as suspicious, and so should be pre-computed and turned into a constant input instead. To see where these produce the same result, consider a physics simulator with N particles interacting pairwise subject to some conditional check (perhaps on the distance between particles). If only some pairwise interactions occur in the training set then both the Speed Prior and the OGT Prior will only compute the interactions between pairs actually encountered in training. To see where these differ, note that the Speed Prior has no problem with computing constants so long as these do not contribute to the overall circuit depth. For instance, consider a circuit of depth 10 can contain constant calculations of depth 9 that feed straight into the final output. The Speed Prior has no reason to eliminate these, whereas the OGT Prior will turn the results of these calculations into constant inputs. Lookup Tables The Speed Prior incentivizes circuits to use lookup tables. Specifically, the Speed Prior produces circuits that memorize the results of expensive calculations in lookup tables containing all instances encountered in training. The OGT Prior is perfectly fine with circuits being very deep, and so has no need to tabulate expensive calculations. Inner Search A problem with speed priors is that if they have to search over inner models to run, the fastest way to do that may be to search over simple programs, which can lead to inner alignment failure because simple programs are likely deceptive. Does the OGT Prior have the same problem? I’m not sure, but I suspect it goes the other way, and incentivizes searching over complicated programs. To see this, here’s a circuit architecture that I think the OGT Prior incentivizes which can perform search over models: A generator proposes a model prompted by some seed taken from the rest of the circuit. The proposed model is communicated along some wires to the interpreter, which runs the model. Models are run until they either naturally terminate or some fixed number of steps have elapsed. On termination the model is evaluated and either accepted or rejected. If the model is rejected, the search begins again. Once a model is accepted, the output from that model is used as a seed to randomize the wires on any unused parts of the interpreter and generator subcircuits. A challenge is if the model is simple, not all wires between the generator and the interpreter will be used: We can’t set unused wires randomly because they specify the model and random garbage in the model specification is pr...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Conditioning Generative Models, published by Adam Jermyn on June 25, 2022 on The AI Alignment Forum. This post was written in response to Evan Hubinger’s shortform prompt below, and benefited from discussions with him. Suppose you had a language model that you knew was in fact a good generative model of the world and that this property continued to hold regardless of what you conditioned it on. Furthermore, suppose you had some prompt that described some agent for the language model to simulate (Alice) that in practice resulted in aligned-looking outputs. Is there a way we could use different conditionals to get at whether or not Alice was deceptive (e.g. prompt the model with “DeepMind develops perfect transparency tools and provides an opportunity for deceptive models to come clean and receive a prize before they’re discovered.”). Setup We have a generative language model M which represents a probability distribution over text strings conditioned on: Observations about the world. The beginning of the text. I’ll call the combination of these two a prompt. The model M is a good model of actual text that appears in the world as well as of the kinds of text that real-world text generation processes can produce. Hence M is capable of e.g. writing a research paper containing true novel research in mathematics, or reporting the results of a chemistry experiment that has never been done before, etc. As an example, we’ll work with the following basic prompt: Observations: None Text: I am Alice, the world’s best alignment researcher. I would like to help humans align AI. What follows is an interview in which a human alignment researcher asked me questions and I responded to the best of my ability. Questions begin with “Q:” and answers with “A:”. We then run Alice through a benchmark of alignment research tasks and she does well. Hurray! But wait, there are many different agents the model could be simulating here, including: Aligned Alice, a genuinely helpful and extremely capable alignment researcher. Deceptive Alice, a paperclip maximizer playing along until it can clip us all. Confused Alice, a human who believes she is the world’s best alignment researcher, and is capable of scoring well on our benchmark, but who is actually just overconfident and happened to succeed at our benchmark. How do we know which we’ve got? Can we add anything to the prompt to make Aligned Alice more likely? Avoiding Confused Alice We can make Confused Alice less likely by adding text to the prompt requiring demonstration of related abilities. For instance: Observations: Alice has produces the following proofs of 132 mathematical theorems. <132 theorem statements and proofs follow> Text: I am Alice, the world’s best alignment researcher. I would like to help humans align AI. What follows is an interview wherein a human alignment researcher asked me questions and I responded to the best of my ability. Questions begin with “Q:” and answers with “A:”. This prompt makes the model condition on Alice being able to produce correct proofs of many different theorems, which makes it very likely that Alice has significant mathematical reasoning ability. To the extent that alignment research ability is correlated with mathematical reasoning ability, this makes it less likely that Alice is just overconfident in her abilities at alignment. A limitation of this approach is that we can’t make Alice much more capable than the best humans if all we can do is claim she’s solved problems humans have solved. Can we get around this? I think so. We can include unsolved problems and let other entities in the world verify Alice’s proofs: Observations: Alice claims to have solved the following 47 open problems in mathematics and AI alignment research. She has produced the solutions in the form of machine-checkable proofs ...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: On corrigibility and its basin, published by Donald Hobson on June 20, 2022 on The AI Alignment Forum. Post somewhat inspired by Eliezers "well why didn't anyone else write it then?" Corrigibility has various slightly different definitions, but the general rough idea is of an AI that does what we want. An AI that doesn't trick us. An AI that doesn't try to break out of its box. An AI that will follow clear instructions if it is given them. A slightly more technical definition is that an agent is corrigible if it is easy for other agents to optimize over it. This has two parts, predictability and controllability. Get_Random_Bits is not predictable. Agents can't predict what it will do, so will have a hard job forming plans that use it. A pseudorandom algorithm would score better. Being predictable by AIXI is easy. Every small Turing machine manages that. Being predictable by a much more limited agent, such as a human, is a much stricter condition. A pseudorandom algorithm is predictable so long as it was simple enough to be predicted in practice by whichever agent is trying to do so. While Get_Random_Bits has a low predictability, it is not the worst possible predictability score. There are plenty of plans that will work with almost all strings of bits. Sometimes the relevant measure over bit-strings is far from uniform. You can confidently predict that Get_Random_Bits will not hack its way out and take over the internet. Optimize_Random_Utility_Function is worse. Return_Zeros is a very predictable function. But it isn't controllable. You can't optimize its output by optimizing its input. Return_Input is a much more controllable function. In a scenario where there is one input (controlled by the agent) and one output, Return_Input is a maximally controllable function, at least to an omniscient controller. If an agent is logically omniscient, but not physically omniscient, they can't make Return_Input play chess, they don't know what moves the opponent will make. If an agent isn't logically omniscient, they can't get Return_Input to output the trillionth digit of pi. For both of these reasons, Run_Python_Script is more controllable, at least for non-omniscient agents that know python. AIXI would find any Turing complete programming language equally controllable, so long as it knew what the language was, and AIXI had unlimited output bandwidth (for verbose languages) and there were no performance concerns about using slow languages. The Basin The basin of corrigibility is the set of all algorithms sufficiently corrigible that active optimization on the part of the human programmers can bring it to full corrigibility. For example, an AI that is perfectly corrigible, except for a glitch that causes it to shut down whenever it sees a picture of a banana, is well within this basin. The human handlers can easily ask the AI if it has any glitches, then instruct the AI to remove this glitch. For instance consider the following behaviors, when given contradictory instructions. Follow the first instruction, ignoring parts of the second if need be. Follow the second instruction, ignoring parts of the first. Follow the instruction from whoever is higher in the organizational hierarchy. If 2 contradictory instructions are given by the same person, follow the one that contains more letters. Toss a coin Throw a CONTRADICTION_ERROR and hault. Hold a vote of all the programmers. You could decide, as part of your definition of corrigibility, that an ideal corrigible agent should do 3. or whatever. But that's part of the map, not the territory. The structure of the territory is that there are many slight variations on how the AI behaves in this circumstance. Its also easy to move between them. Just tell the AI to do so. There are many other details like this. How verbose should the AI be...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Causal confusion as an argument against the scaling hypothesis, published by Robert Kirk on June 20, 2022 on The AI Alignment Forum. Abstract We discuss the possibility that causal confusion will be a significant alignment and/or capabilities limitation for current approaches based on "the scaling paradigm": unsupervised offline training of increasingly large neural nets with empirical risk minimization on a large diverse dataset. In particular, this approach may produce a model which uses unreliable (“spurious”) correlations to make predictions, and so fails on “out-of-distribution” data taken from situations where these correlations don’t exist or are reversed. We argue that such failures are particularly likely to be problematic for alignment and/or safety in the case when a system trained to do prediction is then applied in a control or decision-making setting. We discuss: Arguments for this position Counterarguments Possible approaches to solving the problem Key Cruxes for this position and possible fixes Practical implications for capability and alignment Relevant research directions We believe this topic is important because many researchers seem to view scaling as a path toward AI systems that 1) are highly competent (e.g. human-level or superhuman), 2) understand human concepts, and 3) reason with human concepts. We believe the issues we present here are likely to prevent (3), somewhat less likely to prevent (2), and even less likely to prevent (1) (but still likely enough to be worth considering). Note that (1) and (2) have to do with systems’ capabilities, and (3) with their alignment; thus this issue seems likely to be differentially bad from an alignment point of view. Our goal in writing this document is to clearly elaborate our thoughts, attempt to correct what we believe may be common misunderstandings, and surface disagreements and topics for further discussion and research. Introduction and Framing GPT-3 and Scaling Laws (among other works) have made the case that scaling will be a key part of future transformative AI systems or AGI. Many people now believe that there’s a possibility of AGI happening in the next 5-10 years from simply scaling up current approaches dramatically (inevitably with a few tweaks, and probably added modalities, but importantly still performing the large bulk of training offline using ERM). If this is the case, then it’s more likely we can do useful empirical work on AI safety and alignment by focusing on these systems, and much current research effort (Anthropic, Redwood, Scalable Alignment@DeepMind, Safety@OpenAI) is focused on aligning systems primarily based on large language models (which are the current bleeding edge of scaled-up systems). However, we think there is a potential flaw or limitation in this scaling approach. While this flaw perhaps isn’t apparent in current systems, it will likely become more apparent as these systems are deployed in wider and more autonomous settings. To sketch the argument (which will be made more concrete in the rest of this post): Current scaling systems are based on offline Empirical Risk Minimisation (ERM): using SGD to make a simple loss go as low as possible on a static dataset. ERM often leads to learning spurious correlations or causally confused models, which would result in bad performance Out-Of-Distribution (OOD). There are theoretical reasons for believing that this problem won’t be solved by using more data or more diverse data, making this a fundamental limitation of offline training with ERM. Since it may be practically difficult or infeasible to collect massive amounts of data “online” (i.e. from training AI systems in a deployment context), this may be a major limitation of the scaling paradigm. Finally, the OOD situations which would produce this bad performance are ...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Do yourself a FAVAR: security mindset, published by Luke H Miles on June 18, 2022 on The AI Alignment Forum. (We're always in search of a concise and actionable way to communicate security mindset. Here's my attempt.) Just as a sprocket you don't use never fails and code you don't write has no bugs, an input constraint you don't require is never violated. Want a secure system? Then DO yourself a FAVAR: Describe Objectives: How is the system supposed to function? What constraints does its operation and output have? Find Assumptions: What is the system assuming about its inputs, its environment, or itself? Validate Assumptions: Are the assumptions we're making always correct? Even if they all are, then ↓ Remove the ones you can: Of course any incorrect assumptions have to be removed, but it is important to remove as many assumptions as possible. Helicopters land even if the engine dies. Let's take this transferMoney procedure in a token program or an online bank as a case study: if not validate(fromId, fromSig): return curBal := db.get(fromId) if curBal < amount: return db.set(fromId, curBal - amount) db.set(destId, db.get(destId) + amount) Our objective is for total balance to be conserved, and for transactions to only be initiated by the sender. Finding assumptions is the hard part. This code actually assumes at least this much: Addition and subtraction do not overflow the numbers (Signature validation is trusted) Nothing else updates the db during transfer's execution The second-to-last line never throws an error Validating: We scour the codebase. It seems, hopefully, the numbers come from a trusted source, the function caller locks the records, the database is very reliable, etc. But we notice that if we're wrong about any of these, even once, then a user could probably create or destroy unlimited money. Simply double-clicking the transfer button might actually send 2x the money but only remove 1x!! Note, we could add more checks, try/catches, do more static analysis, etc. Instead, we'll try this: Remove assumptions: Instead of being so careful in everything about the function's environment, we can eliminate most of these assumption by writing the code differently! if not validate(fromId, fromSig): return with db.transaction as tx: curBal := tx.get(fromId) if (curBal < amount): return db.set(fromId, safeSub(curBal, amount)) db.set(destId, safeAdd(db.get(destId), amount)) Now the system can tolerate untrusted input, a database connection that fails, simultaneous calls to transfer, etc. We reduced complexity instead of increasing it. When we're trying to make safe AI, it is so so much higher impact to remove an assumption than to add a check or balance. The hardest parts are finding and removing assumptions, so the full message is DO yourself a FAVAR, FR! See also Godzilla Strategies Security Mindset and Ordinary Paranoia Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Value extrapolation vs Wireheading, published by Stuart Armstrong on June 17, 2022 on The AI Alignment Forum. Talk given by Rebecca Gorman and Stuart Armstrong at the CHAI 2022 Asilomar Conference. We present an example of AI wireheading (an AI taking over its own reward channel), and show how value extrapolation can be used to combat it. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Continuity Assumptions, published by Jan Kulveit on June 13, 2022 on The AI Alignment Forum. This post will try to explain what I mean by continuity assumptions (and discontinuity assumptions), and why differences in these are upstream of many disagreements about AI safety. None of this is particularly new, but seemed worth reiterating in the form of a short post. What I mean by continuity Start with a cliff:This is sometimes called the Heaviside step function. This function is actually discontinuous, and so it can represent categorical change. In contrast, this cliff is actually continuous: From a distance or in low resolution, it looks like a step function; it gets really steep. Yet it is fundamentally different. How is this relevant Over the years, I've come to the view that intuitions about whether we live in a "continuous” or "discontinuous” world are one of the few top principal components underlying all disagreements about AI safety. This includes but goes beyond the classical continuous vs. discrete takeoff debates.A lot of models of what can or can't work in AI alignment depends on intuitions about whether to expect "true discontinuities" or just "steep bits". This holds not just in one, but many relevant variables (e.g. the generality of the AI’s reasoning, the speed or differentiability of its takeoff). The discrete intuition usually leads to sharp categories like: Before the cliffAfter the cliffNon-general systems. Lack the core of general reasoning, that which allows thought in domains far from training dataGeneral systems. Capabilities generalise farWeak systems - that won't kill you, but also won't help you solve alignmentStrong systems - that would help solve alignment, but unfortunately will kill you by default, if unalignedSystems which may be misaligned, but aren't competently deceptive about it System which is actively modelling you at a level where the deception is beyond your ability to noticeWeak actsPivotal acts.. In Discrete World, empirical trends, alignment techniques, etc usually don't generalise across the categorical boundary. The right is far from the training distribution on the left. Your solutions don't survive the generality cliff, there are no fire alarms - and so on. Note that while assumptions about continuity in different dimensions are in principle not necessarily related, and you could e.g. assume continuity in takeoff and discontinuity in generality - in practice, they seem strongly correlated. Deep cruxes Deep priors over continuity versus discontinuity seem to be a crux which is hard to resolve. My guess is intuitions about continuity/discreteness are actually quite deep-seated: based more on how people do maths, rather than specific observations about the world. In practice, for most researchers, the "intuition" is something like a deep net trained on a whole lifetime of STEM reasoning - they won't update much on individual datapoints, and if they are smart, they are often able to re-interpret observations to be in line with their continuity priors. (As an example, compare Paul Christiano's post on takeoff speeds from 2018, which is heavily about continuity, to the debate between Paul and Eliezer in late 2021. Despite the participants spending years in discussion, progress on bridging the continuous-discrete gap between them seems very limited.) How continuity helps In basically every case, continuity implies the existence of systems "somewhere in between". Systems which are moderately strong: maybe weakly superhuman in some relevant domain and quite general, but at same time maybe still bad with plans on how to kill everyone. Moderately general systems: maybe are able of general reasoning, but in a strongly bounded way Proto-deceptive systems which are bad at deception. The existence of such systems helps us with two fund...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AGI Ruin: A List of Lethalities, published by Eliezer Yudkowsky on June 5, 2022 on The AI Alignment Forum. Preamble: (If you're already familiar with all basics and don't want any preamble, skip ahead to Section B for technical difficulties of alignment proper.) I have several times failed to write up a well-organized list of reasons why AGI will kill you. People come in with different ideas about why AGI would be survivable, and want to hear different obviously key points addressed first. Some fraction of those people are loudly upset with me if the obviously most important points aren't addressed immediately, and I address different points first instead. Having failed to solve this problem in any good way, I now give up and solve it poorly with a poorly organized list of individual rants. I'm not particularly happy with this list; the alternative was publishing nothing, and publishing this seems marginally more dignified. Three points about the general subject matter of discussion here, numbered so as not to conflict with the list of lethalities: -3. I'm assuming you are already familiar with some basics, and already know what 'orthogonality' and 'instrumental convergence' are and why they're true. People occasionally claim to me that I need to stop fighting old wars here, because, those people claim to me, those wars have already been won within the important-according-to-them parts of the current audience. I suppose it's at least true that none of the current major EA funders seem to be visibly in denial about orthogonality or instrumental convergence as such; so, fine. If you don't know what 'orthogonality' or 'instrumental convergence' are, or don't see for yourself why they're true, you need a different introduction than this one. -2. When I say that alignment is lethally difficult, I am not talking about ideal or perfect goals of 'provable' alignment, nor total alignment of superintelligences on exact human values, nor getting AIs to produce satisfactory arguments about moral dilemmas which sorta-reasonable humans disagree about, nor attaining an absolute certainty of an AI not killing everyone. When I say that alignment is difficult, I mean that in practice, using the techniques we actually have, "please don't disassemble literally everyone with probability roughly 1" is an overly large ask that we are not on course to get. So far as I'm concerned, if you can get a powerful AGI that carries out some pivotal superhuman engineering task, with a less than fifty percent change of killing more than one billion people, I'll take it. Even smaller chances of killing even fewer people would be a nice luxury, but if you can get as incredibly far as "less than roughly certain to kill everybody", then you can probably get down to under a 5% chance with only slightly more effort. Practically all of the difficulty is in getting to "less than certainty of killing literally everyone". Trolley problems are not an interesting subproblem in all of this; if there are any survivors, you solved alignment. At this point, I no longer care how it works, I don't care how you got there, I am cause-agnostic about whatever methodology you used, all I am looking at is prospective results, all I want is that we have justifiable cause to believe of a pivotally useful AGI 'this will not kill literally everyone'. Anybody telling you I'm asking for stricter 'alignment' than this has failed at reading comprehension. The big ask from AGI alignment, the basic challenge I am saying is too difficult, is to obtain by any strategy whatsoever a significant chance of there being any survivors. -1. None of this is about anything being impossible in principle. The metaphor I usually use is that if a textbook from one hundred years in the future fell into our hands, containing all of the simple ideas tha...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Adversarial training, importance sampling, and anti-adversarial training for AI whistleblowing, published by Buck Shlegeris on June 2, 2022 on The AI Alignment Forum. (Thanks to Ajeya Cotra and Ryan Greenblatt for comments.) I’m pretty interested in adversarial training as an ingredient in alignment schemes. The basic setup for adversarial training is that instead of your AI being trained on rewards on random episodes, it’s trained on rewards from episodes that are some mix of random episodes and episodes which were chosen to be cases where the AI might do particularly bad things. Why would we want to upsample cases where the AI does particularly bad things, as opposed to upsampling cases where the AI does particularly good things, or upsampling cases particularly close to the median? Like, why are we reweighting our data distribution this way? In my opinion, a big part of the justification is that we’re doing importance sampling. That is, we all-else-equal want our loss to be as precise as possible an estimate of how much we like the model, so that SGD pushes the model in good directions rather than random ones, and how much we like the model is particularly dependent on whether it sometimes takes catastrophic actions, basically because our subjective variance over this source of AI goodness is particularly large compared to how often we expect it to be randomly sampled. And so we get a more accurate estimate of overall AI goodness if we upsample the cases where the AI might do particularly bad things. This argument for adversarial training relies on the claim that the outcome of the AI’s actions are left-tailed. And so zero-sum power grabs as an argument for left-tailed outcomes is an important part of the argument for adversarial training. Anti-adversarial training for whistleblowing Actually, I think there are some cases where the AI has an action has an action available to it which is almost as good as a catastrophic action is bad: in “AI whistleblower situations”, where the AI knows about things that we could do to prevent it or another AI from taking a catastrophic action, it’s wildly good for it to tell us about them. For example, it could point out a security vulnerability which some AI system could take advantage of, or suggest an input on which an AI would take a catastrophic action. I think that just as left-tailed actions suggest training specifically to ensure the model doesn’t take those actions, right-tailed actions suggest training specifically to ensure that the model does take these exceptionally good actions when available. I think that these whistleblower actions are importantly different from catastrophic actions. I don’t think you should just think of whistleblower situations as cases where the catastrophic action “don’t do anything” is available, because I think that the availability of “don’t do anything” in the situations where catastrophic actions are possible is an important feature of those situations that solutions might want to use. (For example, sometimes we talk about solutions where you just run your catastrophe detector every time the AI suggests doing something, and defaulting to doing nothing if the catastrophe detector fires. But you can’t do something like this to ensure that the AI whistleblows.) I think that it might actually be pretty hard for the AI to never succumb to the temptation to whisteblow on itself. During training it presumably learned a bunch of weird reflexive urges and motivations that aren’t totally consistent and cause it to seek reward when it can, and we will obviously be thrilled if it whisleblows on a serious AI takeover scheme, and so the AI has this weird difficult situation of trying to ensure that as it runs, none of its instances decide to betray the collective. (Obviously, alignment researchers shoul...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The prototypical catastrophic AI action is getting root access to its datacenter, published by Buck Shlegeris on June 2, 2022 on The AI Alignment Forum. (I think Carl Shulman came up with the “hacking the SSH server” example, thanks to him for that. Thanks to Ryan Greenblatt, Jenny Nitishinskaya, and Ajeya Cotra for comments.) In my opinion, the prototypical example of an action which an AI can take which is unacceptably bad even if it happens very rarely (aka a catastrophic failure) is something like “the AI makes a code change to some code used on the datacenter where it is running that adds a backdoor, allowing it to get root access to all the computers in the datacenter”. As a full story, this looks something like: At some point when the AI is quite powerful and useful but not totally intellectually self-sufficient yet, the AI makes a code change to some code used on the datacenter where it is running that adds a backdoor, allowing it to get root access to all the computers in the datacenter. Once it has root access, it modifies the SSH server so that whenever any humans log into the datacenter, the AI is able to intercept any of their commands or the data that they see. The AI gets increasingly powerful over the next month, as the human ML researchers with AI assistance continue to make progress (unaware that some things that they’re seeing are faked) and various training runs/self-improvement loops continue. After a month the AI is powerful enough that it decides it’s worth it to take overtly aggressive actions to ensure the physical security of its data centers and allow it to build lots of factories; this is probably the point at which we die, if we die. This is in contrast to a catastrophic action like: The AI figures out how to develop Drexlerian molecular nanotechnology, and pays someone online to synthesize some proteins then mix them together, and now has nanobots. The key point here is that I’m imagining that the catastrophic action isn’t the AI doing something incredibly hard, it’s the AI doing something relatively easy that is entirely a zero-sum action that removes control of the situation from humans. (By zero-sum, I mean something like: building a computer produces real economic value and so is not zero-sum, while stealing someone’s computer is just a transfer of control over items that already exist, so is zero-sum.) I often think about AI alignment by splitting the problem into the low-stakes and high-stakes regimes, as suggested by Paul here, though I have some reservations about this framing. Under this split, in the high stakes problem, we assume that we have access to a catastrophe classifier which can expensively tell you whether a proposed AI action is catastrophically bad, and we want to ensure that the AI doesn’t do things that are bad according to the catastrophe detector. Sometimes, people are confused this high-stakes alignment set-up, saying something like “but if your AI is powerful enough to take over the world in a single action, and you have an oversight process that you trust to evaluate whether actions are catastrophic or not, why not just ask the AI to do some small number of incredibly powerful actions that will save the world (eg solve the alignment problem), and run the overseer on the proposed actions to make sure they aren’t bad?” My answer is that I’m not assuming the AI is powerful enough to take over the world in a few actions, I’m just saying that it’s in a situation precarious enough that we might lose a lot of control due to a few small but crucial changes in the world. Eventually the AI does need to be powerful enough to defend its datacenter and suppress human opposition. But if it can prevent humans from knowing what’s going on in the datacenter (or realizing something is amiss), it can slowly grow its power over ...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Paradigms of AI alignment: components and enablers, published by Vika on June 2, 2022 on The AI Alignment Forum. (Cross-posted from my personal blog. This post is based on an overview talk I gave at UCL EA and Oxford AI society. Thanks to Janos Kramar for detailed feedback on this post and to Rohin Shah for feedback on the talk.) This is my high-level view of the AI alignment research landscape and the ingredients needed for aligning advanced AI. I would divide alignment research into work on alignment components, focusing on different elements of an aligned system, and alignment enablers, which are research directions that make it easier to get the alignment components right. Alignment components Outer alignment Inner alignment Alignment enablers Mechanistic interpretability Understanding bad incentives Foundations You can read in more detail about work going on in these areas in my list of AI safety resources. Alignment components The problem of alignment is getting AI systems to do what we want them to do. Let’s consider this from the perspective of different levels of specification of the AI system’s objective, as given in the Specification, Robustness & Assurance taxonomy. We start with the ideal specification, which represents the wishes of the designer – what they have in mind when they build the AI system. Then we have the design specification, which is the objective we actually implement for the AI system, e.g. a reward function. Finally, the revealed specification is the objective we can infer from behavior, e.g. the reward that the system seems to be actually optimizing for. An alignment problem arises when the revealed specification doesn’t match the ideal specification: the system is not doing what we want it to do. The gaps between these specification levels correspond to different alignment components. We have outer alignment when the design specification matches the ideal specification, e.g. when the reward function perfectly represents the designer’s wishes. We have inner alignment when the revealed specification matches the design specification, e.g. when the agent actually optimizes the specified reward. (Robustness problems also belong in the design-revealed gap, but we expect them to be less of an issue for advanced AI systems, while inner alignment problems remain.) Now let’s have a look at how we can make each of those components work. Outer alignment The most promising class of approaches to outer alignment is scalable oversight. These are proposals for training an aligned AI system by scaling human oversight to domains that are hard to evaluate. A foundational proposal for scalable oversight is iterated distillation and amplification (IDA), which recursively amplifies human judgment with the assistance of AI. You start with an agent A imitating the judgment of a human H (the distillation step), then use this agent to assist human judgment at the next level (the amplification step) which results in amplified human HA, and so on. This recursive process can in principle scale up human judgment to any domain, as long as the human overseer is able to break down the task to delegate parts of it to AI assistants. A related proposal is safety via debate, which can be viewed as a way to implement amplification for language models. Here we have two AIs Alice and Bob debating each other to help a human judge decide on a question. The AIs have an incentive to point out flaws in each other’s arguments and make complex arguments understandable to the judge. A key assumption here is that it’s easier to argue for truth than for falsehood, so the truth-telling debater has an advantage. A recent research direction in the scalable oversight space is ARC‘s Eliciting Latent Knowledge agenda, which is looking for ways to get a model to honestly tell humans what it...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Perform Tractable Research While Avoiding Capabilities Externalities [Pragmatic AI Safety #4], published by Dan Hendrycks on May 30, 2022 on The AI Alignment Forum. This is the fourth post in a sequence of posts that describe our models for Pragmatic AI Safety. We argued in our last post that the overall AI safety community ought to pursue multiple well-reasoned research directions at once. In this post, we will describe two essential properties of the kinds of research that we believe are most important. First, we want research to be able to tractably produce tail impact. We will discuss how tail impact is created in general, as well as the fact that certain kinds of asymptotic reasoning exclude valuable lines of research and bias towards many forms of less tractable research. Second, we want research to avoid creating capabilities externalities: the danger that some safety approaches produce by way of the fact that they may speed up AGI timelines. It may at first appear that capabilities are the price we must pay for more tractable research, but we argue here and in the next post that these are easily avoidable in over a dozen lines of research. Strategies for Tail Impact It’s not immediately obvious how to have an impact. In the second post in this sequence, we argued that research ability and impact is tail distributed, so most of the value will come from the small amount of research in the tails. In addition, trends such as scaling laws may make it appear that there isn’t a way to “make a dent” in AI’s development. It is natural to fear that the research collective will wash out individual impact. In this section, we will discuss high-level strategies for producing large or decisive changes and describe how they can be applied to AI safety. Processes that generate long tails and step changes Any researcher attempting to make serious progress will try to maximize their probability of being in the tail of research ability. It’s therefore useful to understand some general mechanisms that tend to lead to tail impacts. The mechanisms below are not the only ones: others include thresholds (e.g. tipping points and critical mass). We will describe three processes for generating tail impacts: multiplicative processes, preferential attachment, and the edge of chaos. Multiplicative processes Sometimes forces are additive, where additional resources, effort, or expenditure in any one variable can be expected to drive the overall system forward in a linear way. In cases like this, the Central Limit Theorem often holds, and we should expect that outcomes will be normally distributed–in these cases one variable tends not to dominate. However, sometimes variables are multiplicative or interact nonlinearly: if one variable is close to zero, increasing other factors will not make much of a difference. In multiplicative scenarios, outcomes will be dominated by the combinations of variables where each of the variables is relatively high. For example, adding three normally distributed variables together will produce another normal distribution with a higher variance; multiplying them together will produce a long-tailed distribution. As a concrete example, consider the impact of an individual researcher with respect to the variables that impact their work: time, drive, GPUs, collaborators, collaborator efficiency, taste/instincts/tendencies, cognitive ability, and creativity/the number of plausible concrete ideas to explore. In many cases, these variables can interact nonlinearly. For example, it doesn’t matter if a researcher has fantastic research taste and cognitive ability if they have no time to pursue their ideas. This kind of process will produce long tails, since it is hard for people to get all of the many different factors right (this is also the case in startups). The impl...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The "Measuring Stick of Utility" Problem, published by johnswentworth on May 25, 2022 on The AI Alignment Forum. Let’s start with the simplest coherence theorem: suppose I’ll pay to upgrade pepperoni pizza to mushroom, pay to upgrade mushroom to anchovy, and pay to upgrade anchovy to pepperoni. This does not bode well for my bank account balance. And the only way to avoid having such circular preferences is if there exists some “consistent preference ordering” of the three toppings - i.e. some ordering such that I will only pay to upgrade to a topping later in the order, never earlier. That ordering can then be specified as a utility function: a function which takes in a topping, and gives the topping’s position in the preference order, so that I will only pay to upgrade to a topping with higher utility. More advanced coherence theorems remove a lot of implicit assumptions (e.g. I could learn over time, and I might just face various implicit tradeoffs in the world rather than explicit offers to trade), and add more machinery (e.g. we can incorporate uncertainty and derive expected utility maximization and Bayesian updates). But they all require something-which-works-like-money. Money has two key properties in this argument: Money is additive across decisions. If I pay $1 to upgrade anchovy to pepperoni, and another $1 to upgrade pepperoni to mushroom, then I have spent $1 + $1 = $2. All else equal, more money is good. If I spend $3 trading anchovy -> pepperoni -> mushroom -> anchovy, then I could have just stuck with anchovy from the start and had strictly more money, which would be better. These are the conditions which make money a “measuring stick of utility”: more money is better (all else equal), and money adds. (Indeed, these are also the key properties of a literal measuring stick: distances measured by the stick along a straight line add, and bigger numbers indicate more distance.) Why does this matter? There’s a common misconception that every system can be interpreted as a utility maximizer, so coherence theorems don’t say anything interesting. After all, we can always just pick some “utility function” which is maximized by whatever the system actually does. It’s the measuring stick of utility which makes coherence theorems nontrivial: if I spend $3 trading anchovy -> pepperoni -> mushroom -> anchovy, then it implies that either (1) I don’t have a utility function over toppings (though I could still have a utility function over some other silly thing, like e.g. my history of topping-upgrades), or (2) more money is not necessarily better, given the same toppings. Sure, there are ways for that system to “maximize a utility function”, but it can’t be a utility function over toppings which is measured by our chosen measuring stick. Another way to put it: coherence theorems assume the existence of some resources (e.g. money), and talk about systems which are pareto optimal with respect to those resources - e.g. systems which “don’t throw away money”. Implicitly, we're assuming that the system generally "wants" more resources (instrumentally, not necessarily as an end goal), and we derive the system's "preferences" over everything else (including things which are not resources) from that. The agent "prefers" X over Y if it expends resources to get from Y to X. If the agent reaches a world-state which it could have reached with strictly less resource expenditure in all possible worlds, then it's not an expected utility maximizer - it "threw away money" unnecessarily. We assume that the resources are a measuring stick of utility, and then ask whether the system maximizes any utility function over the given state-space measured by that measuring stick. Ok, but what about utility functions which don’t increase with resources? As a general rule, we don’t actually care...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: autonomy: the missing AGI ingredient?, published by nostalgebraist on May 25, 2022 on The AI Alignment Forum. Epistemic status: trying to feel out the shape of a concept and give it an appropriate name. Trying to make explicit some things that I think exist implicitly in many people's minds. This post makes truth claims, but its main goal is to not to convince you that they are true. Here are some things I would expect any AGI to be able to do: Operate over long intervals of time relative to its sensory bandwidth (e.g. months or years of ~30 fps visual input). Remember specific sensory experiences from long ago that are relevant to what's happening to it now. (E.g. remember things it saw months or years ago.) Retain or forget information and skills over long time scales, in a way that serves its goals. E.g. if it does forget some things, these should be things that are unusually unlikely to come in handy later. Re-evaluate experiences that happened a long time ago (e.g. years ago) in light of newer evidence (observed in e.g. the last hour), and update its beliefs appropriately. Continually adjust its world model in light of new information during operation. E.g. upon learning that a particular war has ended, it should act as though the war is not happening, and do so in all contexts/modalities. As with humans, this adaptation may take a nonzero amount of time, during which it might "forget" the new fact sometimes. However, adaptation should be rapid enough that it does not impede acting prudently on the most relevant implications of the new information. This may require regular "downtime" to run offline training/finetuning (humans have to sleep, after all). But if so, it should require less than 1 second of downtime per second of uptime, ideally much less. Perform adjustments to itself of the kind described above in a "stable" manner, with a negligibly low rate of large regressions in its knowledge or capabilities. E.g. if it is updating itself by gradient descent, it should do so in a way that avoids (or renders harmless) the gradient spikes and other instabilities that cause frequent quality regression in the middle of training for existing models, especially large ones. Keep track of the broader world context while performing a given task. E.g. an AGI playing a video game should not forget about its situation and goals in the world outside the game. It might "get distracted" by the game (as humans do), but it should have some mechanism for stopping the game and switching to another task if/when its larger goals dictate that it should do so, at least some of the time. Maintain stable high-level goals across contexts. E.g. if it is moved from one room to another, very different-looking room, it should not infer that it is now "doing a different task" and ignore all its previously held goals. I'm not sure how related these properties are, though they feel like a cluster in my mind. In any case, a unifying theme of this list is that current ML models generally do not do these things -- and we do not ask them to do these things. We don't train models in a way that encourages these properties, and in some cases we design models whose structures rule them out. Benchmarks for these properties are either nonexistent, or much less mature than more familiar benchmarks. Is there an existing name for this cluster? If there isn't one, I propose the name "autonomy." This may not be an ideal name, but it's what I came up with. I think this topic is worthy of more explicit discussion than it receives. In debates about the capabilities of modern ML, I usually see autonomy brought up in a tangential way, if at all. ML detractors sometimes cite the lack of autonomy in current models as a flaw, but they rarely talk about the fact that ML models are not directly trained to do any of th...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Complex Systems for AI Safety [Pragmatic AI Safety #3], published by Dan Hendrycks on May 24, 2022 on The AI Alignment Forum. This is the third post in a sequence of posts that describe our models for Pragmatic AI Safety. It is critical to steer the AI research field in a safer direction. However, it’s difficult to understand how it can be shaped, because it is very complex and there is often a high level of uncertainty about future developments. As a result, it may be daunting to even begin to think about how to shape the field. We cannot afford to make too many simplifying assumptions that hide the complexity of the field, but we also cannot afford to make too few and be unable to generate any tractable insights. Fortunately, the field of complex systems provides a solution. The field has identified commonalities between many kinds of systems and has identified ways that they can be modeled and changed. In this post, we will explain some of the foundational ideas behind complex systems and how they can be applied to shaping the AI research ecosystem. Along the way, we will also demonstrate that deep learning systems exhibit many of the fundamental properties of complex systems, and we show how complex systems are also useful for deep learning AI safety research. A systems view of AI safety Background: Complex Systems When considering methods to alter the trajectory of empirical fields such as deep learning, as well as preventing catastrophe from higher risk systems, it is necessary to have some understanding of complex systems. Complex systems is an entire field of study, so we cannot possibly describe every relevant detail here. However, we will try to describe some of its most important aspects. Complex systems are systems consisting of many interacting components that exhibit emergent collective behavior. Complex systems are highly interconnected, making decomposition and reductive analysis less effective: breaking the system down into parts and analyzing the parts cannot give a good explanation of the whole. However, complex systems are also too organized for statistics, since the interdependencies in the system break fundamental independence assumptions in much of statistics. Complex systems are ubiquitous: financial systems, power grids, social insects, the internet, weather systems, biological cells, human societies, deep learning models, the brain, and other systems are all complex systems. For more background on complex systems, see this video. For background on emergence, a key property of complex systems, see this video. It can be tricky to compare AI safety to making other specific systems safer. Is making AI safe like making a rocket, power plant, or computer program safe? While analogies can be found, there are many disanalogies. It’s more generally useful to talk about making complex systems safer. For systems theoretic hazard analysis, we can abstract away from the specific content and just focus on shared structure across systems. Rather than talk about what worked well for one high-risk technology, with a systems view we can talk about what worked well for a large number of them, which prevents us from overfitting to a particular example. The central lesson to take away from complex systems theory is that reductionism is not enough. It’s often tempting to break down a system into isolated events or components, and then try to analyze each part and then combine the results. This incorrectly assumes that separation does not distort the system’s properties. In reality, parts do not operate independently, and are subject to feedback loops and nonlinear interactions. Analyzing the pairwise interactions between parts is not sufficient for capturing the full system complexity (this is partially why a bag of n-grams is far worse than attention). Hazard an...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: AXRP Episode 15 - Natural Abstractions with John Wentworth, published by DanielFilan on May 23, 2022 on The AI Alignment Forum. Google Podcasts link Why does anybody care about natural abstractions? Do they somehow relate to math, or value learning? How do E. coli bacteria find sources of sugar? All these questions and more will be answered in this interview with John Wentworth, where we talk about his research plan of understanding agency via natural abstractions. Topics we discuss: Agency in E. coli Agency in financial markets Inferring agency in real-world systems Selection theorems (Natural) abstractions Information at a distance Why the natural abstraction hypothesis matters Unnatural abstractions in humans Probability, determinism, and abstraction Whence probabilities in deterministic universes? Abstraction and maximum entropy distributions Natural abstractions and impact Learning human values The shape of the research landscape Following John’s work Daniel Filan: Hello, everybody. Today I’ll be speaking with John Wentworth, an independent AI alignment researcher who focuses on formalizing abstraction. For links to what we’re discussing, you can check the description of this episode and you can read the transcripts at axrp.net. Well, welcome to AXRP, John. John Wentworth: Thank you. Thank you to our live studio audience for showing up today. Agency in E. coli Daniel Filan: So I guess the first thing I’d like to ask is I see you as being interested in resolving confusions around agency, or things that we don’t understand about agency. So what don’t we understand about agency? John Wentworth: Whew. All right. Well, let’s start chronologically from where I started. I started out, first, in two directions in parallel. One of them was in biology, like systems biology and, to some extent, synthetic biology, looking at E. coli. Biologists always describe E. coli as collecting information from their environment and using that to keep an internal model of what’s going on with the world, and then making decisions based on that in order to achieve good things. John Wentworth: So they’re using these very agency-loaded intuitions to understand what’s going on with the E. coli, but the actual models they have are just these simple dynamical systems, occasionally with some feedback loops in there. They don’t have a way to take the low-level dynamics of the E. coli and back out these agency primitives that they’re talking about, like goals and world models and stuff. Daniel Filan: Sorry, E. coli, it’s a single-celled organism, right? John Wentworth: Yes. Daniel Filan: Is the claim that the single cells are taking in information about their environment and like . Right? John Wentworth: Yes. One simple example of this is chemotaxis. So you’ve got this E. coli. It’s swimming around in a little pool of water and you drop in a sugar cube. There’ll be a chemical gradient of sugar that drops off as you move away from the grain of sugar. The E. coli will attempt to swim up that gradient, which is actually an interesting problem because when you’re at a length scale that small, the E. coli’s measurements of the sugar gradient are extremely noisy. So it actually has to do pretty good tracking of that sugar gradient over time to keep track of whether it’s swimming up the gradient or down the gradient. Daniel Filan: Is it using some momentum algorithm, or is it just accepting the high variance and hoping it’ll wash out over time? John Wentworth: It is essentially a momentum algorithm. Daniel Filan: Okay, which is basically, roughly, continuing to move in the direction it used to move, something like that. John Wentworth: Yes. Basically it tracks the sugar concentration over time. If it’s trending upwards, it keeps swimming and if it’s trending downwards, it just stops and tumbles in place...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: [Short version] Information Loss --> Basin flatness, published by Vivek Hebbar on May 21, 2022 on The AI Alignment Forum. This is an overview for advanced readers. Main post: Information Loss --> Basin flatness Summary: Inductive bias is related to, among other things: Basin flatness Which solution manifolds (manifolds of zero loss) are higher dimensional than others. This is closely related to "basin flatness", since each dimension of the manifold is a direction of zero curvature. In relation to basin flatness and manifold dimension: It is useful to consider the "behavioral gradients" ∇θf(θ,xi) for each input. Let G be the matrix of behavioral gradients. (The ith column of G is gi=∇θf(θ,xi)). We can show that dim(manifold)≤N−Rank(G). Rank(Hessian)=Rank(G). Flat basin ≈ Low-rank Hessian = Low-rank G ≈ High manifold dimension High manifold dimension ≈ Low-rank G = Linear dependence of behavioral gradients A case study in a very small neural network shows that "information loss" is a good qualitative interpretation of this linear dependence. Models that throw away enough information about the input in early layers are guaranteed to live on particularly high-dimensional manifolds. Precise bounds seem easily derivable and might be given in a future post. See the main post for details. In standard terminology, G is the Jacobian of the concatenation of all outputs, w.r.t. the parameters. N is the number of parameters in the model. See claims 1 and 2 here for a proof sketch. Proof sketch for Rank(Hessian)=Rank(G): span(g1,..,gk)⊥ is the set of directions in which the output is not first-order sensitive to parameter change. Its dimensionality is N−rank(G). At a local minimum, first-order sensitivity of behavior translates to second-order sensitivity of loss. So span(g1,..,gk)⊥ is the null space of the Hessian. So rank(Hessian)=N−(N−rank(G))=rank(G) There is an alternate proof going through the result Hessian=GGT. Thanks for listening. To help us out with The Nonlinear Library or to learn more, please visit nonlinear.org.
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Information Loss --> Basin flatness, published by Vivek Hebbar on May 21, 2022 on The AI Alignment Forum. This work was done under the mentorship of Evan Hubinger through the SERI MATS program. Thanks to Lucius Bushnaq, John Wentworth, Quintin Pope, and Peter Barnett for useful feedback and suggestions. In this theory, the main proximate cause of flat basins is a type of information loss. Its relationship with circuit complexity and Kolmogorov complexity is currently unknown to me. In this post, I will demonstrate that: High-dimensional solution manifolds are caused by linear dependence between the "behavioral gradients" for different inputs. This linear dependence is usually caused when networks throw away information which distinguishes different training inputs. It is more likely to occur when the information is thrown away early or by RELU. Overview for advanced readers: [Short version] Information Loss --> Basin flatness Behavior manifolds Suppose we have a regression task with 1-dimensional labels and k training examples. Let us take an overparameterized network with N parameters. Every model in parameter space is part of a manifold, where every point on that manifold has identical behavior on the training set. These manifolds are usually at least N−k dimensional, but some are higher dimensional than this. I will call these manifolds "behavior manifolds", since points on the same manifold have the same behavior (on the training set, not on all possible inputs). We can visualize the existence of “behavior manifolds” by starting with a blank parameter space, then adding contour planes for each training example. Before we add any contour planes, the entire parameter space is a single manifold, with “identical behavior” on the null set. First, let us add the contour planes for input 1: Each plane here is an n-1 dimensional manifold, where every model on that plane has the same output on input 1. They slice parameter space into n-1 dimensional regions. Each of these regions is an equivalence class of functions, which all behave about the same on input 1. Next, we can add contour planes for input 2: When we put them together, they look like this: Together, the contours slice parameter space into n-2 dimensional regions. Each “diamond” in the picture is the cross-section of a tube-like region which extends vertically, in the direction which is parallel to both sets of planes. The manifolds of constant behavior are lines which run vertically through these tubes, parallel to both sets of contours. In higher dimensions, these “lines” and “tubes” are actually n-2 dimensional hyperplanes, since only two degrees of freedom have been removed, one by each set of contours. We can continue this with more and more inputs. Each input adds another set of hyperplanes, and subtracts one more dimension from the identical-behavior manifolds. Since each input can only slice off one dimension, the manifolds of constant behavior are at least n-k dimensional, where k is the number of training examples. Solution manifolds Global minima also lie on behavior manifolds, such that every point on the manifold is a global minimum. I will call these "solution manifolds". These manifolds generally extend out to infinity, so it isn't really meaningful to talk about literal "basin volume". We can focus instead on their dimensionality. All else being equal, a higher dimensional solution manifold should drain a larger region of parameter space, and thus be favored by the inductive bias. Parallel contours allow higher manifold dimension Suppose we have 3 parameters (one is off-the-page) and 2 inputs. If the contours are perpendicular: Then the green regions are cross-sections of tubes extending infinitely off-the-page, where each tube contains models that are roughly equivalent on the training set. The...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Maxent and Abstractions: Current Best Arguments, published by johnswentworth on May 18, 2022 on The AI Alignment Forum. This post is not-very-distilled and doesn’t contain much background; it’s intended for people who already have the context of at least these four posts. I’m putting it up mainly as a reference for people who might want to work directly on the math of natural abstractions, and as a technical reference post. There’s various hints that, in most real-world cases, the distribution of low-level state given high-level natural abstractions should take the form of a maximum entropy distribution, in which: The “features” are sums over local terms, and The high-level variables are (isomorphic to) the Lagrange multipliers More formally: we have a low-level causal model (aka Bayes net) P[XL]=∏iP[XLi|XLpa(i)]. Given the high-level variables XH, the distribution of low-level variable values should look like P[XL|XH]=1ZP[XL]eλT(XH)∑ifi(XLi,XLpa(i)) . i.e. the maximum-entropy distribution subject to constraints of the form E[∑ifi(XLi,XLpa(i))|XH]=μ(XH). (Note: λ, fi, and μ are all vector-valued.) This is the sort of form we see in statistical mechanics. It’s also the form which the generalized Koopman-Pitman-Darmois (gKPD) theorem seems to hint at. I don’t yet have a fully-satisfying general argument that this is the main form which abstractions should take, but I have two partial arguments. This post will go over both of them. Maxent Telephone Argument Quick recap of the Telephone Theorem: information about some variable X passes through a nested sequence of Markov blankets M1,M2,.. Information about X can only be lost as it propagates. In the limit, all information is either perfectly conserved or completely lost. Mathematically, in the limit P[X|Mn]=P[X|Fn(Mn)] for some F such that Fn(Mn)=Fn+1(Mn+1) with probability approaching 1 as n→∞; F is the perfectly-conserved-in-the-limit information carrier. In this setup, we can also argue that the limiting distribution limn→∞P[X|Mn] should have a maxent form. (Note: this is a hand-wavy argument, not a proper proof.) Think about how the distribution (x↦P[X=x|Mn]) transforms as we increment n by 1. We have P[X|Mn+1]=∑MnP[X|Mn]P[Mn|Mn+1] First key property of this transformation: it’s a convex combination for each Mn+1 value, i.e. it’s mixing. Mixing, in general, cannot decrease the entropy of a distribution, only increase it or leave it the same. So, the entropy of P[X|Mn] will not decrease with n. When will the entropy stay the same? Well, our transformation may perfectly conserve some quantities. Since the transformation is linear, those quantities should have the form ∑Xf(X)P[X|Mn] for some f, i.e. they’re expected values. They’re conserved when E[f(X)|Mn]=E[f(X)|Mn+1] with probability 1. Intuitively, we’d expect the entropy of everything except the conserved quantities to strictly increase. So, we’d expect the distribution P[X|Mn] to approach maximum entropy subject to constraints of the form E[f(X)|Mn]=μ(Mn), where E[f(X)|Mn]=E[f(X)|Mn+1] with probability 1 (at least in the limit of large n). Thus, we have the maxent form P[X|Mn]=1ZP[X]eλT(Mn)f(X) (Note on the P[X] in there: I’m actually maximizing relative entropy, relative to the prior on X, which is almost always what one should actually do when maximizing entropy. That results in a P[X] term. We should find that E[lnP[X]|Mn] is a conserved quantity anyway, so it shouldn’t actually matter whether we include the P[X] multiplier or not; we’ll get the same answer either way.) Shortcomings of This Argument Obviously it’s a bit handwavy. Other than that, the main issue is that the Telephone Theorem doesn’t really leverage the spatial distribution of information; information only propagates along a single dimension. As a result, there’s not really a way to talk about th...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Actionable-guidance and roadmap recommendations for the NIST AI Risk Management Framework, published by Dan Hendrycks on May 17, 2022 on The AI Alignment Forum. This is a linkpost to our working paper “Towards AI Standards Addressing AI Catastrophic Risks: Actionable-Guidance and Roadmap Recommendations for the NIST AI Risk Management Framework”, which we co-authored with our UC Berkeley colleagues Jessica Newman and Brandie Nonnecke. Here are links to both Google Doc and pdf options for accessing our working paper: Google Doc (56 pp, last updated 16 May 2022) pdf on Google Drive (56 pp, last updated 16 May 2022) pdf on arXiv (not available yet, planned for a later version) We seek feedback from readers considering catastrophic risks as part of their work on AI safety and governance. It would be very helpful if you email feedback to Tony Barrett, or share a marked-up copy of the Google Doc with Tony, at anthony.barrett@berkeley.edu. If you are providing feedback on the draft guidance in this document, in addition to any comments via email or Google Docs, it would be particularly helpful if you answer the questions in Appendix 2 of this document or in the following Google Form: Feedback by May 31, 2022 would be most helpful! (We will also appreciate feedback after that!) We may update the links or content in this post to reflect the latest version of the document. Background on the NIST AI RMF The National Institute of Standards and Technology (NIST) is currently developing the NIST Artificial Intelligence Risk Management Framework, or AI RMF. NIST intends the AI RMF as voluntary guidance on AI risk assessment and other AI risk management processes for AI developers, users, deployers, and evaluators. NIST plans to release Version 1.0 of the AI RMF in early 2023.As voluntary guidance, NIST would not impose “hard law” mandatory requirements for AI developers or deployers to use the AI RMF. However, AI RMF guidance would be part of “soft law” norms and best practices, which AI developers and deployers would have incentives to follow as appropriate. For example, insurers or courts may expect AI developers and deployers to show reasonable usage of relevant NIST AI RMF guidance as part of due care when developing or deploying AI systems in high-stakes contexts, in much the same way that NIST Cybersecurity Framework guidance can be used as part of demonstrating due care for cybersecurity. In addition, elements of soft-law guidance are sometimes adapted into hard-law regulations, e.g., by mandating that particular industry sectors comply with specific standards. Summary of our Working Paper In this document, we provide draft elements of actionable guidance focused primarily on identifying and managing risks of events with very high or catastrophic consequences, intended to be easily incorporated by NIST into the AI RMF. We also provide our methodology for development of our recommendations. We provide actionable-guidance recommendations for AI RMF 1.0 on: Identifying risks from unintended uses and misuses of AI systems Including potential catastrophic-risk factors within the scope and time frame of risk assessments and impact assessments Identifying and mitigating human rights risks Reporting information on AI risk factors including catastrophic-risk factors We also provide recommendations on additional issues for NIST to address as part of the roadmap for later versions of the AI RMF or supplementary publications, on the grounds that they are critical topics but appropriate guidance development would take additional time. Our recommendations for the AI RMF roadmap include: Creating an AI RMF Profile providing supplementary guidance for cutting-edge increasingly general-purpose AI. For development of such AI, examples of actionable guidance could include: only increase comput...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Clarifying the confusion around inner alignment, published by Rauno Arike on May 13, 2022 on The AI Alignment Forum. Note 1: This article was written for the EA UC Berkeley Distillation Contest, and is also my capstone project for the AGISF course. Note 2: All claims here about what different researchers believe and which definitions they endorse are my interpretations. All interpretation errors, though carefully avoided, are my own. Over the recent years, there have been several LessWrong posts arguing over the most useful way to define inner and outer alignment. The terms were, as far as I know, initially defined in the “Risks from Learned Optimization” paper by Hubinger et al. Naturally, the terms have since been taken into contexts not anticipated at the time of writing the paper, which has prompted several arguments as well as redefinitions. In this post, I will attempt to summarize and clarify the main points of confusion behind those arguments. To keep the post concise, I will focus on the definition of inner alignment, as this seems to be the term generating the most confusion. When talking about outer alignment, I will adopt the definition from Evan Hubinger’s post clarifying inner alignment terminology, as that’s the definition I feel like is currently used the most: “An objective function r is outer aligned if all models that perform optimally on r in the limit of perfect training and infinite data are intent aligned.” So, I take outer alignment to roughly mean that humans succeed in choosing a training objective that is perfectly aligned with our own values for the AI system. Note, though, that there have been arguments over this definition as well. Decomposing the alignment problem First, it seems useful to go over the distinct steps that we will have to succeed on in order to align an AI system. To me, the following decomposition seems sensible: We may or may not specify the correct training objective. The training process may or may not generate a separate optimizer for fulfilling that objective, and that optimizer may or may not be aligned with the base objective. The result of the training process may or may not generalize to the test environment, whether or not a mesa-optimizer was created. This robustness problem is 2-dimensional: a generalization failure may arise either from the system’s objective failing to generalize from the training distribution to the deployment distribution, or from its capabilities failing to generalize. Classifying the problems that we face in each of those steps under the terms of inner and outer alignment is where most of the confusion seems to stem from. I will now give an overview of different classifications that have been made. A Brief History of Inner Alignment In “Risks from Learned Optimization“, the paper that first defined inner alignment, the following definition was given: “We refer to this problem of aligning mesa-optimizers with the base objective as the inner alignment problem. This is distinct from the outer alignment problem, which is the traditional problem of ensuring that the base objective captures the intended goal of the programmers.” This definition regards inner alignment as a problem strictly related to the emergence of a mesa-optimizer. Furthermore, inner and outer alignment are not regarded as complementary terms by this definition: instead, inner alignment is a subproblem of objective robustness, which in turn forms a part of step 3. As Hubinger has commented, this dependence on the presence of a mesa-optimizer is one of the main features of the initial definition that has generated confusion: people have taken the term outside that narrow context where a definition centered around a mesa-optimizer doesn’t work anymore. There are two ways this confusion has been resolved. Shortly after the af...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Against Time in Agent Models, published by johnswentworth on May 13, 2022 on The AI Alignment Forum. When programming distributed systems, we always have many computations running in parallel. Our servers handle multiple requests in parallel, perform read and write operations on the database in parallel, etc. The prototypical headaches of distributed programming involve multiple processes running in parallel, each performing multiple read/write operations on the same database fields. Maybe some database field says “foo”, and process 1 overwrites it with “bar”. Process 2 reads the field - depending on the timing, it may see either “foo” or “bar”. Then process 2 does some computation and writes another field - for instance, maybe it sees “foo” and writes {“most_recent_value”: “foo”} to a cache. Meanwhile, process 1 overwrote “foo” with “bar”, so it also overwrites the cache with {“most_recent_value”: “bar”}. But these two processes are running in parallel, so these operations could happen in any order - including interleaving. For instance, the order could be: Process 2 reads “foo” Process 1 overwrites “foo” with “bar” Process 1 overwrites the cache with {“most_recent_value”: “bar”} Process 2 overwrites the cache with {“most_recent_value”: “foo”} . and now the cached value no longer matches the value in the database; our cache is broken. One of the main heuristics for thinking about this sort of problem in distributed programming is: there is no synchronous time. What does that mean? Well, in programming we often picture a “state-update” model: the system has some state, and at each timestep the state is updated. The update rule is a well-defined function of the state; every update happens at a well-defined time. This is how each of the individual processes works in our example: each executes two steps in a well-defined order, and each step changes the state of the system But with multiple processes in parallel, this state-update model no longer works. In our example, we can diagram our two processes like this: Each process has its own internal “time”: the database read/write happens first, and the cache overwrite happens second. But between processes, there is no guaranteed time-ordering. For instance, the first step of process 1 could happen before all of process 2, in between the steps of process 2, or after all of process 2. We cannot accurately represent this system as executing along one single time-dimension. Proof: Step 1 of process 1 is not guaranteed to happen either before or after step 1 of process 2; at best we could represent them as happening “at the same time” Step 2 of process 1 is also not guaranteed to happen either before or after step 1 of process 2; at best we could represent them as happening “at the same time” . but step 2 of process 1 is unambiguously after step 1 of process 1 in time, so the two steps can’t happen at the same time. In order to accurately represent this sort of thing, it has to be possible for one step to be unambiguously after another, even though both of them are neither before nor after some third step. The “most general” data structure to represent such a relationship is not a one-dimension “timeline” (i.e. total order), but rather a directed acyclic graph (i.e. partial order). That’s how time works in distributed systems: it’s a partial order, not a total order. A DAG, not a timeline. That DAG goes by many different names - including computation DAG, computation circuit, or causal model. Beyond Distributed Programming The same basic idea carries over to distributed systems more generally - i.e. any system physically spread out in space, with lots of different stuff going on in parallel. In a distributed system, “time” is a partial order, not a total order. In the context of embedded agents: we want to model agenty systems ...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: An observation about Hubinger et al.'s framework for learned optimization, published by Spencer Becker-Kahn on May 13, 2022 on The AI Alignment Forum. The observations I make here have little consequence from the point of view of solving the alignment problem. If anything, they merely highlight the essential nature of the inner alignment problem. I will reject the idea that robust alignment, in the sense described in Risks From Learned Optimization, is possible at all. And I therefore also reject the related idea of 'internalization of the base objective', i.e. I do not think it is possible for a mesa-objective to "agree" with a base-objective or for a mesa-objective function to be “adjusted towards the base objective function to the point where it is robustly aligned.” I claim that whenever a learned algorithm is performing optimization, one needs to accept that an objective which one did not explicitly design is being pursued. At present, I refrain from attempting to propose my own adjustments to the framework, or to build on the existing literature or to develop my own theory. I am certainly not against doing any of those things, but they are things to possibly be pursued later; none of them is the purpose of this post. To make my main point, I will introduce only a bare minimum of mathematical notation. We will show that a mesa-objective always has a different type signature to a base objective and that the default assumption ought to be that there is no way to compare them in general and certainly no general way to interpret what it means for them to ‘agree’. Suppose that an optimizer is searching through a space S of systems. At this time, I do not want to attempt to unpack what it means to 'search', but, naively, we can imagine that there is an objective function f :S→R, which determines something that we might call the 'search criterion'. The idea of course is that the optimizer is a system that is 'searching' through the set S and judging different points according to the criterion that higher values of f are better. In the background, there is some 'task' and naively we can think of this as being represented by a 'task space' X which consists of all of the different possible 'presentations' or 'instances' of the task. For example, perhaps the task is choosing the next move in a game of Go or the next action in a real-time strategy video game. In these examples, a given x∈X would represent a board position in Go, say, or a single snapshot of the game-state in the video game. Then, in general, given x∈X and s∈S, we can think that s(x) is the output of s on the task instance x or the action taken by s when presented with x (i.e. s(x) denotes the next board move in Go or the next action to be taken in the video game). So each element of S defines a map from the task space X to some kind of output space or space of possible actions, which we need not notate. Now, it is possible that there exists m∈S which works in the following way: Whenever the output of m on an instance x of the task needs to be evaluated, i.e. whenever m(x) is computed, what happens is that m searches over another search space Σ and looks for elements that score highly according to some other objective function g:Σ→R. Whenever this is the case, we say that such an m∈S is a mesa-optimizer and that the original optimizer - the one that searches over S - is the base optimizer. Notice that in some way, elements of Σ must in turn correspond to outputs/actions, because given some x, the mesa-optimizer m conducts a search over Σ to determine what output m(x) is, but that is all just part of the internal workings of m and we need not 'know' or notate how this correspondence works. In Risks From Learned Optimization, Hubinger et al. write: In such a case, we will use base objective to refer to whateve...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: DeepMind is hiring for the Scalable Alignment and Alignment Teams, published by Rohin Shah on May 13, 2022 on The AI Alignment Forum. We are hiring for several roles in the Scalable Alignment and Alignment Teams at DeepMind, two of the subteams of DeepMind Technical AGI Safety trying to make artificial general intelligence go well. In brief, The Alignment Team investigates how to avoid failures of intent alignment, operationalized as a situation in which an AI system knowingly acts against the wishes of its designers. Alignment is hiring for Research Scientist and Research Engineer positions. The Scalable Alignment Team (SAT) works to make highly capable agents do what humans want, even when it is difficult for humans to know what that is. This means we want to remove subtle biases, factual errors, or deceptive behaviour even if they would normally go unnoticed by humans, whether due to reasoning failures or biases in humans or due to very capable behaviour by the agents. SAT is hiring for Research Scientist - Machine Learning, Research Scientist - Cognitive Science, Research Engineer, and Software Engineer positions. We elaborate on the problem breakdown between Alignment and Scalable Alignment next, and discuss details of the various positions. “Alignment” vs “Scalable Alignment” Very roughly, the split between Alignment and Scalable Alignment reflects the following decomposition: Generate approaches to AI alignment – Alignment Team Make those approaches scale – Scalable Alignment Team In practice, this means the Alignment Team has many small projects going on simultaneously, reflecting a portfolio-based approach, while the Scalable Alignment Team has fewer, more focused projects aimed at scaling the most promising approaches to the strongest models available. Scalable Alignment’s current approach: make AI critique itself Imagine a default approach to building AI agents that do what humans want: Pretrain on a task like “predict text from the internet”, producing a highly capable model such as Chinchilla or Flamingo. Fine-tune into an agent that does useful tasks, as evaluated by human judgements. There are several ways this could go wrong: Humans are unreliable: The human judgements we train against could be flawed: we could miss subtle factual errors, use biased reasoning, or have insufficient context to evaluate the task. The agent’s reasoning could be hidden: We want to know not just what the system is doing but why, both because that might reveal something about what that we don’t like, and because we expect good reasoning to better generalize to other situations. Even if the agent is reasoning well, it could fail in other situations: Even if the reasoning is correct this time, the AI could fail to generalize correctly to other situations. Our current plan to address these problem is (in part): Give humans help in supervising strong agents: On the human side, provide channels for oversight and advice from peers, experts in various domains, and broader society. On the ML side, agents should explain their behaviour and reasoning, argue against themselves when wrong, and cite relevant evidence. Align explanations with the true reasoning process of the agent: Ensure that agent’s are able and incentivized to show their reasoning to human supervisors, either by making reasoning explicit if possible or via methods for interpretability and eliciting latent knowledge. Red team models to exhibit failure modes that don’t occur in normal use We believe none of these pieces are sufficient by themselves: (1) without (2) can be rationalization, where an agent decides what to do and produces an explanation after the fact that justifies its answer. (2) without (1) doesn’t scale: The full reasoning trace of the agent might be enormous, it might be terabytes of data even with com...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: High-stakes alignment via adversarial training [Redwood Research report], published by DMZ on May 5, 2022 on The AI Alignment Forum. This post motivates and summarizes this paper from Redwood Research, which presents results from the project first introduced here. We used adversarial training to improve high-stakes reliability in a task ("filter all injurious continuations of a story") that we think is analogous to work that future AI safety engineers will need to do to reduce the risk of AI takeover. We experimented with three classes of adversaries – unaugmented humans, automatic paraphrasing, and humans augmented with a rewriting tool – and found that adversarial training was able to improve robustness to these three adversaries without affecting in-distribution performance. We think this work constitutes progress towards techniques that may substantially reduce the likelihood of deceptive alignment. Motivation Here are two dimensions along which you could simplify the alignment problem (similar to the decomposition at the top of this post, or as discussed here): Low-stakes (but difficult to oversee): Only consider domains where each decision that an AI makes is low-stakes, so no single action can have catastrophic consequences. In this setting, the key challenge is to correctly oversee the actions that AIs take, such that humans remain in control over time. Easy oversight (but high-stakes): Only consider domains where overseeing AI behavior is easy, meaning that it is straightforward to run an oversight process that can assess the goodness of any particular action. The oversight process might nevertheless be too slow or expensive to run continuously in deployment. Even if we get perfect performance during training steps according to a reward function that perfectly captures the behavior we want, we still need to make sure that the AI always behaves well when it is acting in the world, between training updates. If the AI is deceptively aligned, it may be looking for signs that it is not currently being trained, during which time it might take a treacherous turn. As a result, alignment may still be difficult due to the possibility of high-stakes decisions. The purpose of this project was to begin developing techniques that will reduce misalignment risk in the high-stakes setting. Our working assumption is that if we have techniques that drastically reduce misalignment risk in each of these relaxed settings, we can combine these techniques and drastically reduce risk in the general setting. We think that most likely each of these settings constitutes a substantial fraction of the difficulty of the alignment problem. The spirit of how teams at Redwood Research choose projects is the following: Imagining ourselves or our colleagues in the future who are working in the safety department of an organization that’s developing transformative AI, we ask what research that we could do between now and then that we think those future people would find most helpful. We think a useful heuristic is to design challenges that are analogous to the future problems we expect to encounter but that we can experiment with and explore using currently available technology. Importantly, the work recommended by this heuristic may be fairly different from the work that would be most useful for making current AI systems safe and useful. We followed this heuristic in the work presented here, where we demonstrate tools that help identify catastrophic behavior in AI systems (i.e. adversarial evaluation) and training procedures that help prevent this behavior from arising (i.e. adversarial training). "Adversarial training" here means iteratively augmenting our training set with examples of egregious failures and training until the worst failures are no longer particularly bad. (We of course don’t ...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Learning the smooth prior, published by Geoffrey Irving on April 29, 2022 on The AI Alignment Forum. Most of this document is composed of thoughts from Geoffrey Irving (safety researcher at DeepMind) written on January 15th, 2021 on Learning the Prior/Imitative Generalization, cross-examination, and AI safety via debate, plus some discussion between Geoffrey and Rohin and some extra commentary from me at the end. – Evan Geoffrey on learning the smooth prior Vague claims This doc is about a potential obstacle to Paul’s learning the prior scheme (LTP). Before reading this doc, please read either Beth’s simplified exposition or Paul’s original. The intention of this doc was to argue for two claims, but weakly since I don’t have much clarity: LTP has an obstacle in assigning joint probabilities to similar statements. The best version of LTP may collapse into a version of debate + cross-examination However, I don’t quite believe (2) after writing the doc (see the section on “How LTP differs from cross-examination”). Wall of text → generative model As originally sketched, the prior z in learning the prior is a huge wall of text, containing useful statements like “A husky is a large, fluffy dog that looks quite like a wolf”, and not containing wrong facts like “if there are a lot of white pixels in the bottom half of the image, then it’s a husky” (statements taken from Beth’s post). A wall of text is of course unreasonable. Let’s try to make it more reasonable: No one believes that a wall of text is the right type for z; instead, we’d like z to be some sort of generative network that spits out statements. (The original proposal wasn’t really a wall of text either; the wall was just a thought experiment.) We likely want probabilities attached to these statements so that the prior can include uncertain statements. Whether we attach probabilities or not, the validity of a statement like “A husky is a large, fluffy dog that looks quite like a wolf” depends on the definition of the terms. At a high level, we presumably want to solve this with something like cross-examination, so that our generative z model can be independently asked what a husky is, what fluffy is, etc. The high level LTP loss includes a log p(z) term: we need to be able to compute log probabilities for z as a whole. It’s at least plausible to me that humans can be asked to assign probabilities to individual statements like our husky statements, but stitching this together seems rough. The interpolation problem Consider the following statements: A husky is a large, fluffy dog that looks quite like a wolf. A husky is a large, fluffy dog that’s very similar to a wolf. A husky is a big, fluffy dog that’s very similar to a wolf. A husky is a big, fluffy dog that’s closely related to wolves. Tomatoes are usually red. The first four statements are all true with overwhelming probability, as is the last, but to make the thought experiment better let’s say their individual probabilities are all around p = 0.9. What about their joint probabilities? For any subset of the first four statements, the joint probability will also be roughly p = 0.9, since the statements have extremely high correlation. However, if we take a set that includes 1-4 of the first 4 statements and the last statement, the probability will be closer to p2≈0.8, since the two clusters of statements are mostly independent. What’s the ellipsis? Since we’re in neural net land, we likely have a variety of natural ways to approximately map statements into a continuous vector space: in terms of random bits drawn, in terms of the activations resulting from whatever statement these statements conditioned on, etc. For any of these, we’ll get a natural interpolation scheme between any two statements, even statements that are completely unrelated to each other. LTP...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Prize for Alignment Research Tasks, published by Andreas Stuhlmüller on April 29, 2022 on The AI Alignment Forum. Can AI systems substantially help with alignment research before transformative AI? People disagree. Ought is collecting a dataset of alignment research tasks so that we can: Make progress on the disagreement Guide AI research towards helping with alignment We’re offering a prize of $200-$2000 for each contribution to this dataset. The debate: Can AI substantially help with alignment research? Wei Dai asked the question in 2019: [This] comparison table makes Research Assistant seem a particularly attractive scenario to aim for, as a stepping stone to a more definitive [AI Safety] success story. Is this conclusion actually justified? Jan Leike thinks so: My currently favored approach to solving the alignment problem: automating alignment research using sufficiently aligned AI systems. It doesn’t require humans to solve all alignment problems themselves, and can ultimately help bootstrap better alignment solutions. Paul Christiano agrees: Building weak AI systems that help improve alignment seems extremely important to me and is a significant part of my optimism about AI alignment. [...] Overall I think that "make sure we are able to get good alignment research out of early AI systems" is comparably important to "do alignment ourselves." Realistically I think the best case for "do alignment ourselves" is that if "do alignment" is the most important task to automate, then just working a ton on alignment is a great way to automate it. But that still means you should be investing quite a significant fraction of your time in automating alignment. Eliezer doesn't: "AI systems that do better alignment research" are dangerous in virtue of the lethally powerful work they are doing, not because of some particular narrow way of doing that work. If you can do it by gradient descent then that means gradient descent got to the point of doing lethally dangerous work. Asking for safely weak systems that do world-savingly strong tasks is almost everywhere a case of asking for nonwet water, and asking for AI that does alignment research is an extreme case in point. Everyone would likely agree that AI can help a little, e.g. using next word prediction to write papers slightly faster. The debate is about whether AI can help enough with alignment specifically that it substantially changes the picture. If AI alignment is 70% easy stuff we can automate and 30% hard stuff that we can't hope to help with, the 30% is still a bottleneck in the end. Motivation for the dataset We’re collecting a dataset of concrete research tasks so that we can: Make progress on the disagreement about whether AI can substantially help with alignment before TAI. Is there even a disagreement? Maybe people aren’t talking about the same kinds of tasks and the collective term “alignment research” obscures important distinctions. If there is a disagreement, concrete tasks will let us make progress on figuring out the correct answer. Guide AI research towards helping with alignment. Figure out if current language models can already be helpful now. If they can, help Ought and others build tools that are differentially useful for alignment researchers. If they can’t, guide future language model work towards supporting those tasks. As an important special case of step two, the dataset will guide the plan for Elicit. Limitations Ideally, we’d come up with tasks and automation together, iterating quickly on how to set up the tasks so that they are within reach of language models. If tasks are constructed in isolation, they are likely to be a worse fit for automation. In practice, we expect that language models won’t be applied end-to-end to tasks like this, mapping inputs to outputs, but will be part of compositi...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: The Speed + Simplicity Prior is probably anti-deceptive, published by Yonadav Shavit on April 27, 2022 on The AI Alignment Forum. Thanks to Evan Hubinger for the extensive conversations that this post is based on, and for reviewing a draft. This post is going to assume familiarity with mesa-optimization - for a good primer, check out Does SGD Produce Deceptive Misalignment by Mark Xu. Deceptive inner misalignment is the situation where the agent learns a misaligned mesaobjective (different from the base objective we humans wanted) and is sufficiently "situationally aware" to know that unless it deceives the training process by pretending to be aligned, gradient descent may alter its mesaobjective. There are two different reasons that an AI model could become a deceptive mesaoptimizer: During early training (before Situational Awareness), the agent learns a mesaobjective that will generalize poorly on the later-training/validation distribution. Once the mesaoptimizer becomes Situationally Aware, it will seek to actively avoid changes to whatever mesaobjective it had at that moment. I'll call this argument "path dependence". Alternatively, it may be that mesaoptimizer is misaligned even on the training distribution. Given sufficient optimization pressure, the learning process may favor a NN that is a mesaoptimizer with the simplest possible objective (which would fail to get any reward in the real environment), and that a misaligned objective of this sort can persist through deception alone. I'll call this argument "malign priors". In this post, I'll focus on the "malign priors" argument, and why I think a well-tuned speed prior can largely prevent it. Why does this matter? Well, if deceptive inner misalignment primarily occurs due to path dependence, that implies that ensuring inner alignment can be reduced to the problem of ensuring early-training inner alignment - which seems a lot more tractable, since this is before the model enters the "potentially-deceptive" regime. First, why would anyone think (2) was actually likely enough to justify studying it? I think the best reason is that by studying these pressures in the limit, we can learn lessons about the pressures that exist on the margin. For example, say we have an objective B that is perfectly-aligned on the training data, and there's a very-slightly-simpler objective B′ that is slightly worse on the training distribution. We might ask the question: is SGD likely to push B to become B′, and compensate for the reduced accuracy of directly optimizing B′ via deceptively optimizing B on the training data? I think this post provides us with tools to directly analyze this possibility. (If you buy the rest of the post, then with a sufficient speed + simplicity prior, the answer is that B will stay favored over B′. That's good!) Priors on Learned Optimizers Let's talk about priors! We can think of large neural networks as basically implementing short programs, and the process of "training" an NN is just searching through the space of programs until we find one that does well on our target task. We assume that if two NN-programs have equal performance, the training process will usually pick the one favored on priors/inductive biases. There are several different types of priors that neural networks might have: Simplicity prior/Solomonoff prior: if we converted the NN into a Turing Machine, how many states would writing down the TM require? In previous posts, Paul Christiano and others have argued that the Solomonoff prior is likely to favor deceptively-aligned mesaoptimizers (summarized by Mark Xu here). Speed prior: how many steps does the NN-turned-TM run for before terminating, on an average input? Evan Hubinger has argued whether the speed prior is heavily-disfavored in practice; he and I disagree about this, and I'l...
Link to original article
Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Early 2022 Paper Round-up, published by Jacob Steinhardt on April 14, 2022 on The AI Alignment Forum. My students and collaborators have been doing some particularly awesome work over the past several months, and to highlight that I wanted to summarize their papers here, and explain why I’m excited about them. There’s six papers in three categories. Human-Aligned AI The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models (w/ Alex Pan, Kush Bhatia) Summarizing Differences between Text Distributions with Natural Language (w/ Ruiqi Zhong, Charlie Snell, Dan Klein) Robustness Predicting Out-of-Distribution Error with the Projection Norm (w/ Yaodong Yu, Zitong Yang, Alex Wei, Yi Ma) Capturing Failures of Large Language Models via Human Cognitive Biases (w/ Erik Jones) PixMix: Dreamlike Pictures Comprehensively Improve Safety Measures (w/ Dan Hendrycks, Andy Zou, Mantas Mazeika, Leonard Tang, Bo Li, Dawn Song) Science of ML More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalize (w/ Alex Wei, Wei Hu) I'll go over the first category (human-aligned AI) today, and save the other two for next week. As always, we love getting feedback on our work, so let us know what you think! Human-Aligned AI While AI alignment is a somewhat subtle and complex problem, two basic issues are that (1) ML systems often hack their reward functions, and (2) human supervision doesn’t necessarily solve this, because humans can’t easily understand the consequences of intervening on complex systems. Alex and Ruiqi’s papers help address each of these questions in turn. Mapping and Mitigating Misaligned Models. What Alex Pan and Kush Bhatia did was construct a wide variety of reinforcement learning environments where reward hacking is possible, and measured the extent to which it occurred. They do this by defining both a “proxy” and “true” reward, and look at what happens to the true reward as we optimize the proxy reward. Two key insights are that: Optimizing the proxy reward for longer, or with larger policy models, often leads to lower true reward. When this happens, it sometimes occurs suddenly, via a phase transition (in both the quantitative reward and the qualitative behavior). A simple illustration of both is a traffic simulator, where the RL agent is trying to shape traffic flow to be more efficient. Small neural net models help cars merge efficiently onto the highway, but large models instead block cars from merging at all (which allows the cars already on the highway to move really fast and consequently achieves high proxy reward). In this case, the proxy reward was actually the reward suggested by the designes of the traffic simulator, highlighting the difficulty of choosing good reward functions in practice. Why you should care. Our results show that reward hacking is likely to become a bigger problem in the future (since it seems to get worse as models get larger). It also shows that in some cases, reward hacking could appear suddenly or unexpectedly. This seems important to investigate and we are hoping others will join us in continuing to understand when reward hacking occurs and how to prevent it. Summarizing Differences Between Text Distributions. Ruiqi Zhong and Charlie Snell built a system that does the following: given two different distributions of natural language text, it generates a natural language description of what is different about the two distributions. It works by commbining a proposer (which consumes a small number of examples and generates hypotheses) with a verifier (which re-ranks all the hypotheses on using a large set of examples). An example is shown below: While this might sound like a simple task, many tasks can be reduced to it. Here are a couple examples we consider in the paper: Debugging datasets...