Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Quantilizers and Generative Models, published by Adam Jermyn on July 18, 2022 on The AI Alignment Forum. Thanks to Evan Hubinger for discussions about quantilizers, and to James Lucassen for discussions about conditioned generative models. Many of these ideas are discussed in Jessica Taylor's Quantilizers: A Safer Alternative to Maximizers for Limited Optimization: this post just expands on a particular thread of ideas in that paper. Throughout I’ll refer to sections of the paper. I have some remaining confusion about the “targeted impact” section, and would appreciate clarifications/corrections! Abstract This post explores the relationship between quantilizers and generative models. My main takeaways are: A natural way to build a quantilizer is by sampling from an appropriately-conditioned generative model. Unfortunately quantilizing doesn’t seem to confer much advantage over the underlying generative model: to the extent that a quantilizer is more powerful than a generative model, it’s more dangerous, and vice versa. Quantilizing is pretty computationally expensive relative to the advantage it brings, making it unclear if this is a competitive approach even if it conferred a net safety advantage at fixed power. Definitions I’ll follow the setup in “Quantilizers: A Safer Alternative to Maximizers for Limited Optimization” (Section 1, Definition 1) and say that a quantilizer is a model with: A base distribution γ, which describes a set of outputs we think are “normal” or “unlikely to lead to doom”. A quantile q∈[0,1]. An ordering over outputs (e.g. a utility or loss function). A quantilizer returns a random output from the top q fraction of outputs from γ according to the ordering. So for instance a 0-quantilizer is a utility maximizer, a 0.1-quantilizer returns outputs in the best 10%, etc. Where does the base distribution come from? One challenge with quantilizers is that they require a base distribution γ that we trust. That is, we want to be able to sample from γ many times and never hit bad outputs. Unfortunately, we don’t know how to explicitly write this distribution down. A natural alternative to explicit construction is to train a generative model on the distribution of actions a humans takes (Section 3.1). Then for any given task γ can be written as This seems quite doable. Train GPT-N on text written by humans, then prompt it with the task: Prompt: You are a good person who wants to accomplish . What follows is your description of how you intend to do this. The output will probably be text describing how a good person would accomplish that task! Unfortunately we might get unlucky and sample a person who isn’t so good (but thinks of themselves as good). That still satisfies the prompt. Robustly getting the agent we want seems hard, but might be possible in the limit where we ask for agents that are generally plausible in the training distribution of the generative model. Power vs Safety Armed with our base distribution, we can start quantilizing. Naively we can do this by sampling q−1 times from γtask, then returning the top output. This produces a tradeoff between capabilities and safety. Single Sampler In the limit q→1 we’re just returning actions a good human would take. That is, our quantilizer just samples once and returns that sample as output, so it’s no more powerful than a random good person. Such a model is really unlikely to take over the world, but it’s also really unlikely to solve quantum chemistry problems. This is because both of these outcomes unlikely according to the base distribution. We can improve things a bit by conditioning on being an unusually capable person, e.g. Prompt: You are a good and extremely capable person who wants to accomplish . What follows is your description of how you intend to do this. Now we can get top-end human ...