Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Conditioning Generative Models with Restrictions, published by Adam Jermyn on July 21, 2022 on The AI Alignment Forum. This is a followup to Conditioning Generative Models based on further discussions with Evan Hubinger, Nicholas Schiefer, Abram Demski, Curtis Huebner, Hoagy Cunningham, Derek Shiller, and James Lucassen, as well as broader conversations with many different people at the recent ARC/ELK retreat. For more background on this general direction see Johannes Treutlein’s “Training goals for large language models”. Background Previously, I wrote about ways we could use a generative language model to produce alignment research. There I proposed two approaches: Simulate a superhuman alignment researcher by conditioning on them generating superhuman machine-verifiable results in a variety of related domains. Try to align this superhuman agent by crafting scenarios where a misaligned agent has an incentive to defect ("Honeypots"). Simulate humans performing alignment research for a long time, by conditioning on observations spaced many years apart ("Patient Research"). I think that both approaches are an improvement over doing nothing, but I suspect that no amount of honeypot construction actually ensures an aligned agent, and so it seems that simulating humans ("Patient Research") is the more promising direction. Overview If we truly have a perfect generative model I think the “Patient Research” approach really does well. We can set up a world where machine learning practice is strictly banned and where researchers spend thousands of years constructing a machine-verifiable scheme for AI alignment. The ban prevents that world from containing malign superintelligences, and the time-scale means that if alignment is something humans can solve, the model should produce that solution. The problem is that we won’t have perfect generative models, so we’ll have to accept some limitations. In this post I’ll explore what changes if we cannot condition on worlds too different from our own. The idea is to understand what happens if our generative model has trouble generalizing too far away from its training data. Failure Modes The main dangerous failure mode with using generative models to produce alignment research is that we accidentally ask for a future that contains a deceptive AGI. If we get such a future, the AGI might spoof observations to trick us into importing it into our world. For instance it could pretend to be a human producing really good alignment research, but produce work which is subtly flawed and, once implemented, allows it to take over the world. There are generally two ways we hit this failure mode. The first is that we ask for a prediction of the distant future. The future 50 years from now has a decent chance of containing a deceptive AGI, and if we naively ask for that future there’s a good chance we’ll simulate that AGI, be deceived, and import that AGI into our world. The second way is to ask for a world that is very unlikely in ways that make deceptive AGI more likely. For instance, we could condition on observing that next year someone produces what appears to be a perfect solution to AI alignment. When we do this, the hypothesis “a deceptive AGI took over and pretended to solve alignment” becomes a lot more likely, and so the generative model is more likely to simulate that scenario. The name of the game, then, is to craft strategies that simultaneously (1) make deceptive AGI’s less likely and (2) accelerate alignment research. Ground Rules Observations We have a generative model trained on observations of the world. The training objective is to predict later observations given earlier ones. We assume that the model is inner-aligned, so that it really does try to predict the future conditioned on observations. Observations are multi-modal, and ...