Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Conditioning Generative Models, published by Adam Jermyn on June 25, 2022 on The AI Alignment Forum. This post was written in response to Evan Hubinger’s shortform prompt below, and benefited from discussions with him. Suppose you had a language model that you knew was in fact a good generative model of the world and that this property continued to hold regardless of what you conditioned it on. Furthermore, suppose you had some prompt that described some agent for the language model to simulate (Alice) that in practice resulted in aligned-looking outputs. Is there a way we could use different conditionals to get at whether or not Alice was deceptive (e.g. prompt the model with “DeepMind develops perfect transparency tools and provides an opportunity for deceptive models to come clean and receive a prize before they’re discovered.”). Setup We have a generative language model M which represents a probability distribution over text strings conditioned on: Observations about the world. The beginning of the text. I’ll call the combination of these two a prompt. The model M is a good model of actual text that appears in the world as well as of the kinds of text that real-world text generation processes can produce. Hence M is capable of e.g. writing a research paper containing true novel research in mathematics, or reporting the results of a chemistry experiment that has never been done before, etc. As an example, we’ll work with the following basic prompt: Observations: None Text: I am Alice, the world’s best alignment researcher. I would like to help humans align AI. What follows is an interview in which a human alignment researcher asked me questions and I responded to the best of my ability. Questions begin with “Q:” and answers with “A:”. We then run Alice through a benchmark of alignment research tasks and she does well. Hurray! But wait, there are many different agents the model could be simulating here, including: Aligned Alice, a genuinely helpful and extremely capable alignment researcher. Deceptive Alice, a paperclip maximizer playing along until it can clip us all. Confused Alice, a human who believes she is the world’s best alignment researcher, and is capable of scoring well on our benchmark, but who is actually just overconfident and happened to succeed at our benchmark. How do we know which we’ve got? Can we add anything to the prompt to make Aligned Alice more likely? Avoiding Confused Alice We can make Confused Alice less likely by adding text to the prompt requiring demonstration of related abilities. For instance: Observations: Alice has produces the following proofs of 132 mathematical theorems. <132 theorem statements and proofs follow> Text: I am Alice, the world’s best alignment researcher. I would like to help humans align AI. What follows is an interview wherein a human alignment researcher asked me questions and I responded to the best of my ability. Questions begin with “Q:” and answers with “A:”. This prompt makes the model condition on Alice being able to produce correct proofs of many different theorems, which makes it very likely that Alice has significant mathematical reasoning ability. To the extent that alignment research ability is correlated with mathematical reasoning ability, this makes it less likely that Alice is just overconfident in her abilities at alignment. A limitation of this approach is that we can’t make Alice much more capable than the best humans if all we can do is claim she’s solved problems humans have solved. Can we get around this? I think so. We can include unsolved problems and let other entities in the world verify Alice’s proofs: Observations: Alice claims to have solved the following 47 open problems in mathematics and AI alignment research. She has produced the solutions in the form of machine-checkable proofs ...