Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Answering questions honestly given world-model mismatches, published by on the AI Alignment Forum. (Warning: this post is rough and in the weeds. I expect most readers should skip it and wait for a clearer synthesis later.) In a recent post I discussed one reason that a naive alignment strategy might go wrong, by learning to “predict what humans would say” rather than “answer honestly.” In this post I want to describe another problem that feels very similar but may require new ideas to solve. In brief, I’m interested in the case where: The simplest way for an AI to answer a question is to first translate from its internal model of the world into the human’s model of the world (so that it can talk about concepts like “tree” that may not exist in its native model of the world). The simplest way to translate between the AI world-model and the human world-model is to use the AI world-model to generate some observations (e.g. video) and then figure out what states in the human world-model could have generated those observations. This leads to bad predictions when the observations are misleading. This is distinct from the failure mode discussed in my recent post — in both cases the AI makes errors because it’s copying “what a human would do,” but in this case we’re worried that “what a human would do” may be simpler than the intended policy of answering questions honestly, even if you didn’t need a predictive model of humans for any other reason. Moreover, I’ll argue below that the algorithm from that post doesn’t appear to handle this case. I want to stress that this post describes an example of a situation that poses a challenge for existing techniques. I don’t actually think that human cognition works the way described in this post, but I believe it highlights a difficulty that would exist in more realistic settings. Formal setup Human world-model I’ll imagine a human who has a simple world model W = (S, P: Δ(S), Ω, O: S → Ω) where: S is a space of trajectories, each describing a sequence of events in the world. For example, a trajectory s ∈ S may specify a set of rigid objects and then specify how they move around over time. P is a probability distribution over trajectories. It includes both a prior over initial states (cars are probably on the road and fish are probably in the ocean) and a dynamics model that tells us how likely a trajectory is under the laws of physics (most trajectories approximately satisfy Newton’s laws). Ω is a space of observations, for example videos. O tells you what you would observe for each possible trajectory. Let Q be the space of natural language questions and A be the space of answers. Natural language has a simple semantics in the human’s world-model, given by a function Answer: Q × Δ(S) → A. For example, we could have Answer(“Is there a cat in the room?”, p) = “there was until recently, but it probably left just now.” Given some observations ω ∈ Ω, an idealized human answers a question q by performing Bayesian inference and then applying Answer to the resulting probability distribution, i.e. HumanAnswer(q, ω) = Answer(q, P(s|O(s) = ω)). Of course in practice the human may make errors and will have cognitive limitations. But if we use the kinds of techniques discussed in Teaching ML to Answer Questions Honestly, we could hope to learn something like HumanAnswer instead of the human’s approximation to it. AI world-model Let’s assume that our AI models the world in the same way as the human, but with a different world model W′ = (S′, P′, Ω, O′: S′ → Ω). This mismatch may make it unclear how to answer questions. Perhaps the human thinks of the world in terms of rigid objects (so that it makes sense to talk about how many objects are in a bag), whereas the AI models objects as composed of small parts that can move continuously bet...