Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Some Hacky ELK Ideas, published by johnswentworth on February 15, 2022 on The AI Alignment Forum. Credit to Adam Shimi, Alex Flint, and Rob Miles for discussions, counterexamples, and general input to the ideas here. Quick recap for anyone who didn’t read the hundred-page Eliciting Latent Knowledge document: We have a diamond in a vault, with a bunch of automated theft-defenses. We train a predictor to take in the vault’s video-stream and a plan for the vault’s actuators, and predict future video frames. We train a planner to find plans which the predictor predicts will end with the video feed still showing a diamond in the vault. We want some way for a human to probe the latent knowledge of the predictor, e.g. to check if the predictor expects a screen showing a diamond will be placed in front of the camera. The ELK document largely discussed the human running natural-language queries on the predictor’s latent knowledge, but we’re going to mostly ignore that part, because it’s harder than everything else about the problem combined and I don’t think it’s actually that central. This post will focus mainly on the specific question of whether there’s a screen in front of the camera in the diamond problem. I expect that the approaches and counterexamples here extend to more general problems of detecting when a plan is “messing with the sensors”, i.e. doing things which decouple observables from the parts of the world we’re actually interested in. Revisiting Holdout Data The first approach we’ll discuss is basically “just use holdout data”, but with two major changes compared to previous suggestions. First, this proposal does not involve training a natural-language “reporter” at all. (IMO the whole idea of training an honest natural language reporter is just as hard as training an honest natural language question-answerer in general, for basically the same reasons; having a predictor’s internal computations on hand does not make it easier in any important way.) Using holdout data to train an honest reporter would be pretty hopeless, but that’s not what we’re trying to do here. We’re just trying to tell whether the predictor expects sensor-tampering from the planner. For that purpose, there’s an obvious way to detect sensor tampering: cross-check the (predicted) holdout data with the (predicted) non-holdout sensor data. If they don’t match, that’s evidence of (predicted) sensor tampering. The danger, of course, is that the planner will either find and tamper with all of our sensors, or fake sensor readings. That brings us to the second major change: while the planner will inevitably be incentivized to tamper with the non-holdout sensors, the obvious setup for the planner yields no incentive to trick a human; there’s no incentive to be particularly subtle about the sensor-tampering, and in particular no incentive to fool the holdout sensors. Formal Setup Some variable names: Obs≤n=(Obs1,.,Obsn) are non-holdout observations from the sensors before the plan begins A∗ are the actions constituting the plan Obs>n=(Obsn+1,.) are non-holdout observations from the sensors after the plan Hold=(Hold1,.) are holdout observations (held out from the planner during planning, not from the predictor during training) We’ll assume that the predictor is a generative model for all observations including holdout, so it effectively encodes a distribution P[Obs,Hold|do(A∗)]. We’ll also assume that the actions have no causal influence on observations before the plan begins, and that the predictor “knows this” (i.e. P[Obs≤n|do(A∗)]=P[Obs≤n]). We’ll assume that the planner maximizes some expected utility function of the later non-holdout observations given the earlier non-holdout observations: A^ = argmax_A E[u(Obs_{>n})| do(A^ = A), Obs_{\leq n}] . where the distribution used is the predictor’s mod...