Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Rephrasing Of and Footnote To An Embedded Agency Proposal, published by Josh Hickman on March 9, 2022 on The AI Alignment Forum. I wanted to clarify some comments made in this post, which proposed a way to resolve the issues brought up in the Action Counterfactuals subsection of the Embedded Agency paper. I noticed that my ideal edits would end up rewriting the post almost completely (it wasn't particularly clear), and I wanted to more coherently lay out the thinking. The Problem The problem is that we'd like agents to consider action counterfactuals, and, perhaps, find a proof that an action leads to the highest reward before taking it. The formulation in the paper leads to some Lobian issues, where agents choose actions by searching over proofs for statements similar to: [The paper actually has a much cleaner phrasing of the statements it searches for proofs for, to make the proof issue much clearer.] This is a rather substantial bummer, because something like that would be great to rely on. It'll find the proofs for what you'd like, but it'll also find proofs for nonsense. The Embedded Agency paper is a great resource for this, but you should also work out why this might happen in your head. The Confusion One potential intuition for why this problem exists is the following: The agent who is taking this argmax always chooses the highest reward thing. If I know the rewards for all actions but the last X, well, choosing X means it was the result of the argmax. It's right there in the source code / function definition. So W must have been the biggest! Therefore, X is the result of the argmax being computed, because even if we don't know W, I know that the agent choosing X means W is bigger than all the other W' values. It must be -- the choice was coming from an agent doing an argmax! This might be a confusing phrasing. The ideal way to explain this might be setting up the 5-10 problem in a theorem prover and just showing people it'll find the bad proof. But I think the narrative version captures something coherent. The confusion we should have is that this is not how we, humans, engage in action counterfactuals! Not even close. Normally that's not an issue, but let's pay more attention to see if the difference is substantive. In the original post, I use the example of a professor considering whether to slash the tires on his car. He ponders what it would imply, if he chose to do that. It would imply he must desperately not want the ability to travel, plausibly. It might imply he's driven into a river and needs access to the air within them for safety, like he's seen in movies. He's a sensible person -- him making the choice must mean there's a very good reason he did so! Much better than the standard "I'm just trying to get to work" reasons to not slash the tires. But the professor, running his weaker, informal decision making process, does not slash his tires. You don't want to know the rewards implied by choosing to take an action. Although the professor has no Lobian issues in his informal decision-making, he views the future differently: he asks what would happen if he was the same type of person who just made the choice, excluding any consideration of why he might be making the decision, and considers what might happen to that imaginary person. The Intuition The intuition I was using was: why can't the agents do something similar? Create a new agent that "teleports them forward in time" by just precommiting to making the decision, and seeing what results? This matches our informal understanding of the phrase "action counterfactual" much better, I think, and because we're creating a new agent (one with this one decision pre-set into it), it's possible the self-reference issues go away. Does it mechanically work to resolve the issue? It does in the specific exam...