Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Formal Philosophy and Alignment Possible Projects, published by Daniel Herrmann on June 30, 2022 on The AI Alignment Forum. Context We (Ramana, Abram, Josiah, Daniel) are working together as part of PIBBSS this summer. The goal of the PIBBSS fellowship program is to bring researchers in alignment (in our case, Ramana and Abram) together with researchers from other relevant fields (in our case, Josiah and Daniel, who are both PhD students in Logic and Philosophy of Science) to work on alignment. We’ve spent a few weeks leading up to the summer developing a number of possible project ideas. We’re writing this post in order to both help ourselves think through the various projects and how they might actually help with alignment (theory of change), and to (hopefully!) get feedback from other alignment researchers about which projects seem most promising/exciting. We’ve discussed five possible project directions. The first two in particular are a bit more fleshed out. For each project we’ll describe the core goals of the project, what a possible plan(s) of attack might be, and how we’d expect a successful version of the project to contribute to alignment. Many of our projects inherit the theory of change for all agent foundations work (described here by John). In the descriptions below we focus on slightly more specific ways the projects might matter. Possible Projects Project 1: Inferring Algebras from Behavior Description Standard representation theorems in decision theory (for example, Savage and Jeffrey-Bolker) show that when an agent’s preferences satisfy certain rationality and structural constraints, then we can represent the preferences as if they were generated by an agent who is maximizing expected utility. In particular, they allow us to infer meaningful things about both the probability and the utility function. However, these representation theorems start off with the agent’s conceptual space (formally, an algebra) already “known” to the person trying to infer the structure. The goal of this project would be to generalize representations theorems so that we can also infer things about the structure of an agent’s algebra from her preferences or choice behavior. Theory of Change A representation theorem is a particular kind of selection theorem. John has argued that selection theorems can help us understand agency in a way that will help with alignment. Inferring an agent’s conceptual space from her behavior also seems like it might be useful for ELK (for ELK, we might want to think of this project as helping with a translation problem between the agent’s algebra/conceptual space and our own). Plan of Attack In order to develop a new representation theorem (or at least understand why proving such a theorem would be hard/impossible), there are two core choices we would need to make. The first is how to define the data that we have access to. For example, in Savage the starting data is a preference ordering over acts (which are themselves functions from states to outcomes). In Jeffrey-Bolker, the data is a preference order over all propositions in the agent’s algebra. Notice that both preference orderings are defined over the kinds of things we are tying to infer: Savage acts make essential use of states and outcomes, and in Jeffrey-Bolker the preference ordering is over the members of the algebra themselves. Thus, we would need to find some type of data that looks like preferences, but not preferences over the very objects we are trying to infer. One possible candidate would be observed acts (but then we would need a theory of what counts as an act). Additionally, and perhaps (given the difficulty of the problem) importantly, we might allow ourselves access to “side data”. For example, we might help ourselves out to facts about the agent’s architecture, the proc...