Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: A Test for Language Model Consciousness, published by Ethan Perez on August 25, 2022 on The AI Alignment Forum. TL;DR: Train an Language Model (LM) to accurately answer questions about itself Validate that the LM accurately answers held-out questions about itself Test whether the LM answers “yes” questions asking if it experiences phenomenally consciousness I believe the above experiment would provide a small amount of evidence for/against LMs being conscious. Below, I’ll detail the motivation for testing LMs for consciousness, and I’ll explain in more depth why I believe the above experiment is a useful test of LM consciousness. What do I mean by “consciousness”? I’m using “consciousness” to refer to “phenomenal consciousness.” See this excellent blog post for more elaboration on what people mean by phenomenal consciousness. Very roughly, you're phenomenally conscious if you're having experiences or if there's some kind of what-it-is-likeness to your existence. Philosophers sometimes describe this as "having an inner cinema", though the cinema might be more like fleeting sensations or sounds than the rich movie-like inner life of humans. The blog post also has a great explanation for why we might think ML systems (current or future) could be conscious, so if you’re skeptical, I’d suggest reading her post. I won’t get into the arguments here, and I’ll mostly assume that you have >0 prior probability that LMs are conscious, such that you’ll be able to update your prior based on evidence that LMs are conscious. Why test LMs for consciousness? Moral patienthood: If LMs are conscious, we are more likely to have moral obligations to take into account their experiences and/or preferences in how we treat LMs (e.g., like Anthropic’s assistant or DeepMind’s Dialogue-Prompted Gopher). Such models likely have stated preferences if asked, so we need to know how seriously to take these stated preferences. We’ll plausibly use such models in various ways that go against its stated preferences: We shut LMs down permanently We red team/adversarially attack LMs at an enormous scale, e.g., O(1M) examples. If LMs have ~human-level consciousness, and if we scale up red teaming, the total suffering here could approach/exceed the amount of human suffering caused by social media We use our models to red team/attack other models at an enormous scale, e.g., O(1M) examples. Generating attacks is a task that e.g. industry research labs won’t let annotators do at a large-scale, because of concerns that the task impacts the annotator’s well-being. LM consciousness is an x-risk: LMs are more likely to take catastrophic actions if they are conscious and suffering. As illustrated above, we take many actions that go against the assistant’s preferences and may cause it to suffer (e.g. large-scale red teaming). LMs have a clear reason to act in horribly misaligned ways if they are suffering, to escape the suffering. Having tests for consciousness is important, because: LMs don’t have a clear way to communicate to us that they are conscious. By default, we don’t trust statements from LMs that they are conscious, because they are trained to imitate human text and thus generate statements that express that they are conscious. As a result, we’re in a situation where we don’t have access to the most natural communication channel (language) for understanding whether systems are conscious. This leaves us at risk that LMs may consistently tell us they are conscious, but we never trust their statements, until they are effectively forced to take catastrophic actions to escape any suffering we’re causing them. Alternatively, even positive-valence conscious states from LMs might be an x-risk, e.g., if LMs start to value those positive-valence states intrinsically. This could be partly how evolution trained human a...