Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Autonomy as taking responsibility for reference maintenance, published by Ramana Kumar on August 17, 2022 on The AI Alignment Forum. I think semantics – specifically, maintaining reference relationships – is a core component of intelligent behaviour. Consequently, I think a better understanding of semantics would enable a better understanding of what machine intelligence that is “trying to do the right thing” ought to look like and how to build it. Unfortunately I’ve not managed to make these ideas particularly clear (either to myself or others). In the spirit of palimpsest-shaped research I’m sharing a draft in the hope of provoking useful thoughts. Summary Particular events can have meaningfulness or significance, and can carry multiple meanings at once. For example, the motion of a dial in a meter might mean that there is electricity flowing through a connected wire, while simultaneously meaning that an alarm condition has been triggered. The “meanings” (or semantics) that can be ascribed to events depend on regularities (stable mechanisms) in the surrounding context. Large-scale regularities are required to sustain large-scale semantics (something carrying meaning about another thing that is far away in time or space). Meanings are generally more robust when their dependencies on the context are fewer and/or more local. However, large-scale semantics can be made robust by managing the contextual dependencies. For example, using mechanisms for error detection and correction. There is compensatory work that can be done to sustain a regularity so something local continues to mean a particular thing about something distal. To be autonomous is to be robust in this sense. Systems that more robustly traffic in large-scale meanings are more autonomous. Thus, being autonomous can be understood as including all the machinery required to hold stable, in a changing world, all the semantic or referential relationships that are required for continued operation. Creating regularities that span larger distances is a way of creating new kinds of semantics, and autonomous systems do this as their conceptual spaces get larger. Local and distal meanings The battery indicator on my laptop reads “84%” when I hover my mouse cursor over it. What does this reading mean? There’s some function, I suppose, in some program running as part of my desktop environment that is designed to produce the text for the popup window. The reading means this function received the number 84 as input. OK, but that’s not all it means. There is, I suppose, a smart battery system in my laptop, and it ran a calculation that produced 84% as an estimate of the battery’s state of charge. More concretely: I imagine there are several quantities in the memory of the smart battery system, such as the last few readings from an ammeter on the battery, the times those readings were made, the last reference point for current when the battery was fully charged, etc. The reading I observed means all these memory locations contained values such that the calculation the smart battery system runs produced the value 84. This is a “more distal” meaning for the reading than “the text rendering function received 84 as input”, because the smart battery system’s calculation happens “before” (or “causally upstream of”) that function’s execution. Most distally, of course, we’d like to say that the reading means that the state of charge of the battery “really is” approximately 84%. We have a collection of meanings for the reading (..., it means the LEDs on my screen were emitting pixels arranged in a shape recognisable as “84%”, ., it means the text-rendering function received 84 as an input, ., it means the smart battery system calculated the state of charge as 84%, ., it means the battery’s state of charge is 84%), and I have just ar...