Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: By Default, GPTs Think In Plain Sight, published by Fabien Roger on November 19, 2022 on The AI Alignment Forum. Epistemic status: Speculation with some factual claims in areas I’m not an expert in. Thanks to Jean-Stanislas Denain, Charbel-Raphael Segerie, Alexandre Variengien, and Arun Jose for helpful feedback on drafts, and thanks to janus, who shared related ideas. Main claims GPTs’ next-token-prediction process roughly matches System 1 (aka human intuition) and is not easily accessible, but GPTs can also exhibit more complicated behavior through chains of thought, which roughly matches System 2 (aka human conscious thinking process). Human will be able understand how a human-level GPTs (trained to do next-token-prediction) complete complicated tasks by reading the chains of thought. GPTs trained with RLHF will bypass this supervision. System 2 and GPTs’ chains of thought are similar A sensible model of the human thinking process Here is what I feel like I’m doing when I’m thinking: Repeat Sample my next thought from my intuition Broadcast this thought to the whole brain When you ask me what is my favorite food, it feels like some thoughts “pop” into consciousness, and the following thoughts deal with previous thoughts. This is also what happens when I try to prove a statement: ideas and intuitions come to my mind, then new thoughts about these intuitions appear. This roughly matches the model described in Consciousness and the Brain by Stanislas Dehaene, and I believe it’s a common model of the brain within neuroscience. How GPTs “think” Autoregressive text models are performing the same kind of process when they generate text. Sampling text is using the following algorithm: Repeat: Do a forward pass, and sample the next token from the output distribution Add the generated token to the input. It makes it part of the input for the next forward pass, which means it can be used by lots of different attention heads, including specialized heads at earlier layers. This looks similar the human thinking process, and the rest of the post will be exploring this similarity and draw some conclusion we might draw from this. System 2 and GPTs’ chains of thought have similar strengths A theoretical reason The sample + broadcast algorithm enables the execution of many serial steps. Neurons take about 1ms to fire, which means that if a thought takes 200ms to be generated, the brain can only do 200 serial operations. This is similar to the hundredths of serial matrix multiplications GPT-3 does in its 96 layers. But by sampling and thought and broadcasting it, both GPT-3 using chains of thought and the brain using System 2 can in principle implement algorithms which require much more serial steps. More precisely, in a Transformer, the number of serial steps used to generate one token is #layers (no matter the prompt length), whereas the number of serial steps used to generate N tokens is N×#layers (where one step is one path trough a block made of an attention layer, an MLP, and a residual connection). As you can see in the figure below, each path the information can take to generate the first token is exactly #layers long, but paths to generate the last token can be up to N×#layers long. (Connections skipping more than 2 positions were omitted to increase readability.) The actual number of FLOPs used is the same when generating N tokens from an M token prompt and when generating 1 tokens from and N+M−1 token prompt because a large prompt enables more parallel computations to take place. But parallel computations are not always the bottleneck, and in particular, I think that planning to deceive humans and taking control is the kind of task which requires a large number of serial steps. Empirical evidence Chains of thought helped complete tasks which human need System 2 for: for ex...