Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Paper: Teaching GPT3 to express uncertainty in words, published by Owain Evans on May 31, 2022 on The AI Alignment Forum. Paper Authors: Stephanie Lin (FHI, Oxford), Jacob Hilton (OpenAI), Owain Evans TLDR: New paper (Arxiv, Twitter) showing that GPT-3 can learn to express calibrated uncertainty using natural language. This is relevant to ARC’s project on Eliciting Latent Knowledge (ELK) and to Truthful AI. The rest of this post is taken from the paper’s Abstract and Introduction. Abstract We show that a GPT-3 model can learn to express uncertainty about its own answers in natural language – without use of model logits. When given a question, the model generates both an answer and a level of confidence (e.g. “90% confidence” or “high confidence”). These levels map to probabilities that are well calibrated. The model also remains moderately calibrated under distribution shift, and is sensitive to uncertainty in its own answers, rather than imitating human examples. To our knowledge, this is the first time a model has been shown to express calibrated uncertainty about its own answers in natural language. For testing calibration, we introduce the CalibratedMath suite of tasks. We compare the calibration of uncertainty expressed in words (“verbalized probability”) to uncertainty extracted from model logits. Both kinds of uncertainty are capable of generalizing calibration under distribution shift. We also provide evidence that GPT-3’s ability to generalize calibration depends on pre-trained latent representations that correlate with epistemic uncertainty over its answers. Introduction Current state-of-the-art language models perform well on a wide range of challenging question-answering tasks. They can even outperform the average human on the MMLU benchmark (which consists of exam-like questions across 57 categories) and on BIG-Bench (which consists of 150+ diverse tasks). Yet when models generate long-form text, they often produce false statements or “hallucinations”. This reduces their value to human users, as users cannot tell when a model is being truthful or not. The problem of truthfulness motivates calibration for language models. If models convey calibrated uncertainty about their statements, then users know how much to trust a given statement. This is important for current models (which often hallucinate falsehoods) but also for any model that makes statements where there is no known ground truth (e.g. economic forecasts, open problems in science or mathematics). Previous work on calibration focuses on the model log-probabilities or “logits”. Yet the log-probabilities of models like GPT-3 represent uncertainty over tokens (ways of expressing a claim) and not epistemic uncertainty over claims themselves. If a claim can be paraphrased in many different ways, then each paraphrase may have a low log-probability. By contrast, when humans express uncertainty, this is epistemic uncertainty about the claim itself. In this paper, we finetune models to express epistemic uncertainty using natural language. We call this “verbalized probability”. The goal of verbalized probability is to express uncertainty in a human-like way but not to directly mimic human training data. Models should be calibrated about their own uncertainty, which differs from human uncertainty. For example, GPT-3 outperforms most humans on a computer security quiz but is much worse at arithmetic questions of the form “2 × 3 × 7 =?”. Thus, we expect pre-trained models will need to be finetuned to produce calibrated verbalized probabilities. Training models in verbalized probability is a component of making models “honest”. We define a model as honest if it can communicate everything it represents internally in natural language (and will not misrepresent any internal states). Honesty helps with AI alignmen...