Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Procedurally evaluating factual accuracy: a request for research, published by Jacob Hilton on March 30, 2022 on The AI Alignment Forum. I am grateful to Daniel Kokotajlo, Beth Barnes and John Schulman for feedback on this post. The purpose of this post is to request research on the design of precise procedures for evaluating how factually accurate pieces of text are. This stands out to me as an area that is potentially valuable for reducing risks from advanced AI, while not requiring detailed knowledge of ML. Problem statement Suppose that you are given: A real-world setting in which text appears, such as "Google search snippets", or "OpenAI API outputs", or "arXiv abstracts" A collection of contexts in that setting, such as a "the search queries made in 2021", or "the questions from the ELI5 dataset", or "all existing papers on the arXiv" A text distribution for each context, such as "actual current Google search snippets", or "responses produced by an ML model", or "the abstract a certain person would write after reading the paper" The problem is to define a procedure that takes in a context and a piece of text from the given distribution, and outputs a numeric score for factual accuracy. The procedure should have the following properties: It is feasible for humans to carry out. It is as unambiguous as possible: different people following the same procedure should end up with the same score, except perhaps for a minimal number of "irreducible" judgment calls. It can be optimized to produce good outcomes: to the extent that factual accuracy is important in the given setting, selecting text with a high score is beneficial, compared to alternative procedures. Included in this is the effect that knowledge of the procedure could have, e.g. people will have more trust in the text if they understand and endorse the procedure. In practice, whether good outcomes are produced will depend on the method and extent of optimization (e.g., Goodhart's law will eventually kick in), and on criteria other than factual accuracy. This makes the problem statement a little fuzzy. Note that the problem statement doesn't mention ML models (except as examples). The problem is of course motivated by ML models, but I think it can be studied relatively independently of ML, at least initially. Motivation The main motivation for this request is that it is a problem that arises very naturally when attempting to train truthful LMs. The most straightforward way to optimize the factual accuracy of a language model is to have humans evaluate the factual accuracy of model outputs, and to then optimize those evaluations using techniques like reinforcement learning. The procedure needs to be unambiguous because label noise hurts both ML training and labeler monitoring (not to mention other benefits). Subject to this constraint, the main criterion for the procedure should be that it produces good outcomes in the given real-world setting. My main reasons for thinking that this research could be important for reducing risks from advanced AI are: Direct benefits. The research could result in procedures that are directly used to train more beneficial AI. The sorts of risks this could help mitigate are discussed in more depth in Truthful AI and Risks from AI persuasion. Preparation for harder versions of the problem. I expect advanced AI systems to pose analogous versions of the same problem, but to require more advanced solutions, in which AI systems assist with human evaluation, for example. Solutions to the simpler problem can serve both as a starting point for more advanced solutions and as a baseline against which they can be compared, and can help build relevant capacity. Supporting ML work on truthful LMs. Currently, the responsibility for this research falls largely on ML researchers trying to improv...