Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Quantifying Uncertainty in GiveWell's GiveDirectly Cost-Effectiveness Analysis, published by Hazelfire on May 27, 2022 on The Effective Altruism Forum. Effort: This took about 40 hours to research and write, excluding time spent developing Squiggle. Disclaimer: Opinions do not represent the Quantified Uncertainty Research Institute nor GiveWell. I talk about GiveWell's intentions with the GiveDirectly model. However, my understanding may not be complete. All mistakes are my own. Epistemic Status: I am uncertain about these uncertainties! Most of them are best guesses. I could also be wrong about the inconsistencies I've identified. A lot of these issues could easily be considered bike-shedding. Target Audiences: I wrote this post for: People who are interested in evaluating interventions. People who are interested in the quantification of uncertainty. EA software developers that are interested in open source projects. TLDR: I've transposed GiveDirectly's Cost-Effectiveness Analysis into an interactive notebook. This format allows us to measure our uncertainty about GiveDirectly's cost-effectiveness. The model finds that GiveDirectly's 95% confidence interval for its effectiveness spans an order of magnitude, which I deem a relatively low level of uncertainty. Additionally, I found an internal inconsistency with the model that increased GiveDirectly's cost-effectiveness by 11%. The notebook is quite long, detailed and technical. Therefore, I present a summary in this post. This model uses Squiggle, an in-development language for estimation and evaluation, developed by myself and others at the Quantified Uncertainty Research Institute. We'll write more about the language itself in future posts, especially as it becomes more stable. GiveWell's cost-effectiveness analyses (CEAs) of top charities are often considered the gold standard. However, they still have room for improvement. One such improvement is the quantification of uncertainty. I created a Squiggle Notebook that investigates this for GiveDirectly CEA. This notebook also serves as an example of Squiggle and what's possible with future CEAs. In GiveWell's CEAs, GiveDirectly is used as a benchmark to evaluate other interventions. All other charities' effectiveness is measured relative to GiveDirectly. For example, as of 2022, the Against Malaria Foundation was calculated to be 7.1x to 15.4x as cost-effective as GiveDirectly. Evidence Action's Deworm the World is considered 5.3x to 38.2x as cost-effective. GiveDirectly makes a good benchmark because unconditional cash transfers have a strong (some might even say tautological) case behind their effectiveness. GiveDirectly being a benchmark makes it a good start for quantifying uncertainty. I also focus on GiveDirectly because it's the most simple CEA. GiveWell CEAs do not include explicit considerations of uncertainty in their analysis. However, quantifying uncertainty has many benefits. It can: Improve people's understanding of how much evidence we have behind interventions. Help us judge the effectiveness of further research on an intervention using the Value of Information. Allows us to forecast parameters and better determine how wrong we were about different parameters to correct them over time. Cole Haus has done similar work quantifying uncertainty on GiveWell models in Python. The primary decision in this work is choosing how much uncertainty each parameter has. I decided on this with two different methods: If there was enough information about the parameter, I performed a formal bayesian update. If there wasn't as much information, I guessed it with the help of Nuño Sempere, a respected forecaster. These estimates are simple, and future researchers could better estimate them. Results Methodology and calculations are in my Squiggle notebook: /@hazelfire/giv...