Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Five steps for quantifying speculative interventions, published by NunoSempere on February 18, 2022 on The Effective Altruism Forum. Summary Currently, we can’t compare the impact of speculative interventions in a principled way. When making a decision about where to work or donate, longtermists or risk-neutral neartermists may have to choose an organization based on status, network effects, or expert opinion. This is, obviously, not ideal. I propose a simple solution, if not an easy one. First, estimate the impact of an intervention in narrow units (such as micro-covids, or estimates of research quality). Then, convert those narrow units to more and more general units (such as QALYs, or percentage reduction in x-risk). Quantifying the value of speculative interventions to a standard similar to GiveWell’s represents a lot of work on a messy problem. In this post, I’ll break it down to these five steps: Create narrow units for specific types of interventions Use narrow units to evaluate interventions Create more general units, and conversion factors from narrow units to general units Resolve or quantify crucial considerations in order to generalize further Scale-up evaluations: do more evaluations, better, more cheaply, about more things. As we make progress on these subproblems, relative value comparisons would become more robust, principled and transparent, which would improve the quality of our decision-making around funding and prioritization. Decisions about where to work or where to donate might still be informed by some subjective factors (e.g., personal fit, value differences), but they would be more grounded in research and expected utility calculations. This proposal grew out of my frustrations with quantitatively evaluating longtermist organizations or EA projects more generally without a developed framework. Nobody has really been doing this kind of evaluation, so the infrastructure and know-how is just not there. It’s not even clear what the bar for funding longtermist interventions should be–we don’t know how much good "the last longtermist dollar" will accomplish. Without that key number, funders have to make grants to the best of their abilities by using heuristics and intuitions, which naturally has limitations. It has been argued that expected utility calculations can be misleading or counter-productive. But these calculations don't have to be perfect, they just have to be better than the alternative—whatever non-quantitative methods people would have used instead. It also doesn’t matter in practice whether one can reach expected value calculations in all their glory, as long as the efforts towards quantification end up paying off (e.g., in terms of better decisions). So from my perspective, one of the most powerful tools in the EA arsenal has been left gathering dust, mostly for unclear reasons. In the short term, intuition or heuristics can fill in the gap. But in the long term, as EA moves billions of additional dollars, we will need to upgrade intuition-based human factors to auditable, scalable and more powerful evaluation methods. Step 1: Create narrow units for specific types of interventions The simplest and cheapest way to start seems with units tailored to one particular intervention or type of intervention. I’m going to call these “narrow units”, as opposed to more general (e.g., QALYs, which could denominate many types of interventions) or abstract ones (e.g., measures of “research value”). With narrow units, we can ask if a unit captures most of what we care about in an intervention, and evaluate a new unit on that metric. In the case of research at EA organizations, we care about how it directly influences decisions, but also about the further research it enables, the mentorship around it, the prestige that the authors attain, etc. A un...