Link to original article

Welcome to The Nonlinear Library, where we use Text-to-Speech software to convert the best writing from the Rationalist and EA communities into audio. This is: Comparing top forecasters and domain experts, published by Gavin on March 6, 2022 on The Effective Altruism Forum. Arb is a new EA research consultancy. You can reach us at arb-consulting@pm.me. The superforecasting phenomenon - that certain teams of forecasters are better than other prediction mechanisms like large crowds and simple statistical rules - seems sound. But serious interest in superforecasting stems from the reported triumph of forecaster generalists over non-forecaster experts. (Another version says that they also outperform analysts with classified information.) So distinguish some claims: "Forecasters > the public" "Forecasters > simple models" "Forecasters > experts" "Forecasters > experts with classified info" "Averaged forecasters > experts" "Aggregated forecasters > experts" Is (3) true? This post reviews all the studies we could find on experts vs forecasters. (We also attempt to cover the related question of prediction markets vs experts.) Summary First, our conclusions. These look pessimistic, but are mostly pretty uncertain: We think claim (1) is true with 99% confidence and claim (2) is true with 95% confidence. But surprisingly few studies compare experts to generalists (i.e. study claim 3). Of those we found, the analysis quality and transparency leave much to be desired. The best study found that forecasters and health professionals performed similarly. In other studies, experts had goals besides accuracy, or there were too few of them to produce a good aggregate prediction. (3a) A common misconception is that superforecasters outperformed intelligence analysts by 30%. Instead: Goldstein et al showed that superforecasters outperformed the intelligence community, but this was partly due to the different aggregation technique used (the GJP weighting algorithm performs better than prediction markets, given the apparently low volumes of the ICPM market). The forecaster prediction market performed about as well as the intelligence analyst prediction market; and in general, prediction pools outperform prediction markets in the current market regime (e.g. low subsidies, low volume, perverse incentives, narrow demographics). [85% confidence] (3b) In the same study, the forecaster average was notably worse than the intelligence community. (3c) Ideally, we would pit a crowd of forecasters against a crowd of experts. Only one study, an unpublished extension of Sell et al. manages this; it found a small (~3%) forecaster advantage. The bar may be low. That is: it doesn't seem that hard to become a top forecaster, at present. Expertise, plus basic forecasting training and active willingness to forecast regularly, were enough to be on par with the best forecasters. [33%] In more complex domains, like ML, there could be significant returns to expertise. So it might be better to shift focus from generalist forecasters to competent ML pros who are excited about forecasting. [40%] Table of studies ComparisonResultNotesCategory: Geopolitics Pandemics As usual, it’s unclear if the panel faced other incentives but forecasting accuracy.Movies SCOTUSAn impressively accurate model built on top of FantasySCOTUS predictions, and from Ruger et al. (2004) we know that simple models outperform experts.FantasySCOTUS The model predicted 75% of the cases correctly, which was more accurate than their experts with 59.1%. Elections Low n and errors are somewhat correlated, so this isn't particularly informative. Miscellaneous Goldstein et al (2015) US Intelligence Community Prediction Market (ICPM) Good Judgement Project (GJP): an average, vs a prediction market (PM), vs the best method (selected post hoc among 20). Participants rewarded for accuracy. ICPM was low stakes: play-money, while GJP participants “were paid a small honorarium for their active participation”. N=193 ...