Investigate Discuss Estimate Aggregate for structured expert judgement

Peer-reviewed paper by Anca Hanea, Bonnie Wintle, Marissa McBride, Mark A Burgman, Fiona Fidler, Louisa Flander, Charles R Twardy, Steven Mascaro, Brian Manning
Published on 10 January 2016


This study presents the results of an approach to the prediction of the outcomes of geopolitical events, which we term the IDEA protocol.

The participants investigate the background and causal factors behind a question, predict the outcome, and discuss their thinking with others. They then make a second, private and anonymous judgement of the probability of the event, which is subsequently aggregated mathematically. The method performed well relative to both an equally weighted linear pool and a prediction market, and is relatively simple to implement. The results indicate the value of discussion for removing arbitrary linguistic uncertainty and for sharing and debating knowledge, thereby improving the judgements. Weighting individual judgements based on prior performance using Cooke’s method improved group judgements. Even though some of the results are not statistically significant, the study may not have had sufficient power to detect some important effects. Nevertheless, the results help us to formulate conjectures, which can then be investigated further.

view full article

Subscribe to our mailing list to get our latest updates