Improving Probability Judgment in Intelligence Analysis: From Structured Analysis to Statistical Aggregation. Issue 5 (17th February 2020)

• Record Type: Journal Article
• Authors: Karvetski, Christopher W.; Mandel, David R.; Irwin, Daniel
• Is Part Of: Risk analysis. Volume 40:Issue 5(2020)
• Page Start: 1040
• Page End: 1057
• Publication Date: 2020-02-17
• Subjects: Accuracy -- aggregation -- Analysis of Competing Hypotheses -- Bayes's theorem -- coherentization -- intelligence analysis -- probability judgment; Technology -- Risk assessment -- Periodicals; 658.403
• DOI: 10.1111/risa.13443
• Languages: English
• ISSNs: 0272-4332

Abstract

Abstract: As in other areas of expert judgment, intelligence analysis often requires judging the probability that hypotheses are true. Intelligence organizations promote the use of structured methods such as "Analysis of Competing Hypotheses" (ACH) to improve judgment accuracy and analytic rigor, but these methods have received little empirical testing. In this experiment, we pitted ACH against a factorized Bayes's theorem (FBT) method, and we examined the value of recalibration (coherentization) and aggregation methods for improving the accuracy of probability judgment. Analytic techniques such as ACH and FBT were ineffective in improving accuracy and handling correlated evidence, and ACH in fact decreased the coherence of probability judgments. In contrast, statistical postanalytic methods (i.e., coherentization and aggregation) yielded large accuracy gains. A wide range of methods for instantiating these techniques were tested. The interactions among the factors considered suggest that prescriptive theorists and interventionists should examine the value of ensembles of judgment‐support methods.