Improving the production and evaluation of structural models using a Delphi process

Bolger, Fergus; Nyberg, Erik P.; Belton, Ian; Crawford, Megan M.; Hamlin, Iain; Nicholson, Ann; Alvandi, Abraham Oshni; Pearson, Ross; Riley, Jeff; Sissons, Aileen; Taylor Browne Lūka, Courtney; Thakur, Shreshth; Vasilichi, Alexandrina; Wright, George

Abstract

Bayes Nets (BNs) are extremely useful for causal and probabilistic modelling in many real-world applications, often built with information elicited from groups of domain experts. But their potential for reasoning and decision support has been limited by two major factors: the need for significant normative knowledge, and the lack of any validated methods or software supporting collaboration. Consequently, we have developed a web-based structured technique – Bayesian Argumentation via Delphi (BARD) – to enable groups of domain experts to receive minimal normative training and then collaborate effectively to produce high-quality BNs. BARD harnesses multiple perspectives on a problem, while minimising biases manifest in freely interacting groups, via a Delphi process: solutions are first produced individually, then shared, followed by an opportunity for individuals to revise their solutions. To test the hypothesis that BNs improve due to Delphi, we conducted an experiment whereby individuals with a little BN training and practice produced structural models using BARD for two Bayesian reasoning problems. Participants then received 6 other structural models for each problem, rated their quality on a 7-point scale, and revised their own models if they wished. Both top-rated and revised models were on average significantly better quality (scored against a gold-standard) than the initial models, with large and medium effect sizes. We conclude that Delphi – and BARD – improves the quality of BNs produced by groups. Further, although rating cannot create new models, rating seems quicker and easier than revision and yielded significantly better models – so, we suggest efficient BN amalgamation should include both.

Citation

Bolger, F., Nyberg, E. P., Belton, I., Crawford, M. M., Hamlin, I., Nicholson, A., Alvandi, A. O., Pearson, R., Riley, J., Sissons, A., Taylor Browne Lūka, C., Thakur, S., Vasilichi, A., & Wright, G. Improving the production and evaluation of structural models using a Delphi process

• Working Paper Type: Preprint
• Deposit Date: Jun 7, 2024
• DOI: https://doi.org/10.31219/osf.io/v6qsp
• Keywords: Bayes Nets, causal models, crowdsourcing, aggregation, group processes, Delphi