Expert elicitation

Eliciting a probability distribution is the process of extracting an expert’s beliefs about some unknown quantity of interest, and representing his/her beliefs with a probability distribution. The challenges are, firstly, to help the expert consider uncertainty carefully, without being excessively overconfident or underconfident, and secondly, to find a way of constructing a full probability distribution based on a small number of simple probability judgements from the expert. Elicitation can be used to construct prior distributions in Bayesian inference, though my interest is more in situations where we are using expert judgement because there is no data.

SHELF

Tony O’Hagan and I have developed the Sheffield Elicitation Framework (SHELF), a package of protocols, templates and guidance documents for conduction expert elicitation. In support of this, I maintain an R package SHELF which is available on CRAN (with a developmement version on GitHub).

Shiny apps for elicitation

The SHELF R package includes shiny apps for implementing various elicitation methods. If you are not an R user, you can try the following apps online. Please note that access to these apps is time-limited (they are hosted on RStudio’s shinyapps.io service). If you want to use the apps in an actual elicitation workshop, I strongly encourage you to use the offline versions in the SHELF R package.

The MATCH elicitation tool

As part of the MATCH project, we produced a web-based elicitation tool which is based on an earlier version of the SHELF R code. Multiple users can log into the same session, which can be useful when the facilitator and expert(s) can’t meet in the same room.

Papers on elicitation

I have also contributed to the following guidance document prepared by the European Food Safety Authority: