Eliciting a probability distribution is the process of extracting an expert’s beliefs about some unknown quantity of interest, and representing his/her beliefs with a probability distribution. The challenges are, firstly, to help the expert consider uncertainty carefully, without being excessively overconfident or underconfident, and secondly, to find a way of constructing a full probability distribution based on a small number of simple probability judgements from the expert. Elicitation can be used to construct prior distributions in Bayesian inference, though my interest is more in situations where we are using expert judgement because there is no data.
Tony O’Hagan and I have developed the Sheffield Elicitation Framework (SHELF), a package of protocols, templates and guidance documents for conduction expert elicitation. In support of this, I maintain an R package
SHELF which is available on CRAN (with a developmement version on GitHub).
Shiny apps for elicitation
SHELF R package includes shiny apps for implementing various elicitation methods. If you are not an R user, you can try the following apps online. Please note that access to these apps is time-limited (they are hosted on RStudio’s shinyapps.io service). If you want to use the apps in an actual elicitation workshop, I strongly encourage you to use the offline versions in the
SHELF R package.
Elicting a single distribution. Elicit a single distribution, directly specifying probabilities or quantiles, or using the roulette method.
Eliciting individual distributions from multiple experts. Includes options for fitting a weighted linear pool.
Eliciting bivariate distributions. Elicit a joint distribution for two dependent variables using a Gaussian copula.
Eliciting Dirichlet distributions. Elicit a Dirichlet distribution for a set of proportions constrained to sum to 1.
The extension method: continuous extension and target variables. This method involves eliciting a distribution for a ‘target’ variable $X$ via a distribution for an ‘extension’ variable $Y$ and a conditional distribution for $X|Y$. In this app, both $X$ and $Y$ are continuous.
The extension method: discrete extension variable and continuous target variable. Similar to above, but with a discrete extension variable $Y$. In particular, this method allows the elicitation of multi-modal distributions: the density for $X$ is of the form $\sum_y f_X(x|Y=y)Pr(Y=y)$.
Assurance for clinical trial design with normally distributed data. This implements the method in Alhussain and Oakley (2019).
The MATCH elicitation tool
As part of the MATCH project, we produced a web-based elicitation tool which is based on an earlier version of the SHELF R code. Multiple users can log into the same session, which can be useful when the facilitator and expert(s) can’t meet in the same room.
Papers on elicitation
Alhussain, Z. A. and Oakley, J. E. (2019). Assurance for clinical trial design with normally distributed outcomes: eliciting uncertainty about variances. In submission. R package (assurance) on GitHub
Ren, S., Oakley, J. E. and Stevens, J. W. (2018). Incorporating genuine prior information about between-study heterogeneity in random effects pairwise and network meta-analyses. Medical Decision Making, 38 (4), 531-542
Morris, D. E., Oakley, J. E. and Crowe, J. A. (2014). A web-based tool for eliciting probability distributions from experts. Environmental Modelling & Software, 52, 1-4.
Ren, S. and Oakley, J. E. (2014). Assurance calculations for planning clinical trials with time-to-event outcomes. Statistics in Medicine 33(1), 31-45. Download supporting R code.
Oakley, J. E. (2010). Eliciting univariate probability distributions, in Rethinking Risk Measurement and Reporting: Volume I, edited by Böcker, K., Risk Books, London.
Daneshkhah A. and Oakley, J.E. (2010). Eliciting multivariate probability distributions. (Supporting R code for this chapter is available here) in Rethinking Risk Measurement and Reporting: Volume I , edited by Böcker, K., Risk Books, London.
Nixon, R.M., O’Hagan, A., Oakley, J. E., Madan, J., Stevens, J.W. Bansback, N. and Brennan, A. (2009). The Rheumatoid Arthritis Drug Development Model: A case study in Bayesian clinical trial simulation. Pharmaceutical Statistics 8(4), 371-389
Gosling, J.P., Oakley, J.E. and O’Hagan, A. (2007). Nonparametric elicitation for heavy-tailed prior distributions. Bayesian Analysis, 2, 693-718.
Oakley, J. and O’Hagan, A. (2007). Uncertainty in prior elicitations: a non-parametric approach. Biometrika 94, 427-441.
O’ Hagan, A., Buck, C. E., Daneshkhah, A., Eiser, J. E., Garthwaite, P. H., Jenkinson, D. J., Oakley, J. E. and Rakow, T. (2006). Uncertain Judgements: Eliciting Expert Probabilities. Chichester: Wiley.
O’Hagan, A. and Oakley, J. E. (2004). Probability is perfect, but we can’t elicit it perfectly. Reliability Engineering and System Safety, 85, 239-248.
Oakley, J. (2002). Eliciting Gaussian process priors for complex computer codes. The Statistician, 51, 81-97.
I have also contributed to the following guidance document prepared by the European Food Safety Authority:
- European Food Safety Authority (2014). Guidance on Expert Knowledge Elicitation in Food and Feed Safety Risk Assessment. EFSA Journal 2014;12(6):3734, 278 pp. doi:10.2903/j.efsa.2014.3734