## Research interests

### Analysing uncertainty in complex computer models

Computer models (also known as process models, mechanistic models, simulation models etc.) are used widely throughout science and engineering for making predictions, and for conducting 'virtual experiments' when physical experiments would be too costly or impractical. There will almost always be uncertainty in any model prediction, caused by uncertainty about what input values to use, and/or uncertainty about how well the model represents reality. We cannot trust a computer model prediction until we have quantified the uncertainty properly.

My interest in this topic began with my PhD, which was on propagating input uncertainty through computationally expensive models, using Gaussian process emulators. I continue to work on work on methods for dealing with input uncertainty, though I think the most important problems now are to do with how we quantify uncertainty about model discrepancy: the difference between a model prediction and reality.

Uncertainty Quantification Research Group

Further reading and publications (show/hide)
For an overview of some of the main statistical issues, see this discussion in
the MUCM Toolkit. To learn more about research activity in the field, see the the MUCM community homepage.

#### Publications on uncertainty in computer models

- Oakley, J. E. and Youngman, B.D. (2017). Calibration of stochastic computer simulators using likelihood emulation. Technometrics, 59, 1, 80-92.
- Strong M., Oakley J. E., Brennan A. and Breeze, P. (2015). Estimating the expected value of sample
information using the probabilistic
sensitivity analysis sample: a fast
nonparametric regression-based method. Medical Decision Making, 35(5), 570-83.
- Andrianakis I, Vernon IR, McCreesh N, McKinley TJ, Oakley JE, et al. (2015) Bayesian History Matching of Complex Infectious Disease Models Using Emulation: A Tutorial and a Case Study on HIV in Uganda. PLoS Comput Biol 11(1): e1003968. doi: 10.1371/journal.pcbi.1003968
- Strong, M. and Oakley, J. E. (2014). When is a model good enough? Deriving the expected value of model improvement via specifying internal model discrepancies. SIAM/ASA Journal on Uncertainty Quantification, 2(1), 106-125.
- Strong M, Oakley J. E., Brennan A. (2014). Estimating multi-parameter partial Expected Value of Perfect Information from a probabilistic sensitivity analysis sample: a non-parametric regression approach. Medical Decision Making, 34(3), 311-26.
- Strong, M. and Oakley, J. E. (2013). An efficient method for computing single parameter partial expected value of perfect information. Medical Decision Making, 33, 755-766.
- Fricker, T. E., Oakley, J. E. and Urban, N. M. (2013). Multivariate Gaussian process emulators with nonseparable covariance structures. Technometrics, 55(1), 47-56.
- Becker, W., Oakley, J. E., Surace, C., Gili, P., Rowson, J., & Worden, K. (2012). Bayesian sensitivity analysis of a nonlinear finite element model. Mechanical Systems and Signal Processing, 32, 18-31.
- Strong, M., Oakley J. E. and Chilcott, J. (2012). Managing structural uncertainty in health economic decision models: a discrepancy approach. Journal of the Royal Statistical Society, Series C, 61(1), 25-45.
- Wilkinson, R. D., Vrettas, M., Cornford, D. and Oakley, J. E. (2011). Quantifying simulator discrepancy in discrete-time dynamical simulators. Journal of Agricultural, Biological, and Environmental Statistics,16(4), 554-570.
- Fricker, T. E., Oakley J. E., Sims, N. D. and Worden, K. and Chilcott, J. (2011). Probabilistic uncertainty analysis of an FRF of a structure using a Gaussian process emulator. Mechanical Systems and Signal Processing, 25(8), 2962-2975.
- Oakley, J. E. (2011). Modelling with deterministic computer models. In Simplicity, Complexity and Modelling, M. Christie, A. Cliffe, P. Dawid and S. Senn (eds.). Chichester: Wiley.
- Becker, W., Rowson, J., Oakley J. E., Yoxall, A., Manson, G. and Worden K. (2011). Bayesian sensitivity analysis of a model of the aortic valve. Journal of Biomechanics 44(8), 1499-1506.
- Oakley, J. E. and Clough, H. E. (2010) Sensitivity analysis in microbial risk assessment: vero-cytotoxigenic E.coli O157 in farm-pasteurised milk. Handbook of Applied Bayesian Analysis, O'Hagan, A. and West, M. (eds). Oxford University Press.
- Conti, S., Gosling, J. P., Oakley, J. E. and O'Hagan, A. (2009). Gaussian process emulation of dynamic computer codes. Biometrika 96, 663-676.
- Oakley, J. E. (2009). Decision-theoretic sensitivity analysis for complex computer models. Technometrics 51, 121-129.
- Oakley, J. (2004). Estimating percentiles of computer code outputs. Journal of the Royal Statistical Society, Series C, 53, 83-93.
- Oakley, J. and O'Hagan, A. (2004). Probabilistic sensitivity analysis of complex models: a Bayesian approach. Journal of the Royal Statistical Society Series B, 66, 751-769. Download example data.
- Oakley, J. and O'Hagan, A. (2002). Bayesian inference for the uncertainty distribution of computer model outputs. Biometrika, 89, 769-784.
- Oakley, J. (2002). Eliciting Gaussian process priors for complex computer codes. The Statistician, 51, 81-97.
- O'Hagan, A., Kennedy. M. C. and Oakley, J. E. (1999). Uncertainty analysis and other inference tools for complex computer codes (with discussion). In Bayesian Statistics 6, J. M. Bernardo et al (eds.). Oxford University Press, 503-524.

#### PhD Thesis

### Eliciting probability distributions from experts

Eliciting a probability distribution is the process of extracting an expert's beliefs about some unknown quantity of interest, and representing his/her beliefs with a probability distribution. The challenges are, firstly, to help the expert consider uncertainty carefully, without being excessively overconfident or underconfident, and secondly, to find a way of constructing a full probability distribution based on a small number of simple probability judgements from the expert. Elicitation can be used to construct prior distributions in Bayesian inference, though my interest is more in situations where we are using expert judgement because there is no data.

Elicitation software and publications (show/hide)
#### Software

- Together with Tony O'Hagan, I have devised the Sheffield Elicitation Framework (SHELF), a package of templates and R functions for conduction expert elicitation. Although the zip file for download from the SHELF website contains R code, I have moved over to maintaining a separate R package SHELF, which is available on CRAN.
- As part of the MATCH project, we have produced a web-based elicitation tool, which has a useful feature for conducting elicitation remotely, when you can't bring your experts together in the same room.

#### Publications on elicitation

- Alhussain, Z. A. and Oakley, J. E. (2017). Eliciting judgements about uncertain population means and variances. In submission.
- Morris, D. E., Oakley, J. E. and Crowe, J. A. (2014). A web-based tool for eliciting probability distributions from experts. Environmental Modelling & Software, 52, 1-4.
- Ren, S. and Oakley, J. E. (2014). Assurance calculations for planning clinical trials with time-to-event outcomes. Statistics in Medicine 33(1), 31-45. Download supporting R code.
- Oakley, J. E. (2010). Eliciting univariate probability distributions, in Rethinking Risk Measurement and Reporting: Volume I, edited by Böcker, K., Risk Books, London.
- Daneshkhah A. and Oakley, J.E. (2010). Eliciting multivariate probability distributions. (Supporting R code for this chapter is available here) in Rethinking Risk Measurement and Reporting: Volume I , edited by Böcker, K., Risk Books, London.
- Nixon, R.M., O'Hagan, A., Oakley, J. E., Madan, J., Stevens, J.W. Bansback, N. and Brennan, A. (2009). The Rheumatoid Arthritis Drug Development Model: A case study in Bayesian clinical trial simulation. Pharmaceutical Statistics 8(4), 371-389
- Gosling, J.P., Oakley, J.E. and O'Hagan, A. (2007). Nonparametric elicitation for heavy-tailed prior distributions. Bayesian Analysis, 2, 693-718.
- Oakley, J. and O'Hagan, A. (2007). Uncertainty in prior elicitations: a non-parametric approach. Biometrika 94, 427-441.
- O' Hagan, A., Buck, C. E., Daneshkhah, A., Eiser, J. E., Garthwaite, P. H., Jenkinson, D. J., Oakley, J. E. and Rakow, T. (2006). Uncertain Judgements: Eliciting Expert Probabilities. Chichester: Wiley.
- O'Hagan, A. and Oakley, J. E. (2004). Probability is perfect, but we can't elicit it perfectly. Reliability Engineering and System Safety, 85, 239-248.
- Oakley, J. (2002). Eliciting Gaussian process priors for complex computer codes. The Statistician, 51, 81-97.
I have also contributed to the following guidance document prepared by the European Food Safety Authority:

- European Food Safety Authority (2014). Guidance on Expert Knowledge Elicitation in Food and Feed Safety Risk Assessment. EFSA Journal 2014;12(6):3734, 278 pp. doi:10.2903/j.efsa.2014.3734

### Bayesian statistical problems in health economics

Health economics is concerned with assessing the cost-effectiveness of medical technologies. Computer models are often used to make cost-effectiveness estimates, and my main interest is in analysing uncertainty in cost-effectiveness model predictions. Other interests include eliciting utilities for different states of health, in particular the design and analysis of discrete choice surveys. I collaborate with researchers in ScHARR through the

Centre for Bayesian Statistics in Health Economics.

Publications (show/hide)

#### Publications on health economics

- Strong M., Oakley J. E., Brennan A. and Breeze, P. (2015). Estimating the expected value of sample
information using the probabilistic
sensitivity analysis sample: a fast
nonparametric regression-based method. To appear in Medical Decision Making.
- Strong M. and Oakley J. E. When is a model good enough? Deriving the expected value of model improvement via specifying internal model discrepancies. To appear in SIAM/ASA Journal on Uncertainty Quantification.
- Strong M, Oakley J. E., Brennan A. Estimating multi-parameter partial Expected Value of Perfect Information from a probabilistic sensitivity analysis sample: a non-parametric regression approach. To appear in Medical Decision Making.
- Strong, M. and Oakley, J. E. (2013). An efficient method for computing single parameter partial expected value of perfect information. Medical Decision Making, 33, 755-766.
- Strong, M., Oakley J. E. and Chilcott, J. (2012). Managing structural uncertainty in health economic decision models: a discrepancy approach. Journal of the Royal Statistical Society, Series C, 61(1), 25-45
- Strong, M. and Oakley J. E. (2011). Bayesian inference for comorbid disease risks using marginal disease risks and correlation information from a separate source. Medical Decision Making 31(4), 571-581.
- Oakley, J.E., Brennan, A., Tappenden, P. and Chilcott, J.B. (2010). Sample sizes for Monte Carlo partial EVPI calculations. Journal of Health Economics 29(3), 468-77. Download example R code.
- Stevenson, M. D., Oakley, J. E., Lloyd Jones, M. Brennan, A., Compston, J. E. , McCloskey E. V. and Selby P. L. (2009). The Cost-Effectiveness of an RCT to Establish Whether 5 or 10 Years of Bisphosphonate Treatment Is the Better Duration for Women With a Prior Fracture. Medical Decision Making 29(6), 678-689.
- Stevenson, M. D., Oakley, J. E., Chick, S. E. and Chalkidou, K. (2009). The cost-effectiveness of surgical instrument management policies to reduce the risk of vCJD transmission to humans. Journal of the Operational Research Society 60, 506-518.
- Coyle D. and Oakley J. (2008) Estimating the expected value of partial perfect information: a review of methods. The Eur. Journal of Health Economics 9, 251-259.
- Karnon J., McIntosh A., Coster J., Bath P., Hutchinson A., Oakley J., Thomas N., Pratt P., Freeman-Parry L., Karsh B. T., Gandhi T. and Tappenden T. (2008). Modelling the expected net benefits of interventions to reduce the burden of medication errors. Journal of Health Services Research and Policy 13, 85-91.
- Karnon, J., McIntosh, A., Bath, P., Dean, J., Hutchinson, A., Oakley, J., Thomas, N., Pratt, P., Freeman-Parry, L., Karsh, B., Gandhi, T. and Tappenden, P. (2007). Medication errors: a prospective hazard and improvement analysis. Safety Science 45, 523-539.
- Stevenson, M. D., Lloyd Jones, M., De Nigris E., Brewer, N., Davis, S. and Oakley, J. (2005). A systematic review and economic evaluation of alendronate, etidronate, risedronate, raloxifene and teriparatide for the prevention and treatment of postmenopausal osteoporosis. Health Technology Assessment, Vol.9: No. 22.
- Stevenson, M. D., Brazier, J. E., Calvert, N.W., Lloyd-Jones M., Oakley, J. and Kanis, J.A. (2005). Description of an individual patient methodology for calculating the cost-effectiveness of treatments for osteoporosis in women. Journal of the Operational Research Society, 56, 214-221.
- Stevenson M.D., Oakley, J. and Chilcott, J.B. (2004). Gaussian process modelling in conjunction with individual patient simulation modelling: A case study describing the calculation of cost-effectiveness ratios for the treatment of osteoporosis. Medical Decision Making, 24(1), 89-100.
- Tappenden, P., Chilcott, J. B., Eggington, S., Oakley, J. and McCabe, C. (2004). Methods for expected value of information analysis in complex health economic models: developments on the health economics of beta-inteferon and glatiramer acetate for multiple sclerosis. Health Technology Assessment, Vol. 8: No. 27.