The Annals of Statistics

Efficient calibration for imperfect computer models

Rui Tuo and C. F. Jeff Wu

Full-text: Open access


Many computer models contain unknown parameters which need to be estimated using physical observations. Tuo and Wu (2014) show that the calibration method based on Gaussian process models proposed by Kennedy and O’Hagan [J. R. Stat. Soc. Ser. B. Stat. Methodol. 63 (2001) 425–464] may lead to an unreasonable estimate for imperfect computer models. In this work, we extend their study to calibration problems with stochastic physical data. We propose a novel method, called the $L_{2}$ calibration, and show its semiparametric efficiency. The conventional method of the ordinary least squares is also studied. Theoretical analysis shows that it is consistent but not efficient. Numerical examples show that the proposed method outperforms the existing ones.

Article information

Ann. Statist., Volume 43, Number 6 (2015), 2331-2352.

Received: April 2014
Revised: January 2015
First available in Project Euclid: 7 October 2015

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62P30: Applications in engineering and industry 62A01: Foundations and philosophical topics
Secondary: 62F12: Asymptotic properties of estimators

Computer experiments uncertainty quantification semiparametric efficiency reproducing kernel Hilbert space


Tuo, Rui; Wu, C. F. Jeff. Efficient calibration for imperfect computer models. Ann. Statist. 43 (2015), no. 6, 2331--2352. doi:10.1214/15-AOS1314.

Export citation


  • Anderson-Cook, C. M. and Prewitt, K. (2005). Some guidelines for using nonparametric methods for modeling data from response surface designs. Journal of Modern Applied Statistical Methods 4 106–119.
  • Bayarri, M. J., Berger, J. O., Cafeo, J., Garcia-Donato, G., Liu, F., Palomo, J., Parthasarathy, R. J., Paulo, R., Sacks, J. and Walsh, D. (2007a). Computer model validation with functional output. Ann. Statist. 35 1874–1906.
  • Bayarri, M. J., Berger, J. O., Paulo, R., Sacks, J., Cafeo, J. A., Cavendish, J., Lin, C.-H. and Tu, J. (2007b). A framework for validation of computer models. Technometrics 49 138–154.
  • Berlinet, A. and Thomas-Agnan, C. (2004). Reproducing Kernel Hilbert Spaces in Probability and Statistics Kluwer, Boston, MA.
  • Bickel, P. J., Klaassen, C. A. J., Ritov, Y. and Wellner, J. A. (1993). Efficient and Adaptive Estimation for Semiparametric Models. Johns Hopkins Univ. Press, Baltimore, MD.
  • Box, G. E. P., Hunter, J. S. and Hunter, W. G. (2005). Statistics for Experimenters: Design, Innovation, and Discovery, 2nd ed. Wiley, Hoboken, NJ.
  • Cressie, N. A. C. (1993). Statistics for Spatial Data. Wiley, New York.
  • Edmunds, D. E. and Triebel, H. (1996). Function Spaces, Entropy Numbers, Differential Operators. Cambridge Univ. Press, Cambridge.
  • Evans, S. N. and Stark, P. B. (2002). Inverse problems as statistics. Inverse Probl. 18 R55–R97.
  • Goh, J., Bingham, D., Holloway, J. P., Grosskopf, M. J., Kuranz, C. C. and Rutter, E. (2013). Prediction and computer model calibration using outputs from multifidelity simulators. Technometrics 55 501–512.
  • Goldstein, M. and Rougier, J. (2004). Probabilistic formulations for transferring inferences from mathematical models to physical systems. SIAM J. Sci. Comput. 26 467–487 (electronic).
  • Gramacy, R. B. and Lee, H. K. H. (2012). Cases for the nugget in modeling computer experiments. Stat. Comput. 22 713–722.
  • Groeneboom, P. and Wellner, J. A. (1992). Information Bounds and Nonparametric Maximum Likelihood Estimation 19. Birkhäuser, Basel.
  • Han, G., Santner, T. J. and Rawlinson, J. J. (2009). Simultaneous determination of tuning and calibration parameters for computer experiments. Technometrics 51 464–474.
  • Higdon, D., Kennedy, M., Cavendish, J. C., Cafeo, J. A. and Ryne, R. D. (2004). Combining field data and computer simulations for calibration and prediction. SIAM J. Sci. Comput. 26 448–466.
  • Higdon, D., Gattiker, J., Williams, B. and Rightley, M. (2008). Computer model calibration using high-dimensional output. J. Amer. Statist. Assoc. 103 570–583.
  • Higdon, D., Gattiker, J., Lawrence, E., Jackson, C., Tobis, M., Pratola, M., Habib, S., Heitmann, K. and Price, S. (2013). Computer model calibration using the ensemble Kalman filter. Technometrics 55 488–500.
  • Joseph, V. R. and Melkote, S. N. (2009). Statistical adjustments to engineering models. Journal of Quality Technology 41 362–375.
  • Kennedy, M. C. and O’Hagan, A. (2001). Bayesian calibration of computer models. J. R. Stat. Soc. Ser. B. Stat. Methodol. 63 425–464.
  • Kosorok, M. R. (2008). Introduction to Empirical Processes and Semiparametric Inference. Springer, New York.
  • Mammen, E. and van de Geer, S. (1997). Penalized quasi-likelihood estimation in partial linear models. Ann. Statist. 25 1014–1035.
  • Murphy, J. M., Booth, B. B. B., Collins, M., Harris, G. R., Sexton, D. M. H. and Webb, M. J. (2007). A methodology for probabilistic predictions of regional climate change from perturbed physics ensembles. Philos. Trans. R. Soc. Lond. Ser. A Math. Phys. Eng. Sci. 365 1993–2028.
  • Myers, R. H. (1999). Response surface methodlogy: Current status and future directions. Journal of Quality Technology 31 30–44.
  • Peng, C.-Y. and Wu, C. F. J. (2014). On the choice of nugget in kriging modeling for deterministic computer experiments. J. Comput. Graph. Statist. 23 151–168.
  • Rasmussen, C. E. and Williams, C. K. I. (2006). Gaussian Processes for Machine Learning. MIT Press, Cambridge, MA.
  • Santner, T. J., Williams, B. J. and Notz, W. I. (2003). The Design and Analysis of Computer Experiments. Springer, New York.
  • Schölkopf, B., Herbrich, R. and Smola, A. J. (2001). A generalized representer theorem. In Computational Learning Theory (Amsterdam, 2001). Lecture Notes in Comput. Sci. 2111 416–426. Springer, Berlin.
  • Shen, X. (1997). On methods of sieves and penalization. Ann. Statist. 25 2555–2591.
  • Stein, M. L. (1999). Interpolation of Spatial Data: Some Theory for Kriging. Springer, New York.
  • Stone, C. J. (1982). Optimal global rates of convergence for nonparametric regression. Ann. Statist. 10 1040–1053.
  • Tuo, R. and Wu, C. F. J. (2014). A theoretical framework for calibration in computer models: Parametrization, estimation and convergence properties. Technical report, Chinese Acad. Sci. and Georgia Inst. Technol.
  • van de Geer, S. (2000). Empirical Processes in M-Estimation 45. Cambridge Univ. Press, Cambridge.
  • van der Vaart, A. W. (1998). Asymptotic Statistics. Cambridge Series in Statistical and Probabilistic Mathematics 3. Cambridge Univ. Press, Cambridge.
  • van der Vaart, A. W. and Wellner, J. A. (1996). Weak Convergence and Empirical Processes: With Applications to Statistics. Springer, New York.
  • Wahba, G. (1990). Spline Models for Observational Data 59. SIAM, Philadelphia, PA.
  • Wang, S., Chen, W. and Tsui, K.-L. (2009). Bayesian validation of computer models. Technometrics 51 439–451.
  • Wendland, H. (2005). Scattered Data Approximation. Cambridge Monographs on Applied and Computational Mathematics 17. Cambridge Univ. Press, Cambridge.
  • Wu, C. J. and Hamada, M. S. (2009). Experiments: Planning, Analysis, and Optimization 2nd ed. Wiley, New York.
  • Xiu, D. (2010). Numerical Methods for Stochastic Computations: A Spectral Method Approach. Princeton Univ. Press, Princeton, NJ.