The Annals of Applied Statistics

Calibrating a large computer experiment simulating radiative shock hydrodynamics

Robert B. Gramacy, Derek Bingham, James Paul Holloway, Michael J. Grosskopf, Carolyn C. Kuranz, Erica Rutter, Matt Trantham, and R. Paul Drake

Full-text: Open access


We consider adapting a canonical computer model calibration apparatus, involving coupled Gaussian process (GP) emulators, to a computer experiment simulating radiative shock hydrodynamics that is orders of magnitude larger than what can typically be accommodated. The conventional approach calls for thousands of large matrix inverses to evaluate the likelihood in an MCMC scheme. Our approach replaces that costly ideal with a thrifty take on essential ingredients, synergizing three modern ideas in emulation, calibration and optimization: local approximate GP regression, modularization, and mesh adaptive direct search. The new methodology is motivated both by necessity—considering our particular application—and by recent trends in the supercomputer simulation literature. A synthetic data application allows us to explore the merits of several variations in a controlled environment and, together with results on our motivating real-data experiment, lead to noteworthy insights into the dynamics of radiative shocks as well as the limitations of the calibration enterprise generally.

Article information

Ann. Appl. Stat., Volume 9, Number 3 (2015), 1141-1168.

Received: October 2014
Revised: June 2015
First available in Project Euclid: 2 November 2015

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Emulator tuning nonparametric regression big data local Gaussian process mesh adaptive direct search (MADS) modularization


Gramacy, Robert B.; Bingham, Derek; Holloway, James Paul; Grosskopf, Michael J.; Kuranz, Carolyn C.; Rutter, Erica; Trantham, Matt; Drake, R. Paul. Calibrating a large computer experiment simulating radiative shock hydrodynamics. Ann. Appl. Stat. 9 (2015), no. 3, 1141--1168. doi:10.1214/15-AOAS850.

Export citation


  • Audet, C. and Dennis, J. E. Jr. (2006). Mesh adaptive direct search algorithms for constrained optimization. SIAM J. Optim. 17 188–217 (electronic).
  • Ba, S. and Joseph, V. R. (2012). Composite Gaussian process models for emulating expensive functions. Ann. Appl. Stat. 6 1838–1860.
  • Bastos, L. S. and O’Hagan, A. (2009). Diagnostics for Gaussian process emulators. Technometrics 51 425–438.
  • Boehly, T. R., Brown, D. L., Craxton, R. S., Keck, R. L., Knauer, J. P., Kelly, J. H., Kessler, T. J., Kumpan, S. A., Loucks, S. J., Letzring, S. A., Marshall, F. J., McCrory, R. L., Morse, S. F. B., Seka, W., Soures, J. M. and Verdon, C. P. (1997). Initial performance results of the OMEGA laser system. Opt. Commun. 133 495–506.
  • Bornn, L., Shaddick, G. and Zidek, J. V. (2012). Modeling nonstationary processes through dimension expansion. J. Amer. Statist. Assoc. 107 281–289.
  • Conn, A. R., Scheinberg, K. and Vicente, L. N. (2009). Introduction to Derivative-Free Optimization. MPS/SIAM Series on Optimization 8. SIAM, Philadelphia, PA.
  • Damblin, G., Barbillon, P., Keller, M., Pasanisi, A. and Parent, E. (2015). Adaptive numerical designs for the calibration of computer models. Technical report, AgroParisTech.
  • Drake, R. P., Doss, F. W., McClarren, R. G., Adams, M. L., Amato, N., Bingham, D., Chou, C. C., DiStefano, C., Fidkowski, K., Fryxell, B., Gombosi, T. I., Grosskopf, M. J., Holloway, J. P., van der Holst, B., Huntington, C. M., Karni, S., Krauland, C. M., Kuranz, C. C., Larsen, E., van Leer, B., Mallick, B., Marion, D., Martin, W., Morel, J. E., Myra, E. S., Nair, V., Powell, K. G., Rauchwerger, L., Roe, P., Rutter, E., Sokolov, I. V., Stout, Q., Torralva, B. R., Toth, G., Thornton, K. and Visco, A. J. (2011). Radiative effects in radiative shocks in shock tubes. Opt. Commun. 7 130–140.
  • Eidsvik, J., Shaby, B. A., Reich, B. J., Wheeler, M. and Niemi, J. (2014). Estimation and prediction in spatial models with block composite likelihoods. J. Comput. Graph. Statist. 23 295–315.
  • Goh, J., Bingham, D., Holloway, J. P., Grosskopf, M. J., Kuranz, C. C. and Rutter, E. (2013). Prediction and computer model calibration using outputs from multifidelity simulators. Technometrics 55 501–512.
  • Gramacy, R. B. (2013). laGP: Local approximate Gaussian process regression. R package version 1.0.
  • Gramacy, R. B. and Apley, D. W. (2015). Local Gaussian process approximation for large computer experiments. J. Comput. Graph. Statist. 24 561–578.
  • Gramacy, R. and Haaland, B. (2015). Speeding up neighborhood search in local Gaussian process prediction. Technometrics. To appear. Available at arXiv:1409.0074.
  • Gramacy, R. B., Niemi, J. and Weiss, R. M. (2014). Massively parallel approximate Gaussian process regression. SIAM/ASA J. Uncertain. Quantificat. 2 564–584.
  • Gramacy, R. B. and Polson, N. G. (2011). Particle learning of Gaussian process models for sequential design and optimization. J. Comput. Graph. Statist. 20 102–118.
  • Haaland, B. and Qian, P. Z. G. (2011). Accurate emulators for large-scale computer experiments. Ann. Statist. 39 2974–3002.
  • Higdon, D., Kennedy, M., Cavendish, J. C., Cafeo, J. A. and Ryne, R. D. (2004). Combining field data and computer simulations for calibration and prediction. SIAM J. Sci. Comput. 26 448–466.
  • Kaufman, C. G., Bingham, D., Habib, S., Heitmann, K. and Frieman, J. A. (2011). Efficient emulators of computer experiments using compactly supported correlation functions, with an application to cosmology. Ann. Appl. Stat. 5 2470–2492.
  • Kennedy, M. C. and O’Hagan, A. (2001). Bayesian calibration of computer models. J. R. Stat. Soc. Ser. B. Stat. Methodol. 63 425–464.
  • Kleijnen, J. P. C. (2014). Simulation-optimization via Kriging and bootstrapping. J. Simul. 8 241–250.
  • Le Digabel, S. (2011). Algorithm 909: NOMAD: Nonlinear optimization with the MADS algorithm. ACM Trans. Math. Software 37 Art. 44, 15.
  • Liu, F., Bayarri, M. J. and Berger, J. O. (2009). Modularization in Bayesian analysis, with emphasis on analysis of computer models. Bayesian Anal. 4 119–150.
  • Loeppky, J., Bingham, D. and Welch, W. (2006). Computer model calibration or tuning in practice. Technical report, Univ. British Columbia.
  • MacDonald, N., Ranjan, P. and Chipman (2012). GPfit: An $\mathsf{R}$ package for Gaussian process model fitting using a new optimization algorithm. Technical report, Acadia Univ., Wolfville, Nova Scotia. Available at arXiv:1305.0759.
  • McClarren, R., Ryub, D., Drake, P., Grosskopf, M., Bingham, D., Chou, C.-C., Fryxell, B., van der Holst, B., Holloway, J., Kuranz, C., Mallick, B., Rutter, E. and Torralva, B. (2011). A physics informed emulator for laser-driven radiating shock simulations. Reliab. Eng. Syst. Saf. 96 1194–1207.
  • McKay, M. D., Beckman, R. J. and Conover, W. J. (1979). A comparison of three methods for selecting values of input variables in the analysis of output from a computer code. Technometrics 21 239–245.
  • Morris, M. D., Mitchell, T. J. and Ylvisaker, D. (1993). Bayesian design and analysis of computer experiments: Use of derivatives in surface prediction. Technometrics 35 243–255.
  • Paciorek, C. J. and Schervish, M. J. (2006). Spatial modelling using a new class of nonstationary covariance functions. Environmetrics 17 483–506.
  • Paciorek, C., Lipshitz, B., Zhuo, W., Prabhat, Kaufman, C. and Thomas, R. (2013). Parallelizing Gaussian process calculations in $\mathsf{R}$. Technical report, Univ. California, Berkeley. Available at arXiv:1305.4886.
  • Racine, J. S. and Nie, Z. (2012). crs: Categorical regression splines. R package version 0.15-18.
  • Sacks, J., Welch, W. J., Mitchell, T. J. and Wynn, H. P. (1989). Design and analysis of computer experiments. Statist. Sci. 4 409–435.
  • Sang, H. and Huang, J. Z. (2012). A full scale approximation of covariance functions for large spatial data sets. J. R. Stat. Soc. Ser. B. Stat. Methodol. 74 111–132.
  • Santner, T. J., Williams, B. J. and Notz, W. I. (2003). The Design and Analysis of Computer Experiments. Springer, New York.
  • Schmidt, A. M. and O’Hagan, A. (2003). Bayesian inference for non-stationary spatial covariance structure via spatial deformations. J. R. Stat. Soc. Ser. B. Stat. Methodol. 65 743–758.
  • Sobol, W. (1993). Analysis of variance of “component stripping” decomposition of multi exponential curves. Comput. Methods Programs Biomed. 39 243–257.
  • Stein, M. L., Chi, Z. and Welty, L. J. (2004). Approximating likelihoods for large spatial data sets. J. R. Stat. Soc. Ser. B. Stat. Methodol. 66 275–296.
  • Vecchia, A. V. (1988). Estimation and model identification for continuous spatial processes. J. R. Stat. Soc. Ser. B. Stat. Methodol. 50 297–312.
  • Wong, R. K., Storlie, C. B. and Lee, T. C. (2014). A frequentist approach to computer model calibration. Technical report, Iowa State Univ., Ames, IA.