Electronic Journal of Statistics

A simple approach to maximum intractable likelihood estimation

F. J. Rubio and Adam M. Johansen

Full-text: Open access

Abstract

Approximate Bayesian Computation (ABC) can be viewed as an analytic approximation of an intractable likelihood coupled with an elementary simulation step. Such a view, combined with a suitable instrumental prior distribution permits maximum-likelihood (or maximum-a-posteriori) inference to be conducted, approximately, using essentially the same techniques. An elementary approach to this problem which simply obtains a nonparametric approximation of the likelihood surface which is then maximised is developed here and the convergence of this class of algorithms is characterised theoretically. The use of non-sufficient summary statistics in this context is considered. Applying the proposed method to four problems demonstrates good performance. The proposed approach provides an alternative for approximating the maximum likelihood estimator (MLE) in complex scenarios.

Article information

Source
Electron. J. Statist., Volume 7 (2013), 1632-1654.

Dates
First available in Project Euclid: 19 June 2013

Permanent link to this document
https://projecteuclid.org/euclid.ejs/1371649229

Digital Object Identifier
doi:10.1214/13-EJS819

Mathematical Reviews number (MathSciNet)
MR3070873

Zentralblatt MATH identifier
1327.62075

Subjects
Primary: 62E17: Approximations to distributions (nonasymptotic) 62F10: Point estimation 62F12: Asymptotic properties of estimators 62G07: Density estimation 65C05: Monte Carlo methods

Keywords
Approximate Bayesian Computation density estimation maximum likelihood estimation Monte Carlo methods

Citation

Rubio, F. J.; Johansen, Adam M. A simple approach to maximum intractable likelihood estimation. Electron. J. Statist. 7 (2013), 1632--1654. doi:10.1214/13-EJS819. https://projecteuclid.org/euclid.ejs/1371649229


Export citation

References

  • Abraham, C., Biau, G. and Cadre, B. (2003). Simple estimation of the mode of a multivariate density., The Canadian Journal of Statistics 31: 23–34.
  • Beaumont, M. A., Zhang, W. and Balding, D. J. (2002). Approximate Bayesian computation in population genetics., Genetics 162: 2025–2035.
  • Besag, J. (1975). Statistical Analysis of Non-Lattice Data., The Statistician 24:179–195.
  • Biau, G., Cérou, F. and Guyader, A. (2012). New Insights into Approximate Bayesian Computation. ArXiv preprint, arXiv:1207.6461.
  • Bickel, D. R. and Früwirth, R. (2006). On a fast, robust estimator of the mode: Comparisons to other robust estimators with applications., Computational Statistics & Data Analysis 50: 3500–3530.
  • Blum, M. G. B. (2010). Approximate Bayesian computation: a nonparametric perspective., Journal of the American Statistical Association 105: 1178–1187.
  • Bretó, C., Daihi, H., Ionides, E. L. and King, A. A. (2009). Time series analysis via mechanistic models., Annals of Applied Statistics 3: 319–348.
  • Cox, D. R. and Kartsonaki, C. (2012). The fitting of complex parametric models., Biometrika 99: 741–747.
  • Cox, D. R. and Reid, N. (2004). A note on pseudolikelihood constructed from marginal densities., Biometrika 91: 729–737.
  • Cox, D. R. and Smith, W. L. (1954). On the superposition of renewal processes., Biometrika 41: 91–9.
  • Cule, M. L., Samworth, R. J. and Stewart, M. I. (2010). Maximum likelihood estimation of a multi-dimensional log-concave density., Journal Royal Statistical Society B 72: 545–600.
  • Dean, T. A., Singh, S. S., Jasra A. and Peters G. W. (2011). Parameter estimation for hidden Markov models with intractable likelihoods. ArXiv preprint, arXiv:1103.5399v1.
  • de Valpine, P. (2004). Monte Carlo state space likelihoods by weighted posterior kernel density estimation., Journal of the American Statistical Association 99: 523–536.
  • Didelot, X., Everitt, R. G., Johansen, A. M. and Lawson, D. J. (2011). Likelihood-free estimation of model evidence., Bayesian Analysis 6: 49–76.
  • Diggle, P. J. and Gratton, R. J. (1984) Monte Carlo Methods of Inference for Implicit Statistical Models., Journal of the Royal Statistical Society B 46:193–227.
  • Duong, T. and Hazleton, M. L. (2005). Cross-validation bandwidth matrices for multivariate kernel density estimation., Scandinavian Journal of Statistics 32:485–506.
  • Duong, T. (2011)., ks: Kernel smoothing. R package version 1.8.5. http://CRAN.R-project.org/package=ks
  • Ehrlich, E., Jasra, A. and Kantas, N. (2012). Static parameter estimation for ABC approximations of hidden Markov models. ArXiv preprint, arXiv:1210.4683.
  • Fan, Y., Nott, D. J. and Sisson, S. A. (2012). Approximate Bayesian Computation via Regression Density Estimation., Stat 2: 34–48.
  • Fearnhead, P. and Prangle, D. (2012). Constructing Summary Statistics for Approximate Bayesian Computation: Semi-automatic ABC (with discussion)., Journal of the Royal Statistical Society B 74: 419–474.
  • Fukunaga, K. and Hostetler, L. D. (1975). The Estimation of the Gradient of a Density Function, with Applications in Pattern Recognition., IEEE Transactions on Information Theory 21: 32–40.
  • Gaetan, C. and Yao, J. F. (2003). A multiple-imputation Metropolis version of the EM algorithm., Biometrika 90: 643–654.
  • Gouriéroux, C., Monfort, A. and Renault, E. (1993). Indirect Inference., Journal of Applied Econometrics 8:S85–S118.
  • Ionides, E. L. (2005). Maximum Smoothed Likelihood Estimation., Statistica Sinica 15: 1003–1014.
  • Jaki, T. and West, R. W. (2008). Maximum Kernel Likelihood Estimation., Journal of Computational and Graphical Statistics 17: 976.
  • Jasra, A., Kantas, N. and Ehrlich, E. (2013). Approximate Inference for Observation Driven Time Series Models with Intractable Likelihoods. ArXiv Preprint, arXiv:1303.7318.
  • Jing, J., Koch, I. and Naito, K. (2012). Polynomial Histograms for Multivariate Density and Mode Estimation., Scandinavian Journal of Statistics 39: 75–96.
  • Johansen, A. M., Doucet, A., and Davy, M. (2008). Particle methods for maximum likelihood parameter estimation in latent variable models., Statistics and Computing 18: 47–57.
  • Konakov, V. D. (1973). On asymptotic normality of the sample mode of multivariate distributions., Theory of Probability and its Applications 18: 836–842.
  • Lehmann, E. and Casella, G. (1998)., Theory of Point Estimation (revised edition). Springer-Verlag, New York.
  • Lele, S. R., Dennis, B. and Lutscher, F. (2007). Data cloning: easy maximum likelihood estimation for complex ecological models using Bayesian Markov chain Monte Carlo methods., Ecology Letters 10: 551–563.
  • Marin, J.-M., Pudlo, P., Robert, C. P. and Ryder, R. (2011). Approximate Bayesian Computational methods., Statistics and Computing 21: 289–291.
  • Marin, J.-M., Pillai, N., Robert, C.P. and Rousseau, J. (2013). Relevant statistics for Bayesian model choice. ArXiv preprint, arXiv:1110.4700v3.
  • Marjoram, P., Molitor, J., Plagnol, V. and Tavaré, S. (2003). Markov chain Monte Carlo without likelihoods., Proceedings of the National Academy of Sciences of the United States of America: 15324–15328.
  • Mengersen, K. L., Pudlo, P. and Robert, C. P. (2013). Bayesian computation via empirical likelihood., Proceedings of the National Academy of Sciences of the United States of America 110: 1321–1326.
  • Nolan, J. P. (2001). Maximum likelihood estimation and diagnostics for stable distributions. In: O.E. Barndorff-Nielsen, T. Mikosh, and S. Resnick, Eds., Lévy Processes, Birkhauser, Boston, 379–400.
  • Parzen, E. (1962). On estimation of a probability density function and mode., Annals of Mathematical Statistics 33: 1065–1076.
  • Peters, G. W., Sisson, S. A. and Fan, Y. (2010). Likelihood-free Bayesian inference for $\alpha-$stable models., Computational Statistics & Data Analysis 56: 3743–3756.
  • Pritchard, J. K., Seielstad, M. T., Perez-Lezaun, A., and Feldman, M. T. (1999). Population Growth of Human Y Chromosomes: A Study of Y Chromosome Microsatellites., Molecular Biology and Evolution 16: 1791–1798.
  • Robert, C. P., Cornuet, J., Marin, J. and Pillai, N. S. (2011). Lack of confidence in ABC model choice., Proceedings of the National Academy of Sciences of the United States of America 108: 15112–15117.
  • Romano, J. P. (1988). On weak convergence and optimality of kernel density estimates of the mode., The Annals of Statistics 16: 629–647.
  • Rubio, F. J. and Johansen, A. M. (March, 2012). On Maximum Intractable Likelihood Estimation. University of Warwick, Dept. of Statistics. CRiSM working paper, 12–04.
  • Rudin, W. (1976)., Principles of Mathematical Analysis. New York: McGraw-Hill.
  • Sisson, S. A., Fan, Y. and Tanaka, M. M. (2007). Sequential Monte Carlo without likelihoods., Proceedings of the National Academy of Sciences of the United States of America, 104: 1760–1765.
  • Sköld, M. and Roberts, G. O. (2003). Density estimation for the Metropolis–Hastings algorithm., Scandinavian Journal of Statistics 30: 699–718.
  • Toni, T., Welch, D., Strelkowa, N., Ipsen, A. and Stumpf, M. P. H. (2009). Approximate Bayesian computation scheme for parameter inference and model selection in dynamical systems., Journal of the Royal Society Interface 6: 187–202.
  • Whitney, K. N. (1991). Uniform Convergence in probability and stochastic equicontinuity., Econometrica 59: 1161–1167.
  • Wilkinson, R. D. (2008). Approximate Bayesian computation (ABC) gives exact results under the assumption of model error. ArXiv preprint, arXiv:0811.3355.
  • Wuertz, D. and R core team members (2010)., fBasics: Rmetrics - Markets and Basic Statistics. R package version 2110.79. http://CRAN.R-project.org/package=fBasics