The Annals of Statistics

Estimating variance of random effects to solve multiple problems simultaneously

Masayo Yoshimori Hirose and Partha Lahiri

Full-text: Access denied (no subscription detected)

We're sorry, but we are unable to provide you with the full text of this article because we are not able to identify you as a subscriber. If you have a personal subscription to this journal, then please login. If you are already logged in, then you may need to update your profile to register your subscription. Read more about accessing full-text


The two-level normal hierarchical model (NHM) has played a critical role in statistical theory for the last several decades. In this paper, we propose random effects variance estimator that simultaneously (i) improves on the estimation of the related shrinkage factors, (ii) protects empirical best linear unbiased predictors (EBLUP) [same as empirical Bayes (EB)] of the random effects from the common overshrinkage problem, (iii) avoids complex bias correction in generating strictly positive second-order unbiased mean square error (MSE) (same as integrated Bayes risk) estimator either by the Taylor series or single parametric bootstrap method. The idea of achieving multiple desirable properties in an EBLUP or EB method through a suitably devised random effects variance estimator is the first of its kind and holds promise in providing good inferences for random effects under the EBLUP or EB framework. The proposed methodology is also evaluated using a Monte Carlo simulation study and real data analysis.

Article information

Ann. Statist., Volume 46, Number 4 (2018), 1721-1741.

Received: November 2016
Revised: May 2017
First available in Project Euclid: 27 June 2018

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62C12: Empirical decision procedures; empirical Bayes procedures
Secondary: 62F40: Bootstrap, jackknife and other resampling methods

Adjusted maximum likelihood method empirical Bayes empirical best linear unbiased prediction linear mixed model second-order unbiasedness


Yoshimori Hirose, Masayo; Lahiri, Partha. Estimating variance of random effects to solve multiple problems simultaneously. Ann. Statist. 46 (2018), no. 4, 1721--1741. doi:10.1214/17-AOS1600.

Export citation


  • Arora, V. and Lahiri, P. (1997). On the superiority of the Bayesian method over the BLUP in small area estimation problems. Statist. Sinica 7 1053–1063.
  • Arora, V., Lahiri, P. and Mukherjee, K. (1997). Empirical Bayes estimation of finite population means from complex surveys. J. Amer. Statist. Assoc. 92 1555–1562.
  • Bell, W. R. (1999). Accounting for Uncertainty About Variances in Small Area Estimation. Bulletin of the International Statistical Institute 52. Session, Helsinki.
  • Bell, W. R. (2008). Examining sensitivity of small area inferences to uncertainty about sampling error variances. In Proceedings of the American Statistical Association, Survey Research Methods Section 327–334. CD-ROM, Amer. Statist. Assoc., Alexandria, VA.
  • Bell, W. R., Basel, W. W. and Maples, J. J. (2015). An Overview of the U.S. Census Bureau’s Small Area Income and Poverty Estimates (SAIPE) program. In Analysis of Poverty Data by Small Area Methods (M. Pratesi, ed.). Wiley, New York.
  • Butar, F. B. and Lahiri, P. (2003). On measures of uncertainty of empirical Bayes small-area estimators. J. Statist. Plann. Inference 112 63–76. Model selection, model diagnostics, empirical Bayes and hierarchical Bayes, Special Issue II (Lincoln, NE, 1999).
  • Butar, B. F. (1997). Empirical Bayes methods in survey sampling. Ph.D. thesis, Department of Mathematics and Statistics, Univ. Nebraska-Lincoln.
  • Carter, G. M. and Rolph, J. F. (1974). Empirical Bayes methods applied to estimating fire alarm probabilities. J. Amer. Statist. Assoc. 69 880–885.
  • Casas-Cordero, C., Encina, J. and Lahiri, P. (2015). Poverty mapping for the Chilean comunas. In Analysis of Poverty Data by Small Area Estimation (M. Pratesi, ed.). Wiley, New York.
  • Chatterjee, S. and Lahiri, P. (2007). A simple computational method for estimating mean squared prediction error in general small-area model. In Proceedings of the Section on Survey Research Methods 3486–3493. American Statistical Association, Alexandria, VA.
  • Citro, C. and Kalton, G., eds. (2000) “Small-Area Income and Poverty Estimates: Priorities for 2000 and Beyond,” Panel on Estimates of Poverty for Small Geogruphic Area. National Academy Press, Washington, DC.
  • Das, K., Jiang, J. and Rao, J. N. K. (2004). Mean squared error of empirical predictor. Ann. Statist. 32 818–840.
  • Datta, G. S. and Lahiri, P. (2000). A unified measure of uncertainty of estimated best linear unbiased predictors in small area estimation problems. Statist. Sinica 10 613–627.
  • Efron, B. (1979). Bootstrap methods: Another look at the jackknife. Ann. Statist. 7 1–26.
  • Efron, B. and Morris, C. (1973). Stein’s estimation rule and its competitors—an empirical Bayes approach. J. Amer. Statist. Assoc. 68 117–130.
  • Efron, B. and Morris, C. (1975). Data analysis using Stein’s estimator and its generalizations. J. Amer. Statist. Assoc. 70 311–319.
  • Fay, R. E. III and Herriot, R. A. (1979). Estimates of income for small places: An application of James-Stein procedures to census data. J. Amer. Statist. Assoc. 74 269–277.
  • Ha, N. S., Lahiri, P. and Parsons, V. (2014). Methods and results for small area estimation using smoking data from the 2008 National Health Interview Survey. Stat. Med. 33 3932–3945.
  • Hall, P. and Maiti, T. (2006). On parametric bootstrap methods for small area prediction. J. R. Stat. Soc. Ser. B Stat. Methodol. 68 221–238.
  • James, W. and Stein, C. (1961). Estimation with quadratic loss. In Proc. 4th Berkeley Sympos. Math. Statist. and Prob., Vol. I 361–379. Univ. California Press, Berkeley, CA.
  • Jiang, J. (2007). Linear and Generalized Linear Mixed Models and Their Applications. Springer, New York.
  • Jiang, J., Lahiri, P. and Nguyen, T. (2016). A unified Monte-Carlo jackknife for small area estimation after model selection. Preprint. Available at arXiv:1602.05238.
  • Laird, N. M. and Louis, T. A. (1987). Empirical Bayes confidence intervals based on bootstrap samples. J. Amer. Statist. Assoc. 82 739–757.
  • Li, H. and Lahiri, P. (2010). An adjusted maximum likelihood method for solving small area estimation problems. J. Multivariate Anal. 101 882–892.
  • Liu, B., Lahiri, P. and Kalton, G. (2014). Hierarchical Bayes modeling of survey-weighted small area proportions. Surv. Methodol. 40 1–13.
  • Mohadjer, L., Rao, J. N. K., Liu, B., Krenzke, T. and Van de Kerckhove, W. (2012). Hierarchical Bayes small area estimates of adult literacy using unmatched sampling and linking models. J. Indian Soc. Agricultural Statist. 66 55–63, 232–233.
  • Morris, C. N. (1983). Parametric empirical Bayes inference: Theory and applications. J. Amer. Statist. Assoc. 78 47–65.
  • Morris, C. and Tang, R. (2011). Estimating random effects via adjustment for density maximization. Statist. Sci. 26 271–287.
  • Otto, M. C. and Bell, W. R. (1995). Sampling error modeling of poverty and income statistics for states. In Proceedings of the Section on Government Statistics 160–165. American Statistical Association, Alexandria, VA.
  • Pfeffermann, D. and Correa, S. (2012). Empirical bootstrap bias correction and estimation of prediction mean square error in small area estimation. Biometrika 99 457–472.
  • Pfeffermann, D. and Glickman, H. (2004). Mean square error approximation in small area estimation by use of parametric and nonparametric bootstrap. In Proceedings of the Section on Survey Research Methods 4167–4178. American Statistical Association, Alexandria, VA.
  • Prasad, N. G. N. and Rao, J. N. K. (1990). The estimation of the mean squared error of small-area estimators. J. Amer. Statist. Assoc. 85 163–171.
  • Rao, J. N. K. and Molina, I. (2015). Small Area Estimation, 2nd ed. Wiley, Hoboken, NJ.
  • Yoshimori, M. and Lahiri, P. (2014). A new adjusted maximum likelihood method for the Fay–Herriot small area model. J. Multivariate Anal. 124 281–294.
  • You, Y. and Chapman, B. (2006). Small area estimation using area level models and estimated sampling variances. Surv. Methodol. 32 97–103.