Electronic Journal of Statistics

Comparison of methods for fixed effect meta-regression of standardized differences of means

Michael J. Malloy, Luke A. Prendergast, and Robert G. Staudte

Full-text: Open access


Given a number of different studies estimating the same effect size, it is often desired to explain heterogeneity of outcomes using concomitant covariates. For very large sample sizes, effect size estimates are approximately normally distributed and a straightforward application of weighted least squares is appropriate. However in practice within study sample variances are often small to moderate, casting doubt on the normality assumption for effect sizes and results based on weighted least squares. One can alternatively variance stabilize the effect size estimates and adopt a generalized linear model. Both methods are compared on two examples when effect sizes are the standardized difference of means. Then simulation studies are conducted to compare the coverage and width of confidence intervals for the meta-regression coefficients. These simulations show that the coverage probability associated with weighted least squares can be well below the nominated level for small to moderate sample sizes. Further empirical investigations reveal a bias in estimation due to using estimated weights which were assumed to be known. For these models, the generalized linear model approach resulted in much improved coverage probabilities.

Article information

Electron. J. Statist., Volume 5 (2011), 83-101.

First available in Project Euclid: 25 February 2011

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62J05: Linear regression
Secondary: 62J12: Generalized linear models

Weighted least squares generalized linear model


Malloy, Michael J.; Prendergast, Luke A.; Staudte, Robert G. Comparison of methods for fixed effect meta-regression of standardized differences of means. Electron. J. Statist. 5 (2011), 83--101. doi:10.1214/11-EJS598. https://projecteuclid.org/euclid.ejs/1298644285

Export citation


  • [1] Azorin, P.F. 1953. Sobre la distribución t no central i, ii., Trabajos de Estadistica, 4, 173–198.
  • [2] Baker, W.L., Michael White, C., Cappelleri, J.C., Kluger, J., & Coleman, C.I. 2009. Understanding heterogeneity in meta-analysis: the role of meta-regression., The International Journal of Clinical Practice, 63(10), 1426–1434.
  • [3] Belanger, H.G., & Vanderploeg, R.D. 2005. The neuropsychological impact of sports-related concussion: A meta-analysis., Journal of the International Neuropsychological Society, 11, 345–357.
  • [4] Gay, D.M. 1990. Usage summary for selected optimization routines., Computing Science Technical Report, 153.
  • [5] Hardin, J.W., & Hilbe, J.M. 2007., Generalized linear models and extensions. Second edn. Stata Press.
  • [6] Hedges, L.V., & Olkin, I. 1985., Statistical methods for meta-analysis. Academic Press Inc. Ltd.
  • [7] Hedges, L.V., Tipton, E., & Johnson, M.C. 2010. Robust variance estimation in meta-regression with dependent effect size estimates., Research Synthesis Methods, 1(1), 39–65.
  • [8] Hemming, K., Hutton, J.L., Maguire, M.G., & Marson, A.G. 2010. Meta-regression with partial information on summary trial or patient characteristics., Statistics in Medicine, 29(12), 1312–1324.
  • [9] Jackson, D. 2008. The significance level of meta-regression’s standard hypothesis test., Communications in Statistics - Theory and Methods, 37, 1576–1590.
  • [10] Knapp, G., & Hartung, J. 2003. Improved tests for a random effects meta-regression with a single covariate., Statistics in Medicine, 22(17), 2693–2710.
  • [11] Kulinskaya, E., Morgenthaler, S., & Staudte, R.G. 2008., Meta analysis: A guide to calibrating and combining statistical evidence. Wiley Series in Probability and Statistics. Chichester: John Wiley & Sons Ltd.
  • [12] McCullagh, P., & Nelder, J.A. 1989., Generalized linear models. Second edn. Monographs on Statistics and Applied Probability. London: Chapman & Hall.
  • [13] Montgomery, D.C., Peck, E.A., & Vining, G.G. 2006., Introduction to linear regression analysis. Fourth edn. Wiley-Interscience.
  • [14] Roberts, C.J. 2005. Issues in meta-regression analysis: An overview., Journal of Economic Surveys, 19(3), 295–298.
  • [15] Stanley, T.D., Doucouliagos, C., & Jarrell, S.B. 2008. Meta-regression analysis as the socio-economics of economics research., Journal of Socio-Economics, 37(1), 276–292.
  • [16] Team, R Development Core. 2008., R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. ISBN 3-900051-07-0.
  • [17] Thompson, S.G., & Higgins, J.P.T. 2002. How should meta-regression analyses be undertaken and interpreted?, Statistics in Medicine, 21, 1559–1573.
  • [18] Viechtbauer, W. 2007. Approximate confidence intervals for standardized effect sizes in the two-independent and two-dependent samples design., Journal of Educational and Behavioral Statistics, 32(1), 39–60.