March 2024 Quantifying replicability of multiple studies in a meta-analysis
Mengli Xiao, Haitao Chu, James S. Hodges, Lifeng Lin
Author Affiliations +
Ann. Appl. Stat. 18(1): 664-682 (March 2024). DOI: 10.1214/23-AOAS1806


For valid scientific discoveries, it is fundamental to evaluate whether research findings are replicable across different settings. While large-scale replication projects across broad research topics are not feasible, systematic reviews and meta-analyses (SRMAs) offer viable alternatives to assess replicability. Due to subjective inclusion and exclusion of studies, SRMAs may contain nonreplicable study findings. However, there is no consensus on rigorous methods to assess the replicability of SRMAs or to explore sources of nonreplicability. Nonreplicability is often misconceived as high heterogeneity. This article introduces a new measure, the externally standardized residuals from a leave-m-studies-out procedure, to quantify replicability. It not only measures the impact of nonreplicability from unknown sources on the conclusion of an SRMA but also differentiates nonreplicability from heterogeneity. A new test statistic for replicability is derived. We explore its asymptotic properties and use extensive simulations and real data to illustrate this measure’s performance. We conclude that replicability should be routinely assessed for all SRMAs and recommend sensitivity analyses, once nonreplicable study results are identified in an SRMA.

Funding Statement

This research was funded in part by the U.S. National Institutes of Health’s National Center for Advancing Translational Sciences grant UL1TR002494 and National Library of Medicine grants R21LM012744 and R01LM012982.


The authors would like to thank the anonymous referees, an Associate Editor, and the Editor for their constructive comments that improved the quality of this paper.


Download Citation

Mengli Xiao. Haitao Chu. James S. Hodges. Lifeng Lin. "Quantifying replicability of multiple studies in a meta-analysis." Ann. Appl. Stat. 18 (1) 664 - 682, March 2024.


Received: 1 January 2022; Revised: 1 May 2023; Published: March 2024
First available in Project Euclid: 31 January 2024

Digital Object Identifier: 10.1214/23-AOAS1806

Keywords: Externally standardized residual , Heterogeneity , Meta-analysis , replicability , statistical power , systematic review

Rights: Copyright © 2024 Institute of Mathematical Statistics


This article is only available to subscribers.
It is not available for individual sale.

Vol.18 • No. 1 • March 2024
Back to Top