August 2023 Can We Reliably Detect Biases that Matter in Observational Studies?
Paul R. Rosenbaum
Author Affiliations +
Statist. Sci. 38(3): 440-457 (August 2023). DOI: 10.1214/23-STS882

Abstract

In an observational study of the effects caused by a treatment, biases from unmeasured covariates remain a concern even after successful adjustments for measured covariates. This concern is partly addressed by demonstrating that the qualitative conclusions of the primary analysis would not be altered by small or moderate biases—that these conclusions are insensitive to small or moderate bias. Additionally, the concern is partly addressed by collecting additional information, such as outcomes known to be unaffected by the treatment, and using this information as a test of various biases. Is there a gap between these two activities? Perhaps the study is insensitive to small biases, and we can detect large biases, but the study is sensitive to moderate biases that cannot be detected—that is an informal description of a gap. The concept of “no gap” is defined formally in Definition 3.1, and the probability of “no gap” is determined under various sampling situations. When there is no gap, ask: Are causal conclusions measurably strengthened? If so, by how much? The answer depends upon the covering design sensitivity, Γ, defined to be the smallest bias that can explain both the ostensible effect of the treatment on the primary outcome and the evidence of bias provided by the unaffected outcome. The covering design sensitivity is calculated in various contexts. A small observational study of the effects of light alcohol consumption on HDL cholesterol is used to illustrate ideas and methods.

Citation

Download Citation

Paul R. Rosenbaum. "Can We Reliably Detect Biases that Matter in Observational Studies?." Statist. Sci. 38 (3) 440 - 457, August 2023. https://doi.org/10.1214/23-STS882

Information

Published: August 2023
First available in Project Euclid: 20 August 2023

MathSciNet: MR4630377
Digital Object Identifier: 10.1214/23-STS882

Keywords: Causal inference , detecting bias , observational study , sensitivity analysis

Rights: Copyright © 2023 Institute of Mathematical Statistics

Vol.38 • No. 3 • August 2023
Back to Top