Abstract
The consequence of a change in school leadership (e.g., principal turnover) on student achievement has important implications for education policy. The impact of such an event can be estimated via the popular difference in difference (DiD) estimator, where those schools with a turnover event are compared to a selected set of schools that did not have such an event. The strength of this comparison depends on the plausibility of the “parallel trends” assumption that the “treated group” of those schools which had leadership turnover, absent such turnover, would have changed “similarly” to those which did not. To bolster such a claim, one might generate a comparison group, via matching, that is similar to the treated group with respect to pretreatment outcomes and/or pretreatment covariates. Unfortunately, as has been previously pointed out, this intuitively appealing approach also has a cost in terms of bias. To assess the trade-offs of matching in our application, we first characterize the bias of matching prior to a DiD analysis under a linear structural model that allows for time-invariant observed and unobserved confounders with time-varying effects on the outcome. Given our framework, we verify that matching on baseline covariates generally reduces bias. We further show how additionally matching on pretreatment outcomes has both cost and benefit. First, matching on pretreatment outcomes partially balances unobserved confounders, which mitigates some bias. This reduction is proportional to the outcome’s reliability, a measure of how coupled the outcomes are with the latent covariates. Offsetting these gains, matching also injects bias into the final estimate by undermining the second difference in the DiD via a regression-to-the-mean effect. Consequently, we provide heuristic guidelines for determining to what degree the bias reduction of matching is likely to outweigh the bias cost. We illustrate our guidelines by reanalyzing a principal turnover study that used matching prior to a DiD analysis and find that matching on both the pretreatment outcomes and observed covariates makes the estimated treatment effect more credible.
Funding Statement
This work was supported by the U.S. Department of Education, Institute for Education Sciences, through Grant R305D200010. The opinions expressed are those of the authors and do not represent views of the Institute or the U.S. Department of Education.
Acknowledgments
We thank the authors of Bartanen, Grissom and Rogers (2019) for supplying their data and replication source code. We also thank the anonymous reviews of our preprint as well as our submission to this journal; their comments have strengthened this paper considerably, pushing us to generalize our initial results, think more about uncertainty quantification, and make connections to other areas of the literature.
Citation
Dae Woong Ham. Luke Miratrix. "Benefits and costs of matching prior to a difference in differences analysis when parallel trends does not hold." Ann. Appl. Stat. 18 (3) 2096 - 2122, September 2024. https://doi.org/10.1214/24-AOAS1872
Information