The Annals of Applied Statistics

Exploiting multiple outcomes in Bayesian principal stratification analysis with application to the evaluation of a job training program

Alessandra Mattei, Fan Li, and Fabrizia Mealli

Full-text: Open access


The causal effect of a randomized job training program, the JOBS II study, on trainees’ depression is evaluated. Principal stratification is used to deal with noncompliance to the assigned treatment. Due to the latent nature of the principal strata, strong structural assumptions are often invoked to identify principal causal effects. Alternatively, distributional assumptions may be invoked using a model-based approach. These often lead to weakly identified models with substantial regions of flatness in the posterior distribution of the causal effects. Information on multiple outcomes is routinely collected in practice, but is rarely used to improve inference. This article develops a Bayesian approach to exploit multivariate outcomes to sharpen inferences in weakly identified principal stratification models. We show that inference for the causal effect on depression is significantly improved by using the re-employment status as a secondary outcome in the JOBS II study. Simulation studies are also performed to illustrate the potential gains in the estimation of principal causal effects from jointly modeling more than one outcome. This approach can also be used to assess plausibility of structural assumptions and sensitivity to deviations from these structural assumptions. Two model checking procedures via posterior predictive checks are also discussed.

Article information

Ann. Appl. Stat., Volume 7, Number 4 (2013), 2336-2360.

First available in Project Euclid: 23 December 2013

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Bayesian causal inference intermediate variables job training program mixture multivariate outcomes noncompliance principal stratification


Mattei, Alessandra; Li, Fan; Mealli, Fabrizia. Exploiting multiple outcomes in Bayesian principal stratification analysis with application to the evaluation of a job training program. Ann. Appl. Stat. 7 (2013), no. 4, 2336--2360. doi:10.1214/13-AOAS674.

Export citation


  • Angrist, J. D., Imbens, G. W. and Rubin, D. B. (1996). Identification of causal effects using instrumental variables. J. Amer. Statist. Assoc. 91 444–455.
  • Barnard, J., Frangakis, C. E., Hill, J. L. and Rubin, D. B. (2003). Principal stratification approach to broken randomized experiments: A case study of school choice vouchers in New York City. J. Amer. Statist. Assoc. 98 299–323.
  • Bayarri, M. J. and Berger, J. O. (2000). $p$ values for composite null models. J. Amer. Statist. Assoc. 95 1127–1142.
  • Chib, S. (1995). Marginal likelihood from the Gibbs output. J. Amer. Statist. Assoc. 90 1313–1321.
  • Chib, S. and Hamilton, B. H. (2000). Bayesian analysis of cross-section and clustered data treatment models. J. Econometrics 97 25–50.
  • Elliott, M. R., Raghunathan, T. E. and Li, Y. (2010). Bayesian inference for causal mediation effects using principal stratification with dichotomous mediators and outcomes. Biostatistics 11 353–372.
  • Frangakis, C. E. and Rubin, D. B. (2002). Principal stratification in causal inference. Biometrics 58 21–29.
  • Gallop, R., Small, D. S., Lin, J. Y., Elliott, M. R., Joffe, M. and Ten Have, T. R. (2009). Mediation analysis with principal stratification. Stat. Med. 28 1108–1130.
  • Gelman, A., Meng, X.-L. and Stern, H. (1996). Posterior predictive assessment of model fitness via realized discrepancies. Statist. Sinica 6 733–807.
  • Gelman, A. E. and Rubin, D. B. (1992). Inference from iterative simulation using multiple sequences. Statist. Sci. 7 457–472.
  • Gilbert, P. B. and Hudgens, M. G. (2008). Evaluating candidate principal surrogate endpoints. Biometrics 64 1146–1154.
  • Gosselin, F. (2011). A new calibrated Bayesian internal goodness-of-fit method: Sampled posterior $p$-values as simple and general $p$-values that allow double use of the data. PLoS ONE 6 1–10.
  • Gustafson, P. (2009). What are the limits of posterior distributions arising from nonidentified models and why should we care? J. Amer. Statist. Assoc. 104 1682–1695.
  • Hirano, K., Imbens, G. W., Rubin, D. B. and Zhou, X. H. (2000). Assessing the effect of an influenza vaccine in an encouragement design. Biostatistics 1 69–88.
  • Hjort, N. L., Dahl, F. A. and Steinbakk, G. H. (2006). Post-processing posterior predictive $p$-values. J. Amer. Statist. Assoc. 101 1157–1174.
  • Imbens, G. W. and Rubin, D. B. (1997). Bayesian inference for causal effects in randomized experiments with noncompliance. Ann. Statist. 25 305–327.
  • Jin, H. and Rubin, D. B. (2008). Principal stratification for causal inference with extended partial compliance. J. Amer. Statist. Assoc. 103 101–111.
  • Jo, B. and Muthen, B. (2001). Modeling of intervention effects with noncompliance: A latent variable approach for randomized trials. In New developments and techniques in structrual equation modeling (G. A. Marcoulides and R. E. Schumacker, eds.) 57–87. Erlbaum Associates, Mahwah, NJ.
  • Johnson, V. E. (2004). A Bayesian $\chi^{2}$ test for goodness-of-fit. Ann. Statist. 32 2361–2384.
  • Johnson, V. E. (2007). Bayesian model assessment using pivotal quantities. Bayesian Anal. 2 719–733.
  • Li, Y., Taylor, J. M. G. and Elliott, M. R. (2010). A Bayesian approach to surrogacy assessment using principal stratification in clinical trials. Biometrics 66 523–531.
  • Li, Y., Taylor, J. M. G. and Elliott, M. R. (2011). Causal assessment of surrogacy in a metanalysis of colorectal cancer trials. Biostatistics 12 478–492.
  • Little, R. J. and Yau, L. H. Y. (1998). Statistical techniques for analyzing data from prevention trials: Treatment of no-shows using Rubin’s causal model. Psychological Methods 3 147–159.
  • Manski, C. F. (1990). Nonparametric bounds on treatment effects. The American Economic Review 80 319–323.
  • Mattei, A., Li, F. and Mealli, F. (2013). Supplement to “Exploiting multiple outcomes in Bayesian principal stratification analysis with application to the evaluation of a job training program.” DOI:10.1214/13-AOAS674SUPP.
  • Mattei, A. and Mealli, F. (2007). Application of the principal stratification approach to the Faenza randomized experiment on breast self-examination. Biometrics 63 437–446.
  • Mealli, F. and Pacini, B. (2008). Comparing principal stratification and selection models in parametric causal inference with nonignorable missingness. Comput. Statist. Data Anal. 53 507–516.
  • Mealli, F. and Pacini, B. (2013). Using secondary outcomes and covariates to sharpen inference in randomized experiments with noncompliance. J. Amer. Statist. Assoc. 108 1120–1131.
  • Mercatanti, A., Li, F. and Mealli, F. (2012). Improving inference of Gaussian mixtures using auxiliary variables. Discussion Paper 12–14. Dept. Statistical Science, Duke Univ., Durham, NC.
  • Rosenbaum, P. R. (1984). The consequences of adjustment for a concomitant variable that has been affected by the treatment. J. Roy. Statist. Soc. Ser. A 147 656–666.
  • Rosenbaum, P. R. and Rubin, D. B. (1983). The central role of the propensity score in observational studies for causal effects. Biometrika 70 41–55.
  • Rubin, D. B. (1974). Estimating causal effects of treatments in randomized and nonrandomized studies. Journal of Educational Psychology 66 688–701.
  • Rubin, D. B. (1978). Bayesian inference for causal effects: The role of randomization. Ann. Statist. 6 34–58.
  • Rubin, D. B. (1980). Comment on “Randomization analysis of experimental data: The Fisher randomization test” by D. Basu. J. Amer. Statist. Assoc. 75 591–593.
  • Rubin, D. B. (1984). Bayesianly justifiable and relevant frequency calculations for the applied statistician. Ann. Statist. 12 1151–1172.
  • Schwartz, S. L., Li, F. and Mealli, F. (2011). A Bayesian semiparametric approach to intermediate variables in causal inference. J. Amer. Statist. Assoc. 106 1331–1344.
  • Schwartz, S., Li, F. and Reiter, J. P. (2012). Sensitivity analysis for unmeasured confounding in principal stratification settings with binary variables. Stat. Med. 31 949–962.
  • Sjölander, A., Humphreys, K., Vansteelandt, S., Bellocco, R. and Palmgren, J. (2009). Sensitivity analysis for principal stratum direct effects, with an application to a study of physical activity and coronary heart disease. Biometrics 65 514–520.
  • Small, D. S. and Cheng, J. (2009). Comment on “Identifiability and estimation of causal effects in randomized trials with noncompliance and completely nonignorable missing data.” Biometrics 65 682–686.
  • Sommer, A. and Zeger, S. L. (1991). On estimating efficacy from clinical trials. Stat. Med. 10 45–52.
  • Ten Have, T. R., Elliott, M. R., Joffe, M. and Zanutto, E. (2004). Causal linear models for non-compliance under randomized treatment with univariate continuous response. J. Amer. Statist. Assoc. 99 16–25.
  • Vinokur, A. D., Caplan, R. D. and Williams, C. C. (1987). Effects of recent and past stress on mental health: Coping with unemployment among Vietnam veterans and non-veterans. Journal of Applied Social Psychology 17 710–730.
  • Vinokur, A. D., Price, R. H. and Schul, Y. (1995). Impact of the JOBS intervention on unemployed workers varying in risk for depression. American Journal of Community Psychology 23 39–74.
  • Zhang, J. L. and Rubin, D. B. (2003). Estimation of causal effects via principal stratification when some outcomes are truncated by “death.” Journal of Educational and Behavioral Statistics 28 353–368.

Supplemental materials

  • Supplementary material: Supplement to “Exploiting multiple outcomes in Bayesian principal stratification analysis with application to the evaluation of a job training program.”. Supplement A: Details of calculation. We describe in detail the Markov Chain Monte Carlo (MCMC) methods used to simulate the posterior distributions of the parameters of the models introduced in Section 5 in the main text. Supplement B: Posterior inference conditional on pretreatment variables. We describe details of calculation and results under the alternative models conditioning on the pretreatment variables. Supplement C: Additional simulation results. We present additional simulations aimed at investigating the role of the partial exclusion restriction assumption and assessing the repeated sampling properties of the proposed approach.