The Annals of Applied Statistics

Random effects models for identifying the most harmful medication errors in a large, voluntary reporting database

Sergio Venturini, Jessica M. Franklin, Laura Morlock, and Francesca Dominici

Full-text: Access denied (no subscription detected)

We're sorry, but we are unable to provide you with the full text of this article because we are not able to identify you as a subscriber. If you have a personal subscription to this journal, then please login. If you are already logged in, then you may need to update your profile to register your subscription. Read more about accessing full-text


Medical errors are a major source of preventable morbidity, mortality and healthcare costs. Voluntary reporting systems are useful data sources that collect detailed information on the circumstances of medical errors occurring in hospitals. Identifying the characteristics of errors that frequently result in patient harm when they occur would allow investigators to prioritize among the many sources of potential errors and design targeted prevention strategies. In this paper, we use data from MEDMARX, a large anonymous and voluntary reporting system for medication errors, to identify the combinations of error characteristics that are more likely to result in harm. To this end, we consider a Bayesian hierarchical model with crossed random effects and a flexible specification of the random effects distribution. We then provide a ranking of the errors using optimal Bayesian ranking based on their probability of harm. The use of optimal Bayesian ranking accounts for the varying amount of uncertainty across the random effects estimates. Finally, we examine the sensitivity of results to different specifications of the random effects distributions. The utility of flexible random effects assumptions is illustrated by empirically comparing results under several choices. We found that errors caused by mistakes in reconciling a patient’s current medication list with the medications prescribed at hospital discharge have an estimated 10.5% probability of harm. These errors had the highest rate of harm of errors that occur during the prescribing stage of medication use. In addition, we found that the results are sensitive to the random effects distribution used in estimation. Thus, an approach that explores this sensitivity is important for accurately comparing the relative harm across errors.

Article information

Ann. Appl. Stat., Volume 11, Number 2 (2017), 504-526.

Received: October 2015
Revised: August 2016
First available in Project Euclid: 20 July 2017

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Bayesian hierarchical model empirical Bayes data mining spontaneous reporting


Venturini, Sergio; Franklin, Jessica M.; Morlock, Laura; Dominici, Francesca. Random effects models for identifying the most harmful medication errors in a large, voluntary reporting database. Ann. Appl. Stat. 11 (2017), no. 2, 504--526. doi:10.1214/16-AOAS974.

Export citation


  • Abad, A. A., Litière, S. and Molenberghs, G. (2010). Testing for misspecification in generalized linear mixed models. Biostat. 11 771–786.
  • Ahmed, I., Bégaud, B. and Tubert-Bitter, P. (2015). Evaluation of post-marketing safety using spontaneous reporting databases. In Statistical Methods for Evaluating Safety in Medical Product Development (A. L. Gould, ed.). Wiley, New York.
  • Ahmed, I., Haramburu, F., Fourrier-Réglat, A., Thiessard, F., Kreft-Jais, C., Miremont-Salamé, G., Bégaud, B. and Tubert-Bitter, P. (2009). Bayesian pharmacovigilance signal detection methods revisited in a multiple comparison setting. Stat. Med. 28 1774–1792.
  • Ahmed, I., Dalmasso, C., Haramburu, F., Thiessard, F., Bröet, P. and Tubert-Bitter, P. (2010). False discovery rate estimation for frequentist pharmacovigilance signal detection methods. Biometrics 66 301–309.
  • Ahmed, I., Thiessard, F., Miremont-Salamé, G., Haramburu, F., Kreft-Jais, C., Bégaud, B. and Tubert-Bitter, P. (2012). Early detection of pharmacovigilance signals with automated methods based on false discovery rates. Drug Saf. 35 495–506.
  • Aspden, P., Corrigan, J. M., Wolcott, J. and Erickson, S. M. (2003). Patient Safety: Achieving a New Standard for Care. The National Academy Press, Washington, DC.
  • Bayarri, M. J. and Castellanos, M. E. (2007). Bayesian checking of the second levels of hierarchical models. Statist. Sci. 22 322–343.
  • Boockvar, K. S., Carlson LaCorte, H., Giambanco, V., Fridman, B. and Siu, A. (2006). Medication reconciliation for reducing drug-discrepancy adverse events. Am. J. Geriatr. Pharmacother. 4 236–243.
  • Brennan, T., Leape, L., Laird, N., Hebert, L., Localio, A., Lawthers, A., Newhouse, J., Weiler, P. and Hiatt, H. (1991). Incidence of adverse events and negligence in hospitalized patients. Results of the Harvard medical practice study I. N. Engl. J. Med. 324 370–376.
  • Browne, W. J. (2004). An illustration of the use of reparameterisation methods for improving MCMC efficiency in crossed random effect models. Multilevel Model. Newsl. 16 13–25.
  • Dey, D. K., Gelfand, A. E., Swartz, T. B. and Vlachos, P. K. (1998). A simulation-intensive approach for checking hierarchical models. TEST 7 325–346.
  • De Iorio, M., Müller, P., Rosner, G. L. and MacEachern, S. N. (2004). An ANOVA model for dependent random measures. J. Amer. Statist. Assoc. 99 205–215.
  • DuMouchel, W. (1999). Bayesian data mining in large frequency tables, with an application to the FDA spontaneous reporting system. Amer. Statist. 53 177–190.
  • Escobar, M. D. and West, M. (1995). Bayesian density estimation and inference using mixtures. J. Amer. Statist. Assoc. 90 577–588.
  • Eugéne, V. P., Egberts, A., Heerdink, E. R. and Leufkens, H. (2000). Detecting drug–drug interactions using a database for spontaneous adverse drug reactions: An example with diuretics and non-steroidal anti-inflammatory drugs. Eur. J. Clin. Pharmacol. 56 733–738.
  • Ferguson, T. S. (1973). A Bayesian analysis of some nonparametric problems. Ann. Statist. 1 209–230.
  • Fernández, C. and Steel, M. F. J. (1998). On Bayesian modeling of fat tails and skewness. J. Amer. Statist. Assoc. 93 359–371.
  • Flegal, J. M., Haran, M. and Jones, G. L. (2008). Markov chain Monte Carlo: Can we trust the third significant figure. Statist. Sci. 23 250–260.
  • GBD 2013 Mortality and Causes of Death Collaborators (2015). Global, regional, and national age–sex specific all-cause and cause-specific mortality for 240 causes of death, 1990–2013: A systematic analysis for the Global Burden of Disease Study 2013. Lancet 385 117–171.
  • Gelman, A. and Hill, J. (2007). Data Analysis Using Regression and Multilevel/Hierarchical Models. Cambridge Univ. Press, Cambridge.
  • Gelman, A., Meng, X.-L. and Stern, H. (1996). Posterior predictive assessment of model fitness via realized discrepancies. Statist. Sinica 6 733–807.
  • Gelman, A., Goegebeur, Y., Tuerlinckx, F. and Van Mechelen, I. (2000). Diagnostic checks for discrete data regression models using posterior predictive simulations. J. R. Stat. Soc. Ser. C. Appl. Stat. 49 247–268.
  • Gelman, A., Carlin, J. B., Stern, H. S., Dunson, D. B., Vehtari, A. and Rubin, D. B. (2014). Bayesian Data Analysis, 3rd ed. CRC Press, Boca Raton, FL.
  • Gibbons, R. D. and Amatya, A. K. (2016). Statistical Methods for Drug Safety. CRC Press, Boca Raton, FL.
  • Gibbons, R. D., Segawa, E., Karabatsos, G., Amatya, A. K., Bhaumik, D. K., Brown, C. H., Kapur, K., Marcus, S. M., Hur, K. and Mann, J. J. (2008). Mixed-effects Poisson regression analysis of adverse event reports: The relationship between antidepressants and suicide. Stat. Med. 27 1814–1833.
  • Haario, H., Saksman, E. and Tamminen, J. (2001). An adaptive Metropolis algorithm. Bernoulli 7 223–242.
  • Huang, X. (2011). Detecting random-effects model misspecification via coarsened data. Comput. Statist. Data Anal. 55 703–714.
  • Huckels-Baumgart, S. and Manser, T. (2014). Identifying medication error chains from critical incident reports: A new analytic approach. J. Clin. Pharmacol. 54 1188–1197.
  • Kohn, L. T., Corrigan, J. and Donaldson, M. S. (2000). To Err Is Human: Building a Safer Health System. The National Academy Press, Washington, DC.
  • The Joint Commission (2000). Reporting of Medical/Health Care Errors: A Position Statement of the Joint Commission.
  • Kwan, J. L., Lo, L., Sampson, M. and Shojania, K. G. (2013). Medication reconciliation during transitions of care as a patient safety strategy: A systematic review. Ann. Intern. Med. 158 397–403.
  • Kyung, M., Gill, J. and Casella, G. (2010). Estimation in Dirichlet random effects models. Ann. Statist. 38 979–1009.
  • Leape, L., Brennan, T., Laird, N., Lawthers, A., Localio, A., Barnes, B., Hebert, L., Newhouse, J., Weiler, P. and Hiatt, H. (1991). The nature of adverse events in hospitalized patients. Results of the Harvard medical practice study II. N. Engl. J. Med. 324 377–384.
  • Lee, K. J. and Thompson, S. G. (2008). Flexible parametric models for random-effects distributions. Stat. Med. 27 418–434.
  • Louis, T. A. and Shen, W. (1999). Innovations in Bayes and empirical Bayes methods: Estimating parameters, populations and ranks. Stat. Med. 18 2493–2505.
  • MacEachern, S. N. and Müller, P. (1998). Estimating mixture of Dirichlet process models. J. Comput. Graph. Statist. 7 223–238.
  • MacEachern, S. N. and Peruggia, M. (2000). Importance link function estimation for Markov chain Monte Carlo methods. J. Comput. Graph. Statist. 9 99–121.
  • Madigan, D., Ryan, P., Simpson, S. and Zorych, I. (2010). Bayesian methods in pharmacovigilance. In Bayesian Statistics, Vol. 9 (J. M. Bernardo, M. J. Bayarri, J. O. Berger, A. P. Dawid, D. Heckerman, A. F. M. Smith and M. West, eds.). Oxford Univ. Press, London.
  • Morlock, L., Dominici, F., Myers, J. A., Shore, A. D., Pronovost, P. J., Dy, S. M. and Cousins, D. D. (2010). Comparing near miss and harmful medication errors: Testing the causal continuum hypothesis using data from the MEDMARX National Reporting System, Technical report, Johns Hopkins Univ., Baltimore, MD.
  • Mueller, S. K., Sponsler, K. C., Kripalani, S. and Schnipper, J. L. (2012). Hospital-based medication reconciliation practices: A systematic review. Arch. Intern. Med. 172 1057–1069.
  • Müller, P., Quintana, F. A., Jara, A. and Hanson, T. (2015). Bayesian Nonparametric Data Analysis. Springer, Cham.
  • National Coordinating Council for Medication Error Reporting and Prevention (NCC MERP) (2001). Medication Error Index.
  • Pronovost, P., Weast, B., Schwarz, M., Wyskiel, R. M., Prow, D., Milanovich, S. N., Berenholtz, S., Dorman, T. and Lipsett, P. (2003). Medication reconciliation: A practical tool to reduce the risk of medication errors. J. Crit. Care 18 201–205.
  • Ramjaun, A., Sudarshan, M., Patakfalvi, L., Tamblyn, R. and Meguerditchian, A. N. (2015). Educating medical trainees on medication reconciliation: A systematic review. BMC Med. Educ. 15 33.
  • Santell, J. P., Hicks, R. W., McMeekin, J. and Cousins, D. D. (2003). Medication errors: Experience of the United States Pharmacopeia (USP) MEDMARX Reporting System. J. Clin. Pharmacol. 43 760–767.
  • Schiff, G. D., Amato, M. G., Eguale, T., Boehne, J. J., Wright, A., Koppel, R., Rashidee, A. H., Elson, R. B., Whitney, D. L., Thach, T.-T., Bates, D. W. and Seger, A. C. (2015). Computerised physician order entry-related medication errors: Analysis of reported errors and vulnerability testing of current systems. BMJ Qual. Saf. 24 264–271.
  • Shen, W. and Louis, T. A. (1998). Triple-goal estimates in two-stage hierarchical models. J. R. Stat. Soc. Ser. B. Stat. Methodol. 60 455–471.
  • Sinharay, S. and Stern, H. S. (2003). Posterior predictive model checking in hierarchical models. J. Statist. Plann. Inference 111 209–221.
  • Stern, H. S. and Cressie, N. (2000). Posterior predictive model checks for disease mapping models. Stat. Med. 19 2377–2397.
  • Tchetgen, E. J. and Coull, B. A. (2006). A diagnostic test for the mixing distribution in a generalised linear mixed model. Biometrika 93 1003–1010.
  • Venturini, S., Franklin, J. M., Morlock, L. and Dominici, F. (2017a). Supplement to “Random effects models for identifying the most harmful medication errors in a large, voluntary reporting database.” DOI:10.1214/16-AOAS974SUPPA.
  • Venturini, S., Franklin, J. M., Morlock, L. and Dominici, F. (2017b). Supplement to “Random effects models for identifying the most harmful medication errors in a large, voluntary reporting database.” DOI:10.1214/16-AOAS974SUPPB.
  • Venturini, S., Franklin, J. M., Morlock, L. and Dominici, F. (2017c). Supplement to “Random effects models for identifying the most harmful medication errors in a large, voluntary reporting database.” DOI:10.1214/16-AOAS974SUPPC.
  • Waagepetersen, R. (2006). A simulation-based goodness-of-fit test for random effects in generalized linear mixed models. Scand. J. Stat. 33 721–731.

Supplemental materials

  • Supplement A: Error Definitions and Results for the Other Nodes. This supplement contains the list of definitions for all potential error types and causes for MEDMARX reports and the results for the Bayesian hierarchical model (BHM) and empirical Bayes data mining (EBDM) approach applied to the data from each of the four other nodes of medication use: documenting, dispensing, administering and monitoring.
  • Supplement B: Empirical Bayes Data Mining Approach. Section 1 of this supplement shows how to adapt the GPS method developed by DuMouchel (1999) and briefly described in Section 4 to the MEDMARX data. Moreover, in Section 2 we provide a brief description of the importance link function estimation as described in MacEachern and Peruggia (2000).
  • Supplement C: Bayesian Hierarchical Model Estimates. This supplement reports more details about the estimation of the BHM described in Section 3.