The Annals of Applied Statistics
- Ann. Appl. Stat.
- Volume 8, Number 4 (2014), 2404-2434.
Detecting duplicates in a homicide registry using a Bayesian partitioning approach
Finding duplicates in homicide registries is an important step in keeping an accurate account of lethal violence. This task is not trivial when unique identifiers of the individuals are not available, and it is especially challenging when records are subject to errors and missing values. Traditional approaches to duplicate detection output independent decisions on the coreference status of each pair of records, which often leads to nontransitive decisions that have to be reconciled in some ad-hoc fashion. The task of finding duplicate records in a data file can be alternatively posed as partitioning the data file into groups of coreferent records. We present an approach that targets this partition of the file as the parameter of interest, thereby ensuring transitive decisions. Our Bayesian implementation allows us to incorporate prior information on the reliability of the fields in the data file, which is especially useful when no training data are available, and it also provides a proper account of the uncertainty in the duplicate detection decisions. We present a study to detect killings that were reported multiple times to the United Nations Truth Commission for El Salvador.
Ann. Appl. Stat., Volume 8, Number 4 (2014), 2404-2434.
First available in Project Euclid: 19 December 2014
Permanent link to this document
Digital Object Identifier
Mathematical Reviews number (MathSciNet)
Zentralblatt MATH identifier
Deduplication duplicate detection distribution on partitions United Nations Truth Commission for El Salvador entity resolution homicide records Hispanic names human rights record linkage string similarity
Sadinle, Mauricio. Detecting duplicates in a homicide registry using a Bayesian partitioning approach. Ann. Appl. Stat. 8 (2014), no. 4, 2404--2434. doi:10.1214/14-AOAS779. https://projecteuclid.org/euclid.aoas/1419001749
- Supplementary material: Supplement to “Detecting duplicates in a homicide registry using a Bayesian partitioning approach”. We provide a Gibbs sampler for the model presented in Section 3, a brief discussion on point estimation of the coreference partition, we explain how we standardized and compared Hispanic names and, finally, we present details on the implementation of the Gibbs sampler for the application in Section 4.