Open Access
2024 Methods for quantifying dataset similarity: a review, taxonomy and comparison
Marieke Stolte, Franziska Kappenberg, Jörg Rahnenführer, Andrea Bommert
Author Affiliations +
Statist. Surv. 18: 163-298 (2024). DOI: 10.1214/24-SS149

Abstract

Quantifying the similarity between datasets has widespread applications in statistics and machine learning. The performance of a predictive model on novel datasets, referred to as generalizability, depends on how similar the training and evaluation datasets are. Exploiting or transferring insights between similar datasets is a key aspect of meta-learning and transfer-learning. In simulation studies, the similarity between distributions of simulated datasets and real datasets, for which the performance of methods is assessed, is crucial. In two- or k-sample testing, it is checked, whether the underlying distributions of two or more datasets coincide.

Extremely many approaches for quantifying dataset similarity have been proposed in the literature. We examine more than 100 methods and provide a taxonomy, classifying them into ten classes. In an extensive review of these methods the main underlying ideas, formal definitions, and important properties are introduced.

We compare the 118 methods in terms of their applicability, interpretability, and theoretical properties, in order to provide recommendations for selecting an appropriate dataset similarity measure based on the specific goal of the dataset comparison and on the properties of the datasets at hand. An online tool facilitates the choice of the appropriate dataset similarity measure.

Funding Statement

This work has been supported (in part) by the Research Training Group “Biostatistical Methods for High-Dimensional Data in Toxicology” (RTG 2624, Project P1) funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation – Project Number 427806116).

Acknowledgments

The authors would like to thank the anonymous referees and the Editor for their constructive comments that improved the quality of this paper.

Citation

Download Citation

Marieke Stolte. Franziska Kappenberg. Jörg Rahnenführer. Andrea Bommert. "Methods for quantifying dataset similarity: a review, taxonomy and comparison." Statist. Surv. 18 163 - 298, 2024. https://doi.org/10.1214/24-SS149

Information

Received: 1 February 2024; Published: 2024
First available in Project Euclid: 11 December 2024

arXiv: 2312.04078
Digital Object Identifier: 10.1214/24-SS149

Subjects:
Primary: 05C90 , 62E99 , 62G10 , 62H15 , 62H30

Keywords: Binary classification , Dataset similarity , Distance measure , divergence , Energy statistic , graph-based distance , inter-point distance , kernel mean embedding , maximum mean discrepancy , meta-learning , Optimal transport , Permutation test , probability metric , similarity measure , theoretical properties , two-sample test

Vol.18 • 2024
Back to Top