Electronic Journal of Statistics
- Electron. J. Statist.
- Volume 10, Number 1 (2016), 380-393.
A note on the use of empirical AUC for evaluating probabilistic forecasts
Scoring functions are used to evaluate and compare partially probabilistic forecasts. We investigate the use of rank-sum functions such as empirical Area Under the Curve (AUC), a widely used measure of classification performance, as a scoring function for the prediction of probabilities of a set of binary outcomes. It is shown that the AUC is not generally a proper scoring function, that is, under certain circumstances it is possible to improve on the expected AUC by modifying the quoted probabilities from their true values. However with some restrictions, or with certain modifications, it can be made proper.
Electron. J. Statist., Volume 10, Number 1 (2016), 380-393.
Received: August 2015
First available in Project Euclid: 17 February 2016
Permanent link to this document
Digital Object Identifier
Mathematical Reviews number (MathSciNet)
Zentralblatt MATH identifier
Primary: 62C99: None of the above, but in this section
Byrne, Simon. A note on the use of empirical AUC for evaluating probabilistic forecasts. Electron. J. Statist. 10 (2016), no. 1, 380--393. doi:10.1214/16-EJS1109. https://projecteuclid.org/euclid.ejs/1455715967
- Supplement to “A note on the use of empirical AUC for evaluating probabilistic forecasts”. Supplementary material contains the calculations for Example 4.