Electronic Journal of Statistics

Linear scoring rules for probabilistic binary classification

Matthew Parry

Full-text: Open access

Abstract

Probabilistic binary classification typically calls for a vector of marginal probabilities where each element gives the probability of assigning the corresponding case to class 1. Scoring rules are principled ways to assess probabilistic forecasts about any outcome that is subsequently observed. We develop a class of proper scoring rules called linear scoring rules that are specifically adapted to probabilistic binary classification. When applied in competition situations, we show that all linear scoring rules essentially balance the needs of organizers and competitors. Linear scoring rules can also be used to train classifiers. Finally, since scoring rules have a statistical decision theoretic foundation, a linear scoring rule can be constructed for any user-defined misclassification loss function.

Article information

Source
Electron. J. Statist., Volume 10, Number 1 (2016), 1596-1607.

Dates
Received: November 2015
First available in Project Euclid: 3 June 2016

Permanent link to this document
https://projecteuclid.org/euclid.ejs/1464966339

Digital Object Identifier
doi:10.1214/16-EJS1150

Mathematical Reviews number (MathSciNet)
MR3509884

Zentralblatt MATH identifier
06600849

Subjects
Primary: 62C99: None of the above, but in this section

Keywords
Scoring rules binary classification probabilistic forecast

Citation

Parry, Matthew. Linear scoring rules for probabilistic binary classification. Electron. J. Statist. 10 (2016), no. 1, 1596--1607. doi:10.1214/16-EJS1150. https://projecteuclid.org/euclid.ejs/1464966339


Export citation

References

  • Banerjee, A., Guo, X. and Wang, H. (2005). On the Optimality of Conditional Expectation as a Bregman Predictor., IEEE Transactions on Information Theory 51 2664–2669.
  • Brier, G. W. (1950). Verification of weather forecasts expressed in terms of probability., Monthly Weather Review 78 1–3.
  • Buja, A., Stuetzle, W. and Shen, Y. (2005). Loss functions for binary class probability estimation and classification: structure and applications. Technical report available at, http://www-stat.wharton.upenn.edu/~buja/.
  • Byrne, S. (2016). A note on the use of empirical AUC for evaluating probabilistic forecasts., Electron. J. Statist. 10 380–393.
  • Dawid, A. P. and Lauritzen, S. L. (2005). The Geometry of Decision Theory. In, Proceedings of the Second International Symposium on Information Geometry and its Applications 22–28. University of Tokyo.
  • Frongillo, R. and Kash, I. (2015). Vector-Valued Property Elicitation., JMLR: Workshop and Conference Proceedings 40 1–18.
  • Gneiting, T. and Raftery, A. E. (2007). Strictly proper scoring rules, prediction, and estimation., Journal of the American Statistical Association 102 359–378.
  • Grünwald, P. D. and Dawid, A. P. (2004). Game theory, maximum entropy, minimum discrepancy, and robust Bayesian decision theory., Annals of Statistics 32 1367–1433.
  • Hand, D. J. (2009). Measuring classifier performance: a coherent alternative to the area under the ROC curve., Machine Learning 77 103–123.
  • Hendrickson, A. D. and Buehler, R. J. (1971). Proper scores for probability forecasters., Ann. Math. Statist. 42 1916–1921.
  • McCarthy, J. (1956). Measures of the value of information., Proc. Nat. Acad. Sci. 42 654–655.
  • Parry, M., Dawid, A. P. and Lauritzen, S. (2012). Proper local scoring rules., Annals of Statistics 40 561–592.