Notes on the bias-variance trade-off phenomenon

Jeesen Chen

Source: Anirban DasGupta, ed., A Festschrift for Herman Rubin (Beachwood, Ohio, USA: Institute of Mathematical Statistics, 2004), 207-217.

Abstract

The main inequality (Theorem 1) here involves the Hellinger distance of a statistical model of an observation $X$, which imposes bounds on the mean of any estimator in terms of its variance. We use this inequality to explain some of the bias-variance trade-off phenomena studied in Doss and Sethuraman (1989) and Liu and Brown (1993). We provide some quantified results about how the reduction of bias would increase the variance of an estimator.