Open Access
November 1998 Instability of least squares, least absolute deviation and least median of squares linear regression, with a comment by Stephen Portnoy and Ivan Mizera and a rejoinder by the author
Steven P. Ellis
Statist. Sci. 13(4): 337-350 (November 1998). DOI: 10.1214/ss/1028905829

Abstract

Say that a regression method is "unstable" at a data set if a small change in the data can cause a relatively large change in the fitted plane. A well-known example of this is the instability of least squares regression (LS) near (multi)collinear data sets. It is known that least absolute deviation (LAD) and least median of squares (LMS) linear regression can exhibit instability at data sets that are far from collinear. Clear-cut instability occurs at a "singularity"--a data set, arbitrarily small changes to which can substantially change the fit. For example, the collinear data sets are the singularities of LS. One way to measure the extent of instability of a regression method is to measure the size of its "singular set" (set of singularities). The dimension of the singular set is a tractable measure of its size that can be estimated without distributional assumptions or asymptotics.

By applying a general theorem on the dimension of singular sets, we find that the singular sets of LAD and LMS are at least as large as that of LS and often much larger. Thus, prima facie, LAD and LMS are frequently unstable. This casts doubt on the trustworthiness of LAD and LMS as exploratory regression tools.

Citation

Download Citation

Steven P. Ellis. "Instability of least squares, least absolute deviation and least median of squares linear regression, with a comment by Stephen Portnoy and Ivan Mizera and a rejoinder by the author." Statist. Sci. 13 (4) 337 - 350, November 1998. https://doi.org/10.1214/ss/1028905829

Information

Published: November 1998
First available in Project Euclid: 9 August 2002

MathSciNet: MR1705266
Digital Object Identifier: 10.1214/ss/1028905829

Subjects:
Primary: 62J05

Keywords: collinearity , singularity , stability

Rights: Copyright © 1998 Institute of Mathematical Statistics

Vol.13 • No. 4 • November 1998
Back to Top