Open Access
2018 On principal components regression, random projections, and column subsampling
Martin Slawski
Electron. J. Statist. 12(2): 3673-3712 (2018). DOI: 10.1214/18-EJS1486

Abstract

Principal Components Regression (PCR) is a traditional tool for dimension reduction in linear regression that has been both criticized and defended. One concern about PCR is that obtaining the leading principal components tends to be computationally demanding for large data sets. While random projections do not possess the optimality properties of the leading principal subspace, they are computationally appealing and hence have become increasingly popular in recent years. In this paper, we present an analysis showing that for random projections satisfying a Johnson-Lindenstrauss embedding property, the prediction error in subsequent regression is close to that of PCR, at the expense of requiring a slightly large number of random projections than principal components. Column sub-sampling constitutes an even cheaper way of randomized dimension reduction outside the class of Johnson-Lindenstrauss transforms. We provide numerical results based on synthetic and real data as well as basic theory revealing differences and commonalities in terms of statistical performance.

Citation

Download Citation

Martin Slawski. "On principal components regression, random projections, and column subsampling." Electron. J. Statist. 12 (2) 3673 - 3712, 2018. https://doi.org/10.1214/18-EJS1486

Information

Received: 1 October 2017; Published: 2018
First available in Project Euclid: 31 October 2018

zbMATH: 06970015
MathSciNet: MR3870509
Digital Object Identifier: 10.1214/18-EJS1486

Vol.12 • No. 2 • 2018
Back to Top