Open Access
June 2020 Just interpolate: Kernel “Ridgeless” regression can generalize
Tengyuan Liang, Alexander Rakhlin
Ann. Statist. 48(3): 1329-1347 (June 2020). DOI: 10.1214/19-AOS1849

Abstract

In the absence of explicit regularization, Kernel “Ridgeless” Regression with nonlinear kernels has the potential to fit the training data perfectly. It has been observed empirically, however, that such interpolated solutions can still generalize well on test data. We isolate a phenomenon of implicit regularization for minimum-norm interpolated solutions which is due to a combination of high dimensionality of the input data, curvature of the kernel function and favorable geometric properties of the data such as an eigenvalue decay of the empirical covariance and kernel matrices. In addition to deriving a data-dependent upper bound on the out-of-sample error, we present experimental evidence suggesting that the phenomenon occurs in the MNIST dataset.

Citation

Download Citation

Tengyuan Liang. Alexander Rakhlin. "Just interpolate: Kernel “Ridgeless” regression can generalize." Ann. Statist. 48 (3) 1329 - 1347, June 2020. https://doi.org/10.1214/19-AOS1849

Information

Received: 1 September 2018; Revised: 1 February 2019; Published: June 2020
First available in Project Euclid: 17 July 2020

zbMATH: 07241593
MathSciNet: MR4124325
Digital Object Identifier: 10.1214/19-AOS1849

Subjects:
Primary: 62G08 , 68Q32

Keywords: data-dependent bounds , high dimensionality , implicit regularization , kernel methods , Minimum-norm interpolation , reproducing kernel Hilbert spaces , spectral decay

Rights: Copyright © 2020 Institute of Mathematical Statistics

Vol.48 • No. 3 • June 2020
Back to Top