## Bernoulli

• Bernoulli
• Volume 25, Number 3 (2019), 1939-1976.

### Hybrid regularisation and the (in)admissibility of ridge regression in infinite dimensional Hilbert spaces

#### Abstract

We consider the problem of estimating the slope function in a functional regression with a scalar response and a functional covariate. This central problem of functional data analysis is well known to be ill-posed, thus requiring a regularised estimation procedure. The two most commonly used approaches are based on spectral truncation or Tikhonov regularisation of the empirical covariance operator. In principle, Tikhonov regularisation is the more canonical choice. Compared to spectral truncation, it is robust to eigenvalue ties, while it attains the optimal minimax rate of convergence in the mean squared sense, and not just in a concentration probability sense. In this paper, we show that, surprisingly, one can strictly improve upon the performance of the Tikhonov estimator in finite samples by means of a linear estimator, while retaining its stability and asymptotic properties by combining it with a form of spectral truncation. Specifically, we construct an estimator that additively decomposes the functional covariate by projecting it onto two orthogonal subspaces defined via functional PCA; it then applies Tikhonov regularisation to the one component, while leaving the other component unregularised. We prove that when the covariate is Gaussian, this hybrid estimator uniformly improves upon the MSE of the Tikhonov estimator in a non-asymptotic sense, effectively rendering it inadmissible. This domination is shown to also persist under discrete observation of the covariate function. The hybrid estimator is linear, straightforward to construct in practice, and with no computational overhead relative to the standard regularisation methods. By means of simulation, it is shown to furnish sizeable gains even for modest sample sizes.

#### Article information

Source
Bernoulli, Volume 25, Number 3 (2019), 1939-1976.

Dates
First available in Project Euclid: 12 June 2019

https://projecteuclid.org/euclid.bj/1560326433

Digital Object Identifier
doi:10.3150/18-BEJ1041

Mathematical Reviews number (MathSciNet)
MR3961236

Zentralblatt MATH identifier
07066245

#### Citation

Chakraborty, Anirvan; Panaretos, Victor M. Hybrid regularisation and the (in)admissibility of ridge regression in infinite dimensional Hilbert spaces. Bernoulli 25 (2019), no. 3, 1939--1976. doi:10.3150/18-BEJ1041. https://projecteuclid.org/euclid.bj/1560326433

#### References

• [1] Alquier, P., Gautier, E. and Stoltz, G., eds. (2011). Inverse Problems and High-Dimensional Estimation. Lecture Notes in Statistics—Proceedings 203. Heidelberg: Springer. Lecture notes from the “Stats in the Château” Summer School held in Jouy-en-Josas, August 31–September 4, 2009.
• [2] Amini, A.A. and Wainwright, M.J. (2012). Sampled forms of functional PCA in reproducing kernel Hilbert spaces. Ann. Statist. 40 2483–2510.
• [3] Bai, Z. and Saranadasa, H. (1996). Effect of high dimension: By an example of a two sample problem. Statist. Sinica 6 311–329.
• [4] Cai, T.T. and Hall, P. (2006). Prediction in functional linear regression. Ann. Statist. 34 2159–2179.
• [5] Cai, T.T. and Yuan, M. (2011). Optimal estimation of the mean function based on discretely sampled functional data: Phase transition. Ann. Statist. 39 2330–2355.
• [6] Cardot, H., Ferraty, F. and Sarda, P. (2003). Spline estimators for the functional linear model. Statist. Sinica 13 571–591.
• [7] Cardot, H. and Johannes, J. (2010). Thresholding projection estimators in functional linear models. J. Multivariate Anal. 101 395–408.
• [8] Cardot, H., Mas, A. and Sarda, P. (2007). CLT in functional linear regression models. Probab. Theory Related Fields 138 325–361.
• [9] Cardot, H. and Sarda, P. (2006). Linear regression models for functional data. In The Art of Semiparametrics. Contrib. Statist. 49–66. Heidelberg: Physica-Verlag/Springer.
• [10] Chakraborty, A. and Panaretos, V.M. (2019). Supplement to “Hybrid regularisation and the (in)admissibility of ridge regression in infinite dimensional Hilbert spaces”. DOI:10.3150/18-BEJ1041SUPP.
• [11] Chen, S.X. and Qin, Y.-L. (2010). A two-sample test for high-dimensional data with applications to gene-set testing. Ann. Statist. 38 808–835.
• [12] Comte, F. and Johannes, J. (2012). Adaptive functional linear regression. Ann. Statist. 40 2765–2797.
• [13] Conway, J.B. (1978). Functions of One Complex Variable, 2nd ed. Graduate Texts in Mathematics 11. New York: Springer.
• [14] Crambes, C., Kneip, A. and Sarda, P. (2009). Smoothing splines estimators for functional linear regression. Ann. Statist. 37 35–72.
• [15] Cuevas, A., Febrero, M. and Fraiman, R. (2002). Linear functional regression: The case of fixed design and functional response. Canad. J. Statist. 30 285–300.
• [16] Ferraty, F. and Vieu, P. (2000). Dimension fractale et estimation de la régression dans des espaces vectoriels semi-normés. C. R. Acad. Sci. Paris Sér. I Math. 330 139–142.
• [17] Grenander, U. (1981). Abstract Inference. Wiley Series in Probability and Mathematical Statistics. New York: Wiley.
• [18] Hall, P. and Horowitz, J.L. (2007). Methodology and convergence rates for functional linear regression. Ann. Statist. 35 70–91.
• [19] Hall, P. and Hosseini-Nasab, M. (2006). On properties of functional principal components analysis. J. R. Stat. Soc. Ser. B. Stat. Methodol. 68 109–126.
• [20] Hocking, R.R. (2003). Methods and Applications of Linear Models: Regression and the Analysis of Variance, 2nd ed. Wiley Series in Probability and Statistics. Hoboken, NJ: Wiley.
• [21] Hoerl, A.E. and Kennard, R.W. (1970). Ridge regression: Biased estimation for nonorthogonal problems. Technometrics 12 55–67. DOI:10.2307/1267351.
• [22] Horváth, L. and Kokoszka, P. (2012). Inference for Functional Data with Applications. Springer Series in Statistics. New York: Springer.
• [23] Hsing, T. and Eubank, R. (2015). Theoretical Foundations of Functional Data Analysis, with an Introduction to Linear Operators. Wiley Series in Probability and Statistics. Chichester: Wiley.
• [24] Jolliffe, I.T. (2002). Principal Component Analysis, 2nd ed. Springer Series in Statistics. New York: Springer.
• [25] Li, Y. and Hsing, T. (2007). On rates of convergence in functional linear regression. J. Multivariate Anal. 98 1782–1804.
• [26] Lukas, M.A. (1993). Asymptotic optimality of generalized cross-validation for choosing the regularization parameter. Numer. Math. 66 41–66.
• [27] Lukas, M.A. (2006). Robust generalized cross-validation for choosing the regularization parameter. Inverse Probl. 22 1883–1902.
• [28] Marx, B.D. and Eilers, P.H. (1999). Generalized linear regression on sampled signals and curves: A P-spline approach. Technometrics 41 1–13.
• [29] Meister, A. (2009). Deconvolution Problems in Nonparametric Statistics. Lecture Notes in Statistics 193. Berlin: Springer.
• [30] Ramsay, J.O. and Dalzell, C.J. (1991). Some tools for functional data analysis. J. Roy. Statist. Soc. Ser. B 53 539–572. With discussion and a reply by the authors.
• [31] Ramsay, J.O. and Silverman, B.W. (2005). Functional Data Analysis, 2nd ed. Springer Series in Statistics. New York: Springer.
• [32] Shao, P.Y.-S. and Strawderman, W.E. (1994). Improving on the James–Stein positive-part estimator. Ann. Statist. 22 1517–1538.
• [33] Theobald, C.M. (1974). Generalizations of mean square error applied to ridge regression. J. Roy. Statist. Soc. Ser. B 36 103–106.
• [34] Tikhonov, A.N. and Arsenin, V.Y. (1977). Solutions of Ill-Posed Problems. Scripta Series in Mathematics. Washington, DC: Wiley, New York: V. H. Winston & Sons. Translated from the Russian; preface by translation editor Fritz John.
• [35] Utreras, F.I. (1987). On generalized cross-validation for multivariate smoothing spline functions. SIAM J. Sci. Statist. Comput. 8 630–643.
• [36] Wahba, G. (1990). Spline Models for Observational Data. CBMS–NSF Regional Conference Series in Applied Mathematics 59. Philadelphia, PA: Society for Industrial and Applied Mathematics (SIAM).
• [37] Yao, F., Müller, H.-G. and Wang, J.-L. (2005). Functional linear regression analysis for longitudinal data. Ann. Statist. 33 2873–2903.
• [38] Yuan, M. and Cai, T.T. (2010). A reproducing kernel Hilbert space approach to functional linear regression. Ann. Statist. 38 3412–3444.

#### Supplemental materials

• Supplement to “Hybrid regularisation and the (in)admissibility of ridge regression in infinite dimensional Hilbert spaces”. A companion supplement contains the proof of Theorem 1, the verification of Assumption (A3$'$) for the case of standard Brownian motion, the results of a simulation study for sample sizes $n=50$ and $n=300$ as well as the case when the functional covariate is observed with error, and comparative results when all three regularisation methods are applied to real data set.