Institute of Mathematical Statistics Collections

Multivariate regression through affinely weighted penalized least squares

Rudolf Beran

Full-text: Open access

Abstract

Stein’s [In Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability 1 (1956) 197–206 University of California Press] asymptotically superior shrinkage estimator of $p$ univariate means may be rederived through adaptive penalized least squares (PLS) estimation in a regression model with one nominal covariate. This paper treats adaptive PLS estimators for $p$ unknown $d$-dimensional mean vectors, each of which depends on $k_{0}$ scalar covariates that may be nominal or ordinal. The initial focus is on complete regression designs, not necessarily balanced: for every $k_{0}$-tuple of possible covariate values, at least one observation is made on the corresponding mean vector. The results include definition of suitable candidate classes of PLS estimators in multivariate regression problems, comparison of these candidate estimators through their estimated quadratic risks under a general model that makes no assumptions about the regression function, and supporting asymptotic theory as the number $p$ of covariate-value combinations observed tends to infinity. Empirical process theory establishes that: (a) estimated risks converge to their intended targets uniformly over large classes of candidate PLS estimators; (b) a candidate PLS estimator that minimizes estimated risk within such a class has risk that converges to the minimal possible risk within the class. Extension of the results to incomplete regression designs is outlined. The Efron-Morris [ Journal of the American Statistical Association 68 (1973) 117–130] and Beran [ Annals of the Institute of Statistical Mathematics 60 (2008) 843–864] estimators for multivariate means in balanced complete designs are seen as special cases.

Chapter information

Source
Banerjee, M., Bunea, F., Huang, J., Koltchinskii, V., and Maathuis, M. H., eds., From Probability to Statistics and Back: High-Dimensional Models and Processes -- A Festschrift in Honor of Jon A. Wellner, (Beachwood, Ohio, USA: Institute of Mathematical Statistics, 2013) , 33-46

Dates
First available in Project Euclid: 8 March 2013

Permanent link to this document
https://projecteuclid.org/euclid.imsc/1362751178

Digital Object Identifier
doi:10.1214/12-IMSCOLL904

Mathematical Reviews number (MathSciNet)
MR3186747

Zentralblatt MATH identifier
1327.62419

Subjects
Primary: 62J07: Ridge regression; shrinkage estimators 62F12: Asymptotic properties of estimators
Secondary: 62H12: Estimation

Keywords
unbalanced design quadratic risk adaptation multiple penalties

Rights
Copyright © 2010, Institute of Mathematical Statistics

Citation

Beran, Rudolf. Multivariate regression through affinely weighted penalized least squares. From Probability to Statistics and Back: High-Dimensional Models and Processes -- A Festschrift in Honor of Jon A. Wellner, 33--46, Institute of Mathematical Statistics, Beachwood, Ohio, USA, 2013. doi:10.1214/12-IMSCOLL904. https://projecteuclid.org/euclid.imsc/1362751178


Export citation

References

  • [1] Beran, R. (2005). ASP fits to multi-way layouts. Annals of the Institute of Statistical Mathematics 57 201–220.
  • [2] Beran, R. (2007). Multiple penalty regression: Fitting and extrapolating a discrete incomplete multi-way layout. Annals of the Institute of Statistical Mathematics 59 171–195.
  • [3] Beran, R. (2008). Estimating a mean matrix: Boosting efficiency by multiple affine shrinkage. Annals of the Institute of Statistical Mathematics 60 843–864.
  • [4] Beran, R. and Dümbgen, L. (1998). Modulation of estimators and confidence sets. Annals of Statistics 26 1826–1856.
  • [5] Buja, A., Hastie, T., Tibshirani, R. (1989). Linear smoothers and additive models (with discussion). Annals of Statistics 17 453–555.
  • [6] Chatterjee, S., Handcock, M. S. and Simonoff, J. S. (1995). A Casebook for a First Course in Statistics and Data Analysis. Wiley, New York.
  • [7] Efron, B. and Morris, C. (1973). Stein’s estimation rule and its competitors—an empirical Bayes approach. Journal of the American Statistical Association 68 117–130.
  • [8] Green, P., Jennison, C. and Seheult, A. (1985). Analysis of field experiments by least squares smoothing. Journal of the Royal Statistical Society, Series B 47 299–315.
  • [9] Hoerl, A. E. and Kennard, R. W. (1970). Ridge regression: Biased estimation for nonorthogonal problems. Technometrics 12 55–67.
  • [10] Kneip, A. (1994). Ordered linear smoothers. Annals of Statistics 22 835–866.
  • [11] Mallows, C. L. (1973). Some comments on $C_p$. Technometrics 15 661–676.
  • [12] Stein, C. (1956). Inadmissibility of the usual estimator for the mean of a multivariate normal distribution. In Proceedings of the Third Berkeley Symposium on Mathematical Statistics and Probability (J. Neyman, ed.) 1 197–206. University of California Press.
  • [13] Stein, C. (1966). An approach to the recovery of inter-block information in balanced incomplete block designs. In Festschrift for Jerzy Neyman (F. N. David, ed.) 351–364. Wiley, New York.
  • [14] Stein, C. (1981). Estimation of the mean of a multivariate normal distribution. Annals of Statistics 9 1135–1151.
  • [15] Tukey, J. W. (1980). Methodological comments focused on opportunities. In Multivariate Techniques in Communication Research (P. R. Monge and J. Cappella, eds.) 489–528. Academic Press, New York.
  • [16] Wood, S. N. (2000). Modelling and smoothing parameter estimation with multiple quadratic penalties. Journal of the Royal Statistical Society, Series B 62 413–428.