## The Annals of Statistics

### Sharp oracle inequalities for Least Squares estimators in shape restricted regression

Pierre C. Bellec

#### Abstract

The performance of Least Squares (LS) estimators is studied in shape-constrained regression models under Gaussian and sub-Gaussian noise. General bounds on the performance of LS estimators over closed convex sets are provided. These results have the form of sharp oracle inequalities that account for the model misspecification error. In the presence of misspecification, these bounds imply that the LS estimator estimates the projection of the true parameter at the same rate as in the well-specified case.

In isotonic and unimodal regression, the LS estimator achieves the nonparametric rate $n^{-2/3}$ as well as a parametric rate of order $k/n$ up to logarithmic factors, where $k$ is the number of constant pieces of the true parameter. In univariate convex regression, the LS estimator satisfies an adaptive risk bound of order $q/n$ up to logarithmic factors, where $q$ is the number of affine pieces of the true regression function. This adaptive risk bound holds for any collection of design points. While Guntuboyina and Sen [Probab. Theory Related Fields 163 (2015) 379–411] established that the nonparametric rate of convex regression is of order $n^{-4/5}$ for equispaced design points, we show that the nonparametric rate of convex regression can be as slow as $n^{-2/3}$ for some worst-case design points. This phenomenon can be explained as follows: Although convexity brings more structure than unimodality, for some worst-case design points this extra structure is uninformative and the nonparametric rates of unimodal regression and convex regression are both $n^{-2/3}$. Higher order cones, such as the cone of $\beta$-monotone sequences, are also studied.

#### Article information

Source
Ann. Statist., Volume 46, Number 2 (2018), 745-780.

Dates
Revised: March 2017
First available in Project Euclid: 3 April 2018

https://projecteuclid.org/euclid.aos/1522742435

Digital Object Identifier
doi:10.1214/17-AOS1566

Mathematical Reviews number (MathSciNet)
MR3782383

Zentralblatt MATH identifier
06870278

Subjects
Primary: 62G08: Nonparametric regression 62C20: Minimax procedures

#### Citation

Bellec, Pierre C. Sharp oracle inequalities for Least Squares estimators in shape restricted regression. Ann. Statist. 46 (2018), no. 2, 745--780. doi:10.1214/17-AOS1566. https://projecteuclid.org/euclid.aos/1522742435

#### References

• [1] Amelunxen, D., Lotz, M., McCoy, M. B. and Tropp, J. A. (2014). Living on the edge: Phase transitions in convex programs with random data. Inf. Inference 3 224–294.
• [2] Bartlett, P. L. and Mendelson, S. (2006). Empirical minimization. Probab. Theory Related Fields 135 311–334.
• [3] Bellec, P. C. (2018). Supplement to “Sharp oracle inequalities for Least Squares estimators in shape restricted regression.” DOI:10.1214/17-AOS1566SUPP.
• [4] Bellec, P. C., Lecué, G. and Tsybakov, A. B. (2016). Slope meets lasso: Improved oracle bounds and optimality. ArXiv preprint. Available at arXiv:1605.08651.
• [5] Bellec, P. C. and Tsybakov, A. B. (2015). Sharp oracle bounds for monotone and convex regression through aggregation. J. Mach. Learn. Res. 16 1879–1892.
• [6] Boucheron, S., Lugosi, G. and Massart, P. (2013). Concentration Inequalities. A Nonasymptotic Theory of Independence. Oxford Univ. Press, Oxford.
• [7] Chandrasekaran, V. and Jordan, M. I. (2013). Computational and statistical tradeoffs via convex relaxation. Proc. Natl. Acad. Sci. USA 110 E1181–E1190.
• [8] Chandrasekaran, V., Recht, B., Parrilo, P. A. and Willsky, A. S. (2012). The convex geometry of linear inverse problems. Found. Comput. Math. 12 805–849.
• [9] Chatterjee, S. (2014). A new perspective on least squares under convex constraint. Ann. Statist. 42 2340–2381.
• [10] Chatterjee, S. (2016). An improved global risk bound in concave regression. Electron. J. Stat. 10 1608–1629.
• [11] Chatterjee, S., Guntuboyina, A. and Sen, B. (2015). On risk bounds in isotonic and other shape restricted regression problems. Ann. Statist. 43 1774–1800.
• [12] Chatterjee, S., Guntuboyina, A. and Sen, B. (2015). On matrix estimation under monotonicity constraints. ArXiv preprint. Available at arXiv:1506.03430.
• [13] Chatterjee, S. and Lafferty, J. (2017). Adaptive risk bounds in unimodal regression. Bernoulli. To appear. Available at arXiv:1512.02956.
• [14] Flammarion, N., Mao, C. and Rigollet, P. (2016). Optimal rates of statistical seriation. ArXiv preprint. Available at arXiv:1607.02435.
• [15] Gao, F. and Wellner, J. A. (2007). Entropy estimate for high-dimensional monotonic functions. J. Multivariate Anal. 98 1751–1764.
• [16] Guntuboyina, A. and Sen, B. (2015). Global risk bounds and adaptation in univariate convex regression. Probab. Theory Related Fields 163 379–411.
• [17] Ledoux, M. and Talagrand, M. (1991). Probability in Banach Spaces: Isoperimetry and Processes. Ergebnisse der Mathematik und Ihrer Grenzgebiete (3) [Results in Mathematics and Related Areas (3)] 23. Springer, Berlin.
• [18] Meyer, M. and Woodroofe, M. (2000). On the degrees of freedom in shape-restricted regression. Ann. Statist. 28 1083–1104.
• [19] Oymak, S. and Hassibi, B. (2016). Sharp MSE bounds for proximal denoising. Found. Comput. Math. 16 965–1029.
• [20] Oymak, S., Recht, B. and Soltanolkotabi, M. (2015). Sharp time–data tradeoffs for linear inverse problems. ArXiv preprint. Available at arXiv:1507.04793.
• [21] Plan, Y., Vershynin, R. and Yudovina, E. (2017). High-dimensional estimation with geometric constraints. Inf. Inference 6 1–40.
• [22] Tsybakov, A. B. (2009). Introduction to Nonparametric Estimation. Springer Series in Statistics. Springer, New York. Revised and extended from the 2004 French original, translated by Vladimir Zaiats.
• [23] van de Geer, S. and Wainwright, M. (2015). On concentration for (regularized) empirical risk minimization. ArXiv preprint. Available at arXiv:1512.00677.
• [24] Vershynin, R. (2015). Estimation in high dimensions: A geometric perspective. In Sampling Theory, A Renaissance. Appl. Numer. Harmon. Anal. 3–66. Birkhäuser, Basel.
• [25] Zhang, C.-H. (2002). Risk bounds in isotonic regression. Ann. Statist. 30 528–555.

#### Supplemental materials

• Supplement to “Sharp oracle inequalities for Least Squares estimators in shape restricted regression”. The supplementary material contains generalizations of the results in isotonic and convex regression to higher order cones. Theorems 1, 2 and 3 in the supplementary material generalize Theorems 3.2, 4.1 and 4.3 to the cones $\mathcal{S}^{[\beta ]}$ for $\beta \ge 3$.