The Annals of Statistics

Sharp oracle inequalities for Least Squares estimators in shape restricted regression

Pierre C. Bellec

Full-text: Access denied (no subscription detected)

We're sorry, but we are unable to provide you with the full text of this article because we are not able to identify you as a subscriber. If you have a personal subscription to this journal, then please login. If you are already logged in, then you may need to update your profile to register your subscription. Read more about accessing full-text


The performance of Least Squares (LS) estimators is studied in shape-constrained regression models under Gaussian and sub-Gaussian noise. General bounds on the performance of LS estimators over closed convex sets are provided. These results have the form of sharp oracle inequalities that account for the model misspecification error. In the presence of misspecification, these bounds imply that the LS estimator estimates the projection of the true parameter at the same rate as in the well-specified case.

In isotonic and unimodal regression, the LS estimator achieves the nonparametric rate $n^{-2/3}$ as well as a parametric rate of order $k/n$ up to logarithmic factors, where $k$ is the number of constant pieces of the true parameter. In univariate convex regression, the LS estimator satisfies an adaptive risk bound of order $q/n$ up to logarithmic factors, where $q$ is the number of affine pieces of the true regression function. This adaptive risk bound holds for any collection of design points. While Guntuboyina and Sen [Probab. Theory Related Fields 163 (2015) 379–411] established that the nonparametric rate of convex regression is of order $n^{-4/5}$ for equispaced design points, we show that the nonparametric rate of convex regression can be as slow as $n^{-2/3}$ for some worst-case design points. This phenomenon can be explained as follows: Although convexity brings more structure than unimodality, for some worst-case design points this extra structure is uninformative and the nonparametric rates of unimodal regression and convex regression are both $n^{-2/3}$. Higher order cones, such as the cone of $\beta $-monotone sequences, are also studied.

Article information

Ann. Statist., Volume 46, Number 2 (2018), 745-780.

Received: October 2015
Revised: March 2017
First available in Project Euclid: 3 April 2018

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62G08: Nonparametric regression 62C20: Minimax procedures

Shape restricted regression convexity minimax rates Gaussian width concentration


Bellec, Pierre C. Sharp oracle inequalities for Least Squares estimators in shape restricted regression. Ann. Statist. 46 (2018), no. 2, 745--780. doi:10.1214/17-AOS1566.

Export citation


  • [1] Amelunxen, D., Lotz, M., McCoy, M. B. and Tropp, J. A. (2014). Living on the edge: Phase transitions in convex programs with random data. Inf. Inference 3 224–294.
  • [2] Bartlett, P. L. and Mendelson, S. (2006). Empirical minimization. Probab. Theory Related Fields 135 311–334.
  • [3] Bellec, P. C. (2018). Supplement to “Sharp oracle inequalities for Least Squares estimators in shape restricted regression.” DOI:10.1214/17-AOS1566SUPP.
  • [4] Bellec, P. C., Lecué, G. and Tsybakov, A. B. (2016). Slope meets lasso: Improved oracle bounds and optimality. ArXiv preprint. Available at arXiv:1605.08651.
  • [5] Bellec, P. C. and Tsybakov, A. B. (2015). Sharp oracle bounds for monotone and convex regression through aggregation. J. Mach. Learn. Res. 16 1879–1892.
  • [6] Boucheron, S., Lugosi, G. and Massart, P. (2013). Concentration Inequalities. A Nonasymptotic Theory of Independence. Oxford Univ. Press, Oxford.
  • [7] Chandrasekaran, V. and Jordan, M. I. (2013). Computational and statistical tradeoffs via convex relaxation. Proc. Natl. Acad. Sci. USA 110 E1181–E1190.
  • [8] Chandrasekaran, V., Recht, B., Parrilo, P. A. and Willsky, A. S. (2012). The convex geometry of linear inverse problems. Found. Comput. Math. 12 805–849.
  • [9] Chatterjee, S. (2014). A new perspective on least squares under convex constraint. Ann. Statist. 42 2340–2381.
  • [10] Chatterjee, S. (2016). An improved global risk bound in concave regression. Electron. J. Stat. 10 1608–1629.
  • [11] Chatterjee, S., Guntuboyina, A. and Sen, B. (2015). On risk bounds in isotonic and other shape restricted regression problems. Ann. Statist. 43 1774–1800.
  • [12] Chatterjee, S., Guntuboyina, A. and Sen, B. (2015). On matrix estimation under monotonicity constraints. ArXiv preprint. Available at arXiv:1506.03430.
  • [13] Chatterjee, S. and Lafferty, J. (2017). Adaptive risk bounds in unimodal regression. Bernoulli. To appear. Available at arXiv:1512.02956.
  • [14] Flammarion, N., Mao, C. and Rigollet, P. (2016). Optimal rates of statistical seriation. ArXiv preprint. Available at arXiv:1607.02435.
  • [15] Gao, F. and Wellner, J. A. (2007). Entropy estimate for high-dimensional monotonic functions. J. Multivariate Anal. 98 1751–1764.
  • [16] Guntuboyina, A. and Sen, B. (2015). Global risk bounds and adaptation in univariate convex regression. Probab. Theory Related Fields 163 379–411.
  • [17] Ledoux, M. and Talagrand, M. (1991). Probability in Banach Spaces: Isoperimetry and Processes. Ergebnisse der Mathematik und Ihrer Grenzgebiete (3) [Results in Mathematics and Related Areas (3)] 23. Springer, Berlin.
  • [18] Meyer, M. and Woodroofe, M. (2000). On the degrees of freedom in shape-restricted regression. Ann. Statist. 28 1083–1104.
  • [19] Oymak, S. and Hassibi, B. (2016). Sharp MSE bounds for proximal denoising. Found. Comput. Math. 16 965–1029.
  • [20] Oymak, S., Recht, B. and Soltanolkotabi, M. (2015). Sharp time–data tradeoffs for linear inverse problems. ArXiv preprint. Available at arXiv:1507.04793.
  • [21] Plan, Y., Vershynin, R. and Yudovina, E. (2017). High-dimensional estimation with geometric constraints. Inf. Inference 6 1–40.
  • [22] Tsybakov, A. B. (2009). Introduction to Nonparametric Estimation. Springer Series in Statistics. Springer, New York. Revised and extended from the 2004 French original, translated by Vladimir Zaiats.
  • [23] van de Geer, S. and Wainwright, M. (2015). On concentration for (regularized) empirical risk minimization. ArXiv preprint. Available at arXiv:1512.00677.
  • [24] Vershynin, R. (2015). Estimation in high dimensions: A geometric perspective. In Sampling Theory, A Renaissance. Appl. Numer. Harmon. Anal. 3–66. Birkhäuser, Basel.
  • [25] Zhang, C.-H. (2002). Risk bounds in isotonic regression. Ann. Statist. 30 528–555.

Supplemental materials

  • Supplement to “Sharp oracle inequalities for Least Squares estimators in shape restricted regression”. The supplementary material contains generalizations of the results in isotonic and convex regression to higher order cones. Theorems 1, 2 and 3 in the supplementary material generalize Theorems 3.2, 4.1 and 4.3 to the cones $\mathcal{S}^{[\beta ]}$ for $\beta \ge 3$.