Statistics Surveys

Additive monotone regression in high and lower dimensions

Solveig Engebretsen and Ingrid K. Glad

Full-text: Open access


In numerous problems where the aim is to estimate the effect of a predictor variable on a response, one can assume a monotone relationship. For example, dose-effect models in medicine are of this type. In a multiple regression setting, additive monotone regression models assume that each predictor has a monotone effect on the response. In this paper, we present an overview and comparison of very recent frequentist methods for fitting additive monotone regression models. Three of the methods we present can be used both in the high dimensional setting, where the number of parameters $p$ exceeds the number of observations $n$, and in the classical multiple setting where $1<p\leq n$. However, many of the most recent methods only apply to the classical setting. The methods are compared through simulation experiments in terms of efficiency, prediction error and variable selection properties in both settings, and they are applied to the Boston housing data. We conclude with some recommendations on when the various methods perform best.

Article information

Statist. Surv., Volume 13 (2019), 1-51.

Received: November 2018
First available in Project Euclid: 20 June 2019

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62G08: Nonparametric regression

Monotone regression shape constrained regression regression splines additive regression

Creative Commons Attribution 4.0 International License.


Engebretsen, Solveig; Glad, Ingrid K. Additive monotone regression in high and lower dimensions. Statist. Surv. 13 (2019), 1--51. doi:10.1214/19-SS124.

Export citation


  • [1] Antoniadis, A., Bigot, J., and Gijbels, I. (2007). Penalized wavelet monotone regression. Statistics & Probability Letters, 77(16):1608–1621.
  • [2] Bacchetti, P. (1989). Additive isotonic models. Journal of the American Statistical Association, 84(405):289–294.
  • [3] Barlow, R. E., Bartholomew, D. J., Bremner, J. M., and Brunk, H. D. (1972). Statistical Inference Under Order Restrictions: The Theory and Application of Isotonic Regression. Hoboken, NJ: Wiley.
  • [4] Barlow, R. E. and Brunk, H. D. (1972). The isotonic regression problem and its dual. Journal of the American Statistical Association, 67(337):140–147.
  • [5] Bergersen, L. C., Tharmaratnam, K., and Glad, I. K. (2014). Monotone splines lasso. Computational Statistics & Data Analysis, 77:336–351.
  • [6] Bollaerts, K., Eilers, P. H., and Mechelen, I. (2006). Simple and multiple P-splines regression with shape constraints. British Journal of Mathematical and Statistical Psychology, 59(2):451–469.
  • [7] Bornkamp, B. and Ickstadt, K. (2008). Bayesian nonparametric estimation of continuous monotone functions with applications to dose-response analysis. Biometrics, 65(1):198–205.
  • [8] Bornkamp, B., Ickstadt, K., and Dunson, D. (2010). Stochastically ordered multiple regression. Biostatistics, 11(3):419–431.
  • [9] Brezger, A. and Steiner, W.J. (2008). Monotonic regression based on Bayesian P-splines: an application to estimating price response functions from store-level scanner data. Journal of business & economic statistics, 26(1):90 –104.
  • [10] Bühlmann, P. and Yu, B. (2006). Sparse boosting. The Journal of Machine Learning Research, 7:1001–1024.
  • [11] Chen, Y. and Samworth, R. J. (2016). Generalized additive and index models with shape constraints. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 78(4):729 – 754.
  • [12] Chipman, H. A., George, E., I, McCulloch, R. E. and Shively, T. S. (2016). High-dimensional nonparametric monotone function estimation using BART. arXiv preprint arXiv:1612.01619.
  • [13] Chiquet, J., Grandvalet, Y., and Charbonnier, C. (2012). Sparsity with sign-coherent groups of variables via the cooperative-lasso. The Annals of Applied Statistics, 6(2):795–830.
  • [14] Dette, H. and Scheder, R. (2006). Strictly monotone and smooth nonparametric regression for two or more variables. Canadian Journal of Statistics, 34(4):535–561.
  • [15] Du, P., Cheng, G. and Liang, H. (2012). Semiparametric regression models with additive nonparametric components and high dimensional parametric components. Computational Statistics & Data Analysis, 56(6):2006–2017.
  • [16] Du, P., Parmeter, C. F. and Racine, J. S. (2013). Nonparametric kernel regression with multiple predictors and multiple shape constraints Statistica Sinica, 23(3):1347–1371.
  • [17] Eilers, P. H. and Marx, B. D. (1996). Flexible smoothing with B-splines and penalties. Statistical Science, 11:89–102.
  • [18] Fang, Z. and Meinshausen, N. (2012). Lasso isotone for high-dimensional additive isotonic regression. Journal of Computational and Graphical Statistics, 21(1):72–91.
  • [19] Gunter, L., Zhu, J. and Murphy, S. (2007). Variable selection for optimal decision making. Conference on Artificial Intelligence in Medicine in Europe, 2007:149–154. Springer.
  • [20] Guo, J., Tang, M., Tian, M. and Zhu, K. (2013). Variable selection in high-dimensional partially linear additive models for composite quantile regression Computational Statistics & Data Analysis, 65:56–67.
  • [21] Harrison, D. and Rubinfeld, D. L. (1978). Hedonic housing prices and the demand for clean air. Journal of Environmental Economics and Management, 5(1):81–102.
  • [22] Hastie, T. and Tibshirani, R. (1986). Generalized additive models. Statistical Science, 1(3):297–310.
  • [23] He, X. and Shi, P. (1998). Monotone B-spline smoothing. Journal of the American statistical Association, 93(442):643–650.
  • [24] Hesterberg, T., Choi, N. H., Meier, L., and Fraley, C. (2008). Least angle and $L_{1}$ penalized regression: A review. Statistics Surveys, 2:61–93.
  • [25] Hofner, B., Kneib, T., and Hothorn, T. (2016). A unified framework of constrained regression. Statistics and Computing, 26(1-2):1–14.
  • [26] Hothorn, T., Buehlmann, P., Kneib, T., Schmid, M., and Hofner, B. (2017). mboost: Model-Based Boosting. R package version 2.8-1.
  • [27] Härdle, W. and Liang, H. (2007). Partially linear models. In Statistical methods for biostatistics and related fields, 87–103. Springer, Berlin, Heidelberg.
  • [28] Leitenstorfer, F. and Tutz, G. (2006). Generalized monotonic regression based on B-splines with an application to air pollution data. Biostatistics, 8(3):654–673.
  • [29] Lian, H., Liang, H. and Ruppert, D. (2015). Separation of covariates into nonparametric and parametric parts in high-dimensional partially linear additive models. Statistica Sinica, 25(2):591–607.
  • [30] Lin, L. and Dunson, D. B. (2014). Bayesian monotone regression using Gaussian process projection Biometrika, 101(2):303–317.
  • [31] Lin, Y. and Zhang, H. H. (2006). Component selection and smoothing in multivariate nonparametric regression. The Annals of Statistics, 34(5):2272–2297.
  • [32] Liu, X., Wang, L. and Liang, H. (2011). Estimation and variable selection for semiparametric additive partial linear models Statistica Sinica, 21(3):1225–1248.
  • [33] Lou, Y., Bien, J., Caruana, R. and Gehrke, J. (2016). Sparse partially linear additive models. Journal of Computational and Graphical Statistics,25(4): 1126–1140.
  • [34] Luss, R., Rosset, S., and Shahar, M. (2012). Efficient regularized isotonic regression with application to gene–gene interaction search. The Annals of Applied Statistics, 6(1):253–283.
  • [35] Lv, J., Yang, H. and Guo, C. (2017). Variable selection in partially linear additive models for modal regression Communications in Statistics-Simulation and Computation 46(7): 5646 – 5665.
  • [36] Meyer, M. C. (2008). Inference using shape-restricted regression splines. The Annals of Applied Statistics, 2(3):1013–1033.
  • [37] Meyer, M. C. (2013). Semi-parametric additive constrained regression. Journal of Nonparametric Statistics, 25(3):715–730.
  • [38] Meyer, M. C., Hackstadt, A. J., and Hoeting, J. A. (2011). Bayesian estimation and inference for generalised partial linear models using shape-restricted splines. Journal of Nonparametric Statistics, 23(4):867–884.
  • [39] Pya, N. and Wood, S. N. (2015). Shape constrained additive models. Statistics and Computing, 25(3):543–559.
  • [40] Ramsay, J. O. (1988). Monotone regression splines in action. Statistical Science, 3(4):425–441.
  • [41] Ruppert, D. (2002). Selecting the number of knots for penalized splines. Journal of Computational and Graphical Statistics, 11(4):735–757.
  • [42] Saarela, O. and Arjas, E. (2011). A method for Bayesian monotonic multiple regression. Scandinavian Journal of Statistics, 38(3):499–513.
  • [43] Schell, M. J. and Singh, B. (1997). The reduced monotonic regression method. Journal of the American Statistical Association, 92(437):128–135.
  • [44] Tutz, G. and Leitenstorfer, F. (2007). Generalized smooth monotonic regression in additive modeling. Journal of Computational and Graphical Statistics, 16(1):165–188.
  • [45] Wang, L. and Xue, L. (2015). Constrained polynomial spline estimation of monotone additive models. Journal of Statistical Planning and Inference, 167:27–40.
  • [46] Wei, F. (2012). Group selection in high-dimensional partially linear additive models Brazilian Journal of Probability and Statistics, 26(3): 219–243.
  • [47] Wood, S. N. (2006). Generalized additive models: an introduction with R. Chapman and Hall/CRC.
  • [48] Zhang, H. H., Cheng, G. and Liu, Y. (2011). Linear or nonlinear? Automatic structure discovery for partially linear models Journal of the American Statistical Association, 106(495):1099–1112.
  • [49] Zou H, Hastie T. (2005). Regularization and variable selection via the elastic net. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 67(2):301–20.