The Annals of Statistics

Bayesian manifold regression

Yun Yang and David B. Dunson

Full-text: Open access

Enhanced PDF (379 KB)

Abstract

There is increasing interest in the problem of nonparametric regression with high-dimensional predictors. When the number of predictors $D$ is large, one encounters a daunting problem in attempting to estimate a $D$-dimensional surface based on limited data. Fortunately, in many applications, the support of the data is concentrated on a $d$-dimensional subspace with $d\ll D$. Manifold learning attempts to estimate this subspace. Our focus is on developing computationally tractable and theoretically supported Bayesian nonparametric regression methods in this context. When the subspace corresponds to a locally-Euclidean compact Riemannian manifold, we show that a Gaussian process regression approach can be applied that leads to the minimax optimal adaptive rate in estimating the regression function under some conditions. The proposed model bypasses the need to estimate the manifold, and can be implemented using standard algorithms for posterior computation in Gaussian processes. Finite sample performance is illustrated in a data analysis example.

Article information

Source
Ann. Statist., Volume 44, Number 2 (2016), 876-905.

Dates
Received: December 2014
Revised: September 2015
First available in Project Euclid: 17 March 2016

Permanent link to this document
https://projecteuclid.org/euclid.aos/1458245738

Digital Object Identifier
doi:10.1214/15-AOS1390

Mathematical Reviews number (MathSciNet)
MR3476620

Zentralblatt MATH identifier
1341.62196

Subjects
Primary: 62H30: Classification and discrimination; cluster analysis [See also 68T10, 91C20] 62-07: Data analysis
Secondary: 65U05 68T05: Learning and adaptive systems [See also 68Q32, 91E40]

Keywords
Asymptotics contraction rates dimensionality reduction Gaussian process manifold learning nonparametric Bayes subspace learning

Citation

Yang, Yun; Dunson, David B. Bayesian manifold regression. Ann. Statist. 44 (2016), no. 2, 876--905. doi:10.1214/15-AOS1390. https://projecteuclid.org/euclid.aos/1458245738


Export citation

References

  • [1] Aronszajn, N. (1950). Theory of reproducing kernels. Trans. Amer. Math. Soc. 68 337–404.
    Mathematical Reviews (MathSciNet): MR51437
    Zentralblatt MATH: 0037.20701
    Digital Object Identifier: doi:10.1090/S0002-9947-1950-0051437-7
  • [2] Belkin, M. (2003). Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput. 15 1373–1396.
  • [3] Bhattacharya, A., Pati, D. and Dunson, D. (2014). Anisotropic function estimation using multi-bandwidth Gaussian processes. Ann. Statist. 42 352–381.
    Mathematical Reviews (MathSciNet): MR3189489
    Zentralblatt MATH: 06291844
    Digital Object Identifier: doi:10.1214/13-AOS1192
    Project Euclid: euclid.aos/1395234981
  • [4] Bickel, P. J. and Kleijn, B. J. K. (2012). The semiparametric Bernstein–von Mises theorem. Ann. Statist. 40 206–237.
    Mathematical Reviews (MathSciNet): MR3013185
    Zentralblatt MATH: 1246.62081
    Digital Object Identifier: doi:10.1214/11-AOS921
    Project Euclid: euclid.aos/1333029963
  • [5] Bickel, P. J. and Li, B. (2007). Local polynomial regression on unknown manifolds. In Complex Datasets and Inverse Problems. Institute of Mathematical Statistics Lecture Notes—Monograph Series 54 177–186. IMS, Beachwood, OH.
    Mathematical Reviews (MathSciNet): MR2459188
    Digital Object Identifier: doi:10.1214/074921707000000148
  • [6] Binev, P., Cohen, A., Dahmen, W. and DeVore, R. (2007). Universal algorithms for learning theory. II. Piecewise polynomial functions. Constr. Approx. 26 127–152.
    Mathematical Reviews (MathSciNet): MR2327596
    Digital Object Identifier: doi:10.1007/s00365-006-0658-z
  • [7] Binev, P., Cohen, A., Dahmen, W., DeVore, R. and Temlyakov, V. (2005). Universal algorithms for learning theory. I. Piecewise constant functions. J. Mach. Learn. Res. 6 1297–1321.
    Mathematical Reviews (MathSciNet): MR2249856
    Zentralblatt MATH: 1191.62068
  • [8] Camastra, F. and Vinviarelli, A. (2002). Estimating the intrinsic dimension of data with a fractal-based method. IEEE P.A.M.I. 24 1404–1407.
  • [9] Carter, K. M., Raich, R. and Hero, A. O. III (2010). On local intrinsic dimension estimation and its applications. IEEE Trans. Signal Process. 58 650–663.
    Mathematical Reviews (MathSciNet): MR2750463
    Digital Object Identifier: doi:10.1109/TSP.2009.2031722
  • [10] Castillo, I., Kerkyacharian, G. and Picard, D. (2013). Thomas Bayes’ walk on manifolds. Probab. Theory Related Fields 158 665–710.
    Mathematical Reviews (MathSciNet): MR3176362
    Zentralblatt MATH: 1285.62028
    Digital Object Identifier: doi:10.1007/s00440-013-0493-0
  • [11] Chen, M., Silva, J., Paisley, J., Wang, C., Dunson, D. and Carin, L. (2010). Compressive sensing on manifolds using a nonparametric mixture of factor analyzers: Algorithm and performance bounds. IEEE Trans. Signal Process. 58 6140–6155.
    Mathematical Reviews (MathSciNet): MR2790088
    Digital Object Identifier: doi:10.1109/TSP.2010.2070796
  • [12] Farahmand, A. M., Szepesvái, C. and Audibert, J. (2007). Manifold-adaptive dimension estimation. In ICML 2007 265–272. ACM Press, New York.
  • [13] Ghosal, S., Ghosh, J. K. and van der Vaart, A. W. (2000). Convergence rates of posterior distributions. Ann. Statist. 28 500–531.
    Mathematical Reviews (MathSciNet): MR1790007
    Digital Object Identifier: doi:10.1214/aos/1016218228
    Project Euclid: euclid.aos/1016218228
  • [14] Ghosal, S. and van der Vaart, A. (2007). Convergence rates of posterior distributions for non-i.i.d. observations. Ann. Statist. 35 192–223.
    Mathematical Reviews (MathSciNet): MR2332274
    Digital Object Identifier: doi:10.1214/009053606000001172
    Project Euclid: euclid.aos/1181100186
  • [15] Giné, E. and Nickl, R. (2011). Rates on contraction for posterior distributions in $L^{r}$-metrics, $1\leq r\leq\infty$. Ann. Statist. 39 2883–2911.
    Mathematical Reviews (MathSciNet): MR3012395
    Digital Object Identifier: doi:10.1214/11-AOS924
    Project Euclid: euclid.aos/1327413772
  • [16] Kpotufe, S. (2009). Escaping the curse of dimensionality with a tree-based regressor. In COLT 2009—The 22nd Conference on Learning Theory, June 18–21. Montreal, QC.
  • [17] Kpotufe, S. and Dasgupta, S. (2012). A tree-based regressor that adapts to intrinsic dimension. J. Comput. System Sci. 78 1496–1515.
    Mathematical Reviews (MathSciNet): MR2926146
    Zentralblatt MATH: 06067743
    Digital Object Identifier: doi:10.1016/j.jcss.2012.01.002
  • [18] Kundu, S. and Dunson, D. B. (2011). Latent factor models for density estimation. Available at arXiv:1108.2720v2.
    arXiv: 1108.2720v2
    Mathematical Reviews (MathSciNet): MR3254906
    Zentralblatt MATH: 06351235
    Digital Object Identifier: doi:10.1093/biomet/asu019
  • [19] Lawrence, N. D. (2003). Gaussian process latent variable models for visualisation of high dimensional data. Neural Information Processing Systems 16 329–336.
  • [20] Levina, E. and Bickel, P. (2004). Maximun likelihood estimation of intrinsic dimension. In Advances in Neural Information Processing Systems 17. MIT Press, Cambridge, MA.
  • [21] Lin, L. and Dunson, D. B. (2014). Bayesian monotone regression using Gaussian process projection. Biometrika 101 303–317.
    Mathematical Reviews (MathSciNet): MR3215349
    Zentralblatt MATH: 06329733
    Digital Object Identifier: doi:10.1093/biomet/ast063
  • [22] Little, A. V., Lee, J., Jung, Y. M. and Maggioni, M. (2009). Estimation of intrinsic dimensionality of samples from noisy low-dimensional manifolds in high dimensions with multiscale SVD. In 2009 IEEE/SP 15th Workshop on Statistical Signal Processing 85–88. IEEE, Cardiff.
  • [23] Nene, S. A., Nayar, S. K. and Murase, H. (1996). Columbia object image library (COIL-100). Technical report, Columbia Univ., New York.
  • [24] Page, G., Bhattacharya, A. and Dunson, D. (2013). Classification via Bayesian nonparametric learning of affine subspaces. J. Amer. Statist. Assoc. 108 187–201.
    Mathematical Reviews (MathSciNet): MR3174612
    Zentralblatt MATH: 06158335
    Digital Object Identifier: doi:10.1080/01621459.2013.763566
  • [25] Reich, B. J., Bondell, H. D. and Li, L. (2011). Sufficient dimension reduction via Bayesian mixture modeling. Biometrics 67 886–895.
    Mathematical Reviews (MathSciNet): MR2829263
    Digital Object Identifier: doi:10.1111/j.1541-0420.2010.01501.x
  • [26] Roweis, S. T. and Saul, L. K. (2000). Nonlinear dimensionality reduction by locally linear embedding. Science 290 2323–2326.
  • [27] Savitsky, T., Vannucci, M. and Sha, N. (2011). Variable selection for nonparametric Gaussian process priors: Models and computational strategies. Statist. Sci. 26 130–149.
    Mathematical Reviews (MathSciNet): MR2849913
    Digital Object Identifier: doi:10.1214/11-STS354
    Project Euclid: euclid.ss/1307626570
  • [28] Stone, C. J. (1982). Optimal global rates of convergence for nonparametric regression. Ann. Statist. 10 1040–1053.
    Mathematical Reviews (MathSciNet): MR673642
    Zentralblatt MATH: 0511.62048
    Digital Object Identifier: doi:10.1214/aos/1176345969
    Project Euclid: euclid.aos/1176345969
  • [29] Tenenbaum, J. B., de Silva, V. and Langford, J. C. (2000). A global geometric framework for nonlinear dimensionality reduction. Science 290 2319–2323.
  • [30] Tibshirani, R. (1996). Regression shrinkage and selection via the lasso. J. R. Stat. Soc. Ser. B. Stat. Methodol. 73 273–282.
    Mathematical Reviews (MathSciNet): MR2815776
    Digital Object Identifier: doi:10.1111/j.1467-9868.2011.00771.x
  • [31] Tokdar, S. T., Zhu, Y. M. and Ghosh, J. K. (2010). Bayesian density regression with logistic Gaussian process and subspace projection. Bayesian Anal. 5 319–344.
    Mathematical Reviews (MathSciNet): MR2719655
    Digital Object Identifier: doi:10.1214/10-BA605
    Project Euclid: euclid.ba/1340218341
  • [32] van de Geer, S. (2000). Empirical Processes in M-Estimation. Cambridge Univ. Press, Cambridge.
  • [33] van der Vaart, A. and van Zanten, H. (2011). Information rates of nonparametric Gaussian process methods. J. Mach. Learn. Res. 12 2095–2119.
    Mathematical Reviews (MathSciNet): MR2819028
    Zentralblatt MATH: 1280.68228
  • [34] van der Vaart, A. W. and van Zanten, J. H. (2008). Reproducing kernel Hilbert spaces of Gaussian priors. In Pushing the Limits of Contemporary Statistics: Contributions in Honor of Jayanta K. Ghosh. Inst. Math. Stat. Collect. 3 200–222. IMS, Beachwood, OH.
    Mathematical Reviews (MathSciNet): MR2459226
    Digital Object Identifier: doi:10.1214/074921708000000156
  • [35] van der Vaart, A. W. and van Zanten, J. H. (2009). Adaptive Bayesian estimation using a Gaussian random field with inverse gamma bandwidth. Ann. Statist. 37 2655–2675.
    Mathematical Reviews (MathSciNet): MR2541442
    Zentralblatt MATH: 1173.62021
    Digital Object Identifier: doi:10.1214/08-AOS678
    Project Euclid: euclid.aos/1247836664
  • [36] Yang, Y. and Dunson, D. B. (2015). Supplement to “Bayesian manifold regression.” DOI:10.1214/15-AOS1390SUPP.
  • [37] Ye, G.-B. and Zhou, D.-X. (2008). Learning and approximation by Gaussians on Riemannian manifolds. Adv. Comput. Math. 29 291–310.
    Mathematical Reviews (MathSciNet): MR2438346
    Zentralblatt MATH: 1156.68045
    Digital Object Identifier: doi:10.1007/s10444-007-9049-0
  • [38] Zou, H. and Hastie, T. (2005). Regularization and variable selection via the elastic net. J. R. Stat. Soc. Ser. B. Stat. Methodol. 67 301–320.
    Mathematical Reviews (MathSciNet): MR2137327
    Zentralblatt MATH: 1069.62054
    Digital Object Identifier: doi:10.1111/j.1467-9868.2005.00503.x

Supplemental materials

  • Reviews of geometric properties and proofs of Theorems 2.1, 2.2, 2.4 and 3.2. Concepts and results in differential and Riemannian geometry were reviewed in Section 7, where new results are included with proofs. Then proofs of Theorems 2.1, 2.2, 2.4 and 3.2 are provided in Section 8.
    Digital Object Identifier: doi:10.1214/15-AOS1390SUPP
    Supplemental PDF (182 KB)

The Institute of Mathematical Statistics

The Annals of Statistics

New content alerts

Email RSS ToC RSS Article
  • You have access to this content.
  • You have partial access to this content.
  • You do not have access to this content.
Turn Off MathJax

What is MathJax?

More like this

+ See more