The Annals of Applied Probability

Universality in polytope phase transitions and message passing algorithms

Abstract

We consider a class of nonlinear mappings $\mathsf{F}_{A,N}$ in $\mathbb{R}^{N}$ indexed by symmetric random matrices $A\in\mathbb{R}^{N\times N}$ with independent entries. Within spin glass theory, special cases of these mappings correspond to iterating the TAP equations and were studied by Bolthausen [Comm. Math. Phys. 325 (2014) 333–366]. Within information theory, they are known as “approximate message passing” algorithms.

We study the high-dimensional (large $N$) behavior of the iterates of $\mathsf{F}$ for polynomial functions $\mathsf{F}$, and prove that it is universal; that is, it depends only on the first two moments of the entries of $A$, under a sub-Gaussian tail condition. As an application, we prove the universality of a certain phase transition arising in polytope geometry and compressed sensing. This solves, for a broad class of random projections, a conjecture by David Donoho and Jared Tanner.

Article information

Source
Ann. Appl. Probab., Volume 25, Number 2 (2015), 753-822.

Dates
First available in Project Euclid: 19 February 2015

https://projecteuclid.org/euclid.aoap/1424355130

Digital Object Identifier
doi:10.1214/14-AAP1010

Mathematical Reviews number (MathSciNet)
MR3313755

Zentralblatt MATH identifier
1322.60207

Subjects
Primary: 60F05: Central limit and other weak theorems

Citation

Bayati, Mohsen; Lelarge, Marc; Montanari, Andrea. Universality in polytope phase transitions and message passing algorithms. Ann. Appl. Probab. 25 (2015), no. 2, 753--822. doi:10.1214/14-AAP1010. https://projecteuclid.org/euclid.aoap/1424355130

References

• [1] Adamczak, R., Litvak, A. E., Pajor, A. and Tomczak-Jaegermann, N. (2011). Restricted isometry property of matrices with independent columns and neighborly polytopes by random sampling. Constr. Approx. 34 61–88.
• [2] Affentranger, F. and Schneider, R. (1992). Random projections of regular simplices. Discrete Comput. Geom. 7 219–226.
• [3] Anderson, G. W., Guionnet, A. and Zeitouni, O. (2010). An Introduction to Random Matrices. Cambridge Studies in Advanced Mathematics 118. Cambridge Univ. Press, Cambridge.
• [4] Bai, Z. and Silverstein, J. W. (2009). Spectral Analysis of Large Dimensional Random Matrices, 2nd ed. Springer, New York.
• [5] Bai, Z. D. and Silverstein, J. W. (1998). No eigenvalues outside the support of the limiting spectral distribution of large-dimensional sample covariance matrices. Ann. Probab. 26 316–345.
• [6] Bayati, M. and Montanari, A. (2012). The LASSO risk for Gaussian matrices. IEEE Trans. Inform. Theory 58 1997–2017.
• [7] Bolthausen, E. (2014). An iterative construction of solutions of the TAP equations for the Sherrington–Kirkpatrick model. Comm. Math. Phys. 325 333–366.
• [8] Bürgisser, P. and Cucker, F. (2010). Smoothed analysis of Moore–Penrose inversion. SIAM J. Matrix Anal. Appl. 31 2769–2783.
• [9] Donoho, D. and Tanner, J. (2009). Observed universality of phase transitions in high-dimensional geometry, with implications for modern data analysis and signal processing. Philos. Trans. R. Soc. Lond. Ser. A Math. Phys. Eng. Sci. 367 4273–4293.
• [10] Donoho, D. L. (2005). Neighborly polytopes and sparse solution of underdetermined linear equations. Technical report, Statistics Dept., Stanford Univ., Stanford, CA.
• [11] Donoho, D. L. (2006). High-dimensional centrally symmetric polytopes with neighborliness proportional to dimension. Discrete Comput. Geom. 35 617–652.
• [12] Donoho, D. L., Javanmard, A. and Montanari, A. (2013). Information-theoretically optimal compressed sensing via spatial coupling and approximate message passing. IEEE Trans. Inform. Theory 59 7434–7464.
• [13] Donoho, D. L., Johnstone, I. and Montanari, A. (2013). Accurate prediction of phase transitions in compressed sensing via a connection to minimax denoising. IEEE Trans. Inform. Theory 59 3396–3433.
• [14] Donoho, D. L., Maleki, A. and Montanari, A. (2009). Message passing algorithms for compressed sensing. Proc. Natl. Acad. Sci. USA 106 18914–18919.
• [15] Donoho, D. L., Maleki, A. and Montanari, A. (2011). The noise-sensitivity phase transition in compressed sensing. IEEE Trans. Inform. Theory 57 6920–6941.
• [16] Donoho, D. L. and Tanner, J. (2005). Neighborliness of randomly projected simplices in high dimensions. Proc. Natl. Acad. Sci. USA 102 9452–9457 (electronic).
• [17] Donoho, D. L. and Tanner, J. (2005). Sparse nonnegative solution of underdetermined linear equations by linear programming. Proc. Natl. Acad. Sci. USA 102 9446–9451 (electronic).
• [18] Donoho, D. L. and Tanner, J. (2009). Counting faces of randomly projected polytopes when the projection radically lowers dimension. J. Amer. Math. Soc. 22 1–53.
• [19] Javanmard, A. and Montanari, A. (2013). State evolution for general approximate message passing algorithms, with applications to spatial coupling. Information and Inference 2 115–144.
• [20] Kabashima, Y., Wadayama, T. and Tanaka, T. (2009). A typical reconstruction limit for compressed sensing based on lp-norm minimization. J. Stat. Mech. L09003.
• [21] Krzakala, F., Mézard, M., Sausset, F., Sun, Y. F. and Zdeborová, L. (2012). Statistical-physics-based reconstruction in compressed sensing. Physical Review X 2 021005.
• [22] Lubinsky, D. S. (2007). A survey of weighted polynomial approximation with exponential weights. Surv. Approx. Theory 3 1–105.
• [23] Maleki, A., Anitori, L., Yang, A. and Baraniuk, R. (2011). Asymptotic analysis of complex LASSO via complex approximate message passing (CAMP). Available at arXiv:1108.0477.
• [24] Mézard, M., Parisi, G. and Virasoro, M. A. (1987). Spin Glass Theory and Beyond. World Scientific Lecture Notes in Physics 9. World Scientific, Teaneck, NJ.
• [25] Rangan, S. (2011). Generalized approximate message passing for estimation with random linear mixing. In IEEE Intl. Symp. on Inform. Theory (St. Perersbourg), August 2011. IEEE, Piscataway, NJ.
• [26] Rangan, S., Fletcher, A. K. and Goyal, V. K. (2009). Asymptotic analysis of map estimation via the replica method and applications to compressed sensing. In Neural Information Processing Systems (NIPS). Vancouver.
• [27] Schniter, P. (2010). Turbo reconstruction of structured sparse signals. In Proceedings of the Conference on Information Sciences and Systems. Princeton, NJ.
• [28] Schniter, P. and Rangan, S. (2012). Compressive phase retrieval via generalized approximate message passing. In Communication, Control, and Computing (Allerton), 2012 50th Annual Allerton Conference on 815–822. IEEE, Piscataway, NJ.
• [29] Tao, T. and Vu, V. (2012). Random matrices: The Universality phenomenon for Wigner ensembles. Available at arXiv:1202.0068.
• [30] Thouless, D. J., Anderson, P. W. and Palmer, R. G. (1977). Solution of “solvable model of a spin glass.” Philosophical Magazine 35 593–601.
• [31] Vershik, A. M. and Sporyshev, P. V. (1992). Asymptotic behavior of the number of faces of random polyhedra and the neighborliness problem. Selecta Math. Soviet. 11 181–201. Selected translations.