The Annals of Statistics

Impact of regularization on spectral clustering

Antony Joseph and Bin Yu

Full-text: Open access

Abstract

The performance of spectral clustering can be considerably improved via regularization, as demonstrated empirically in Amini et al. [Ann. Statist. 41 (2013) 2097–2122]. Here, we provide an attempt at quantifying this improvement through theoretical analysis. Under the stochastic block model (SBM), and its extensions, previous results on spectral clustering relied on the minimum degree of the graph being sufficiently large for its good performance. By examining the scenario where the regularization parameter $\tau$ is large, we show that the minimum degree assumption can potentially be removed. As a special case, for an SBM with two blocks, the results require the maximum degree to be large (grow faster than $\log n$) as opposed to the minimum degree. More importantly, we show the usefulness of regularization in situations where not all nodes belong to well-defined clusters. Our results rely on a ‘bias-variance’-like trade-off that arises from understanding the concentration of the sample Laplacian and the eigengap as a function of the regularization parameter. As a byproduct of our bounds, we propose a data-driven technique DKest (standing for estimated Davis–Kahan bounds) for choosing the regularization parameter. This technique is shown to work well through simulations and on a real data set.

Article information

Source
Ann. Statist., Volume 44, Number 4 (2016), 1765-1791.

Dates
Received: July 2014
Revised: January 2016
First available in Project Euclid: 7 July 2016

Permanent link to this document
https://projecteuclid.org/euclid.aos/1467894715

Digital Object Identifier
doi:10.1214/16-AOS1447

Mathematical Reviews number (MathSciNet)
MR3519940

Zentralblatt MATH identifier
1357.62229

Subjects
Primary: 62F12: Asymptotic properties of estimators
Secondary: 62H99: None of the above, but in this section

Keywords
Spectral clustering regularization network analysis community detection stochastic block model

Citation

Joseph, Antony; Yu, Bin. Impact of regularization on spectral clustering. Ann. Statist. 44 (2016), no. 4, 1765--1791. doi:10.1214/16-AOS1447. https://projecteuclid.org/euclid.aos/1467894715


Export citation

References

  • [1] Adamic, L. A. and Glance, N. (2005). The political blogosphere and the 2004 US election: Divided they blog. In Proceedings of the 3rd International Workshop on Link Discovery 36–43. ACM, New York.
  • [2] Amini, A. A., Chen, A., Bickel, P. J. and Levina, E. (2013). Pseudo-likelihood methods for community detection in large sparse networks. Ann. Statist. 41 2097–2122.
  • [3] Awasthi, P. and Sheffet, O. (2012). Improved spectral-norm bounds for clustering. In Approximation, Randomization, and Combinatorial Optimization. Lecture Notes in Computer Science 7408 37–49. Springer, Heidelberg.
  • [4] Belkin, M. and Niyogi, P. (2003). Laplacian eigenmaps for dimensionality reduction and data representation. Neural Comput. 15 1373–1396.
  • [5] Bhatia, R. (1997). Matrix Analysis. Graduate Texts in Mathematics 169. Springer, New York.
  • [6] Bickel, P. J. and Chen, A. (2009). A nonparametric view of network models and Newman–Girvan and other modularities. Proc. Natl. Acad. Sci. USA 106 21068–21073.
  • [7] Chaudhuri, K., Chung, F. and Tsiatas, A. Spectral clustering of graphs with general degrees in the extended planted partition model. J. Mach. Learn. Res. 2012 1–23.
  • [8] Chen, A., Amini, A., Bickel, P. and Levina, L. (2012). Fitting community models to large sparse networks. In Joint Statistical Meetings, San Diego.
  • [9] Dhillon, I. S. (2001). Co-clustering documents and words using bipartite spectral graph partitioning. In Proc. Seventh ACM SIGKDD Inter. Conf. on Know. Disc. and Data Mining 269–274. ACM, New York.
  • [10] Fishkind, D. E., Sussman, D. L., Tang, M., Vogelstein, J. T. and Priebe, C. E. (2013). Consistent adjacency-spectral partitioning for the stochastic block model when the model parameters are unknown. SIAM J. Matrix Anal. Appl. 34 23–39.
  • [11] Hagen, L. and Kahng, A. B. (1992). New spectral methods for ratio cut partitioning and clustering. IEEE Trans. Comput.-Aided Des. Integr. Circuits Syst. 11 1074–1085.
  • [12] Holland, P. W., Laskey, K. B. and Leinhardt, S. (1983). Stochastic blockmodels: First steps. Soc. Netw. 5 109–137.
  • [13] Joseph, A. and Yu, B. (2016). Supplement to “Impact of regularization on spectral clustering.” DOI:10.1214/16-AOS1447SUPP.
  • [14] Karrer, B. and Newman, M. E. J. (2011). Stochastic blockmodels and community structure in networks. Phys. Rev. E (3) 83 016107.
  • [15] Kumar, A. and Kannan, R. (2010). Clustering with spectral norm and the $k$-means algorithm. In 2010 IEEE 51st Annual Symposium on Foundations of Computer Science FOCS 2010 299–308. IEEE Computer Soc., Los Alamitos, CA.
  • [16] Kwok, T. C., Lau, L. C., Lee, Y. T., Oveis Gharan, S. and Trevisan, L. (2013). Improved Cheeger’s inequality: Analysis of spectral partitioning algorithms through higher order spectral gap. In STOC’13—Proceedings of the 2013 ACM Symposium on Theory of Computing 11–20. ACM, New York.
  • [17] Le, C. M. and Vershynin, R. (2015). Concentration and regularization of random graphs. Available at arXiv:1506.00669.
  • [18] Mackey, L., Jordan, M. I., Chen, R. Y., Farrell, B. and Tropp, J. A. (2012). Matrix concentration inequalities via the method of exchangeable pairs. Available at arXiv:1201.6002.
  • [19] McSherry, F. (2001). Spectral partitioning of random graphs. In 42nd IEEE Symposium on Foundations of Computer Science (Las Vegas, NV, 2001) 529–537. IEEE Computer Soc., Los Alamitos, CA.
  • [20] Newman, M. E. and Girvan, M. (2004). Finding and evaluating community structure in networks. Phys. Rev. E 69 026113.
  • [21] Ng, A. Y., Jordan, M. I., Weiss, Y. et al. (2002). On spectral clustering: Analysis and an algorithm. Adv. Neural Inf. Process. Syst. 2 849–856.
  • [22] Oliveira, R. I. (2009). Concentration of the adjacency matrix and of the Laplacian in random graphs with independent edges. Available at arXiv:0911.0600.
  • [23] Qin, T. and Rohe, K. (2013). Regularized spectral clustering under the degree-corrected stochastic blockmodel. Available at arXiv:1309.4111.
  • [24] Rohe, K., Chatterjee, S. and Yu, B. (2011). Spectral clustering and the high-dimensional stochastic blockmodel. Ann. Statist. 39 1878–1915.
  • [25] Shi, J. and Malik, J. (2000). Normalized cuts and image segmentation. IEEE Trans. Pattern Anal. Mach. Intell. 22 888–905.
  • [26] Sussman, D. L., Tang, M., Fishkind, D. E. and Priebe, C. E. (2012). A consistent adjacency spectral embedding for stochastic blockmodel graphs. J. Amer. Statist. Assoc. 107 1119–1128.
  • [27] von Luxburg, U. (2007). A tutorial on spectral clustering. Stat. Comput. 17 395–416.

Supplemental materials

  • Supplementary Material: Supplement to “Impact of regularization on spectral clustering”. The supplementary file contains the proof of the claims in the paper that were not included in the main body.