Abstract
Modern statistical applications often involve minimizing an objective function that may be nonsmooth and/or nonconvex. This paper focuses on a broad Bregman-surrogate algorithm framework including the local linear approximation, mirror descent, iterative thresholding, DC programming and many others as particular instances. The recharacterization via generalized Bregman functions enables us to construct suitable error measures and establish global convergence rates for nonconvex and nonsmooth objectives in possibly high dimensions. For sparse learning problems with a composite objective, under some regularity conditions, the obtained estimators as the surrogate’s fixed points, though not necessarily local minimizers, enjoy provable statistical guarantees, and the sequence of iterates can be shown to approach the statistical truth within the desired accuracy geometrically fast. The paper also studies how to design adaptive momentum based accelerations without assuming convexity or smoothness by carefully controlling stepsize and relaxation parameters.
Funding Statement
The work is partially supported by the National Science Foundation.
Acknowledgments
The authors would like to thank the Editor, Associate Editor and referees for suggestions that significantly improved the paper.
Funding Statement
The work is partially supported by the National Science Foundation.
Acknowledgments
The authors would like to thank the Editor, Associate Editor and referees for suggestions that significantly improved the paper.
Citation
Yiyuan She. Zhifeng Wang. Jiuwu Jin. "Analysis of generalized Bregman surrogate algorithms for nonsmooth nonconvex statistical learning." Ann. Statist. 49 (6) 3434 - 3459, December 2021. https://doi.org/10.1214/21-AOS2090
Information