Open Access
2023 Central limit theorems for stochastic gradient descent with averaging for stable manifolds
Steffen Dereich, Sebastian Kassing
Author Affiliations +
Electron. J. Probab. 28: 1-48 (2023). DOI: 10.1214/23-EJP947


In this article, we establish new central limit theorems for Ruppert-Polyak averaged stochastic gradient descent schemes. Compared to previous work we do not assume that convergence occurs to an isolated attractor but instead allow convergence to a stable manifold. On the stable manifold the target function is constant and the oscillations of the iterates in the tangential direction may be significantly larger than the ones in the normal direction. We still recover a central limit theorem for the averaged scheme in the normal direction with the same rates as in the case of isolated attractors. In the setting where the magnitude of the random perturbation is of constant order, our research covers step-sizes γn=Cγnγwith Cγ>0 and γ(34,1). In particular, we show that the beneficial effect of averaging prevails in more general situations.

Funding Statement

Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy EXC 2044–390685587, Mathematics Münster: Dynamics–Geometry–Structure.


The authors would like to thank an anonymous referee for his valuable comments.


Download Citation

Steffen Dereich. Sebastian Kassing. "Central limit theorems for stochastic gradient descent with averaging for stable manifolds." Electron. J. Probab. 28 1 - 48, 2023.


Received: 19 December 2019; Accepted: 16 April 2023; Published: 2023
First available in Project Euclid: 28 April 2023

MathSciNet: MR4580893
zbMATH: 07707073
arXiv: 1912.09187
MathSciNet: MR4529085
Digital Object Identifier: 10.1214/23-EJP947

Primary: 62L20
Secondary: 60J05 , 65C05

Keywords: deep learning , Robbins-Monro , Ruppert-Polyak average , stable manifold , stochastic approximation

Vol.28 • 2023
Back to Top