In this article, we establish new central limit theorems for Ruppert-Polyak averaged stochastic gradient descent schemes. Compared to previous work we do not assume that convergence occurs to an isolated attractor but instead allow convergence to a stable manifold. On the stable manifold the target function is constant and the oscillations of the iterates in the tangential direction may be significantly larger than the ones in the normal direction. We still recover a central limit theorem for the averaged scheme in the normal direction with the same rates as in the case of isolated attractors. In the setting where the magnitude of the random perturbation is of constant order, our research covers step-sizes with and . In particular, we show that the beneficial effect of averaging prevails in more general situations.
Funded by the Deutsche Forschungsgemeinschaft (DFG, German Research Foundation) under Germany’s Excellence Strategy EXC 2044–390685587, Mathematics Münster: Dynamics–Geometry–Structure.
The authors would like to thank an anonymous referee for his valuable comments.
"Central limit theorems for stochastic gradient descent with averaging for stable manifolds." Electron. J. Probab. 28 1 - 48, 2023. https://doi.org/10.1214/23-EJP947