Central limit theorems for stochastic gradient descent with averaging for stable manifolds

Abstract

In this article, we establish new central limit theorems for Ruppert-Polyak averaged stochastic gradient descent schemes. Compared to previous work we do not assume that convergence occurs to an isolated attractor but instead allow convergence to a stable manifold. On the stable manifold the target function is constant and the oscillations of the iterates in the tangential direction may be significantly larger than the ones in the normal direction. We still recover a central limit theorem for the averaged scheme in the normal direction with the same rates as in the case of isolated attractors. In the setting where the magnitude of the random perturbation is of constant order, our research covers step-sizes ${\mathit{\gamma }}_n=C_{\mathit{\gamma }}{n}^{-\mathit{\gamma }}$with $C_{\mathit{\gamma }}>0$ and $\mathit{\gamma }\in (\frac{3}{4},1)$. In particular, we show that the beneficial effect of averaging prevails in more general situations.