The Annals of Statistics

A Robbins–Monro procedure for estimation in semiparametric regression models

Bernard Bercu and Philippe Fraysse

Full-text: Open access


This paper is devoted to the parametric estimation of a shift together with the nonparametric estimation of a regression function in a semiparametric regression model. We implement a very efficient and easy to handle Robbins–Monro procedure. On the one hand, we propose a stochastic algorithm similar to that of Robbins–Monro in order to estimate the shift parameter. A preliminary evaluation of the regression function is not necessary to estimate the shift parameter. On the other hand, we make use of a recursive Nadaraya–Watson estimator for the estimation of the regression function. This kernel estimator takes into account the previous estimation of the shift parameter. We establish the almost sure convergence for both Robbins–Monro and Nadaraya–Watson estimators. The asymptotic normality of our estimates is also provided. Finally, we illustrate our semiparametric estimation procedure on simulated and real data.

Article information

Ann. Statist., Volume 40, Number 2 (2012), 666-693.

First available in Project Euclid: 17 May 2012

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62G05: Estimation
Secondary: 62G20: Asymptotic properties

Semiparametric estimation estimation of a shift estimation of a regression function asymptotic properties


Bercu, Bernard; Fraysse, Philippe. A Robbins–Monro procedure for estimation in semiparametric regression models. Ann. Statist. 40 (2012), no. 2, 666--693. doi:10.1214/12-AOS969.

Export citation


  • [1] Bercu, B. and Portier, B. (2008). Kernel density estimation and goodness-of-fit test in adaptive tracking. SIAM J. Control Optim. 47 2440–2457.
  • [2] Bickel, P. J., Klaassen, C. A. J., Ritov, Y. and Wellner, J. A. (1998). Efficient and Adaptive Estimation for Semiparametric Models. Springer, New York.
  • [3] Castillo, I. and Loubes, J. M. (2009). Estimation of the distribution of random shifts deformation. Math. Methods Statist. 18 21–42.
  • [4] Chen, H. F., Lei, G. and Gao, A. J. (1988). Convergence and robustness of the Robbins–Monro algorithm truncated at randomly varying bounds. Stochastic Process. Appl. 27 217–231.
  • [5] Choi, E., Hall, P. and Rousson, V. (2000). Data sharpening methods for bias reduction in nonparametric regression. Ann. Statist. 28 1339–1355.
  • [6] Clifford, G. D., Azuaje, F. and McSharry, P. (2006). Advanced Methods and Tools for ECG Data Analysis. Artech House, Boston.
  • [7] Dalalyan, A. S., Golubev, G. K. and Tsybakov, A. B. (2006). Penalized maximum likelihood and semiparametric second-order efficiency. Ann. Statist. 34 169–201.
  • [8] Devroye, L. and Lugosi, G. (2001). Combinatorial Methods in Density Estimation. Springer, New York.
  • [9] Duflo, M. (1997). Random Iterative Models. Applications of Mathematics (New York) 34. Springer, Berlin.
  • [10] Fabian, V. (1973). Asymptotically efficient stochastic approximation; the RM case. Ann. Statist. 1 486–495.
  • [11] Gamboa, F., Loubes, J.-M. and Maza, E. (2007). Semi-parametric estimation of shifts. Electron. J. Stat. 1 616–640.
  • [12] Gapoškin, V. F. and Krasulina, T. P. (1974). The law of the iterated logarithm in stochastic approximation processes. Teor. Verojatnost. i Primenen. 19 879–886.
  • [13] Hall, P. and Heyde, C. C. (1980). Martingale Limit Theory and Its Application. Academic Press [Harcourt Brace Jovanovich Publishers], New York.
  • [14] Hall, P. and Huang, L.-S. (2001). Nonparametric kernel regression subject to monotonicity constraints. Ann. Statist. 29 624–647.
  • [15] Härdle, W. (1984). A law of the iterated logarithm for nonparametric regression function estimators. Ann. Statist. 12 624–635.
  • [16] Härdle, W., Janssen, P. and Serfling, R. (1988). Strong uniform consistency rates for estimators of conditional functionals. Ann. Statist. 16 1428–1449.
  • [17] Härdle, W. and Marron, J. S. (1990). Semiparametric comparison of regression curves. Ann. Statist. 18 63–89.
  • [18] Härdle, W. and Tsybakov, A. B. (1988). Robust nonparametric regression with simultaneous scale curve estimation. Ann. Statist. 16 120–135.
  • [19] Hürtgen, H. and Gervini, D. (2009). Semiparametric shape-invariant models for periodic data. J. Appl. Stat. 36 1055–1065.
  • [20] Kneip, A. and Engel, J. (1995). Model estimation in nonlinear regression under shape invariance. Ann. Statist. 23 551–570.
  • [21] Kneip, A. and Gasser, T. (1988). Convergence and consistency results for self-modeling nonlinear regression. Ann. Statist. 16 82–112.
  • [22] Kushner, H. J. and Yin, G. G. (2003). Stochastic Approximation and Recursive Algorithms and Applications. Applications of Mathematics (New York) 35. Springer, New York.
  • [23] Lassen, K. and Friis-Christensen, E. (1995). Variability of the solar cycle length during the past five centuries and the apparent association with terrestrial climate. J. Atmospheric and Terrestrial Physics 57 835–845.
  • [24] Lawton, W. H., Sylvestre, E. A. and Maggio, M. S. (1972). Self modeling nonlinear regression. Technometrics 14 513–532.
  • [25] Lelong, J. (2008). Almost sure convergence for randomly truncated stochastic algorithms under verifiable conditions. Statist. Probab. Lett. 78 2632–2636.
  • [26] McDonald, J. A. (1986). Periodic smoothing of time series. SIAM J. Sci. Statist. Comput. 7 665–688.
  • [27] Mokkadem, A. and Pelletier, M. (2007). A companion for the Kiefer–Wolfowitz–Blum stochastic approximation algorithm. Ann. Statist. 35 1749–1772.
  • [28] Nadaraja, È. A. (1964). On a regression estimate. Teor. Verojatnost. i Primenen. 9 157–159.
  • [29] Noda, K. (1976). Estimation of a regression function by the Parzen kernel-type density estimators. Ann. Inst. Statist. Math. 28 221–234.
  • [30] Parzen, E. (1962). On estimation of a probability density function and mode. Ann. Math. Statist. 33 1065–1076.
  • [31] Pelletier, M. (1998). On the almost sure asymptotic behaviour of stochastic algorithms. Stochastic Process. Appl. 78 217–244.
  • [32] Robbins, H. and Monro, S. (1951). A stochastic approximation method. Ann. Math. Statist. 22 400–407.
  • [33] Robbins, H. and Siegmund, D. (1971). A convergence theorem for non negative almost supermartingales and some applications. In Optimizing Methods in Statistics (Proc. Sympos., Ohio State Univ., Columbus, Ohio, 1971) 233–257. Academic Press, New York.
  • [34] Schuster, E. F. (1972). Joint asymptotic distribution of the estimated regression function at a finite number of distinct points. Ann. Math. Statist. 43 84–88.
  • [35] Stone, C. J. (1975). Adaptive maximum likelihood estimators of a location parameter. Ann. Statist. 3 267–284.
  • [36] Trigano, T., Isserles, U. and Ritov, Y. (2011). Semiparametric curve alignment and shift density estimation for biological data. IEEE Trans. Signal Process. 59 1970–1984.
  • [37] Tsybakov, A. B. (2004). Introduction à L’estimation Non-Paramétrique. Mathématiques and Applications (Berlin) 41. Springer, Berlin.
  • [38] Vimond, M. (2010). Efficient estimation for a subclass of shape invariant models. Ann. Statist. 38 1885–1912.
  • [39] Wang, Y. and Brown, M. B. (1996). A flexible model for human circadian rhythms. Biometrics 52 588–596.
  • [40] Watson, G. S. (1964). Smooth regression analysis. Sankhyā Ser. A 26 359–372.