The Annals of Applied Probability

When can the two-armed bandit algorithm be trusted?

Damien Lamberton, Gilles Pagès, and Pierre Tarrès

Full-text: Open access


We investigate the asymptotic behavior of one version of the so-called two-armed bandit algorithm. It is an example of stochastic approximation procedure whose associated ODE has both a repulsive and an attractive equilibrium, at which the procedure is noiseless. We show that if the gain parameter is constant or goes to 0 not too fast, the algorithm does fall in the noiseless repulsive equilibrium with positive probability, whereas it always converges to its natural attractive target when the gain parameter goes to zero at some appropriate rates depending on the parameters of the model. We also elucidate the behavior of the constant step algorithm when the step goes to 0. Finally, we highlight the connection between the algorithm and the Polya urn. An application to asset allocation is briefly described.

Article information

Ann. Appl. Probab., Volume 14, Number 3 (2004), 1424-1454.

First available in Project Euclid: 13 July 2004

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62L20: Stochastic approximation
Secondary: 93C40: Adaptive control 91E40: Memory and learning [See also 68T05] 68T05: Learning and adaptive systems [See also 68Q32, 91E40] 91B32: Resource and cost allocation

Two-armed bandit algorithm stochastic approximation learning automata Pólya urn asset allocation


Lamberton, Damien; Pagès, Gilles; Tarrès, Pierre. When can the two-armed bandit algorithm be trusted?. Ann. Appl. Probab. 14 (2004), no. 3, 1424--1454. doi:10.1214/105051604000000350.

Export citation


  • Baldi, P., Mazliak, L. and Priouret, P. (2000). Martingales et chaînes de Markov. Hermann, 2ē éd. Paris, or (2002). Martingales and Markov Chains. Chapman and Hall/CRC, London.
  • Benaïm, M. and Hirsch, M. W. (1996). Asymptotic pseudo-trajectories, chain recurrent flows and stochastic approximations. J. Dynam. Differential Equations 8 141–176.
  • Brandière, O. and Duflo, M. (1996). Les algorithmes stochastiques contournent-ils les pièges? Ann. Inst. H. Poincaré Probab. Statist. 32 395–427.
  • Dacunha-Castelle, D. and Duflo, M. (1983). Probabilités et Statistiques II. Problèmes à temps mobile. Masson, Paris, or (1986). Probability and Statistics II. Springer, New York.
  • Duflo, M. (1996). Algorithmes Stochastiques. Springer, Berlin.
  • Fort, J. C. and Pagès, G. (2002). Decreasing step stochastic algorithms: A.s. behaviour of weighted empirical measures. Monte Carlo Methods Appl. 8 221–320.
  • Hall, P. and Heyde, C. C. (1980). Martingale Limit Theory and Its Application. Academic Press, New York.
  • Kushner, H. J. and Clark, D. S. (1978). Stochastic Approximation for Constrained and Unconstrained Systems. Springer, New York.
  • Lazarev, V. A. (1992). Convergence of stochastic approximation procedures in the case of a regression equation with several roots.
  • Problemy Pederachi Informatsii
  • Narendra, K. S. and Thathachar, M. A. L. (1974). Learning automata–-a survey. IEEE Trans. Systems Man. Cybernetics S.M.C-4 323–334.
  • Narendra, K. S. and Thathachar, M. A. L. (1989). Learning Automata–-An Introduction. Prentice-Hall, Englewood Cliffs, NJ.
  • Niang, M. (1999). Algorithme de Narendra et application à l'allocation d'actifs. Rapport de stage de DEA, Olympia Capital Management et Univ. Marne-la-Vallée, France.
  • Norman, M. F. (1968). On linear models with two absorbing barriers. J. Math. Psych. 5 225–241.
  • Pemantle, R. (1990). Nonconvergence to unstable points in urn models and stochastic approximations. Ann. Probab. 18 698–712.
  • Shapiro, I. J. and Narendra, K. S. (1969). Use of stochastic automata for parameter self-optimization with multi-modal perfomance criteria. IEEE Trans. Syst. Sci. and Cybern. SSC-5 352–360.\goodbreak
  • Tarrès, P. (2000). Pièges répulsifs. C. R. Acad. Sci. Paris Sér. I 330 125–130.
  • Tarrès, P. (2001). Piēges des Algorithmes stochastiques et marches aléatoires renforcées par sommets. Thèse de l'ENS Cachan, France.