The Annals of Applied Probability

When can the two-armed bandit algorithm be trusted?

Damien Lamberton, Gilles Pagès, and Pierre Tarrès

Full-text: Open access

Abstract

We investigate the asymptotic behavior of one version of the so-called two-armed bandit algorithm. It is an example of stochastic approximation procedure whose associated ODE has both a repulsive and an attractive equilibrium, at which the procedure is noiseless. We show that if the gain parameter is constant or goes to 0 not too fast, the algorithm does fall in the noiseless repulsive equilibrium with positive probability, whereas it always converges to its natural attractive target when the gain parameter goes to zero at some appropriate rates depending on the parameters of the model. We also elucidate the behavior of the constant step algorithm when the step goes to 0. Finally, we highlight the connection between the algorithm and the Polya urn. An application to asset allocation is briefly described.

Article information

Source
Ann. Appl. Probab., Volume 14, Number 3 (2004), 1424-1454.

Dates
First available in Project Euclid: 13 July 2004

Permanent link to this document
https://projecteuclid.org/euclid.aoap/1089736291

Digital Object Identifier
doi:10.1214/105051604000000350

Mathematical Reviews number (MathSciNet)
MR2071429

Zentralblatt MATH identifier
1048.62079

Subjects
Primary: 62L20: Stochastic approximation
Secondary: 93C40: Adaptive control 91E40: Memory and learning [See also 68T05] 68T05: Learning and adaptive systems [See also 68Q32, 91E40] 91B32: Resource and cost allocation

Keywords
Two-armed bandit algorithm stochastic approximation learning automata Pólya urn asset allocation

Citation

Lamberton, Damien; Pagès, Gilles; Tarrès, Pierre. When can the two-armed bandit algorithm be trusted?. Ann. Appl. Probab. 14 (2004), no. 3, 1424--1454. doi:10.1214/105051604000000350. https://projecteuclid.org/euclid.aoap/1089736291


Export citation

References

  • Baldi, P., Mazliak, L. and Priouret, P. (2000). Martingales et chaînes de Markov. Hermann, 2ē éd. Paris, or (2002). Martingales and Markov Chains. Chapman and Hall/CRC, London.
  • Benaïm, M. and Hirsch, M. W. (1996). Asymptotic pseudo-trajectories, chain recurrent flows and stochastic approximations. J. Dynam. Differential Equations 8 141–176.
  • Brandière, O. and Duflo, M. (1996). Les algorithmes stochastiques contournent-ils les pièges? Ann. Inst. H. Poincaré Probab. Statist. 32 395–427.
  • Dacunha-Castelle, D. and Duflo, M. (1983). Probabilités et Statistiques II. Problèmes à temps mobile. Masson, Paris, or (1986). Probability and Statistics II. Springer, New York.
  • Duflo, M. (1996). Algorithmes Stochastiques. Springer, Berlin.
  • Fort, J. C. and Pagès, G. (2002). Decreasing step stochastic algorithms: A.s. behaviour of weighted empirical measures. Monte Carlo Methods Appl. 8 221–320.
  • Hall, P. and Heyde, C. C. (1980). Martingale Limit Theory and Its Application. Academic Press, New York.
  • Kushner, H. J. and Clark, D. S. (1978). Stochastic Approximation for Constrained and Unconstrained Systems. Springer, New York.
  • Lazarev, V. A. (1992). Convergence of stochastic approximation procedures in the case of a regression equation with several roots.
  • Problemy Pederachi Informatsii
  • Narendra, K. S. and Thathachar, M. A. L. (1974). Learning automata–-a survey. IEEE Trans. Systems Man. Cybernetics S.M.C-4 323–334.
  • Narendra, K. S. and Thathachar, M. A. L. (1989). Learning Automata–-An Introduction. Prentice-Hall, Englewood Cliffs, NJ.
  • Niang, M. (1999). Algorithme de Narendra et application à l'allocation d'actifs. Rapport de stage de DEA, Olympia Capital Management et Univ. Marne-la-Vallée, France.
  • Norman, M. F. (1968). On linear models with two absorbing barriers. J. Math. Psych. 5 225–241.
  • Pemantle, R. (1990). Nonconvergence to unstable points in urn models and stochastic approximations. Ann. Probab. 18 698–712.
  • Shapiro, I. J. and Narendra, K. S. (1969). Use of stochastic automata for parameter self-optimization with multi-modal perfomance criteria. IEEE Trans. Syst. Sci. and Cybern. SSC-5 352–360.\goodbreak
  • Tarrès, P. (2000). Pièges répulsifs. C. R. Acad. Sci. Paris Sér. I 330 125–130.
  • Tarrès, P. (2001). Piēges des Algorithmes stochastiques et marches aléatoires renforcées par sommets. Thèse de l'ENS Cachan, France.