The Annals of Applied Probability

Stochastic approximation, cooperative dynamics and supermodular games

Michel Benaïm and Mathieu Faure

Full-text: Open access


This paper considers a stochastic approximation algorithm, with decreasing step size and martingale difference noise. Under very mild assumptions, we prove the nonconvergence of this process toward a certain class of repulsive sets for the associated ordinary differential equation (ODE). We then use this result to derive the convergence of the process when the ODE is cooperative in the sense of Hirsch [SIAM J. Math. Anal. 16 (1985) 423–439]. In particular, this allows us to extend significantly the main result of Hofbauer and Sandholm [Econometrica 70 (2002) 2265–2294] on the convergence of stochastic fictitious play in supermodular games.

Article information

Ann. Appl. Probab., Volume 22, Number 5 (2012), 2133-2164.

First available in Project Euclid: 12 October 2012

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 62L20: Stochastic approximation 37C50: Approximate trajectories (pseudotrajectories, shadowing, etc.)
Secondary: 37C65: Monotone flows 91A12: Cooperative games

Stochastic approximation cooperative systems stochastic fictitious play supermodular games


Benaïm, Michel; Faure, Mathieu. Stochastic approximation, cooperative dynamics and supermodular games. Ann. Appl. Probab. 22 (2012), no. 5, 2133--2164. doi:10.1214/11-AAP816.

Export citation


  • Beggs, A. (2009). Learning in Bayesian games with binary actions. B. E. J. Theor. Econ. 9 Art. 33, 30.
  • Benaïm, M. (1996). A dynamical system approach to stochastic approximations. SIAM J. Control Optim. 34 437–472.
  • Benaïm, M. (1999). Dynamics of stochastic approximation algorithms. In Séminaire de Probabilités, XXXIII. Lecture Notes in Math. 1709 1–68. Springer, Berlin.
  • Benaïm, M. (2000). Convergence with probability one of stochastic approximation algorithms whose average is cooperative. Nonlinearity 13 601–616.
  • Benaïm, M. and Hirsch, M. W. (1995). Dynamics of Morse-Smale urn processes. Ergodic Theory Dynam. Systems 15 1005–1030.
  • Benaïm, M. and Hirsch, M. W. (1996). Asymptotic pseudotrajectories and chain recurrent flows, with applications. J. Dynam. Differential Equations 8 141–176.
  • Benaïm, M. and Hirsch, M. W. (1999a). Mixed equilibria and dynamical systems arising from fictitious play in perturbed games. Games Econom. Behav. 29 36–72.
  • Benaïm, M. and Hirsch, M. W. (1999b). Stochastic approximation algorithms with constant step size whose average is cooperative. Ann. Appl. Probab. 9 216–241.
  • Benveniste, A., Metivier, M. and Priouret, P. (1990). Stochastic Approximations and Adaptive Algorithms. Springer, Berlin.
  • Brandière, O. and Duflo, M. (1996). Les algorithmes stochastiques contournent-ils les pièges? Ann. Inst. Henri Poincaré Probab. Stat. 32 395–427.
  • Chow, Y. and Teicher, H. (1998). Probability: Independence, Interchangeability, Martingales. Springer, New York.
  • Conley, C. (1978). Isolated Invariant Sets and the Morse Index. CBMS Regional Conference Series in Mathematics 38. Amer. Math. Soc., Providence, RI.
  • Duflo, M. (1996). Algorithmes Stochastiques. Mathématiques & Applications (Berlin) [Mathematics & Applications] 23. Springer, Berlin.
  • Fudenberg, D. and Kreps, D. M. (1993). Learning mixed equilibria. Games Econom. Behav. 5 320–367.
  • Fudenberg, D. and Levine, D. K. (1998). The Theory of Learning in Games. MIT Press Series on Economic Learning and Social Evolution 2. MIT Press, Cambridge, MA.
  • Hall, P. and Heyde, C. (1980). Martingale Limit Theorem and Its Application. Academic Press, New York.
  • Harsanyi, J. C. (1973). Games with randomly disturbed payoffs: A new rationale for mixed-strategy equilibrium points. Internat. J. Game Theory 2 1–23.
  • Hirsch, M. W. (1985). Systems of differential equations that are competitive or cooperative. II. Convergence almost everywhere. SIAM J. Math. Anal. 16 423–439.
  • Hirsch, M. W. (1988). Systems of differential equations which are competitive or cooperative. III. Competing species. Nonlinearity 1 51–71.
  • Hirsch, M. W. (1999). Chain transitive sets for smooth strongly monotone dynamical systems. Dynam. Contin. Discrete Impuls. Systems 5 529–543.
  • Hirsch, M. W. and Smith, H. L. (2006). Asymptotically stable equilibria for monotone semiflows. Discrete Contin. Dyn. Syst. 14 385–398.
  • Hofbauer, J. and Sandholm, W. H. (2002). On the global convergence of stochastic fictitious play. Econometrica 70 2265–2294.
  • Jiang, J. F. (1991). Attractors in strongly monotone flows. J. Math. Anal. Appl. 162 210–222.
  • Kiefer, J. and Wolfowitz, J. (1952). Stochastic estimation of the maximum of a regression function. Ann. Math. Statist. 23 462–466.
  • Kushner, H. J. and Clark, D. S. (1978). Stochastic Approximation Methods for Constrained and Unconstrained Systems. Applied Mathematical Sciences 26. Springer, New York.
  • Kushner, H. J. and Whiting, P. A. (2004). Convergence of proportional-fair sharing algorithms under general conditions. IEEE Transactions on Wireless Communications 3 1250–1259.
  • Kushner, H. J. and Yin, G. G. (2003). Stochastic Approximation and Recursive Algorithms and Applications, 2nd ed. Applications of Mathematics (New York) 35. Springer, New York.
  • Ljung, L. (1977). Analysis of recursive stochastic algorithms. IEEE Trans. Automat. Control AC-22 551–575.
  • Milgrom, P. and Roberts, J. (1990). Rationalizability, learning, and equilibrium in games with strategic complementarities. Econometrica 58 1255–1277.
  • Pemantle, R. (1990). Nonconvergence to unstable points in urn models and stochastic approximations. Ann. Probab. 18 698–712.
  • Pemantle, R. (2007). A survey of random processes with reinforcement. Probab. Surv. 4 1–79.
  • Robbins, H. and Monro, S. (1951). A stochastic approximation method. Ann. Math. Statist. 22 400–407.
  • Sadeghi, A. A. (1998). Asymptotic behavior of self-organizing maps with nonuniform stimuli distribution. Ann. Appl. Probab. 8 281–299.
  • Tarrès, P. (2001). Pièges des algorithmes stochastiques et marches aléatoires renforcées par sommets. Ph.D. thesis, ENS Cachan.
  • Terescak, I. (1996). Dynamics of C 1 smooth strongly monotone discrete-time dynamical systems. Preprint.
  • Topkis, D. M. (1979). Equilibrium points in nonzero-sum $n$-person submodular games. SIAM J. Control Optim. 17 773–787.