• Bernoulli
  • Volume 25, Number 1 (2019), 112-147.

Verifiable conditions for the irreducibility and aperiodicity of Markov chains by analyzing underlying deterministic models

Alexandre Chotard and Anne Auger

Full-text: Access denied (no subscription detected)

We're sorry, but we are unable to provide you with the full text of this article because we are not able to identify you as a subscriber. If you have a personal subscription to this journal, then please login. If you are already logged in, then you may need to update your profile to register your subscription. Read more about accessing full-text


We consider Markov chains that obey the following general non-linear state space model: $\Phi_{k+1}=F(\Phi_{k},\alpha(\Phi_{k},U_{k+1}))$ where the function $F$ is $C^{1}$ while $\alpha$ is typically discontinuous and $\{U_{k}:k\in\mathbb{Z}_{>0}\}$ is an independent and identically distributed process. We assume that for all $x$, the random variable $\alpha(x,U_{1})$ admits a density $p_{x}$ such that $(x,w)\mapsto p_{x}(w)$ is lower semi-continuous.

We generalize and extend previous results that connect properties of the underlying deterministic control model to provide conditions for the chain to be $\varphi$-irreducible and aperiodic. By building on those results, we show that if a rank condition on the controllability matrix is satisfied for all $x$, there is equivalence between the existence of a globally attracting state for the control model and $\varphi$-irreducibility of the Markov chain. Additionally, under the same rank condition on the controllability matrix, we prove that there is equivalence between the existence of a steadily attracting state and the $\varphi$-irreducibility and aperiodicity of the chain. The notion of steadily attracting state is new. We additionally derive practical conditions by showing that the rank condition on the controllability matrix needs to be verified only at a globally attracting state (resp. steadily attracting state) for the chain to be a $\varphi$-irreducible $T$-chain (resp. $\varphi$-irreducible aperiodic $T$-chain).

Those results hold under considerably weaker assumptions on the model than previous ones that would require $(x,u)\mapsto F(x,\alpha(x,u))$ to be $C^{\infty}$ (while it can be discontinuous here). Additionally the establishment of a necessary and sufficient condition on the control model for the $\varphi$-irreducibility and aperiodicity without a structural assumption on the control set is novel – even for Markov chains where $(x,u)\mapsto F(x,\alpha(x,u))$ is $C^{\infty}$.

We illustrate that the conditions are easy to verify on a non-trivial and non-artificial example of Markov chain arising in the context of adaptive stochastic search algorithms to optimize continuous functions in a black-box scenario.

Article information

Bernoulli, Volume 25, Number 1 (2019), 112-147.

Received: September 2015
Revised: May 2017
First available in Project Euclid: 12 December 2018

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

aperiodicity controllability matrix deterministic control model Evolution Strategies globally attracting state irreducibility Markov chains $T$-chain


Chotard, Alexandre; Auger, Anne. Verifiable conditions for the irreducibility and aperiodicity of Markov chains by analyzing underlying deterministic models. Bernoulli 25 (2019), no. 1, 112--147. doi:10.3150/17-BEJ970.

Export citation


  • [1] Auger, A. and Hansen, N. (2016). Linear convergence of comparison-based step-size adaptive randomized search via stability of Markov chains. SIAM J. Optim. 26 1589–1624.
  • [2] Chan, K.S. (1990). Deterministic Stability, Stochastic Stability and Ergodicity, Appendix, Volume Non-linear Time Series: A Dynamical System Approach. Oxford: Oxford Univ. Press.
  • [3] Chotard, A. and Auger, A. (2015). Verifiable conditions for irreducibility, aperiodicity and T-chain property of a general Markov Chain. Preprint. Available at arXiv:1508.01644.
  • [4] Cline, D.B. (2011). Irreducibility and continuity assumptions for positive operators with application to threshold garch time series models. Adv. in Appl. Probab. 49–76.
  • [5] Cline, D.B.H. and Pu, H.H. (1998). Verifying irreducibility and continuity of a nonlinear time series. Statist. Probab. Lett. 40 139–148.
  • [6] Glasmachers, T., Schaul, T., Sun, Y., Wierstra, D. and Schmidhuber, J. (2010). Exponential natural evolution strategies. In Genetic and Evolutionary Computation Conference (GECCO 2010) 393–400. New York: ACM Press.
  • [7] Hansen, N., Arnold, D.V. and Auger, A. (2015). Evolution strategies. In Springer Handbook of Computational Intelligence 871–898. Berlin: Springer.
  • [8] Jakubczyk, B. and Sontag, E.D. (1990). Controllability of nonlinear discrete time systems: A lie-algebraic approach. SIAM J. Control Optim. 28 1–33.
  • [9] Meyn, S.P. and Caines, P.E. (1989). Stochastic controllability and stochastic Lyapunov functions with applications to adaptive and nonlinear systems. In Stochastic Differential Systems 235–257. Berlin: Springer.
  • [10] Meyn, S.P. and Caines, P.E. (1991). Asymptotic behavior of stochastic systems possessing Markovian realizations. SIAM J. Control Optim. 29 535–561.
  • [11] Meyn, S.P. and Tweedie, R.L. (1993). Markov Chains and Stochastic Stability, 2nd ed. Cambridge: Cambridge Univ. Press.
  • [12] Nummelin, E. (1984). General Irreducible Markov Chains and Nonnegative Operators. Cambridge Tracts in Mathematics 83. Cambridge: Cambridge Univ. Press.
  • [13] Ollivier, Y., Arnold, L., Auger, A. and Hansen, N. (2017). Information-geometric optimization algorithms: A unifying picture via invariance principles. J. Mach. Learn. Res. 18 Paper No. 18, 65.
  • [14] Pham, D.T. (1986). The mixing property of bilinear and generalised random coefficient autoregressive models. Stochastic Process. Appl. 23 291–300.
  • [15] Puterman, M.L. (1994). Markov Decision Processes: Discrete Stochastic Dynamic Programming, 1st ed. New York: Wiley.
  • [16] Rechenberg, I. (1973). Evolutionstrategie: Optimierung Technischer Systeme Nach Prinzipien der Biologischen Evolution. Stuttgart: Frommann-Holzboog Verlag.
  • [17] Sternberg, S. (1964). Lectures on Differential Geometry. New York: Prentice Hall, 390 pp.