Bernoulli

• Bernoulli
• Volume 25, Number 1 (2019), 112-147.

Verifiable conditions for the irreducibility and aperiodicity of Markov chains by analyzing underlying deterministic models

Abstract

We consider Markov chains that obey the following general non-linear state space model: $\Phi_{k+1}=F(\Phi_{k},\alpha(\Phi_{k},U_{k+1}))$ where the function $F$ is $C^{1}$ while $\alpha$ is typically discontinuous and $\{U_{k}:k\in\mathbb{Z}_{>0}\}$ is an independent and identically distributed process. We assume that for all $x$, the random variable $\alpha(x,U_{1})$ admits a density $p_{x}$ such that $(x,w)\mapsto p_{x}(w)$ is lower semi-continuous.

We generalize and extend previous results that connect properties of the underlying deterministic control model to provide conditions for the chain to be $\varphi$-irreducible and aperiodic. By building on those results, we show that if a rank condition on the controllability matrix is satisfied for all $x$, there is equivalence between the existence of a globally attracting state for the control model and $\varphi$-irreducibility of the Markov chain. Additionally, under the same rank condition on the controllability matrix, we prove that there is equivalence between the existence of a steadily attracting state and the $\varphi$-irreducibility and aperiodicity of the chain. The notion of steadily attracting state is new. We additionally derive practical conditions by showing that the rank condition on the controllability matrix needs to be verified only at a globally attracting state (resp. steadily attracting state) for the chain to be a $\varphi$-irreducible $T$-chain (resp. $\varphi$-irreducible aperiodic $T$-chain).

Those results hold under considerably weaker assumptions on the model than previous ones that would require $(x,u)\mapsto F(x,\alpha(x,u))$ to be $C^{\infty}$ (while it can be discontinuous here). Additionally the establishment of a necessary and sufficient condition on the control model for the $\varphi$-irreducibility and aperiodicity without a structural assumption on the control set is novel – even for Markov chains where $(x,u)\mapsto F(x,\alpha(x,u))$ is $C^{\infty}$.

We illustrate that the conditions are easy to verify on a non-trivial and non-artificial example of Markov chain arising in the context of adaptive stochastic search algorithms to optimize continuous functions in a black-box scenario.

Article information

Source
Bernoulli, Volume 25, Number 1 (2019), 112-147.

Dates
Revised: May 2017
First available in Project Euclid: 12 December 2018

https://projecteuclid.org/euclid.bj/1544605241

Digital Object Identifier
doi:10.3150/17-BEJ970

Mathematical Reviews number (MathSciNet)
MR3892314

Zentralblatt MATH identifier
07007202

Citation

Chotard, Alexandre; Auger, Anne. Verifiable conditions for the irreducibility and aperiodicity of Markov chains by analyzing underlying deterministic models. Bernoulli 25 (2019), no. 1, 112--147. doi:10.3150/17-BEJ970. https://projecteuclid.org/euclid.bj/1544605241

References

• [1] Auger, A. and Hansen, N. (2016). Linear convergence of comparison-based step-size adaptive randomized search via stability of Markov chains. SIAM J. Optim. 26 1589–1624.
• [2] Chan, K.S. (1990). Deterministic Stability, Stochastic Stability and Ergodicity, Appendix, Volume Non-linear Time Series: A Dynamical System Approach. Oxford: Oxford Univ. Press.
• [3] Chotard, A. and Auger, A. (2015). Verifiable conditions for irreducibility, aperiodicity and T-chain property of a general Markov Chain. Preprint. Available at arXiv:1508.01644.
• [4] Cline, D.B. (2011). Irreducibility and continuity assumptions for positive operators with application to threshold garch time series models. Adv. in Appl. Probab. 49–76.
• [5] Cline, D.B.H. and Pu, H.H. (1998). Verifying irreducibility and continuity of a nonlinear time series. Statist. Probab. Lett. 40 139–148.
• [6] Glasmachers, T., Schaul, T., Sun, Y., Wierstra, D. and Schmidhuber, J. (2010). Exponential natural evolution strategies. In Genetic and Evolutionary Computation Conference (GECCO 2010) 393–400. New York: ACM Press.
• [7] Hansen, N., Arnold, D.V. and Auger, A. (2015). Evolution strategies. In Springer Handbook of Computational Intelligence 871–898. Berlin: Springer.
• [8] Jakubczyk, B. and Sontag, E.D. (1990). Controllability of nonlinear discrete time systems: A lie-algebraic approach. SIAM J. Control Optim. 28 1–33.
• [9] Meyn, S.P. and Caines, P.E. (1989). Stochastic controllability and stochastic Lyapunov functions with applications to adaptive and nonlinear systems. In Stochastic Differential Systems 235–257. Berlin: Springer.
• [10] Meyn, S.P. and Caines, P.E. (1991). Asymptotic behavior of stochastic systems possessing Markovian realizations. SIAM J. Control Optim. 29 535–561.
• [11] Meyn, S.P. and Tweedie, R.L. (1993). Markov Chains and Stochastic Stability, 2nd ed. Cambridge: Cambridge Univ. Press.
• [12] Nummelin, E. (1984). General Irreducible Markov Chains and Nonnegative Operators. Cambridge Tracts in Mathematics 83. Cambridge: Cambridge Univ. Press.
• [13] Ollivier, Y., Arnold, L., Auger, A. and Hansen, N. (2017). Information-geometric optimization algorithms: A unifying picture via invariance principles. J. Mach. Learn. Res. 18 Paper No. 18, 65.
• [14] Pham, D.T. (1986). The mixing property of bilinear and generalised random coefficient autoregressive models. Stochastic Process. Appl. 23 291–300.
• [15] Puterman, M.L. (1994). Markov Decision Processes: Discrete Stochastic Dynamic Programming, 1st ed. New York: Wiley.
• [16] Rechenberg, I. (1973). Evolutionstrategie: Optimierung Technischer Systeme Nach Prinzipien der Biologischen Evolution. Stuttgart: Frommann-Holzboog Verlag.
• [17] Sternberg, S. (1964). Lectures on Differential Geometry. New York: Prentice Hall, 390 pp.