Verifiable conditions for the irreducibility and aperiodicity of Markov chains by analyzing underlying deterministic models

Abstract

We consider Markov chains that obey the following general non-linear state space model: $\Phi_{k+1}=F(\Phi_{k},\alpha(\Phi_{k},U_{k+1}))$ where the function $F$ is $C^{1}$ while $\alpha$ is typically discontinuous and $\{U_{k}:k\in\mathbb{Z}_{>0}\}$ is an independent and identically distributed process. We assume that for all $x$, the random variable $\alpha(x,U_{1})$ admits a density $p_{x}$ such that $(x,w)\mapsto p_{x}(w)$ is lower semi-continuous.

We generalize and extend previous results that connect properties of the underlying deterministic control model to provide conditions for the chain to be $\varphi$-irreducible and aperiodic. By building on those results, we show that if a rank condition on the controllability matrix is satisfied for all $x$, there is equivalence between the existence of a globally attracting state for the control model and $\varphi$-irreducibility of the Markov chain. Additionally, under the same rank condition on the controllability matrix, we prove that there is equivalence between the existence of a steadily attracting state and the $\varphi$-irreducibility and aperiodicity of the chain. The notion of steadily attracting state is new. We additionally derive practical conditions by showing that the rank condition on the controllability matrix needs to be verified only at a globally attracting state (resp. steadily attracting state) for the chain to be a $\varphi$-irreducible $T$-chain (resp. $\varphi$-irreducible aperiodic $T$-chain).

Those results hold under considerably weaker assumptions on the model than previous ones that would require $(x,u)\mapsto F(x,\alpha(x,u))$ to be $C^{\infty}$ (while it can be discontinuous here). Additionally the establishment of a necessary and sufficient condition on the control model for the $\varphi$-irreducibility and aperiodicity without a structural assumption on the control set is novel – even for Markov chains where $(x,u)\mapsto F(x,\alpha(x,u))$ is $C^{\infty}$.

We illustrate that the conditions are easy to verify on a non-trivial and non-artificial example of Markov chain arising in the context of adaptive stochastic search algorithms to optimize continuous functions in a black-box scenario.