Abstract
Consider a stochastic process $\boldsymbol{X}$ on a finite state space $\mathsf{X}=\{1,\dots,d\}$. It is conditionally Markov, given a real-valued “input process” $\boldsymbol{\zeta}$. This is assumed to be small, which is modeled through the scaling, \[\zeta_{t}=\varepsilon\zeta^{1}_{t},\qquad0\le\varepsilon\le1,\] where $\boldsymbol{\zeta}^{1}$ is a bounded stationary process. The following conclusions are obtained, subject to smoothness assumptions on the controlled transition matrix and a mixing condition on $\boldsymbol{\zeta}$:
(i) A stationary version of the process is constructed, that is coupled with a stationary version of the Markov chain $\boldsymbol{X}^{\bullet}$ obtained with $\boldsymbol{\zeta} \equiv0$. The triple $(\boldsymbol{X} ,\boldsymbol{X}^{\bullet},\boldsymbol{\zeta} )$ is a jointly stationary process satisfying
\[\mathsf{P}\{X(t)\neq X^{\bullet}(t)\}=O(\varepsilon).\] Moreover, a second-order Taylor-series approximation is obtained:
\[\mathsf{P}\{X(t)=i\}=\mathsf{P}\{X^{\bullet}(t)=i\}+\varepsilon^{2}\pi^{(2)}(i)+o(\varepsilon^{2}),\qquad1\le i\le d,\] with an explicit formula for the vector $\pi^{(2)}\in\mathbb{R}^{d}$.
(ii) For any $m\ge1$ and any function $f:\{1,\dots,d\}\times\mathbb{R}\to\mathbb{R}^{m}$, the stationary stochastic process $Y(t)=f(X(t),\zeta(t))$ has a power spectral density $\mathrm{S}_{f}$ that admits a second-order Taylor series expansion: A function $\mathrm{S}^{(2)}_{f}:[-\pi,\pi]\to\mathbb{C}^{m\times m}$ is constructed such that
\[\mathrm{S}_{f}(\theta)=\mathrm{S}^{\bullet}_{f}(\theta)+\varepsilon^{2}\mathrm{S}^{(2)}_{f}(\theta)+o(\varepsilon^{2}),\qquad\theta\in[-\pi,\pi ]\] in which the first term is the power spectral density obtained with $\varepsilon=0$. An explicit formula for the function $\mathrm{S}^{(2)}_{f}$ is obtained, based in part on the bounds in (i).
The results are illustrated with two general examples: mean field games, and a version of the timing channel of Anantharam and Verdu.
Citation
Yue Chen. Ana Bušić. Sean Meyn. "Ergodic theory for controlled Markov chains with stationary inputs." Ann. Appl. Probab. 28 (1) 79 - 111, February 2018. https://doi.org/10.1214/17-AAP1300
Information