The Annals of Applied Probability

Ergodic theory for controlled Markov chains with stationary inputs

Abstract

Consider a stochastic process $\boldsymbol{X}$ on a finite state space $\mathsf{X}=\{1,\dots,d\}$. It is conditionally Markov, given a real-valued “input process” $\boldsymbol{\zeta}$. This is assumed to be small, which is modeled through the scaling, $\zeta_{t}=\varepsilon\zeta^{1}_{t},\qquad0\le\varepsilon\le1,$ where $\boldsymbol{\zeta}^{1}$ is a bounded stationary process. The following conclusions are obtained, subject to smoothness assumptions on the controlled transition matrix and a mixing condition on $\boldsymbol{\zeta}$:

(i) A stationary version of the process is constructed, that is coupled with a stationary version of the Markov chain $\boldsymbol{X}^{\bullet}$ obtained with $\boldsymbol{\zeta} \equiv0$. The triple $(\boldsymbol{X} ,\boldsymbol{X}^{\bullet},\boldsymbol{\zeta} )$ is a jointly stationary process satisfying

$\mathsf{P}\{X(t)\neq X^{\bullet}(t)\}=O(\varepsilon).$ Moreover, a second-order Taylor-series approximation is obtained:

$\mathsf{P}\{X(t)=i\}=\mathsf{P}\{X^{\bullet}(t)=i\}+\varepsilon^{2}\pi^{(2)}(i)+o(\varepsilon^{2}),\qquad1\le i\le d,$ with an explicit formula for the vector $\pi^{(2)}\in\mathbb{R}^{d}$.

(ii) For any $m\ge1$ and any function $f:\{1,\dots,d\}\times\mathbb{R}\to\mathbb{R}^{m}$, the stationary stochastic process $Y(t)=f(X(t),\zeta(t))$ has a power spectral density $\mathrm{S}_{f}$ that admits a second-order Taylor series expansion: A function $\mathrm{S}^{(2)}_{f}:[-\pi,\pi]\to\mathbb{C}^{m\times m}$ is constructed such that

$\mathrm{S}_{f}(\theta)=\mathrm{S}^{\bullet}_{f}(\theta)+\varepsilon^{2}\mathrm{S}^{(2)}_{f}(\theta)+o(\varepsilon^{2}),\qquad\theta\in[-\pi,\pi ]$ in which the first term is the power spectral density obtained with $\varepsilon=0$. An explicit formula for the function $\mathrm{S}^{(2)}_{f}$ is obtained, based in part on the bounds in (i).

The results are illustrated with two general examples: mean field games, and a version of the timing channel of Anantharam and Verdu.

Article information

Source
Ann. Appl. Probab., Volume 28, Number 1 (2018), 79-111.

Dates
Revised: April 2017
First available in Project Euclid: 3 March 2018

Permanent link to this document
https://projecteuclid.org/euclid.aoap/1520046084

Digital Object Identifier
doi:10.1214/17-AAP1300

Mathematical Reviews number (MathSciNet)
MR3770873

Zentralblatt MATH identifier
06873680

Citation

Chen, Yue; Bušić, Ana; Meyn, Sean. Ergodic theory for controlled Markov chains with stationary inputs. Ann. Appl. Probab. 28 (2018), no. 1, 79--111. doi:10.1214/17-AAP1300. https://projecteuclid.org/euclid.aoap/1520046084