## The Annals of Applied Probability

- Ann. Appl. Probab.
- Volume 28, Number 1 (2018), 79-111.

### Ergodic theory for controlled Markov chains with stationary inputs

Yue Chen, Ana Bušić, and Sean Meyn

#### Abstract

Consider a stochastic process $\boldsymbol{X}$ on a finite state space $\mathsf{X}=\{1,\dots,d\}$. It is conditionally Markov, given a real-valued “input process” $\boldsymbol{\zeta}$. This is assumed to be small, which is modeled through the scaling, \[\zeta_{t}=\varepsilon\zeta^{1}_{t},\qquad0\le\varepsilon\le1,\] where $\boldsymbol{\zeta}^{1}$ is a bounded stationary process. The following conclusions are obtained, subject to smoothness assumptions on the controlled transition matrix and a mixing condition on $\boldsymbol{\zeta}$:

(i) A stationary version of the process is constructed, that is coupled with a stationary version of the Markov chain $\boldsymbol{X}^{\bullet}$ obtained with $\boldsymbol{\zeta} \equiv0$. The triple $(\boldsymbol{X} ,\boldsymbol{X}^{\bullet},\boldsymbol{\zeta} )$ is a jointly stationary process satisfying

\[\mathsf{P}\{X(t)\neq X^{\bullet}(t)\}=O(\varepsilon).\] Moreover, a second-order Taylor-series approximation is obtained:

\[\mathsf{P}\{X(t)=i\}=\mathsf{P}\{X^{\bullet}(t)=i\}+\varepsilon^{2}\pi^{(2)}(i)+o(\varepsilon^{2}),\qquad1\le i\le d,\] with an explicit formula for the vector $\pi^{(2)}\in\mathbb{R}^{d}$.

(ii) For any $m\ge1$ and any function $f:\{1,\dots,d\}\times\mathbb{R}\to\mathbb{R}^{m}$, the stationary stochastic process $Y(t)=f(X(t),\zeta(t))$ has a power spectral density $\mathrm{S}_{f}$ that admits a second-order Taylor series expansion: A function $\mathrm{S}^{(2)}_{f}:[-\pi,\pi]\to\mathbb{C}^{m\times m}$ is constructed such that

\[\mathrm{S}_{f}(\theta)=\mathrm{S}^{\bullet}_{f}(\theta)+\varepsilon^{2}\mathrm{S}^{(2)}_{f}(\theta)+o(\varepsilon^{2}),\qquad\theta\in[-\pi,\pi ]\] in which the first term is the power spectral density obtained with $\varepsilon=0$. An explicit formula for the function $\mathrm{S}^{(2)}_{f}$ is obtained, based in part on the bounds in (i).

The results are illustrated with two general examples: mean field games, and a version of the timing channel of Anantharam and Verdu.

#### Article information

**Source**

Ann. Appl. Probab., Volume 28, Number 1 (2018), 79-111.

**Dates**

Received: June 2016

Revised: April 2017

First available in Project Euclid: 3 March 2018

**Permanent link to this document**

https://projecteuclid.org/euclid.aoap/1520046084

**Digital Object Identifier**

doi:10.1214/17-AAP1300

**Mathematical Reviews number (MathSciNet)**

MR3770873

**Zentralblatt MATH identifier**

06873680

**Subjects**

Primary: 60J20: Applications of Markov chains and discrete-time Markov processes on general state spaces (social mobility, learning theory, industrial processes, etc.) [See also 90B30, 91D10, 91D35, 91E40]

Secondary: 60G10: Stationary processes 68M20: Performance evaluation; queueing; scheduling [See also 60K25, 90Bxx] 94A15: Information theory, general [See also 62B10, 81P94]

**Keywords**

Controlled Markov chain ergodic theory information theory

#### Citation

Chen, Yue; Bušić, Ana; Meyn, Sean. Ergodic theory for controlled Markov chains with stationary inputs. Ann. Appl. Probab. 28 (2018), no. 1, 79--111. doi:10.1214/17-AAP1300. https://projecteuclid.org/euclid.aoap/1520046084