## The Annals of Applied Probability

### Ergodic theory for controlled Markov chains with stationary inputs

#### Abstract

Consider a stochastic process $\boldsymbol{X}$ on a finite state space $\mathsf{X}=\{1,\dots,d\}$. It is conditionally Markov, given a real-valued “input process” $\boldsymbol{\zeta}$. This is assumed to be small, which is modeled through the scaling, $\zeta_{t}=\varepsilon\zeta^{1}_{t},\qquad0\le\varepsilon\le1,$ where $\boldsymbol{\zeta}^{1}$ is a bounded stationary process. The following conclusions are obtained, subject to smoothness assumptions on the controlled transition matrix and a mixing condition on $\boldsymbol{\zeta}$:

(i) A stationary version of the process is constructed, that is coupled with a stationary version of the Markov chain $\boldsymbol{X}^{\bullet}$ obtained with $\boldsymbol{\zeta} \equiv0$. The triple $(\boldsymbol{X} ,\boldsymbol{X}^{\bullet},\boldsymbol{\zeta} )$ is a jointly stationary process satisfying

$\mathsf{P}\{X(t)\neq X^{\bullet}(t)\}=O(\varepsilon).$ Moreover, a second-order Taylor-series approximation is obtained:

$\mathsf{P}\{X(t)=i\}=\mathsf{P}\{X^{\bullet}(t)=i\}+\varepsilon^{2}\pi^{(2)}(i)+o(\varepsilon^{2}),\qquad1\le i\le d,$ with an explicit formula for the vector $\pi^{(2)}\in\mathbb{R}^{d}$.

(ii) For any $m\ge1$ and any function $f:\{1,\dots,d\}\times\mathbb{R}\to\mathbb{R}^{m}$, the stationary stochastic process $Y(t)=f(X(t),\zeta(t))$ has a power spectral density $\mathrm{S}_{f}$ that admits a second-order Taylor series expansion: A function $\mathrm{S}^{(2)}_{f}:[-\pi,\pi]\to\mathbb{C}^{m\times m}$ is constructed such that

$\mathrm{S}_{f}(\theta)=\mathrm{S}^{\bullet}_{f}(\theta)+\varepsilon^{2}\mathrm{S}^{(2)}_{f}(\theta)+o(\varepsilon^{2}),\qquad\theta\in[-\pi,\pi ]$ in which the first term is the power spectral density obtained with $\varepsilon=0$. An explicit formula for the function $\mathrm{S}^{(2)}_{f}$ is obtained, based in part on the bounds in (i).

The results are illustrated with two general examples: mean field games, and a version of the timing channel of Anantharam and Verdu.

#### Article information

Source
Ann. Appl. Probab., Volume 28, Number 1 (2018), 79-111.

Dates
Revised: April 2017
First available in Project Euclid: 3 March 2018

Permanent link to this document
https://projecteuclid.org/euclid.aoap/1520046084

Digital Object Identifier
doi:10.1214/17-AAP1300

Mathematical Reviews number (MathSciNet)
MR3770873

Zentralblatt MATH identifier
06873680

#### Citation

Chen, Yue; Bušić, Ana; Meyn, Sean. Ergodic theory for controlled Markov chains with stationary inputs. Ann. Appl. Probab. 28 (2018), no. 1, 79--111. doi:10.1214/17-AAP1300. https://projecteuclid.org/euclid.aoap/1520046084

#### References

• [1] Adlakha, S., Johari, R., Weintraub, G. and Goldsmith, A. (2009). Oblivious equilibrium: An approximation to large population dynamic games with concave utility. In 2009 International Conference on Game Theory for Networks (GameNets’ 09) 68–69.
• [2] Anantharam, V. and Verdu, S. (2006). Bits through queues. IEEE Trans. Inform. Theory 42 4–18.
• [3] Caines, P. E. (1988). Linear Stochastic Systems. Wiley, New York.
• [4] Chen, Y. (2016). Markovian demand dispatch design for virtual energy storage to support renewable energy integration. Ph.D. thesis, Univ. Florida, Gainesville, FL, USA.
• [5] Chen, Y., Bušić, A. and Meyn, S. (2017). Estimation and control of quality of service in demand dispatch. IEEE Trans. Smart Grid. To appear.
• [6] Chen, Y., Bušić, A. and Meyn, S. P. (2017). State estimation for the individual and the population in mean field control with application to demand dispatch. IEEE Trans. Automat. Control 62 1138–1149.
• [7] Chen, Y., Bušić, A. and Meyn, S. (2014). Individual risk in mean field control with application to automated demand response. In 53rd IEEE Conference on Decision and Control 6425–6432.
• [8] Cover, T. M. and Thomas, J. A. (1991). Elements of Information Theory. Wiley, New York.
• [9] Dembo, A. and Zeitouni, O. (1998). Large Deviations Techniques and Applications, 2nd ed. Applications of Mathematics (New York) 38. Springer, New York.
• [10] Foss, S. and Konstantopoulos, T. (2004). An overview of some stochastic stability methods. J. Oper. Res. Soc. Japan 47 275–303.
• [11] Froyland, G. and González-Tokman, C. (2016). Stability and approximation of invariant measures of Markov chains in random environments. Stoch. Dyn. 16 Art. ID 1650003.
• [12] Huang, M., Caines, P. E. and Malhamé, R. P. (2007). Large-population cost-coupled LQG problems with nonuniform agents: Individual-mass behavior and decentralized $\epsilon$-Nash equilibria. IEEE Trans. Automat. Control 52 1560–1571.
• [13] Huang, M., Malhame, R. P. and Caines, P. E. (2006). Nash certainty equivalence in large population stochastic dynamic games: Connections with the physics of interacting particle systems. In Proceedings of the 45th IEEE Conference on Decision and Control 4921-4926.
• [14] Krylov, N. V., Lipster, R. S. and Novikov, A. A. (1985). Kalman filter for Markov processes. In Statistics and Control of Stochastic Processes (Moscow, 1984) 197–213. Optimization Software, New York.
• [15] Lasry, J.-M. and Lions, P.-L. (2006). Jeux à champ moyen. I. Le cas stationnaire. C. R. Math. Acad. Sci. Paris 343 619–625.
• [16] Ma, Z., Callaway, D. S. and Hiskens, I. A. (2013). Decentralized charging control of large populations of plug-in electric vehicles. IEEE Trans. Control Syst. Technol. 21 67–78.
• [17] Mathieu, J. L. (2012). Modeling, Analysis, and Control of Demand Response Resources. Ph.D. thesis, Univ. California, Berkeley.
• [18] Meyn, S. P., Barooah, P., Bušić, A., Chen, Y. and Ehren, J. (2015). Ancillary service to the grid using intelligent deferrable loads. IEEE Trans. Automat. Control 60 2847–2862.
• [19] Schweitzer, P. J. (1968). Perturbation theory and finite Markov chains. J. Appl. Probab. 5 401–413.
• [20] Weintraub, G. Y., Benkard, C. L. and Van Roy, B. (2005). Oblivious equilibrium: A mean field approximation for large-scale dynamic games. In NIPS 1489–1496.
• [21] Yin, G. G. and Zhang, Q. (2005). Discrete-Time Markov Chains: Two-Time-Scale Methods and Applications. Applications of Mathematics (New York): Stochastic Modelling and Applied Probability 55. Springer, New York.