The Annals of Applied Probability

Ergodic theory for controlled Markov chains with stationary inputs

Yue Chen, Ana Bušić, and Sean Meyn

Full-text: Access denied (no subscription detected)

We're sorry, but we are unable to provide you with the full text of this article because we are not able to identify you as a subscriber. If you have a personal subscription to this journal, then please login. If you are already logged in, then you may need to update your profile to register your subscription. Read more about accessing full-text


Consider a stochastic process $\boldsymbol{X}$ on a finite state space $\mathsf{X}=\{1,\dots,d\}$. It is conditionally Markov, given a real-valued “input process” $\boldsymbol{\zeta}$. This is assumed to be small, which is modeled through the scaling, \[\zeta_{t}=\varepsilon\zeta^{1}_{t},\qquad0\le\varepsilon\le1,\] where $\boldsymbol{\zeta}^{1}$ is a bounded stationary process. The following conclusions are obtained, subject to smoothness assumptions on the controlled transition matrix and a mixing condition on $\boldsymbol{\zeta}$:

(i) A stationary version of the process is constructed, that is coupled with a stationary version of the Markov chain $\boldsymbol{X}^{\bullet}$ obtained with $\boldsymbol{\zeta} \equiv0$. The triple $(\boldsymbol{X} ,\boldsymbol{X}^{\bullet},\boldsymbol{\zeta} )$ is a jointly stationary process satisfying

\[\mathsf{P}\{X(t)\neq X^{\bullet}(t)\}=O(\varepsilon).\] Moreover, a second-order Taylor-series approximation is obtained:

\[\mathsf{P}\{X(t)=i\}=\mathsf{P}\{X^{\bullet}(t)=i\}+\varepsilon^{2}\pi^{(2)}(i)+o(\varepsilon^{2}),\qquad1\le i\le d,\] with an explicit formula for the vector $\pi^{(2)}\in\mathbb{R}^{d}$.

(ii) For any $m\ge1$ and any function $f:\{1,\dots,d\}\times\mathbb{R}\to\mathbb{R}^{m}$, the stationary stochastic process $Y(t)=f(X(t),\zeta(t))$ has a power spectral density $\mathrm{S}_{f}$ that admits a second-order Taylor series expansion: A function $\mathrm{S}^{(2)}_{f}:[-\pi,\pi]\to\mathbb{C}^{m\times m}$ is constructed such that

\[\mathrm{S}_{f}(\theta)=\mathrm{S}^{\bullet}_{f}(\theta)+\varepsilon^{2}\mathrm{S}^{(2)}_{f}(\theta)+o(\varepsilon^{2}),\qquad\theta\in[-\pi,\pi ]\] in which the first term is the power spectral density obtained with $\varepsilon=0$. An explicit formula for the function $\mathrm{S}^{(2)}_{f}$ is obtained, based in part on the bounds in (i).

The results are illustrated with two general examples: mean field games, and a version of the timing channel of Anantharam and Verdu.

Article information

Ann. Appl. Probab., Volume 28, Number 1 (2018), 79-111.

Received: June 2016
Revised: April 2017
First available in Project Euclid: 3 March 2018

Permanent link to this document

Digital Object Identifier

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 60J20: Applications of Markov chains and discrete-time Markov processes on general state spaces (social mobility, learning theory, industrial processes, etc.) [See also 90B30, 91D10, 91D35, 91E40]
Secondary: 60G10: Stationary processes 68M20: Performance evaluation; queueing; scheduling [See also 60K25, 90Bxx] 94A15: Information theory, general [See also 62B10, 81P94]

Controlled Markov chain ergodic theory information theory


Chen, Yue; Bušić, Ana; Meyn, Sean. Ergodic theory for controlled Markov chains with stationary inputs. Ann. Appl. Probab. 28 (2018), no. 1, 79--111. doi:10.1214/17-AAP1300.

Export citation


  • [1] Adlakha, S., Johari, R., Weintraub, G. and Goldsmith, A. (2009). Oblivious equilibrium: An approximation to large population dynamic games with concave utility. In 2009 International Conference on Game Theory for Networks (GameNets’ 09) 68–69.
  • [2] Anantharam, V. and Verdu, S. (2006). Bits through queues. IEEE Trans. Inform. Theory 42 4–18.
  • [3] Caines, P. E. (1988). Linear Stochastic Systems. Wiley, New York.
  • [4] Chen, Y. (2016). Markovian demand dispatch design for virtual energy storage to support renewable energy integration. Ph.D. thesis, Univ. Florida, Gainesville, FL, USA.
  • [5] Chen, Y., Bušić, A. and Meyn, S. (2017). Estimation and control of quality of service in demand dispatch. IEEE Trans. Smart Grid. To appear.
  • [6] Chen, Y., Bušić, A. and Meyn, S. P. (2017). State estimation for the individual and the population in mean field control with application to demand dispatch. IEEE Trans. Automat. Control 62 1138–1149.
  • [7] Chen, Y., Bušić, A. and Meyn, S. (2014). Individual risk in mean field control with application to automated demand response. In 53rd IEEE Conference on Decision and Control 6425–6432.
  • [8] Cover, T. M. and Thomas, J. A. (1991). Elements of Information Theory. Wiley, New York.
  • [9] Dembo, A. and Zeitouni, O. (1998). Large Deviations Techniques and Applications, 2nd ed. Applications of Mathematics (New York) 38. Springer, New York.
  • [10] Foss, S. and Konstantopoulos, T. (2004). An overview of some stochastic stability methods. J. Oper. Res. Soc. Japan 47 275–303.
  • [11] Froyland, G. and González-Tokman, C. (2016). Stability and approximation of invariant measures of Markov chains in random environments. Stoch. Dyn. 16 Art. ID 1650003.
  • [12] Huang, M., Caines, P. E. and Malhamé, R. P. (2007). Large-population cost-coupled LQG problems with nonuniform agents: Individual-mass behavior and decentralized $\epsilon$-Nash equilibria. IEEE Trans. Automat. Control 52 1560–1571.
  • [13] Huang, M., Malhame, R. P. and Caines, P. E. (2006). Nash certainty equivalence in large population stochastic dynamic games: Connections with the physics of interacting particle systems. In Proceedings of the 45th IEEE Conference on Decision and Control 4921-4926.
  • [14] Krylov, N. V., Lipster, R. S. and Novikov, A. A. (1985). Kalman filter for Markov processes. In Statistics and Control of Stochastic Processes (Moscow, 1984) 197–213. Optimization Software, New York.
  • [15] Lasry, J.-M. and Lions, P.-L. (2006). Jeux à champ moyen. I. Le cas stationnaire. C. R. Math. Acad. Sci. Paris 343 619–625.
  • [16] Ma, Z., Callaway, D. S. and Hiskens, I. A. (2013). Decentralized charging control of large populations of plug-in electric vehicles. IEEE Trans. Control Syst. Technol. 21 67–78.
  • [17] Mathieu, J. L. (2012). Modeling, Analysis, and Control of Demand Response Resources. Ph.D. thesis, Univ. California, Berkeley.
  • [18] Meyn, S. P., Barooah, P., Bušić, A., Chen, Y. and Ehren, J. (2015). Ancillary service to the grid using intelligent deferrable loads. IEEE Trans. Automat. Control 60 2847–2862.
  • [19] Schweitzer, P. J. (1968). Perturbation theory and finite Markov chains. J. Appl. Probab. 5 401–413.
  • [20] Weintraub, G. Y., Benkard, C. L. and Van Roy, B. (2005). Oblivious equilibrium: A mean field approximation for large-scale dynamic games. In NIPS 1489–1496.
  • [21] Yin, G. G. and Zhang, Q. (2005). Discrete-Time Markov Chains: Two-Time-Scale Methods and Applications. Applications of Mathematics (New York): Stochastic Modelling and Applied Probability 55. Springer, New York.