### Perturbation analysis of inhomogeneous finite Markov chains

#### Abstract

In this paper we provide a perturbation analysis of finite time-inhomogeneous Markov processes. We derive closed-form representations for the derivative of the transition probability at time t, with t > 0. Elaborating on this result, we derive simple gradient estimators for transient performance characteristics either taken at some fixed point in time t, or for the integrated performance over a time interval [0 , t]. Bounds for transient performance sensitivities are presented as well. Eventually, we identify a structural property of the derivative of the generator matrix of a Markov chain that leads to a significant simplification of the estimators.

#### Article information

Source
Adv. in Appl. Probab., Volume 48, Number 1 (2016), 255-273.

Dates
First available in Project Euclid: 8 March 2016

https://projecteuclid.org/euclid.aap/1457466165

Mathematical Reviews number (MathSciNet)
MR3473577

Zentralblatt MATH identifier
1337.60181

Subjects
Primary: 60J27: Continuous-time Markov processes on discrete state spaces
Secondary: 65C05: Monte Carlo methods

#### Citation

Heidergott, Bernd; Leahu, Haralambie; Löpker, Andreas; Pflug, Georg. Perturbation analysis of inhomogeneous finite Markov chains. Adv. in Appl. Probab. 48 (2016), no. 1, 255--273. https://projecteuclid.org/euclid.aap/1457466165

#### References

• Abdalla, N. and Boucherie, R. J. (2002). Blocking probabilities in mobile communications networks with time-varying rates and redialing subscribers. Ann. Operat. Res. 112, 15–34.
• Abramov, V. and Liptser, R. (2004). On the existence of limiting distributions for time-nonhomogeneous countable Markov process. Queueing Systems 46, 353–361.
• Andreychenko, A., Crouzen, P. and Wolf, V. (2011). On-the-fly uniformization of time-inhomogeneous infinite Markov population models. In Proc. QAPL EPTCS, pp. 1–15. Available at http://published.eptcs.org
• Artalejo, J. R. and Gómez-Corral, A. (2008). Retrial Queueing Systems. Springer, Berlin.
• Banasiak, J. and Arlotti, L. (2006). Perturbations of Positive Semigroups with Applications. Springer, London.
• Bellman, R. (1997). Introduction to Matrix Analysis, 2nd edn. SIAM, Philadelphia, PA.
• Brémaud, P. (1992). Maximal coupling and rare perturbation sensitivity analysis. Queueing Systems Theory Appl. 11, 307–333.
• Brémaud, P. and Vázquez-Abad, F. J. (1992). On the pathwise computation of derivatives with respect to the rate of a point process: the phantom RPA method. Queueing Systems Theory Appl. 10, 249–269.
• Cao, X.-R. (2007). Stochastic Learning and Optimization: A Sensitivity-Based Approach. Springer, New York.
• Dollard, J. D. and Friedman, C. N. (1978). On strong product integration. J. Funct. Anal. 28, 309–354.
• Falin, G. (1990). A survey of retrial queues. Queueing Systems Theory Appl. 7, 127–167.
• Falin, G. I. and Templeton, J. G. C. (1997). Retrial Queues. Chapman & Hall, London.
• Feldman, Z., Mandelbaum, A., Massey, W. A. and Whitt, W. (2008). Staffing of time-varying queues to achieve time-stable performance. Manag. Sci. 54, 324–338.
• Fu, M. C. (2006). Gradient estimation. In Handbook on Operations Research and Management Science: Simulation, eds S. G. Henderson and B. L. Nelson, Elsevier, Amsterdam, pp. 575–616.
• Fu, M. and Hu, J.-Q. (1997). Conditional Monte Carlo: Gradient Estimation and Optimization Applications. Kluwer, Boston, MA.
• Gans, N., Koole, G. and Mandelbaum, A. (2003). Telephone call centers: tutorial, review, and research prospects. Manufacturing Service Operat. Manag. 5, 79–141.
• Gill, R. D. and Johansen, S. (1990). A survey of product-integration with a view toward application in survival analysis. Ann. Statist. 18, 1501–1555.
• Glasserman, P. (1991). Gradient Estimation via Perturbation Analysis. Kluwer, Boston, MA.
• Glasserman, P. and Gong, W.-B. (1990). Smoothed perturbation analysis for a class of discrete-event systems. IEEE Trans. Automatic Control 35, 1218–1230.
• Gong, W.-B. and Ho, Y.-C. (1987). Smoothed (conditional) perturbation analysis of discrete event dynamical systems. IEEE Trans. Automatic Control 32, 858–866.
• Green, L. V., Kolesar, P. J. and Whitt, W. (2007). Coping with time-varying demand when setting staffing requirements for a service system. Production Operat. Manag. 16, 13–39.
• Heidergott, B. and Hordijk, A. (2003). Taylor series expansions for stationary Markov chains. Adv. Appl. Prob. 35, 1046–1070. (Correction: 36 (2004), 1300.)
• Heidergott, B. and Leahu, H. (2010). Weak differentiability of product measures. Math. Operat. Res. 35, 27–51.
• Heidergott, B. and Vázquez-Abad, F. J. (2008). Measure-valued differentiation for Markov chains. J. Optimization Theory Appl. 136, 187–209.
• Ho, Y.-C. and Cao, X.-R. (1991). Perturbation Analysis of Discrete Event Dynamic Systems. Kluwer, Boston, MA.
• Johansen, S. (1986). Product integrals and Markov processes. CWI Newslett. 12, 3–13.
• Kato, T. (1984). Perturbation Theory for Linear Operators, 2nd edn. Springer, Berlin.
• Kulkarni, V. G. and Liang, H. M. (1997). Retrial queues revisited. In Frontiers in Queueing, CRC, Boca Raton, FL, pp. 19–34.
• Massey, W. A. (1985). Asymptotic analysis of the time dependent $M/M/1$ queue. Math. Operat. Res. 10, 305–327.
• Massey, W. A. (2002). The analysis of queues with time-varying rates for telecommunication models. Telecommun. Systems 21, 173–204.
• Massey, W. A. and Whitt, W. (1998). Uniform acceleration expansions for Markov chains with time-varying rates. Ann. Appl. Prob. 8, 1130–1155.
• Otte, P. (1999). An integral formula for section determinants of semi-groups of linear operators. J. Phys. A 32, 3793–3803.
• Pflug, G. C. (1996). Optimization of Stochastic Models. Kluwer, Boston, MA.
• Rubinstein, R. Y. (1992). Sensitivity analysis of discrete event systems by the `push out' method. Ann. Operat. Res. 39, 229–250.
• Rubinstein, R. Y. and Melamed, B. (1998). Modern Simulation and Modeling. John Wiley, New York.
• Rubinstein, R. Y. and Kroese, D. P. (2008). Simulation and the Monte Carlo Method, 2nd edn. John Wiley, Hoboken, NJ.
• Van Dijk, N. M. (1992). Uniformization for nonhomogeneous Markov chains. Operat. Res. Lett. 12, 283–291.
• Van Loan, C. (1977). The sensitivity of the matrix exponential. SIAM J. Numer. Anal. 14, 971–981.
• Vázquez-Abad, F. J. (1999). Strong points of weak convergence: a study using RPA gradient estimation for automatic learning. Automatica 35, 1255–1274.
• Yang, T. and Templeton, J. G. C. (1987). A survey on retrial queues. Queueing Systems 2, 201–233. (Correction: 4 (1989), 94.)
• Yin, G. and Zhang, Q. (1989). Continuous Time Markov Chains and Applications. Springer, New York.
• Zeifman, A. and Korotysheva, A. (2011). Weak ergodicity of M$_{t}/$M$_{t}/N/N+R$ queue. Pliska Stud. Math. Bulgar. 20, 243–254.