Advances in Applied Probability

Perturbation analysis of inhomogeneous finite Markov chains

Bernd Heidergott, Haralambie Leahu, Andreas Löpker, and Georg Pflug

Full-text: Access denied (no subscription detected)

We're sorry, but we are unable to provide you with the full text of this article because we are not able to identify you as a subscriber. If you have a personal subscription to this journal, then please login. If you are already logged in, then you may need to update your profile to register your subscription. Read more about accessing full-text


In this paper we provide a perturbation analysis of finite time-inhomogeneous Markov processes. We derive closed-form representations for the derivative of the transition probability at time t, with t > 0. Elaborating on this result, we derive simple gradient estimators for transient performance characteristics either taken at some fixed point in time t, or for the integrated performance over a time interval [0 , t]. Bounds for transient performance sensitivities are presented as well. Eventually, we identify a structural property of the derivative of the generator matrix of a Markov chain that leads to a significant simplification of the estimators.

Article information

Adv. in Appl. Probab., Volume 48, Number 1 (2016), 255-273.

First available in Project Euclid: 8 March 2016

Permanent link to this document

Mathematical Reviews number (MathSciNet)

Zentralblatt MATH identifier

Primary: 60J27: Continuous-time Markov processes on discrete state spaces
Secondary: 65C05: Monte Carlo methods

Finite Markov process inhomogeneous Markov process sensitivity analysis infinitesimal generator


Heidergott, Bernd; Leahu, Haralambie; Löpker, Andreas; Pflug, Georg. Perturbation analysis of inhomogeneous finite Markov chains. Adv. in Appl. Probab. 48 (2016), no. 1, 255--273.

Export citation


  • Abdalla, N. and Boucherie, R. J. (2002). Blocking probabilities in mobile communications networks with time-varying rates and redialing subscribers. Ann. Operat. Res. 112, 15–34.
  • Abramov, V. and Liptser, R. (2004). On the existence of limiting distributions for time-nonhomogeneous countable Markov process. Queueing Systems 46, 353–361.
  • Andreychenko, A., Crouzen, P. and Wolf, V. (2011). On-the-fly uniformization of time-inhomogeneous infinite Markov population models. In Proc. QAPL EPTCS, pp. 1–15. Available at
  • Artalejo, J. R. and Gómez-Corral, A. (2008). Retrial Queueing Systems. Springer, Berlin.
  • Banasiak, J. and Arlotti, L. (2006). Perturbations of Positive Semigroups with Applications. Springer, London.
  • Bellman, R. (1997). Introduction to Matrix Analysis, 2nd edn. SIAM, Philadelphia, PA.
  • Brémaud, P. (1992). Maximal coupling and rare perturbation sensitivity analysis. Queueing Systems Theory Appl. 11, 307–333.
  • Brémaud, P. and Vázquez-Abad, F. J. (1992). On the pathwise computation of derivatives with respect to the rate of a point process: the phantom RPA method. Queueing Systems Theory Appl. 10, 249–269.
  • Cao, X.-R. (2007). Stochastic Learning and Optimization: A Sensitivity-Based Approach. Springer, New York.
  • Dollard, J. D. and Friedman, C. N. (1978). On strong product integration. J. Funct. Anal. 28, 309–354.
  • Falin, G. (1990). A survey of retrial queues. Queueing Systems Theory Appl. 7, 127–167.
  • Falin, G. I. and Templeton, J. G. C. (1997). Retrial Queues. Chapman & Hall, London.
  • Feldman, Z., Mandelbaum, A., Massey, W. A. and Whitt, W. (2008). Staffing of time-varying queues to achieve time-stable performance. Manag. Sci. 54, 324–338.
  • Fu, M. C. (2006). Gradient estimation. In Handbook on Operations Research and Management Science: Simulation, eds S. G. Henderson and B. L. Nelson, Elsevier, Amsterdam, pp. 575–616.
  • Fu, M. and Hu, J.-Q. (1997). Conditional Monte Carlo: Gradient Estimation and Optimization Applications. Kluwer, Boston, MA.
  • Gans, N., Koole, G. and Mandelbaum, A. (2003). Telephone call centers: tutorial, review, and research prospects. Manufacturing Service Operat. Manag. 5, 79–141.
  • Gill, R. D. and Johansen, S. (1990). A survey of product-integration with a view toward application in survival analysis. Ann. Statist. 18, 1501–1555.
  • Glasserman, P. (1991). Gradient Estimation via Perturbation Analysis. Kluwer, Boston, MA.
  • Glasserman, P. and Gong, W.-B. (1990). Smoothed perturbation analysis for a class of discrete-event systems. IEEE Trans. Automatic Control 35, 1218–1230.
  • Gong, W.-B. and Ho, Y.-C. (1987). Smoothed (conditional) perturbation analysis of discrete event dynamical systems. IEEE Trans. Automatic Control 32, 858–866.
  • Green, L. V., Kolesar, P. J. and Whitt, W. (2007). Coping with time-varying demand when setting staffing requirements for a service system. Production Operat. Manag. 16, 13–39.
  • Heidergott, B. and Hordijk, A. (2003). Taylor series expansions for stationary Markov chains. Adv. Appl. Prob. 35, 1046–1070. (Correction: 36 (2004), 1300.)
  • Heidergott, B. and Leahu, H. (2010). Weak differentiability of product measures. Math. Operat. Res. 35, 27–51.
  • Heidergott, B. and Vázquez-Abad, F. J. (2008). Measure-valued differentiation for Markov chains. J. Optimization Theory Appl. 136, 187–209.
  • Ho, Y.-C. and Cao, X.-R. (1991). Perturbation Analysis of Discrete Event Dynamic Systems. Kluwer, Boston, MA.
  • Johansen, S. (1986). Product integrals and Markov processes. CWI Newslett. 12, 3–13.
  • Kato, T. (1984). Perturbation Theory for Linear Operators, 2nd edn. Springer, Berlin.
  • Kulkarni, V. G. and Liang, H. M. (1997). Retrial queues revisited. In Frontiers in Queueing, CRC, Boca Raton, FL, pp. 19–34.
  • Massey, W. A. (1985). Asymptotic analysis of the time dependent $M/M/1$ queue. Math. Operat. Res. 10, 305–327.
  • Massey, W. A. (2002). The analysis of queues with time-varying rates for telecommunication models. Telecommun. Systems 21, 173–204.
  • Massey, W. A. and Whitt, W. (1998). Uniform acceleration expansions for Markov chains with time-varying rates. Ann. Appl. Prob. 8, 1130–1155.
  • Otte, P. (1999). An integral formula for section determinants of semi-groups of linear operators. J. Phys. A 32, 3793–3803.
  • Pflug, G. C. (1996). Optimization of Stochastic Models. Kluwer, Boston, MA.
  • Rubinstein, R. Y. (1992). Sensitivity analysis of discrete event systems by the `push out' method. Ann. Operat. Res. 39, 229–250.
  • Rubinstein, R. Y. and Melamed, B. (1998). Modern Simulation and Modeling. John Wiley, New York.
  • Rubinstein, R. Y. and Kroese, D. P. (2008). Simulation and the Monte Carlo Method, 2nd edn. John Wiley, Hoboken, NJ.
  • Van Dijk, N. M. (1992). Uniformization for nonhomogeneous Markov chains. Operat. Res. Lett. 12, 283–291.
  • Van Loan, C. (1977). The sensitivity of the matrix exponential. SIAM J. Numer. Anal. 14, 971–981.
  • Vázquez-Abad, F. J. (1999). Strong points of weak convergence: a study using RPA gradient estimation for automatic learning. Automatica 35, 1255–1274.
  • Yang, T. and Templeton, J. G. C. (1987). A survey on retrial queues. Queueing Systems 2, 201–233. (Correction: 4 (1989), 94.)
  • Yin, G. and Zhang, Q. (1989). Continuous Time Markov Chains and Applications. Springer, New York.
  • Zeifman, A. and Korotysheva, A. (2011). Weak ergodicity of M$_{t}/$M$_{t}/N/N+R$ queue. Pliska Stud. Math. Bulgar. 20, 243–254.