A Potential Theoretic Proof of a Theorem of Derman and Veinott

Abstract

The purpose of this note is to provide alternative proofs of results of Derman and Veinott [1]. The method of proof uses the potential theory for Markov chains developed by Kemeny and Snell in [2], [3], [4], [5]. In [2], Kemeny and Snell treat transient and recurrent chains separately, whereas Derman and Veinott consider chains having one positive recurrent class, $C, (0 \epsilon C)$, and a set of transient states, $T$, with $T \cup C = \{0, 1, 2, \cdots\}$. This difference does not cause any great difficulty. Kemeny and Snell also assume in [2] that their recurrent chains are noncyclic. The recurrent class, $C$, will therefore be assumed noncyclic in this note, the extension to the cyclic case being handled in the usual way. By relabelling the states, one may write the transition matrix, $P$, as \begin{equation*}\tag{1.1}P = \begin{pmatrix}P^1 R 0 \\ R Q\end{pmatrix}\end{equation*} where $P^1$ is the transition matrix of the recurrent class and $Q$ is the (substochastic) transition matrix of the set of transient states. In this note, the notation will be that of Derman and Veinott [1] and all references to the work of Kemeny and Snell will be to [2]. In matrix notation, the Derman-Veinott equation is \begin{equation*}\tag{1.2} (I - P)v = \omega^{\ast}\end{equation*} where $\omega{^\ast} = \omega - g\mathbf{1}, \omega = (\omega_0, \omega_1, \omega_2, \cdots)^T$ is a known vector, $\mathbf{1} = (1, 1, 1, \cdots)^T, v = (v_0, v_1, v_2, \cdots)^T$ is an unknown vector and $g$ is an unknown constant.