Open Access
2010 Optimal Open Loop Markov Decision Rules May Require Parametric Excitation
Roger Brockett
Commun. Inf. Syst. 10(4): 279-292 (2010).


We present here a general theory, and give a specific example, showing that there exist time invariant Markov decision problems, with no time variation in the model which, when optimized over an infinite interval, have optimal closed loop control laws that are time varying. Although similar behavior was observed much earlier for specific problems arising in chemical and aeronautical engineering, this work is not applicable to Markov decision problems because of the specific form of the constraints involving the action of the semigroup of stochastic matrices on the standard simplex and the bilinear structure that goes along with rate control for Markov processes. The results given here are especially interesting insofar as they are analogous to the optimal solutions of stochastic control problems associated with Carnot cycles. As in some earlier work, the conditions under which time varying controls are optimal are characterized in terms of the the second variation about a singular solution. In this case the second variation is expressible in terms of a kernel function and conditions under which the second variation is positive definite can be checked by determining if the transform of this kernel is positive real or not.


Download Citation

Roger Brockett. "Optimal Open Loop Markov Decision Rules May Require Parametric Excitation." Commun. Inf. Syst. 10 (4) 279 - 292, 2010.


Published: 2010
First available in Project Euclid: 24 November 2010

zbMATH: 1230.90108
MathSciNet: MR2737880

Rights: Copyright © 2010 International Press of Boston

Vol.10 • No. 4 • 2010
Back to Top