We define the notion of a Markov strategy for the general optimal control problem where the index set is partially ordered. We prove that the supremum over all strategies is always attained by a Markov strategy if and only if the structure of the probability space is that of a Markov random field.
G. F. Lawler. R. J. Vanderbei. "Markov Strategies for Optimal Control Problems Indexed by a Partially Ordered Set." Ann. Probab. 11 (3) 642 - 647, August, 1983. https://doi.org/10.1214/aop/1176993508