Markov Decision Process - Problem

Problem

The core problem of MDPs is to find a "policy" for the decision maker: a function that specifies the action that the decision maker will choose when in state . Note that once a Markov decision process is combined with a policy in this way, this fixes the action for each state and the resulting combination behaves like a Markov chain.

The goal is to choose a policy that will maximize some cumulative function of the random rewards, typically the expected discounted sum over a potentially infinite horizon:

(where we choose )

where is the discount factor and satisfies . (For example, when the discount rate is r.) is typically close to 1.

Because of the Markov property, the optimal policy for this particular problem can indeed be written as a function of only, as assumed above.