Markov Chain - Markov Chains - Steady-state Analysis and Limiting Distributions

Steady-state Analysis and Limiting Distributions

If the Markov chain is a time-homogeneous Markov chain, so that the process is described by a single, time-independent matrix, then the vector is called a stationary distribution (or invariant measure) if its entries are non-negative and sum to 1 and if it satisfies

An irreducible chain has a stationary distribution if and only if all of its states are positive recurrent. In that case, π is unique and is related to the expected return time:

where is the normalizing constant. Further, if the chain is both irreducible and aperiodic, then for any i and j,

Note that there is no assumption on the starting distribution; the chain converges to the stationary distribution regardless of where it begins. Such π is called the equilibrium distribution of the chain.

If a chain has more than one closed communicating class, its stationary distributions will not be unique (consider any closed communicating class in the chain; each one will have its own unique stationary distribution . Extending these distributions to the overall chain, setting all values to zero outside the communication class, yields that the set of invariant measures of the original chain is the set of all convex combinations of the 's). However, if a state j is aperiodic, then

and for any other state i, let f_ij be the probability that the chain ever visits state j if it starts at i,

If a state i is periodic with period k > 1 then the limit

does not exist, although the limit

does exist for every integer r.