Continuous-time Markov Process - Embedded Markov Chain

Embedded Markov Chain

One method of finding the stationary probability distribution, π, of an ergodic continuous-time Markov process, Q, is by first finding its embedded Markov chain (EMC). Strictly speaking, the EMC is a regular discrete-time Markov chain, sometimes referred to as a jump process. Each element of the one-step transition probability matrix of the EMC, S, is denoted by sij, and represents the conditional probability of transitioning from state i into state j. These conditional probabilities may be found by

s_{ij} = \begin{cases} \frac{q_{ij}}{\sum_{k \neq i} q_{ik}} & \text{if } i \neq j \\ 0 & \text{otherwise}. \end{cases}

From this, S may be written as

where DQ = diag{Q} is the diagonal matrix of Q.

To find the stationary probability distribution vector, we must next find such that

with being a row vector, such that all elements in are greater than 0 and = 1, and the 0 on the right side also being a row vector of 0's. From this, π may be found as

Note that S may be periodic, even if Q is not. Once π is found, it must be normalized to a unit vector.

Another discrete-time process that may be derived from a continuous-time Markov chain is a δ-skeleton—the (discrete-time) Markov chain formed by observing X(t) at intervals of δ units of time. The random variables X(0), X(δ), X(2δ), ... give the sequence of states visited by the δ-skeleton.

Read more about this topic:  Continuous-time Markov Process

Famous quotes containing the words embedded and/or chain:

    “The idea of feminine authority is so deeply embedded in the human subconscious that even after all these centuries of father-right the young child instinctively regards the mother as the supreme authority. He looks upon the father as equal with himself, equally subject to the woman’s rule. Children have to be taught to love, honor, and respect the father.”
    —Elizabeth Gould Davis (b. 1910)

    “The name of the town isn’t important. It’s the one that’s just twenty-eight minutes from the big city. Twenty-three if you catch the morning express. It’s on a river and it’s got houses and stores and churches. And a main street. Nothing fancy like Broadway or Market, just plain Broadway. Drug, dry good, shoes. Those horrible little chain stores that breed like rabbits.”
    —Joseph L. Mankiewicz (1909–1993)