Class: [Link].

SEM 3
Subject: Statistical and Risk Modelling 3
Chapter: Unit 2 Chapter 1
Chapter Name: Markov Chains

1
Today’s Agenda
1. Introduction to Markov chains
1.1 Transition graph and transition matrix

2. Chapman-Kolmogorov equations

3. Time homogenous and Time in-homogenous

4. Long Term Stationary Distribution of a Markov Chain

5. Modelling using Markov Chains

2
1 Markov Chains
The Markov property means that the future value of a process is independent of the past history
and only depends on the current value. Any process satisfying the Markov property is a Markov
process.

The term Markov chain refers to Markov processes in discrete time and with a discrete state space.

3
1 Example
A simple model of a No Claims Discount (NCD) policy.

The No Claims Discount system (NCD) in motor insurance, whereby the premium charged
depends on the driver’s claim record, is a prime application of Markov chains.

A motor insurance company grants its customers either no discount (state 0) or 25% discount
(state 1) or 50% discount (state 2).

A claim-free year results in a transition to the next higher state the following year (or in the
retention of the maximum discount); similarly, a year with one claim or more causes a transition to
the next lower state (or the retention of the zero-discount status). This is an example of a discrete-
time process that satisfies the Markov property (because the future only depends on the current
level, and not on the past history). Hence this is a Markov chain.

Under these rules, the discount status of a policyholder forms a Markov chain with state space S =
{0,1,2}: consider that the probability of a claim-free year is ¾.
4
1.1 Transition Graph
One way of representing a Markov chain is by its transition graph. In case of the example stated
above, the transition graph will be given as:

The states are represented by the circles, and each arrow represents a possible transition.
Staying in a state for one time period is also a possible transition, so we need an arrow to show
this. Next to each arrow is written the corresponding transition probability.

5
1.2 Transition Matrix
The transition probability between two states in unit time is therefore given explicitly. These can be
written in the form of a transition matrix:

The (i, j)th entry in the matrix, ie: i-th row and j-th column, gives the probability of moving in one
step from state i to state j .

The entries in each row sum to 1. This can be interpreted as saying that at the end of each year,
some transition into another state must occur or the process stays where it is. The sum of the
probabilities of the mutually exclusive and exhaustive events that can happen when the process is in
state i is 1.

It turns out that to calculate transition probabilities over 2 steps, we use the matrix 𝑃2 , and for three
steps we use 𝑃3 , and so on.
6
2 Chapman-Kolmogorov equation
A Markov chain is a sequence of random variables 𝑋0 , 𝑋1 , …. , 𝑋𝑁 , .. with the following property:

for all integer times n > m and states 𝑖0 , 𝑖1 , …. , 𝑖𝑚−1 , i, j in S.

The Markov property has the following interpretation - given the present state of the process 𝑋𝑚 = i,
the additional knowledge of the past is irrelevant for the calculation of the probability distribution of
future values of the process.

The conditional probabilities on the right-hand side of the property above are the key objects for the
description of a Markov chain; we call them transition probabilities, and we denote them by:

(𝒎,𝒏)
P [𝑿𝒏 = 𝐣 | 𝑿𝒎 = i ] = 𝒑𝒊 𝒋

(𝑚,𝑛)
So 𝑝𝑖 𝑗 is the probability of being in state j at time n having been in state i at time m .
7
2 Chapman-Kolmogorov equation
The transition probabilities of a discrete-time Markov chain obey the Chapman-Kolmogorov
equations:
(𝒎,𝒏) (𝒎,𝒍) (𝒍,𝒏)
𝒑𝒊 𝒋 = 𝑘 ∈𝑆 𝒑𝒊 𝒌 𝒑𝒌 𝒋
σ

for all states i, j in S and all integer times m < l < n.

8
3 Time homogenous and Time inhomogenous
Time homogenous – Transition probabilities are independent of time, i.e. only depend upon
duration and not on starting time.
(𝒏, 𝒏+𝟏)
The transition probabilities are given as: 𝒑𝒊 𝒋 = pi j

Eg: [pij(5,7) = pij(0,2) = pij(10,12) = pij(k,k+2)]

Time in-homogenous – Transition probabilities are dependent on starting time as well as

duration.
(𝒏, 𝒏+𝟏)
The transition probabilities are given as: 𝒑𝒊 𝒋

Eg: [pij(5,7) ≠ pij(0,2) ]

9
4 The long-term distribution of a Markov Chain
The stationary probability distribution
We say that 𝜋𝑗 , 𝑗 ∈ 𝑆 is a stationary probability distribution for a Markov chain with transition matrix
𝑃 if the following conditions hold for all 𝑗 in 𝑆 :
𝜋𝑗 = σ𝑖∈𝑆 𝜋𝑖 𝑝𝑖𝑗
𝜋𝑗 ≥ 0
σ𝑗∈𝑆 𝜋𝑗 = 1

First condition can be stated in the compact form 𝜋 = 𝜋𝑃 where 𝜋 is viewed as a row vector.
The interpretation is that, if we take 𝜋 as our initial probability distribution, that is to say 𝑃 𝑋0 = 𝑖
= 𝜋𝑖 , then the distribution at time 1 is again given by 𝜋 :

𝑃 𝑋1 = 𝑗 = ෍ 𝑃 𝑋1 = 𝑗 ∣ 𝑋0 = 𝑖 𝑃 𝑋0 = 𝑖 = ෍ 𝑝𝑖𝑗 𝜋𝑖 = 𝜋𝑗
𝑖∈𝑆 𝑖∈𝑆

Stationary distribution result (1) - A Markov chain with a finite state space has at least one stationary
probability distribution.
10
4 The long-term distribution of a Markov Chain
The method and steps to compute the stationary probability for a Markov model are:
Step 1: Apply 𝜋 = 𝜋𝑃 and get the system of equations.
Step 2: Discard one of the equations. [Preferably complicated one]
Step 3: Apply normalization theory. ie. σ𝑗∈𝑆 𝜋𝑗 = 1
Step 4: Select one of the 𝜋𝑗 's as working variable.
Step 5: Rewrite remaining equations in terms of the working variable. (It is good practice to solve
for the components of 𝜋 in terms of one of them which we will refer to as the working variable).
Step 6: Solve the equations in terms of the working variable.
Step 7: Solve for the working variable.
Step 8: Find the stationary distribution by substituting the working variable.

11
4.1 Irreducible chains
Irreducibility

A Markov chain is said to be irreducible if any state ‘j’ can be reached from any other state i in
any number of steps.

This is a property that can normally be judged from the transition graph alone.

Stationary distribution result (2) - An irreducible Markov chain with a finite state space has
a unique stationary probability distribution.

12
Question
Identify whether these chains are irreducible or not

13
Question

4) State A State B

State C

5) State A State B

14
4 The long-term distribution of a Markov Chain
It is natural to expect the distribution of a Markov chain to tend to the invariant distribution 𝜋 for
large times. This is why the stationary distribution is so important.

(𝑛)
If the above convergence holds, 𝑝𝑖 𝑗 will be close to 𝜋𝑗 for an overwhelming fraction of the time in
the long run.

The period of a state

A state 𝑖 is said to be periodic with period 𝑑 > 1 if a return to 𝑖 is possible only in a number of
(𝑛)
steps that is a multiple of 𝑑 (ie 𝑝𝑖𝑖 = 0 unless 𝑛 = 𝑚𝑑 for some integer 𝑚 ).
A state is said to be aperiodic if it is not periodic.

Periodicity result - If a Markov chain is irreducible all its states have the same period (or all are
aperiodic).

Aperiodic Markov chains - A Markov chain is said to be aperiodic if all its states are aperiodic.
15
4 The long-term distribution of a Markov Chain
Stationary distribution result (3)
(𝑛)
Let 𝑝𝑖𝑗 be the 𝑛-step transition probability of an irreducible aperiodic Markov chain on a finite
state space. Then for every 𝑖 and 𝑗 :
(𝑛)
lim 𝑝𝑖𝑗 = 𝜋𝑗
𝑛→∞

where 𝜋 is the stationary probability distribution.

Summary
• A Markov chain with a finite state space has at least one stationary distribution.
• An irreducible Markov chain with a finite state space has a unique stationary distribution.
• An irreducible, aperiodic Markov chain with a finite state space will settle down to its unique
stationary distribution in the long run.

16
Question
A Markov Chain with state space {A, B, C} has the following properties:
• it is irreducible
• it is periodic
• the probability of moving from A to B equals the probability of moving from A to C

(i) Show that these properties uniquely define the process. [4]

(ii) Sketch a transition diagram for the process. [1]

[Total 5]

17
Solution

18
Solution

19
5 Modelling using Markov Chains
Estimating transition probabilities
The transition probability 𝑝𝑖𝑗 is estimated by:
𝑛𝑖𝑗 number of transitions from state 𝑖 to state 𝑗
𝑝ƶ𝑖𝑗 = =
𝑛𝑖 number of transitions out of state 𝑖

Testing the Markov assumption

We can test the Markov assumption using a triplets test. The formula for the test statistic is:
2
𝑛𝑖𝑗𝑘 − 𝑛𝑖𝑗 𝑝ƶ𝑗𝑘
෍
𝑛𝑖𝑗 𝑝ƶ𝑗𝑘
𝑖,𝑗,𝑘

20
Question
CT4 A2017 Q9

A journalist has just been appointed as a

foreign correspondent covering news
stories in three island countries Atlantis,
Beachy and Coral.
He will spend each day in the country likely
to have the most exciting news, taking the
flights available between each country
which go once per day at the end of the
day.
The previous foreign correspondent tells
him “If you want to know how many flights
you are likely to take, I estimate my
movements have been like this” and she
drew this diagram showing the transition
probabilities: 21
Question
(i) Give the transition matrix for this process. [1]

On his first day in the job the new foreign correspondent will be in Atlantis.
(ii) Calculate the probability that the foreign correspondent will be in each country in his third day
in the job. [2]

The previous correspondent also reported that Beachy must be the most interesting of the islands
in terms of news because she spent 41.9% of her time there compared with 32.6% on Atlantis and
25.6% on Coral.

(iii) Sketch a graph showing the probability that the journalist is in each country over time, using
the information above. [3]

(iv) Calculate the proportion of days on which the foreign correspondent will take a flight. [1]

22
Question
The first time the foreign correspondent visits each of the countries he takes a photograph to mark
the occasion.

(v) Identify a suitable state space for modelling as a Markov chain which countries he has visited so
far. [2]

(vi) Draw a transition diagram for the possible transitions between these states. [3]
[Total 12]

23
Solution

24
Solution

25
Solution

26
Solution

27