This is equivalent to rotating the coordinate axes through an angle of ⇡/4.

First,
P(Yn+1 Yn = (1, 1)) = P(An+1 An = 1) = P(Xn+1 Xn = (1, 0)) = 1/4,
similarly for the other three possibilities for Yn+1 Yn , namely ( 1, 1), (1, 1), and ( 1, 1).
Therefore, Yn+1 Yn takes values in (1, 1), ( 1, 1), (1, 1), and ( 1, 1), all with proba-
bility 1/4. It follows that the coordinates of (Yn )n 0 , which are (Un )n 0 and (Vn )n 0 , are
independent symmetric random walks on Z. Also, Xn = (0, 0) if and only if Yn = (0, 0).

Simple Symmetric Random walk on Z3 . Consider S = Z3 and for any a, b, c 2 Z,

1
P(a,b,c),(a±1,b,c) = P(a,b,c),(a,b±1,c) = P(a,b,c),(a,b,c±1) = .
6
In this case,
X (2n)! ⇣ 1 ⌘2n
(P 2n )(0,0,0),(0,0,0) =
i!i!j!j!k!k! 6
i+j+k=n
⇣ 1 ⌘2n X (2n)!
=
6 (i!j!k!)2
i+j+k=n
⇣ 1 ⌘2n ✓2n◆ X ✓ n! ◆2
=
2 n 3n i!j!k!
i+j+k=n
⇣ 1 ⌘2n ✓2n◆ n n! o X n!
 max n :i+j+k =n n
2 n 3 i!j!k! 3 i!j!k!
i+j+k=n
✓
⇣ 1 ⌘2n 2n ◆ n n! o
= max n :i+j+k =n
2 n 3 i!j!k!
⇣ 1 ⌘2n ✓2n◆ n!
= (assuming n is divisible by 3)
2 n 3n n ! 3
3
(2n)!
=  Cn 3/2 (by Stirling’s formula),
n n 3
12 n! 3 !
where C is a constant that does not depend on n. So, the return probabilities are summable.
Therefore, the MC is transient.
Remark 1.32. One can show that 2n-step return probability simple symmetric random walk on Zd
behaves approximately like Cd n d/2 .
To summarize, we have shown

Theorem 1.33 (Pólya, 1921). The simple symmetric random walk

on Zd is recurrent if d = 1, 2 and transient if d 3.
George Pólya 1887-1985

1.5 Stationary Distribution

Definition 1.34. Consider a MC with transition
P matrix P on state space S. Let ⇡ be probability
vector on S, i.e., ⇡(x) 0 for all x 2 S and x ⇡(x) = 1. We call ⇡ a stationary distribution if
⇡P = ⇡ (⇡ is a row vector).

17
That is, X
⇡(x)p(x, y) = ⇡(y) for all y 2 S. (11)
x

(a) Why is ⇡ called stationary distribution? If the initial distribution of X0 is ⇡, i.e., X0 ⇠ ⇡, then
Xn ⇠ ⇡P n = ⇡ for any n 1. That means X0 and Xn have the same marginal distribution
for all n.
Also, suppose that starting from an initial distribution µ, Xn converges in distribution for
some probability distribution ⇡ on S, which is equivalent to saying that Pµ (Xn = y) ! ⇡(y)
for each y. Then
⇡ = lim µP n = (lim µP n 1 )P = ⇡P,
n n

i.e., ⇡ must satisfy the stationary equation ⇡ = ⇡P .

(b) For finite state MC, the stationary distribution always exists, but it may not be unique. For
example, S = {0, 1}, P0,0 = 1 = P1,1 . Then any ⇡ = (↵, 1 ↵) for all ↵ 2 [0, 1] is a stationary
distribution.

(c) For infinite MC, stationary distribution may not even exist. Consider SSRW on Z. If ⇡P = ⇡,
then for all i 2 Z,
1
⇡(i 1) + ⇡(i + 1) = ⇡(i).
2
This implies ⇡(i + 1) ⇡(i) = ⇡(i) ⇡(i 1) for all i, i.e, ⇡ : Z ! R+ has constant slope, or
⇡(i) = c for all i. Then c = 0, a contradiction.

Example 1.35. Consider a MC on S = {1, 2} with transition matrix


1 ↵ ↵
P = ,
1

where ↵, 2 (0, 1). The stationary equations are

⇡(1)(1 ↵) + ⇡(2) = ⇡(1) (12)

⇡(1)↵ + ⇡(2)(1 ) = ⇡(2) (13)

(12) yields

↵⇡(1) = ⇡(2) ) ⇡(1) = c/↵ and ⇡(2) = c/ for some constant c.

Since ⇡(1) + ⇡(2) = 1, we obtain

✓ ◆
1 1 ↵
c + =1)c= .
↵ ↵+

Therefore, ✓ ◆
↵
(⇡(1), ⇡(2)) = , .
↵+ ↵+

The following lemma (proof omitted) provides a situation where the stationary distribution exists
and is unique.

18
Lemma 1.36. An irreducible finite state MC has a unique stationary distribution.

Proof (existence part, under a stronger assumption p(x, y) > 0 for all x, y 2 S.)
Since (P I)1 = 0, rank(P I) < |S|. So, there exists ⇡ 6⌘ 0 such that

⇡(P I) = 0, i.e., ⇡P = ⇡.

Now we would like to show that the entries of ⇡ are either all non-negative or all non-positive. If
we can show that then by taking negative of ⇡ if necessary, we can assume that ⇡(x) 0 for all
x 2 S. Then we normalize ⇡ so that it becomes a probability vector, i.e., define
⇡(x)
¯ (x) = P
⇡ .
y2S ⇡(y)

Clearly, ⇡
¯P = ⇡
¯ , as desired. We now prove the claim. Suppose, if possible, ⇡ contains both positive
and negative entries. Then
X X X XX
|⇡(y)| = | ⇡(x)p(x, y)| < |⇡(x)|p(x, y)
y2S y2S x2S y2S x2S
X X X
= |⇡(x)| p(x, y) = |⇡(x)|,
x2S y2S x2S

which is a contradiction. The crucial strict inequality in the middle appears due to the hypotheses
that ⇡ has both positive and negative entries and the transition probabilities are strictly positive. 2

Consider an irreducible Markov chain on state space S = {1, 2, 3} with transition matrix

1 2 3
2 3
1 0.8 0.1 0.1
P = 2 4 0.2 0.6 0.2 5.
3 0.3 0.3 0.4

Let us compute the stationary distribution ⇡. We need to solve the equation ⇡P = ⇡ or ⇡(P I) = 0,
I being the identity matrix.
2 3
0.2 0.1 0.1
(⇡(1) ⇡(2) ⇡(3)) 4 0.2 0.4 0.2 5 = (0 0 0).
0.3 0.3 0.6

Writing out the linear equations, we have

0.2⇡(1) + 0.2⇡(2) + 0.3⇡(3) = 0

0.1⇡(1) 0.4⇡(2) + 0.3⇡(3) = 0
0.1⇡(1) + 0.2⇡(2) 0.6⇡(3) = 0.

These three equations are not ‘linearly independent’ as summing them yields 0 = 0. To get rid of
the redundancy, we remove one of them, say the third one to have
2 3
0.2 0.1
(⇡(1) ⇡(2) ⇡(3)) 4 0.2 0.45 = (0 0)
0.3 0.3

19
 However, we have an additional equation in the form ⇡(1) + ⇡(2) + ⇡(3) = 1, which we add to the
above matrix equation as
2 3
0.2 0.1 1
(⇡(1) ⇡(2) ⇡(3)) 4 0.2 0.4 15 = (0 0 1).
0.3 0.3 1

Therefore, we conclude that

2 3 1 2 3 1
0.2 0.1 1 0.2 0.1 1
4
(⇡(1) ⇡(2) ⇡(3)) = (0 0 1) 0.2 0.4 15 4
= the last row of 0.2 0.4 15 .
0.3 0.3 1 0.3 0.3 1

Below we record this strategy as an algorithm for computing the stationary distribution of a finite
irreducible MC:

Step 1. Compute P I and replace the last column by all 1’s. Call the new matrix A.

Step 2. Compute A 1.

Step 3. ⇡ is the last row of A 1.

Finding stationary distributions beyond irreducibility in finite MC.

Lemma 1.37 (stationary probability vanishes on transient states). Let ⇡ be a stationary distribu-
tion of a (possibility infinite) MC. If x is transient, then ⇡(x) = 0.

Proof.
⇣ [
1 ⌘
⇡(x) = P⇡ (Xn = x)  P⇡ {Xm = x} .
m=n

Taking n ! 1, we have
⇣ [
1 ⌘ ⇣\
1 [
1 ⌘
⇡(x)  lim P⇡ {Xm = x} = P⇡ {Xm = x} = P⇡ (Xn = x infinitely often) = 0.
n
m=n n=1 m=n

t
u
The above lemma shows that a stationary distribution is always supported only on recurrent
states. This rules out the possibility of existence of stationary distribution for the biased (p 6= 1/2)
simple random walk on Z as in that case every state is transient.
We now describe, through the following example, how to find all stationary distributions of a
finite MC with more than one recurrent class (along with some transient classes).
Consider a MC on S = {1, 2, 3, 4, 5} with the transition diagram. It has two recurrent classes
(closed, irreducible sets) {1, 2} and {4, 5} whereas 3 is a transient state. The transition matrix is
given by
Note that the 2 ⇥ 2 submatrices P1 and P2 are themselves irreducible transition matrices on
{1, 2} and {4, 5}. By Lemma 1.37, ⇡(3) = 0. The stationary equation becomes

(⇡(1) ⇡(2) 0 ⇡(4) ⇡(5))P = (⇡(1) ⇡(2) 0 ⇡(4) ⇡(5)).

20
This implies that

(⇡(1) ⇡(2))P1 = (⇡(1) ⇡(2)) and (⇡(4) ⇡(5))P2 = (⇡(4) ⇡(5)).

By Example 1.35,
✓ ◆ ✓ ◆
1 0.6 3 5
(⇡(1) ⇡(2)) = = and
0.6 + 1 0.6 + 1 8 8
✓ ◆ ✓ ◆
0.3 0.8 3 8
(⇡(4) ⇡(5)) = = .
0.8 + 0.3 0.8 + 0.3 11 11
Note that both (⇡(1) ⇡(2) 0 0 0) and (0 0 0 ⇡(4) ⇡(5)) are stationary distributions of P .
Therefore, any convex combination of them ↵(⇡(1) ⇡(2) 0 0 0) + (1 ↵)(0 0 0 ⇡(4) ⇡(5)) for
any ↵ 2 [0, 1] is also a stationary distribution. So, for any ↵ 2 [0, 1],
✓ ◆
3↵ 5↵ 3(1 ↵) 8(1 ↵)
0
8 8 11 11

is a stationary distribution for P .

1.5.1 Positive and null recurrence

Definition 1.38. For a recurrent state y, we have Py (Ty < 1) = 1. If we also have Ey [Ty ] < 1,
then y is called a positive recurrent state. Otherwise, if Ey [Ty ] = 1 is called a null recurrent state.

21