9231. Statistics. Probability Generating Functions

We are going to look at finding probability generating functions, PGFs, of discrete probability distributions, which can give us an efficient way to find expected values and variance, as well as allowing for greater analysis.

Let X represent a discrete random variable, which can take values x_i for i from 1 to n, with probabilities as per the following table:

x	x₁	x₂	x₃	…	x_n
P(X=x)	P(X=x₁)	P(X=x₂)	P(X=x₃)	…	P(X=x_n)

Using the table we can form the following function:

G_X(t) = P(X=x₁)t^x₁ + P(X=x₂)t^x₂ + P(X=x₃)t^x₃ + … P(X=x_n)t^x_n, which can be more concisely written as G_X(t) = Σt^x_iP(X=x_i) (known as the closed form of this PGF).

Notice that G_X(t) is also the same as the expectation function for t^x, so E(t^x) = G_X(t) = Σt^x_iP(X=x_i).

The t we are using is effectively a dummy variable that is useful for us.

Worked Example. PGF

Consider the following probability distribution:

x	0	1	2	3	4	5	6
P(X = x)	0.1	0.2	0.3	0.15	0.1	0.1	0.05

Write down the PGF for the random variable X.

Worked Example. PGF 2

Consider the following probability distribution:

x	2	4	5	10
P(X = x)	0.1	0.2	0.3	0.4

Write down the PGF for the random variable X.

You probably noticed in the above examples that the probabilities are simply the coefficients of the t terms. Hence these coefficients sum to 1. So G_X(1) = 1.

If we differentiate the PGF with respect to t, we will cause each term to be multiplied by the value x_i, giving G’_X(t) = Σx_i(t)^x_i-1P(X=x_i).

So G’_X(1) = Σx_i(1)^x_i-1P(X=x_i) = Σx_iP(X=x_i), which is the same as E(X). Hence G’_X(1) = E(X).

Worked Example. PGF 3

Let X be a discrete random variable, as showing in the probability distribution given by:

x	1	2	3	4	5
P(X = x)	0.2	0.2	0.2	0.2	0.2

Find the probability generating function for X.

Standard discrete distributions

Discrete uniform distribution: If X is a discrete random variable, with a uniform distribution, that is P(X = x_i) = 1/n for i = 1,2, .., n, then $G_X(t) = \frac{t(1-t^n)}{n(1-t)}$

Binomial distribution: Let X ~ Bi(n,p). Then $G_X(t) = (q+pt)^n$

Geometric distribution: Let X ~ Geo(p). Then $G_X(t) = \frac{pt}{1-qt}$

Poisson distribution: Let X ~ Po(𝞴). Then $G_X(t) = e^{ \lambda (t-1)}$

Worked Example Binary Distribution

Let x ~ Bi(5,0.2). Find the probability generating function for X.

Worked Example Geometric Distribution

Let x ~ Geo(1/5). Find the probability generating function for X.

Exercise 1

Answers to Exercise 1

Worked solutions to Exercise 1

Using the Probability Generating Function to calculate Mean and Variance

We have seen above that G_X(t) = Σt^xP(X=x).

And from this it follows that G’_X(t) = Σxt^x-1P(X=x) and

G”_X(t) = Σx(x-1)t^x-2P(X=x) = Σ(x²-x)t^x-2P(X=x) = Σx²t^x-2P(X=x) – Σxt^x-2P(X=x)

If we evaluate at t=1, we get G”_X(1) = Σx²P(X=x) – ΣxtP(X=x)

So G”_X(1) = E(X²) – E(X) which we rearrange to give: E(X²) = G”_X(1) + G’_X(1) ( as we showed above that G’_X(1) = E(X).

Hence, Var(X) = E(X²) – [E(X)]²= G”_X(1) + G’_X(1) – [G’_X(1)]²

Worked Example. Mean and Variance using PGF

A bag contains 5 red balls and 3 green balls. The balls are taken out one at a time, the colour is noted, and then it is replaced. Let X be the number of times that a ball is removed until a green ball is chosen.

(a) State the PGF of X.

(b) Calculate the mean and variance of X.

Worked Example. Mean and Variance using PGF (Poisson)

Prove that for X ~ Po(𝞴):

(a) E(X) = 𝞴

(b) Var(X) = 𝞴

Worked Example PGF

A discrete random variable has the following probability distribution:

x	0	1	2
P(X = x)	a	b	c

The mean is 2/3 and the variance is 5/9. Find a, b and c.

Exercise 2

Answers to Exercise 2

Worked Solutions to Exercise 2

Sum of Independent Random Variables

Statistics 2 looked at the situation where we have random variable X with Normal distribution N(μ₁,𝞼₁²) and random variable Y with Normal distribution N(μ₂,𝞼₂²) and we are interested in the distribution of X+Y, which is N(μ₁ + μ₂, 𝞼₁² + 𝞼₂²). The same applies for the Poisson distribution.

We are now interested in finding the PGF of X+Y for independent random variables X and Y. We will consider discrete RVs.

Let us consider X, which has the following probability distribution:

x	0	1	2
P(X=x)	p₀	p₁	p₂

Let us also consider Y, which has the following probability distribution:

x	0	1	2
P(X=x)	q₀	q₁	q₂

We can hence see that the distribution of X+Y is:

x+y	P(X+Y = x+y)
0	P (X = 0 ∩ Y = 0)
1	P (X = 0 ∩ Y = 1) + P (X = 1 ∩ Y = 0)
2	P (X = 0 ∩ Y = 2) + P (X = 1 ∩ Y = 1) + P (X = 2 ∩ Y = 0)
3	P (X = 1 ∩ Y = 2) + P (X = 2 ∩ Y = 1)
4	P (X = 2 ∩ Y = 2)

If X and Y are independent, then P(X=x_i ∩ Y=y_j) = P(X=x_i) x P(Y=y_j) = p_iq_j, so we can simplify the table as:

x+y	P(X+Y = x+y)
0	p₀q₀
1	p₁q₀ + p₀q₁
2	p₂q₀ + p₁q₁ + p₀q₂
3	p₂q₁ + p₁q₂
4	p₂q₂

So we have the PGF G_X+Y(t) = p₀q₀ +(p₁q₀ + p₀q₁)t + (p₂q₀ + p₁q₁ + p₀q₂)t² + (p₂q₁ + p₁q₂)t³ +p₂q₂t⁴ which can be rewritten as G_X+Y(t) = (p₀ + p₁t + p₂t²)(q₀ + q₁t + q₂t²), which are the PGFs of X and Y, so in fact G_X+Y(t) = G_X(t) x G_Y(t), which we call the convolution theorem.