Moment Generating Function And Probability Generating Function

1. Core Concept Overview & Intuition

The moment generating function (m.g.f.) and the probability generating function (p.g.f.) are two transforms that encode the entire probability distribution of a random variable into a single function of an auxiliary real (or complex) variable. Their practical value is that they convert the operation of computing moments—which normally requires summing or integrating powers of $x$ against the probability distribution—into the mechanical operation of repeated differentiation evaluated at a single point.

The central intuition is that the m.g.f. is a "moment factory." If $M_X(t)$ is the m.g.f. of $X$, then the $r$-th derivative of $M_X$ evaluated at $t=0$ produces the $r$-th raw moment $E[X^r]$. Consequently, the mean and variance are recovered from the first two derivatives at zero. The p.g.f. plays the same role for non-negative integer-valued random variables, but it is built from powers of a generating variable $s$ rather than the exponential $e^{tX}$; its derivatives at $s=1$ produce factorial moments, from which ordinary moments follow.

Both transforms are defined through expectation, so they inherit the linearity and independence properties of expectation. This makes them especially convenient for deriving moments of distributions specified by a probability mass function (discrete case) or a probability density function (continuous case), and for handling affine transformations such as $aX+b$.

A second intuition concerns the domain of the transform. The m.g.f. is defined wherever the expectation $E[e^{tX}]$ is finite; for many distributions this holds on an open interval of $t$ containing zero, and the moments are then recovered by differentiation inside that interval. The p.g.f. is defined for $s$ in a neighbourhood of $1$ (and at least on $[0,1]$ for bounded non-negative integer variables), with moments recovered at $s=1$.

The two functions are linked: for a non-negative integer-valued random variable, substituting $s=e^{t}$ into the p.g.f. yields the m.g.f., because $E[s^{X}]$ becomes $E[e^{tX}]$. This relationship allows a single distribution to be studied through whichever transform is algebraically simpler.

2. Mathematical Formalization & Core Definitions

> Definition (Moment Generating Function). Let $X$ be a random variable. The moment generating function of $X$, when it exists, is the function

$$
M_X(t)=E\left[e^{tX}\right]
$$

> defined for all real $t$ for which the expectation is finite.

For a discrete random variable with probability mass function $f(x)=P(X=x)$, the definition expands to a sum:

$$
M_X(t)=\sum_{x} e^{tx} f(x)
$$

For a continuous random variable with probability density function $f(x)$, the definition expands to an integral:

$$
M_X(t)=\int_{-\infty}^{\infty} e^{tx} f(x)\,dx
$$

In both cases the sum or integral is taken over the support of $X$, that is, the set of values where $f(x)>0$.

> Definition (Probability Generating Function). Let $X$ be a random variable taking values in the non-negative integers ${0,1,2,\dots}$ with probability mass function $f(x)=P(X=x)$. The probability generating function of $X$ is

$$
G_X(s)=E\left[s^{X}\right]=\sum_{x=0}^{\infty} s^{x} f(x)
$$

> defined for all real $s$ for which the series converges.

The p.g.f. is thus a power series in $s$ whose coefficients are the probabilities $f(x)$. This is the reason for the name: the function "generates" the probability distribution as the coefficient sequence of its power-series expansion.

> Definition (Raw Moment). For a positive integer $r$, the $r$-th raw moment of $X$ is $\mu_r'=E[X^{r}]$, whenever this expectation exists.

> Definition (Factorial Moment). For a positive integer $r$, the $r$-th factorial moment of a non-negative integer-valued random variable $X$ is $E[X(X-1)\cdots(X-r+1)]$, whenever this expectation exists.

The factorial moments are the natural quantities produced by differentiating the p.g.f., because the $r$-th derivative of $s^{x}$ is $x(x-1)\cdots(x-r+1)s^{x-r}$.

2.1 Recovering Moments from the Moment Generating Function

Differentiating $M_X(t)=E[e^{tX}]$ with respect to $t$ inside the expectation gives

$$
\frac{d}{dt}M_X(t)=E\left[Xe^{tX}\right]
$$

and, more generally, the $r$-th derivative is

$$
M_X^{(r)}(t)=E\left[X^{r}e^{tX}\right]
$$

Evaluating at $t=0$ removes the exponential factor and yields the moment-recovery formula:

$$
M_X^{(r)}(0)=E\left[X^{r}\right]=\mu_r'
$$

In particular, the first two raw moments are

$$
\mu_1'=M_X'(0)
$$

$$
\mu_2'=M_X''(0)
$$

and the variance follows from the standard identity

$$
\operatorname{Var}(X)=\mu_2'-(\mu_1')^{2}=M_X''(0)-\left[M_X'(0)\right]^{2}
$$

2.2 Recovering Moments from the Probability Generating Function

Differentiating $G_X(s)=\sum_{x=0}^{\infty}s^{x}f(x)$ term by term gives

$$
G_X'(s)=\sum_{x=1}^{\infty}x\,s^{x-1}f(x)
$$

and evaluating at $s=1$ produces the first factorial moment:

$$
G_X'(1)=E[X]
$$

Differentiating a second time gives

$$
G_X''(s)=\sum_{x=2}^{\infty}x(x-1)s^{x-2}f(x)
$$

so that

$$
G_X''(1)=E\left[X(X-1)\right]=E[X^{2}]-E[X]
$$

Hence the second raw moment is recovered as

$$
E[X^{2}]=G_X''(1)+G_X'(1)
$$

and the variance becomes

$$
\operatorname{Var}(X)=G_X''(1)+G_X'(1)-\left[G_X'(1)\right]^{2}
$$

2.3 Relationship Between the Two Generating Functions

For a non-negative integer-valued random variable, setting $s=e^{t}$ in the p.g.f. gives

$$
G_X(e^{t})=\sum_{x=0}^{\infty}e^{tx}f(x)=E\left[e^{tX}\right]=M_X(t)
$$

so the m.g.f. is the p.g.f. composed with the exponential map. Conversely, the p.g.f. is the m.g.f. evaluated at $t=\ln s$ wherever this is defined.

2.4 Effect of an Affine Transformation

If $Y=aX+b$ for constants $a$ and $b$, then by linearity of expectation inside the exponential,

$$
M_Y(t)=E\left[e^{t(aX+b)}\right]=e^{bt}E\left[e^{(at)X}\right]=e^{bt}M_X(at)
$$

This shift-and-scale rule is the m.g.f. analogue of the change of origin and change of scale used when computing moments of grouped data: multiplying $X$ by $a$ rescales the argument of $M_X$, while adding $b$ multiplies the whole function by $e^{bt}$.