Multinomial Distribution

1. Core Concept Overview & Intuition

The multinomial distribution is the multivariate generalization of the binomial distribution. In a binomial experiment each of $n$ independent trials results in one of exactly two mutually exclusive outcomes (success or failure). In a multinomial experiment each of $n$ independent trials results in one of $k+1$ mutually exclusive and exhaustive classes, with class probabilities $p_1,p_2,\dots,p_k,p_{k+1}$ satisfying $p_i>0$ and $\sum_{i=1}^{k+1}p_i=1$.

The random vector of interest is $\mathbf{X}=(X_1,X_2,\dots,X_k)'$, where $X_i$ records the number of trials whose outcome falls in class $i$. Because every trial falls in exactly one class, the counts are constrained by

$$
X_1+X_2+\cdots+X_k+X_{k+1}=n,
$$

where $X_{k+1}=n-\sum_{i=1}^{k}X_i$ is the count in the residual class and $p_{k+1}=1-\sum_{i=1}^{k}p_i$ is its probability. Only $k$ components are recorded because the last is determined by the others; this is why the vector has dimension $k$ although the experiment has $k+1$ classes.

The essential intuition is a counting argument. A particular outcome with counts $(x_1,\dots,x_k,x_{k+1})$ can occur in $\frac{n!}{x_1!x_2!\cdots x_k!x_{k+1}!}$ distinguishable orderings of the $n$ trials, and each ordering has probability $p_1^{x_1}p_2^{x_2}\cdots p_k^{x_k}p_{k+1}^{x_{k+1}}$ by independence. Multiplying gives the probability function.

Two structural facts drive all the theory that follows. First, the joint probability function is the general term of the multinomial expansion of $(p_1+p_2+\cdots+p_k+p_{k+1})^n$, which equals $1$; this both verifies that the function is a genuine probability function and produces the moment generating function and probability generating function in closed form. Second, the marginal distribution of any single component is binomial, and the joint marginal of any pair of components is trinomial. These marginal facts are read off directly from the generating functions by setting the appropriate arguments to zero (for the mgf) or to one (for the pgf).

The dependence structure is negative: because the counts must sum to $n$, a large count in one class forces smaller counts elsewhere, so $\operatorname{cov}(X_i,X_j)=-np_ip_j<0$ for $i\neq j$. The full variance-covariance matrix is therefore singular, with every row and every column summing to the zero vector.

2. Mathematical Formalization & Core Definitions

2.1 The Multinomial Random Vector

> Definition (Multinomial distribution). The random variables $X_1,X_2,\dots,X_k$ are said to have a multinomial distribution with parameters $n;p_1,p_2,\dots,p_k$, written $\mathbf{X}\sim\operatorname{multi}(n;p_1,p_2,\dots,p_k)$, if their joint probability function is

$$
> f(x_1,x_2,\dots,x_k)=\frac{n!}{x_1!\,x_2!\cdots x_k!\,x_{k+1}!}\,p_1^{x_1}p_2^{x_2}\cdots p_k^{x_k}p_{k+1}^{x_{k+1}},
$$

> where $x_i$ are non-negative integers with $\sum_{i=1}^{k}x_i\le n$, where $0<p_i<1$ for $i=1,2,\dots,k$, and where

$$
> p_{k+1}=1-\sum_{i=1}^{k}p_i,\qquad x_{k+1}=n-\sum_{i=1}^{k}x_i.
$$

Here $n$ is the fixed number of independent trials, $p_i$ is the probability that a single trial falls in class $i$, $X_i$ is the number of trials falling in class $i$, and $x_i$ is a realized value of $X_i$. The factorial in the denominator is the multinomial coefficient, which counts the number of distinct orderings of the $n$ trials consistent with the given class counts.

2.2 The Probability Function as a Multinomial Expansion

Because $f$ is a probability function it must sum to one over its support:

$$
\sum_{x_1}\sum_{x_2}\cdots\sum_{x_k}f(x_1,x_2,\dots,x_k)=1.
$$

Substituting the probability function, the left-hand side is exactly the multinomial expansion of a power of a sum, so the normalization condition reads

$$
(p_1+p_2+\cdots+p_k+p_{k+1})^n=1.
$$

This identity holds because the class probabilities sum to one, and it confirms that the probability function is correctly normalized.

2.3 Special Case: One Class

When $k=1$ the vector reduces to the single count $X_1$ in class $1$ against the residual class, and the probability function becomes

$$
f(x_1)=\frac{n!}{x_1!\,(n-x_1)!}\,p_1^{x_1}(1-p_1)^{n-x_1},\qquad x_1=0,1,\dots,n,
$$

which is the probability function of the binomial distribution $\operatorname{bin}(n;p_1)$. The multinomial therefore contains the binomial as the case of a single recorded class.

2.4 The Trinomial Case

When $k=2$ the vector is $\mathbf{X}=(X_1,X_2)'$ with $X_3=n-X_1-X_2$ and $p_3=1-p_1-p_2$, and the joint probability function is

$$
f(x_1,x_2)=\frac{n!}{x_1!\,x_2!\,(n-x_1-x_2)!}\,p_1^{x_1}p_2^{x_2}(1-p_1-p_2)^{\,n-x_1-x_2},
$$

for $x_1,x_2\ge 0$ with $x_1+x_2\le n$, and $f(x_1,x_2)=0$ otherwise. This is the trinomial distribution $\operatorname{Trinomial}(n;p_1,p_2)$, and it is the form in which most examination questions on this topic are posed.

2.5 The Moment Generating Function

> Definition (mgf of the multinomial). For $\mathbf{X}\sim\operatorname{multi}(n;p_1,\dots,p_k)$ the moment generating function is

$$
> M(t_1,t_2,\dots,t_k)=E\left{e^{t_1X_1+t_2X_2+\cdots+t_kX_k}\right}.
$$

Evaluating the expectation against the probability function and collecting powers of $e^{t_i}$ gives the closed form

$$
M(t_1,t_2,\dots,t_k)=\left(p_1e^{t_1}+p_2e^{t_2}+\cdots+p_ke^{t_k}+p_{k+1}\right)^n.
$$

The derivation proceeds by writing the expectation as a multiple sum, grouping the factor $e^{t_ix_i}$ with $p_i^{x_i}$ to obtain $(p_ie^{t_i})^{x_i}$ in each term, and recognizing the resulting sum as the multinomial expansion of the displayed power.

2.6 The Probability Generating Function

> Definition (pgf of the multinomial). For $\mathbf{X}\sim\operatorname{multi}(n;p_1,\dots,p_k)$ the probability generating function is

$$
> G(s_1,s_2,\dots,s_k)=E\left{s_1^{X_1}s_2^{X_2}\cdots s_k^{X_k}\right}.
$$

Evaluating the expectation and applying the same multinomial expansion argument gives

$$
G(s_1,s_2,\dots,s_k)=\left(s_1p_1+s_2p_2+\cdots+s_kp_k+p_{k+1}\right)^n.
$$

The mgf and the pgf carry the same information: substituting $s_i=e^{t_i}$ in $G$ recovers $M$, and conversely. Either may be used to obtain moments and marginal distributions; the choice is a matter of convenience.

2.7 The Variance-Covariance Matrix

> Definition (variance-covariance matrix). For the multinomial random vector the variance-covariance matrix $\Sigma$ has $(i,j)$ entry $\operatorname{cov}(X_i,X_j)$, with $\operatorname{var}(X_i)$ on the diagonal.

Writing out all entries using the results of Section 3 gives

$$
\Sigma=n\begin{pmatrix} p_1(1-p_1) & -p_1p_2 & \cdots & -p_1p_k & -p_1p_{k+1}\ -p_2p_1 & p_2(1-p_2) & \cdots & -p_2p_k & -p_2p_{k+1}\ \vdots & \vdots & \ddots & \vdots & \vdots\ -p_{k+1}p_1 & -p_{k+1}p_2 & \cdots & -p_{k+1}p_k & p_{k+1}(1-p_{k+1}) \end{pmatrix}.
$$

Summing all the rows (equivalently all the columns) of $\Sigma$ produces the zero vector, so $\Sigma$ is singular and has no inverse. The singularity is the algebraic expression of the linear constraint $\sum_{i=1}^{k+1}X_i=n$: the components cannot vary freely.