The Weak Law Of Large Numbers

1. Core Concept Overview & Intuition

The central object of study in this unit is a random variable $X$ together with its first two moments, the mean $\mu=E{X}$ and the variance $\sigma^{2}=\operatorname{var}{X}$. The guiding theme is that knowledge of these two moments alone, without any further distributional assumption, already constrains the probability that $X$ deviates far from its mean. This is the content of the two moment inequalities, Markov and Chebychev, and it is precisely this moment-based control that makes the Weak Law of Large Numbers accessible.

The intuition behind the inequalities is a simple averaging argument. If a nonnegative quantity has a small average, then it cannot be large with high probability, because a large value occurring with appreciable probability would force the average upward. Markov's inequality formalizes this for an arbitrary nonnegative function $U(X)$ of the random variable. Chebychev's inequality is the special case obtained by applying Markov's inequality to the squared deviation $U(X)=(X-\mu)^{2}$, which converts a statement about the average squared deviation (the variance) into a statement about the probability of large absolute deviations.

The second theme is stochastic convergence, that is, convergence in probability. A sequence of random variables ${X_{n}}$ converges in probability to a limit when, for every fixed tolerance $\varepsilon>0$, the probability that $X_{n}$ lies farther than $\varepsilon$ from the limit tends to zero as $n\to\infty$. This is a statement about the bulk behaviour of the sequence rather than about individual realizations, and it is the correct mode of convergence for the law of large numbers.

Combining these two ideas yields the Weak Law of Large Numbers. For independent random variables with common mean $\mu$ and finite common variance $\sigma^{2}$, the sample mean $\overline{X}{n}$ has mean $\mu$ and variance $\sigma^{2}/n$. Chebychev's inequality then bounds the probability that $\overline{X})$, which vanishes as $n\to\infty$. Hence the sample mean converges in probability to the population mean, which is the tendency of the sample mean to converge to the true population mean under the restriction that the population variance is finite.}$ differs from $\mu$ by more than $\varepsilon$ by the quantity $\sigma^{2}/(n\varepsilon^{2

2. Mathematical Formalization & Core Definitions

2.1 Moments and the Sample Mean

Let $X$ be a random variable with expectation $E{X}=\mu$ and variance $\operatorname{var}{X}=\sigma^{2}<\infty$. The variance admits the computational identity

$$
\sigma^{2}=\operatorname{var}{X}=E{X^{2}}-\left[E{X}\right]^{2}.
$$

For independent random variables $X_{1},X_{2},\dots,X_{n}$ with common mean $\mu$ and common variance $\sigma^{2}$, the sample mean is defined by

$$
\overline{X}{n}=\frac{1}{n}\sum.}^{n}X_{i
$$

Its first two moments are

$$
E{\overline{X}{n}}=\mu,
\qquad
\operatorname{var}{\overline{X}.}}=\frac{\sigma^{2}}{n
$$

The variance of the sample mean is obtained from the independence assumption, which makes all covariance terms vanish, and from the scaling property of variance under division by the constant $n$.

2.2 Stochastic Convergence to Zero

> Definition 6.1 (Convergence in probability to zero). A sequence of random variables ${X_{n}}$ is said to converge stochastically to zero if for every $\varepsilon>0$ the relation

$$
\lim_{n\to\infty}\Pr{|X_{n}|>\varepsilon}=0
$$

> holds. In this case we say that the random variable converges in probability to zero and write $\lim_{n\to\infty}X_{n}=0$, or $X_{n}\xrightarrow{P}0$, or $X_{n}\xrightarrow{\text{c.i.p}}0$.

2.3 Stochastic Convergence to a Constant

> Definition 6.2 (Convergence in probability to a constant). A sequence of random variables ${X_{n}}$ is said to converge stochastically to a constant $c$ if for every $\varepsilon>0$ the relation

$$
\lim_{n\to\infty}\Pr{|X_{n}-c|>\varepsilon}=0
$$

> holds. We denote this by $X_{n}\xrightarrow{P}c$.

2.4 Stochastic Convergence to a Random Variable

> Definition 6.3 (Convergence in probability to a random variable). A sequence of random variables ${X_{n}}$ is said to converge stochastically to a random variable $X$ if for every $\varepsilon>0$ the relation

$$
\lim_{n\to\infty}\Pr{|X_{n}-X|>\varepsilon}=0
$$

> holds. We denote this by $X_{n}\xrightarrow{P}X$.

The three definitions are nested: Definition 6.1 is the special case of Definition 6.2 with $c=0$, and Definition 6.2 is the special case of Definition 6.3 in which the limit random variable is degenerate at the constant $c$. In each case the defining condition is a statement about the tail probability of the absolute deviation exceeding an arbitrary fixed tolerance $\varepsilon$.

2.5 The Weak Law of Large Numbers

> Definition 6.4 (WLLN). A sequence of random variables ${X_{n}}$ with $E{X_{n}}=\mu$ is said to obey the Weak Law of Large Numbers if the sequence ${\overline{X}_{n}}$ converges in probability to $\mu$. That is,

$$
\overline{X}{n}\xrightarrow{P}\mu
\qquad\text{or}\qquad
\overline{X}\mu.}\xrightarrow{\text{c.i.p}
$$