Random And Mean Vectors

1. Core Concept Overview & Intuition

1.1 Objectives of the Topic

By the end of this topic the learner should be able to:

• Define a random vector and a mean vector and distinguish between the two.
• Define and evaluate a variance-covariance matrix of a random vector.
• Evaluate the mean and variance of a linear function of a random vector.
• Define and evaluate the expectation of a quadratic form.

1.2 The Central Idea

A random vector is a vector whose components are random variables. It is therefore itself random: before observation, each coordinate is uncertain, and the coordinates may be statistically dependent. A mean vector is the corresponding vector of constants obtained by taking the expectation of the random vector component by component. The distinction is fundamental: the random vector is a random object living in $p$-dimensional space, whereas the mean vector is a fixed point in that same space, summarising the centre of the distribution.

Because the components of a random vector vary jointly, a single set of variances is insufficient to describe their dispersion. One must also record how each pair of components co-varies. The variance-covariance matrix collects all variances on its diagonal and all pairwise covariances off its diagonal, and it is the multivariate analogue of the univariate variance.

The topic then extends the familiar univariate rules for linear functions and quadratic functions to the vector setting. For a scalar linear combination $Y=a'X$ of the components, the mean is $a'\mu$ and the variance is $a'\Sigma a$. For a vector of linear combinations $Y=AX$, the mean vector is $A\mu$ and the variance-covariance matrix is $A\Sigma A'$. For a quadratic form $Q=X'AX$ with $A$ symmetric, the expectation is $\operatorname{trace}(A\Sigma)+\mu'A\mu$. These three results are the working tools of the entire topic.

1.3 Intuition for the Variance-Covariance Matrix

The matrix $\Sigma$ is symmetric because the covariance operation is symmetric: $\operatorname{cov}(X_i,X_j)=\operatorname{cov}(X_j,X_i)$. Its diagonal entries are non-negative because they are variances. Its off-diagonal entries may be positive, negative, or zero; a zero entry means the corresponding pair of components is uncorrelated. The correlation matrix rescales $\Sigma$ so that every diagonal entry equals one, making pairwise dependence comparable across variables measured on different scales.

1.4 Intuition for the Quadratic Form Expectation

A quadratic form $Q=X'AX$ is a scalar random variable built from products $X_iX_j$. Its expectation therefore requires the second moments $E(X_i^2)$ and $E(X_iX_j)$. These are recovered from the variance-covariance structure through the identities $E(X_i^2)=\sigma_{ii}+\mu_i^2$ and $E(X_iX_j)=\sigma_{ij}+\mu_i\mu_j$. Summing the resulting terms and recognising the trace structure yields the compact formula $\operatorname{trace}(A\Sigma)+\mu'A\mu$. The trace term measures the contribution of dispersion, while the term $\mu'A\mu$ measures the contribution of location.

2. Mathematical Formalization & Core Definitions

2.1 Random Vector

> Definition (Random Vector). Let $X_1,X_2,\dots,X_p$ be random variables defined on a common probability space. Then the column vector

$$
X=(X_1,X_2,\dots,X_p)'
$$

> is called a $p\times 1$ random vector. Its components are random variables, so $X$ is itself random.

The prime denotes transposition, so $X$ is a column vector of dimension $p\times 1$. The dimension of a random vector is the number of random variables it contains.

2.2 Mean Vector

> Definition (Mean Vector). Let $X=(X_1,X_2,\dots,X_p)'$ be a random vector with $E(X_i)=\mu_i$ for $i=1,2,\dots,p$. The mean vector of $X$ is

$$
E(X)=\big(E(X_1),E(X_2),\dots,E(X_p)\big)'=(\mu_1,\mu_2,\dots,\mu_p)'=\mu.
$$

> The components of $\mu$ are constants.

The mean vector is obtained by applying the expectation operator component by component. It has the same dimension $p\times 1$ as the random vector, but it is non-random. This is the essential distinction between the two objects.

2.3 Variance-Covariance Matrix

> Definition (Variance-Covariance Matrix). Let $X=(X_1,\dots,X_p)'$ be a random vector with mean vector $\mu$. The variance-covariance matrix of $X$ is

$$
\operatorname{var}(X)=E{(X-\mu)(X-\mu)'}.
$$

Expanding the outer product entry by entry gives the matrix display

$$
\operatorname{var}(X)=
\begin{pmatrix}
\operatorname{var}(X_1) & \operatorname{cov}(X_1,X_2) & \cdots & \operatorname{cov}(X_1,X_p)\
\operatorname{cov}(X_2,X_1) & \operatorname{var}(X_2) & \cdots & \operatorname{cov}(X_2,X_p)\
\vdots & \vdots & \ddots & \vdots\
\operatorname{cov}(X_p,X_1) & \operatorname{cov}(X_p,X_2) & \cdots & \operatorname{var}(X_p)
\end{pmatrix}.
$$

Writing $\operatorname{var}(X_i)=\sigma_i^2$ and $\operatorname{cov}(X_i,X_j)=\sigma_{ij}$ for $i\neq j$, and setting $\sigma_{ii}=\sigma_i^2$, the matrix becomes

$$
\Sigma=
\begin{pmatrix}
\sigma_{11} & \sigma_{12} & \cdots & \sigma_{1p}\
\sigma_{21} & \sigma_{22} & \cdots & \sigma_{2p}\
\vdots & \vdots & \ddots & \vdots\
\sigma_{p1} & \sigma_{p2} & \cdots & \sigma_{pp}
\end{pmatrix}.
$$

The following properties hold.

• Symmetry: $\sigma_{ij}=\sigma_{ji}$, hence $\Sigma=\Sigma'$.
• Diagonal entries: $\sigma_{ii}=\sigma_i^2\ge 0$.
• Dimension: $\Sigma$ is a $p\times p$ matrix.

2.4 Correlation Matrix

> Definition (Correlation Coefficient and Correlation Matrix). The correlation coefficient between $X_i$ and $X_j$ is

$$
r_{ij}=\frac{\sigma_{ij}}{\sigma_i\sigma_j},
$$

> and the matrix $R=((r_{ij}))$ is the correlation matrix of $X$.

The diagonal entries satisfy $r_{ii}=1$ for every $i$, since $\sigma_{ii}=\sigma_i^2$ and therefore $r_{ii}=\sigma_i^2/(\sigma_i\sigma_i)=1$. The correlation matrix is symmetric with unit diagonal.

2.5 Linear Function of a Random Vector

> Definition (Scalar Linear Combination). Let $a=(a_1,a_2,\dots,a_p)'$ be a vector of constants and let $Y=a_1X_1+a_2X_2+\cdots+a_pX_p$. Then $Y=a'X$ is a scalar random variable.

> Definition (Vector Linear Transformation). Let $A$ be a $q\times p$ matrix of constants and let $Y=AX$. Then $Y=(Y_1,\dots,Y_q)'$ is a $q\times 1$ random vector whose components are

$$
Y_i=a_{i1}X_1+a_{i2}X_2+\cdots+a_{ip}X_p,\qquad i=1,\dots,q.
$$

2.6 Quadratic Form

> Definition (Quadratic Form). Let $X=(X_1,\dots,X_p)'$ be a random vector and let $A=((a_{ij}))$ be a symmetric $p\times p$ matrix of constants. Then

$$
Q=X'AX
$$

> is called a quadratic form in $X$.

Expanding the product gives

$$
X'AX=\sum_{i=1}^{p}a_{ii}X_i^2+\sum_{i}\sum_{j\neq i}a_{ij}X_iX_j.
$$