Stochastic Processes With Discrete States In Continuous Time

1. Core Concept Overview & Intuition

1.1 Course Description and Scope

This unit treats stochastic processes whose state space is discrete but whose time parameter is continuous. The central object is a counting process $N(t)$ recording the number of events that have occurred in the interval $(0,t]$. The prototype is the Poisson process, which models events occurring singly in time at a constant average rate $\lambda$.

The unit is organised so that the Poisson process is first defined axiomatically through local (infinitesimal) probability statements, then analysed by deriving a system of difference differential equations, and finally solved by two techniques: a recursive method and the probability generating function (p.g.f.) technique combined with Lagrange's linear partial differential equation method. The mean and variance of the number of events are then obtained from the p.g.f.

1.2 Learning Outcomes

On completing this unit the learner should be able to:

• distinguish a random phenomenon from a stochastic process, and identify the time set, the state space, and the sample paths of a given process;
• state the defining conditions of a Poisson process of rate $\lambda$ in terms of the probabilities of $0$, $1$, and more than $1$ event in a short interval $(t,t+\Delta t]$;
• derive the difference differential equations $P_n'(t)=-\lambda P_n(t)+\lambda P_{n-1}(t)$ for $n\ge 1$ and $P_0'(t)=-\lambda P_0(t)$;
• solve these equations by the p.g.f. technique using Lagrange's auxiliary equations, and recover $P_n(t)$ as the coefficient of $s^n$;
• compute the mean $\lambda t$ and variance $\lambda t$ of $N(t)$ from the p.g.f.;
• recognise a compound Poisson distribution through a generating function of the form $e^{-\lambda(1-P(s))}$.

1.3 Random Phenomena

A random phenomenon is an empirical situation whose outcome cannot be predicted with certainty before it is observed, although the collection of possible outcomes is known. Examples include the arrival of calls at a telephone exchange, the emission of particles from a radioactive source, and the number of car accidents in a city within a given time interval. In each case individual outcomes are unpredictable, yet regularities emerge in aggregate, and probability theory supplies the language for describing those regularities.

1.4 Stochastic Process or Random Process

A stochastic process (equivalently, a random process) is a family of random variables indexed by a parameter, usually time. Instead of a single random variable, one observes an entire collection ${X(t): t \in T}$, where $T$ is the time set and each $X(t)$ takes values in a state space $S$. The essential intuition is that randomness is now indexed by time: the process describes how a random quantity evolves as time advances.

1.5 Time, States, and State Space

Three ingredients specify any stochastic process:

• the time set $T$, which may be discrete (for example $T={0,1,2,\dots}$) or continuous (for example $T=[0,\infty)$);
• the state space $S$, the set of values the process can take, which may be discrete (for example $S={0,1,2,\dots}$) or continuous;
• the probabilistic law governing how the process moves between states as time changes.

In this unit the time set is continuous, $t\ge 0$, while the state space is discrete: the state is the number of events or individuals counted, so $S={0,1,2,\dots}$. A realisation of the process, called a sample path, is a step function of time that jumps upward by one unit at each event.

1.6 Examples of Stochastic Processes

The approved material identifies the following empirical situations modelled by the Poisson process:

• the arrival of calls at a telephone exchange;
• the emission of particles from a radioactive source;
• the number of car accidents in a city within a given time interval.

In each case events occur singly in time, the chance of an event in a very short interval is proportional to the length of that interval, and occurrences in disjoint time intervals are independent.

1.7 Common and Important Stochastic Processes

Among the common processes, the Poisson process is described in the source as one of the simplest continuous-time processes with discrete state space, and it serves as the building block for the wider class of exponential models. The compound Poisson distribution, whose generating function has the form $e^{-\lambda(1-P(s))}$ with $P(s)$ itself a generating function, is the natural extension obtained by summing a random number of independent claims or jumps.

1.8 Definition (Informal Statement of the Unit's Central Object)

> A Poisson process of rate $\lambda$ is a stochastic process of point events occurring singly in time such that, as $\Delta t \to 0$, the probability of no event in $(t,t+\Delta t]$ is $1-\lambda\Delta t+o(\Delta t)$, the probability of exactly one event is $\lambda\Delta t+o(\Delta t)$, the probability of more than one event is $o(\Delta t)$, and the number of events in $(t,t+\Delta t]$ is completely independent of occurrences in $(0,t]$.

2. Mathematical Formalization & Core Definitions

2.1 The Little-o Notation

The symbol $o(\Delta t)$ denotes a function of $\Delta t$ that tends to zero faster than $\Delta t$ itself; by definition

$$
\lim_{\Delta t \to 0}\frac{o(\Delta t)}{\Delta t}=0.
$$

This notation allows the local behaviour of the process over a short interval to be summarised without carrying exact remainder terms.

2.2 Definition of the Poisson Process

> Consider events occurring singly in time. Let $N(t,t+\Delta t)$ denote the number of events in the time interval $(t,t+\Delta t]$. Suppose that for some positive constant $\lambda$, as $\Delta t \to 0$,

$$
\Pr{N(t,t+\Delta t)=0}=1-\lambda\Delta t+o(\Delta t),
$$

$$
\Pr{N(t,t+\Delta t)=1}=\lambda\Delta t+o(\Delta t),
$$

$$
\Pr{N(t,t+\Delta t)>1}=o(\Delta t).
$$

> Furthermore $N(t,t+\Delta t)$ is completely independent of occurrences in the time interval $(0,t]$. Then the stochastic process of point events satisfying these conditions is a Poisson process of rate $\lambda$.

The constant $\lambda$ is the rate of the process, measured in events per unit time. The three probability statements are labelled ①, ②, and ③ in the source and are used repeatedly in the derivation that follows.

2.3 The Sequence of State Probabilities

Let $N(t)$ be the number of events in an interval of fixed length, say $(0,t]$, and define

$$
P_n(t)=\Pr{N(t)=n},
$$

the probability that there are $n$ events at time $t$. The probabilities corresponding to negative states are by definition taken to be zero, so $P_{-1}(t)=0$.

2.4 The Difference Differential Equations

For $\Delta t>0$, decompose the event ${N(t+\Delta t)=n}$ according to the number of events in the final sub-interval:

$$
P_n(t+\Delta t)=\Pr{N(t+\Delta t)=n}.
$$

The event occurs if either $N(t)=n$ and $N(t,t+\Delta t)=0$, or $N(t)=n-1$ and $N(t,t+\Delta t)=1$, or $N(t)=n-L$ and $N(t,t+\Delta t)=L$ for $L=2,3,\dots$. In probability, "or" corresponds to addition, and the joint probabilities are written through conditional probabilities because $N(t)$ and $N(t,t+\Delta t)$ are not independent. Hence

$$
\begin{aligned}
P_n(t+\Delta t) &= \Pr{N(t)=n}\cdot\Pr{N(t,t+\Delta t)=0 \mid N(t)=n} \
&\quad + \Pr{N(t)=n-1}\cdot\Pr{N(t,t+\Delta t)=1 \mid N(t)=n-1} \
&\quad + \sum_{L\ge 2}\Pr{N(t)=n-L}\cdot\Pr{N(t,t+\Delta t)=L \mid N(t)=n-L}.
\end{aligned}
$$

For the Poisson process the conditional probabilities are constant and are given by ①, ②, and ③. Letting $\Delta t \to 0$,

$$
P_n(t+\Delta t)=P_n(t)\left[1-\lambda\Delta t+o(\Delta t)\right]+P_{n-1}(t)\left[\lambda\Delta t+o(\Delta t)\right]+o(\Delta t),
$$

which simplifies to

$$
P_n(t+\Delta t)=P_n(t)-P_n(t)\lambda\Delta t+P_{n-1}(t)\lambda\Delta t+o(\Delta t).
$$

The derivative is then obtained as a limit of the difference quotient:

$$
\begin{aligned}
P_n'(t) &= \frac{d}{dt}P_n(t) \
&= \lim_{\Delta t \to 0}\frac{P_n(t+\Delta t)-P_n(t)}{\Delta t} \
&= \lim_{\Delta t \to 0}\frac{P_n(t)-P_n(t)\lambda\Delta t+P_{n-1}(t)\lambda\Delta t+o(\Delta t)-P_n(t)}{\Delta t} \
&= \lim_{\Delta t \to 0}\left{-\lambda P_n(t)+\lambda P_{n-1}(t)+\frac{o(\Delta t)}{\Delta t}\right}.
\end{aligned}
$$

Since $\lim_{\Delta t \to 0}o(\Delta t)/\Delta t=0$, this yields the difference differential equation

$$
P_n'(t)=-\lambda P_n(t)+\lambda P_{n-1}(t),\qquad n=1,2,3,\dots
$$

and, for $n=0$ with $P_{-1}(t)=0$,

$$
P_0'(t)=-\lambda P_0(t).
$$

2.5 Initial (Boundary) Conditions

The system is solved subject to an initial condition, which is always available. For the standard Poisson process the condition is

$$
P_n(0)=1 \text{ for } n=0,\qquad P_n(0)=0 \text{ for } n\ne 0,
$$

expressing that no events have occurred at time zero.

2.6 The Probability Generating Function of the Process

Define the p.g.f. of the sequence ${P_n(t)}$ as

$$
G(s,t)=\sum_{n=0}^{\infty}P_n(t)s^n.
$$

Partial differentiation gives

$$
\frac{\partial}{\partial t}G(s,t)=\sum_{n=0}^{\infty}P_n'(t)s^n,\qquad \frac{\partial}{\partial s}G(s,t)=\sum_{n=0}^{\infty}nP_n(t)s^{n-1}.
$$

2.7 Lagrange's Linear Partial Differential Equation (Recap)

> Suppose we have a linear partial differential equation (Lagrange's linear equation)

$$
P\frac{\partial z}{\partial x}+Q\frac{\partial z}{\partial y}=R
$$

> subject to some boundary conditions, where $P$, $Q$, and $R$ are functions of $x$, $y$, and $z$. Then

$$
\frac{\partial x}{P}=\frac{\partial y}{Q}=\frac{\partial z}{R}
$$

> is known as the auxiliary (subsidiary) set of equations. From it one forms $\partial x/P=\partial y/Q$ and $\partial y/Q=\partial z/R$. If $u(x,y,z)=\text{constant}$ and $v(x,y,z)=\text{constant}$ are any two solutions of the auxiliary equations, the general solution is $\Phi(u,v)=0$ or $u=\phi(v)$.

2.8 Compound Poisson Distribution

> A distribution whose generating function has the form

$$
H(s)=G(P(s))=e^{-\lambda(1-P(s))},
$$

> where $P(s)$ is itself a generating function, is called a compound Poisson distribution.

If $S_N=X_1+X_2+\cdots+X_N$ is a random sum in which $N$ is Poisson with mean $\lambda$ and the $X_i$ are independent of $N$ with common mean $E(X_i)$, then the mean of the random sum is

$$
E[S_N]=E(X_i)\cdot E(N)=\lambda E(X_i).
$$