Numerical Differentiation

1. Core Concept Overview & Intuition

Numerical differentiation is the branch of numerical analysis concerned with approximating the derivative of a function when the function is known only through a table of discrete values, or when its analytic derivative is inconvenient or impossible to evaluate directly. The central idea is to replace the exact limiting process of the calculus definition of the derivative with a finite difference quotient built from sampled function values. Because the limit is replaced by a finite step, the resulting approximation carries a truncation error, and because the sampled values themselves are usually rounded, the approximation also carries a propagated error inherited from the data. A rigorous treatment of numerical differentiation therefore requires the same error vocabulary developed for numerical computations in general: absolute error, relative error, percentage error, round-off error, truncation error, propagated error, and accumulated error.

The intuition is geometric. The derivative $f'(x)$ is the slope of the tangent line to the graph of $y=f(x)$ at the point $(x,f(x))$. If only two nearby points $(x_0,f(x_0))$ and $(x_1,f(x_1))$ are available, the slope of the secant line joining them is computable, and for a sufficiently smooth function this secant slope is close to the tangent slope. The forward difference uses the point to the right, the backward difference uses the point to the left, and the central difference uses one point on each side. The central difference is generally the most accurate of the three because the leading error terms of the one-sided approximations cancel.

Two distinct sources of inaccuracy must be kept separate throughout. The first is the truncation error, which arises because a finite difference quotient is only an approximation to the limit; it is governed by the step size $h$ and shrinks as $h$ shrinks. The second is the round-off error, which arises because the tabulated values $f(x_i)$ are themselves approximations; it grows as $h$ shrinks, because the subtraction $f(x+h)-f(x)$ then involves nearly equal numbers and the absolute errors in the data are amplified by division by the small quantity $h$. This tension means that making $h$ arbitrarily small does not make the computed derivative arbitrarily accurate; there is an optimal step size balancing the two effects. Understanding this balance is the conceptual heart of the topic.

The lesson material also situates numerical differentiation within a structured learning cycle: learners first meet the definition and the computation of errors, then list in point form the key words of the definition, then make short notes on the causes of errors and how to avoid them, then compute propagated errors, and throughout the cycle the instructor directs group discussion, encourages the participation of all learners, provides feedback, and summarises the lesson. These pedagogical elements are treated below as part of the revision record because they are explicitly part of the syllabus for this unit.

2. Mathematical Formalization & Core Definitions

2.1 Errors in Numerical Computation

> Definition (Round-off error). Round-off errors arise because it is impossible to represent all real numbers exactly on a machine with finite memory. The process involves dropping decimal digits or significant figures, and may be carried out either by chopping (truncating) or by symmetric rounding.

> Definition (Absolute error). Let $x$ be an exact number and let $x_r$ be an approximate value of $x$. The absolute error is the absolute difference between the exact number and the approximation:

$$
e_{\mathrm{abs}}=\lvert x-x_r\rvert.
$$

> If $\Delta x$ is an upper bound such that $e_{\mathrm{abs}}=\lvert x-x_r\rvert=\Delta x$, then $\Delta x$ is called the upper bound on the absolute error.

> Definition (Relative error). The relative error $e_r$ is the absolute error measured in units of the exact quantity:

$$
e_r=\frac{\lvert x-x_r\rvert}{\lvert x\rvert}.
$$

The percentage error is the relative error expressed as a percentage, obtained by multiplying $e_r$ by $100$.

> Definition (Propagated error). The error in the result of an arithmetic operation arising from initial errors in the numbers used in that operation is called the propagated error.

> Definition (Accumulated error). The total cumulative error at any step of a procedure is called the accumulated error.

2.2 Computation of Errors from Decimal Representation

When a number is quoted correct to a stated number of decimal places, the exact value is known only to lie in an interval centred on the quoted value. The half-width of that interval is the absolute error bound.

If $x_r$ is correct to one decimal place, then the exact value satisfies an interval of the form $x\in(x_r-0.05,\;x_r+0.05)$, so that $e_{\mathrm{abs}}\le 0.05$. For the sample value $x_r=2.4$ correct to one decimal place, the exact value lies in $(2.35,2.45)$, and subtracting $2.4$ throughout gives

$$
\begin{aligned}
2.35-2.4 &\le x-x_r \le 2.45-2.4, \
\lvert -0.05\rvert &\le \lvert x-x_r\rvert \le \lvert 0.05\rvert, \
e_{\mathrm{abs}} &= 0.05 = 5.0\times 10^{-2}.
\end{aligned}
$$

For a negative value such as $x_r=-10.788$ correct to three decimal places, the exact value lies in $[-10.7885,\,-10.7875]$, and the same interval argument yields

$$
\begin{aligned}
-10.7885+10.788 &\le x-x_r \le -10.7875+10.788, \
e_{\mathrm{abs}} &= 0.0005 = 5\times 10^{-4}.
\end{aligned}
$$

> Result (General bound for $n$ decimal places). If a number is correct to $n$ decimal places, then the maximum absolute error is

$$
e_{\mathrm{abs}}=0.5\times 10^{-n}.
$$

Thus a value such as $8.35678$ quoted to five decimal places carries the bound $e_{\mathrm{abs}}\le 0.5\times 10^{-5}$.

2.3 Key Words in the Definition of Error

In point form, the key words of the definition of error, as required by the lesson activity, are:

• Exact number $x$: the true value, generally unknown.
• Approximate number $x_r$: the computed or quoted value.
• Difference: the operation $\lvert x-x_r\rvert$ that measures the discrepancy.
• Absolute value: the error is a magnitude, hence non-negative.
• Upper bound $\Delta x$: a guaranteed ceiling on the discrepancy.
• Relative error: the discrepancy scaled by $\lvert x\rvert$.
• Percentage error: the relative error scaled by $100$.
• Round-off: loss of digits through finite representation.
• Truncation: replacement of an exact process by a finite one.
• Propagation: transmission of initial errors through arithmetic operations.
• Accumulation: growth of error over a sequence of steps.

2.4 Causes of Errors and How to Avoid Them

Short notes on the causes of errors and the measures taken to minimise them:

• Finite machine memory. Real numbers cannot all be stored exactly, so representation error is unavoidable; the remedy is to work with a controlled number of significant figures and to be aware of the machine precision.
• Chopping versus symmetric rounding. Chopping discards trailing digits and biases the result in one direction; symmetric rounding discards trailing digits while adjusting the last retained digit, and is preferred because it keeps the error centred near zero.
• Loss of significance through subtraction. Subtracting nearly equal numbers destroys significant figures; the remedy is to rearrange expressions algebraically before evaluation so that subtraction of nearly equal quantities is avoided.
• Division by small quantities. Dividing a small absolute error by a small step size amplifies the error; the remedy is to choose the step size to balance truncation against round-off rather than to make it arbitrarily small.
• Truncation of infinite processes. Replacing a limit or an infinite series by finitely many terms introduces truncation error; the remedy is to retain enough terms to meet a stated tolerance and to estimate the remainder.
• Error propagation through arithmetic. Errors in inputs are transmitted to outputs; the remedy is to track error bounds through each operation and to prefer numerically stable orderings of operations.

2.5 Propagation of Errors in Arithmetic Operations

Let $x$ and $y$ be exact numbers and let $x^{}$ and $y^{}$ be approximations with initial errors $\varepsilon$ and $\eta$ respectively, so that

$$
\begin{aligned}
x &= x^{}+\varepsilon, \
y &= y^{}+\eta.
\end{aligned}
$$

Let $\omega$ denote an arithmetic operation. The discrepancy between the exact operation and the operation performed on the approximations decomposes into two parts:

$$
\begin{aligned}
(x\,\omega\,y)-(x^{}\,\omega\,y^{}) &= \bigl[(x\,\omega\,y)-(x^{}\,\omega\,y)\bigr]+\bigl[(x^{}\,\omega\,y)-(x^{}\,\omega\,y^{})\bigr].
\end{aligned}
$$

The first bracket is the propagated error, the part of the discrepancy caused by the initial error in $x$; the second bracket is the generated error, the part caused by the initial error in $y$. Their sum is the accumulated error of the operation. This decomposition is the formal content of the lesson on propagation of errors and is the tool used to bound the error in any computed derivative.