Efficiency and the Cramer-Rao Bound

Two unbiased estimators of the same parameter can behave very differently. Both are correct on average, and one can be reliably close while the other swings wildly. The distinguishing quantity is variance.

Definition. Let $T_1$ and $T_2$ be unbiased estimators of the same parameter $\theta$. Then $T_2$ is more efficient than $T_1$ when $\mathrm{Var}(T_2) < \mathrm{Var}(T_1)$ irrespective of the value of $\theta$. The relative efficiency of $T_2$ with respect to $T_1$ is $\frac{\mathrm{Var}(T_1)}{\mathrm{Var}(T_2)}$

A relative efficiency of 3 means $T_1$ has three times the variance of $T_2$, so $T_2$ extracts more information from the same data.

Two conditions in the definition are easy to skip and both matter. The estimators must be unbiased, since otherwise comparing variances ignores the systematic error and mean squared error is the right instrument. And the inequality must hold for every $\theta$, since an estimator that wins for some parameter values and loses for others is not more efficient in this sense.

1. Comparing two unbiased estimators

The procedure is direct: compute both variances and compare.

Worked example. Estimating $\theta$, the number of tanks, from serial numbers, with two unbiased estimators: $T_1 = 2\bar{X}_n - 1$ from the method of moments, and $T_2 = \frac{n+1}{n}X_{(n)} - 1$ from the corrected maximum.

Simulating at $\theta = 1000$ with $n = 25$ captures shows two clearly different sampling distributions. Both are centred at 1000, confirming unbiasedness. $T_1$ is broad and roughly symmetric; $T_2$ is narrow and sharply peaked with a short left tail. So $\mathrm{Var}(T_2) < \mathrm{Var}(T_1)$ and $T_2$ is more efficient.

The mechanism is worth extracting, because it explains when the maximum beats the mean. The sample mean uses all $n$ observations but each one contributes only its own magnitude, and low and high draws partially cancel. The maximum uses effectively one observation, but for a uniform distribution the largest of $n$ draws sits close to the ceiling and rarely far from it. The information about $\theta$ is concentrated at the top of the range, and the maximum is the statistic that reads it.

2. Is there a best possible estimator?

Given a stream of unbiased estimators, each more efficient than the last, a natural question is whether the improvement can continue indefinitely. It cannot. There is a floor on the variance of any unbiased estimator, determined by the model itself.

Theorem (Cramér-Rao inequality). Let $X_1, \dots, X_n$ be a random sample from a distribution with density $f_\theta$, and let $T$ be an unbiased estimator of $\theta$. Under smoothness conditions on $f_\theta$, $\mathrm{Var}(T) \ge \frac{1}{n\,\mathbb{E}\left[\left(\frac{\partial}{\partial\theta}\log f_\theta(X)\right)^2\right]}$ for all $\theta$.

The right-hand side is the Cramér-Rao lower bound. No unbiased estimator can have smaller variance.

Definition. An unbiased estimator whose variance attains the Cramér-Rao bound is a minimum variance unbiased estimator, abbreviated MVUE.

An MVUE cannot be beaten among unbiased estimators. It can still be beaten on mean squared error by a biased estimator, which is not a contradiction, because the bound only constrains estimators that are unbiased.

Reading the bound

The denominator is $n$ times the Fisher information

$I(\theta) = \mathbb{E}\left[\left(\frac{\partial}{\partial\theta}\log f_\theta(X)\right)^2\right]$

The quantity being squared and averaged is the derivative of the log-density with respect to the parameter, which measures how sharply the density changes as $\theta$ moves. A density that responds strongly to a change in $\theta$ makes different parameter values easy to tell apart, so the data are informative and the floor is low. A density barely affected by $\theta$ makes them hard to distinguish and the floor is high.

The factor $n$ says information adds across independent observations, so the bound falls like $1/n$, matching the rate at which the variance of a sample mean falls.

3. Establishing that an estimator is an MVUE

Three steps, and the exam expects all three.

1. Compute $\log f_\theta(x)$ and differentiate with respect to $\theta$.
2. Square and take the expectation to get $I(\theta)$, then form the bound $\frac{1}{nI(\theta)}$.
3. Compute $\mathrm{Var}(T)$ and show it equals the bound.

Worked example: the exponential, parameterized by the mean

$X_1, \dots, X_n \overset{\text{i.i.d.}}{\sim} \mathrm{Exp}(\lambda)$ with $\mu = 1/\lambda$. Write the density in terms of $\mu$:

$f_\mu(x) = \frac{1}{\mu}e^{-x/\mu}, \qquad x \ge 0$

Show that $\bar{X}_n$ is an MVUE of $\mu$.

Step 1: the log-density and its derivative.

$\log f_\mu(x) = \log\left(\frac{1}{\mu}\right) - \frac{x}{\mu} = -\log\mu - \frac{x}{\mu}$

$\frac{\partial}{\partial\mu}\log f_\mu(x) = -\frac{1}{\mu} + \frac{x}{\mu^2} = \frac{x - \mu}{\mu^2}$

The last simplification is worth doing, because it makes the next step immediate.

Step 2: square, take the expectation.

$\mathbb{E}\left[\left(\frac{X-\mu}{\mu^2}\right)^2\right] = \frac{\mathbb{E}\left[(X-\mu)^2\right]}{\mu^4} = \frac{\mathrm{Var}(X)}{\mu^4} = \frac{\mu^2}{\mu^4} = \frac{1}{\mu^2}$

using $\mathrm{Var}(X) = 1/\lambda^2 = \mu^2$ for an exponential. The numerator $\mathbb{E}[(X-\mu)^2]$ is the variance by definition, since $\mu = \mathbb{E}[X]$.

So $I(\mu) = 1/\mu^2$ and the bound is

$\mathrm{Var}(T) \ge \frac{1}{n \cdot \frac{1}{\mu^2}} = \frac{\mu^2}{n}$

Step 3: compare with the variance of the sample mean.

$\mathrm{Var}(\bar{X}_n) = \frac{\mathrm{Var}(X_1)}{n} = \frac{\mu^2}{n}$

The sample mean attains the bound, so it is a minimum variance unbiased estimator of $\mu$. No unbiased estimator of the mean lifetime does better, whatever cleverness is applied.

The same computation in the rate parameterization

Writing $\theta = 1/\lambda$ and expanding rather than simplifying gives the same answer by a longer route:

$\left[\frac{\partial}{\partial\theta}\log f_\theta(X)\right]^2 = \left[-\frac{1}{\theta} + \frac{X}{\theta^2}\right]^2 = \frac{1}{\theta^2} - \frac{2X}{\theta^3} + \frac{X^2}{\theta^4}$

Taking expectations term by term with $\mathbb{E}[X] = \theta$ and $\mathbb{E}[X^2] = \mathrm{Var}(X) + (\mathbb{E}[X])^2 = \theta^2 + \theta^2 = 2\theta^2$:

$\frac{1}{\theta^2} - \frac{2\theta}{\theta^3} + \frac{2\theta^2}{\theta^4} = \frac{1}{\theta^2} - \frac{2}{\theta^2} + \frac{2}{\theta^2} = \frac{1}{\theta^2}$

Same information, same bound. Grouping the derivative into $(X-\mu)/\mu^2$ first turns three expectations into one and is the version worth doing under time pressure.

4. What the bound does and does not say

It applies only to unbiased estimators. A biased estimator can have a variance below the bound. The shrunken exponential estimator $\frac{1}{n+1}\sum X_i$ has smaller variance than $\bar{X}_n$ and a smaller mean squared error, and no theorem is violated, because it is not unbiased.

It requires smoothness conditions. The density must be differentiable in $\theta$ with the support not depending on $\theta$. The uniform family fails this, since its support is $[0,\theta]$, which is why the Cramér-Rao machinery cannot be applied there. That is the same structural feature that broke the maximum likelihood recipe for the uniform.

Attaining it is sufficient but not necessary for being best. For some models no unbiased estimator reaches the bound, and a minimum variance unbiased estimator still exists above it. Attaining the bound settles the question immediately; failing to attain it settles nothing.

5. Estimating efficiency by simulation

Where the variances are not available in closed form, simulate both estimators and compare the sample variances.

m <- 1000
lambda <- 5
n <- 365
t1 <- numeric(m)
t2 <- numeric(m)
for (i in seq(m)) {
  sales <- rpois(n, lambda)
  t1[i] <- mean(sales == 0)
  t2[i] <- exp(-mean(sales))
}
var(t1)
var(t2)

Output:

[1] 1.812066e-05
[1] 6.149703e-07

$\mathrm{Var}(T_1)$ is roughly 30 times $\mathrm{Var}(T_2)$, so $T_2$ is far less variable.

The comparison is not a statement about efficiency as defined here, because $T_2 = e^{-\bar{X}_n}$ is biased and the definition requires both estimators to be unbiased. What the simulation licenses is a mean squared error comparison, which needs the bias as well. Stating that $T_2$ is “more efficient” on these numbers alone is the error the definition exists to prevent.

Vocabulary to deploy

• More efficient: smaller variance among unbiased estimators, for every $\theta$.
• Relative efficiency $\mathrm{Var}(T_1)/\mathrm{Var}(T_2)$.
• Cramér-Rao lower bound $\frac{1}{nI(\theta)}$; Fisher information $I(\theta) = \mathbb{E}\left[\left(\frac{\partial}{\partial\theta}\log f_\theta(X)\right)^2\right]$.
• Minimum variance unbiased estimator: unbiased and attaining the bound.
• Smoothness conditions, which fail when the support depends on the parameter.