Mean Squared Error

Unbiasedness is not the same as accuracy. An unbiased estimator can be wildly wrong on every sample and still average out to the truth. Mean squared error measures how far the estimator lands from the parameter, counting both kinds of error at once.

Definition. Let $T$ be an estimator for a parameter $\theta$. The mean squared error of $T$ is $\mathrm{MSE}(T) = \mathbb{E}\left[(T - \theta)^2\right]$

The squared distance from the target, averaged over all possible samples. Squaring makes errors in both directions count as error, and penalizes large misses more than small ones.

Definition. Given two estimators $T_1$ and $T_2$ for $\theta$, $T_1$ is more efficient than $T_2$ when $\mathrm{MSE}(T_1) < \mathrm{MSE}(T_2)$ irrespective of the value of $\theta$.

The qualifier “irrespective of $\theta$” matters. If one estimator wins for some parameter values and loses for others, neither is more efficient in this sense.

1. The decomposition

Theorem. For any estimator $T$ of $\theta$, $\mathrm{MSE}(T) = \mathrm{Var}(T) + \left[\mathrm{bias}(T)\right]^2$

This identity is the most-used result in the course. Total error splits into a random part and a systematic part, and they are separately controllable.

Proof. Write $b = \mathrm{bias}(T) = \mathbb{E}[T] - \theta$, so $\mathbb{E}[T] = \theta + b$. Add and subtract $\mathbb{E}[T]$ inside the square:

$\mathrm{MSE}(T) = \mathbb{E}\left[\left(T - \mathbb{E}[T] + \mathbb{E}[T] - \theta\right)^2\right] = \mathbb{E}\left[\left((T - \mathbb{E}[T]) + b\right)^2\right]$

Expanding the square:

$= \mathbb{E}\left[(T - \mathbb{E}[T])^2\right] + 2b\,\mathbb{E}\left[T - \mathbb{E}[T]\right] + b^2$

The first term is $\mathrm{Var}(T)$ by definition, and the middle term vanishes because $\mathbb{E}[T - \mathbb{E}[T]] = 0$. So $\mathrm{MSE}(T) = \mathrm{Var}(T) + b^2$. $\blacksquare$

The immediate corollary. For an unbiased estimator the bias term is zero, so

$\mathrm{MSE}(T) = \mathrm{Var}(T) \iff \mathrm{bias}(T) = 0$

which is why comparing unbiased estimators reduces to comparing variances.

2. Variance and bias as two kinds of error

The two components fail differently, and the distinction is worth holding.

Variance is scatter. The estimator is centred correctly but individual estimates land far from the centre. More data reduces it, since variance typically falls like $1/n$.

Bias is displacement. The estimator is tightly concentrated but around the wrong value. More data does not necessarily reduce it, since bias can persist at any $n$.

The dartboard picture: high variance is a wide spray around the bullseye, high bias is a tight cluster off to one side. Both are error, and mean squared error adds them on one scale.

3. Deriving an MSE

The standard exam problem gives a family of estimators indexed by a constant and asks for the MSE as a function of that constant.

Worked example. For $X_1, \dots, X_n \overset{\text{i.i.d.}}{\sim} \mathrm{Exp}(\lambda)$ with $\mu = 1/\lambda$, consider $T_c = c\left(X_1 + \cdots + X_n\right)$ for $c > 0$. Show that

$\mathrm{MSE}(T_c) = \frac{c^2 n + (cn-1)^2}{\lambda^2}$

For an exponential, $\mathbb{E}[X_i] = 1/\lambda$ and $\mathrm{Var}(X_i) = 1/\lambda^2$.

Variance. By independence,

$\mathrm{Var}(T_c) = c^2 \sum_{i=1}^n \mathrm{Var}(X_i) = c^2 n \cdot \frac{1}{\lambda^2} = \frac{c^2 n}{\lambda^2}$

Bias. $\mathbb{E}[T_c] = cn\mathbb{E}[X_1] = \frac{cn}{\lambda}$, so

$\mathrm{bias}(T_c) = \frac{cn}{\lambda} - \frac{1}{\lambda} = \frac{cn - 1}{\lambda}$

Combine.

$\mathrm{MSE}(T_c) = \frac{c^2 n}{\lambda^2} + \frac{(cn-1)^2}{\lambda^2} = \frac{c^2 n + (cn-1)^2}{\lambda^2}$

4. Minimizing over the constant

Having the MSE as a function of $c$, minimize it. The parameter $\lambda$ enters only through the factor $1/\lambda^2$, which does not depend on $c$, so minimize the numerator.

Let $g(c) = c^2 n + (cn-1)^2$. Differentiate:

$g'(c) = 2cn + 2n(cn-1) = 2n\left[c + cn - 1\right] = 2n\left[c(n+1) - 1\right]$

Setting $g'(c) = 0$ gives

$c^* = \frac{1}{n+1}$

Verify it is a minimum:

$g''(c) = 2n + 2n^2 > 0$

so the critical point is a minimum.

The second-order check is worth a mark on its own and is the step most often skipped. A vanishing first derivative locates a stationary point, not necessarily a minimum.

Reading the answer. The MSE-optimal estimator is

$T_{c^*} = \frac{1}{n+1}\sum_{i=1}^n X_i$

against the unbiased $\bar{X}_n = \frac{1}{n}\sum X_i$. Dividing by $n+1$ rather than $n$ shrinks every estimate toward zero, so $T_{c^*}$ is biased low. It is still preferred under mean squared error, because the shrinkage cuts the variance by more than the bias adds. Trading a little bias for a larger reduction in variance is worthwhile whenever total squared error is the criterion.

5. Estimating MSE by simulation

When the expectation is intractable, simulate and use the decomposition. This is the standard R-output question.

Fix the true $\theta$, generate many samples, compute the estimator each time, and read off the simulated mean and variance.

$\widehat{\mathrm{MSE}}(T) = \widehat{\mathrm{Var}}(T) + \left(\widehat{\mathbb{E}[T]} - \theta\right)^2$

Worked example. Raffle tickets numbered $1, \dots, N$, five drawn with replacement, two estimators: $T_1 = 2\bar{X}_n - 1$ from the method of moments, and $T_2 = \max\{X_1, \dots, X_5\}$.

set.seed(238)
t1 <- numeric(10000)
t2 <- numeric(10000)
for (i in 1:10000) {
  x <- sample(1:100, 5, replace = TRUE)
  t1[i] <- 2 * mean(x) - 1
  t2[i] <- max(x)
}
round(c(mean(t1), mean(t2)), 1)
round(c(var(t1), var(t2)), 1)

Output, at a true $N = 100$:

[1] 100.0  83.8
[1] 666.4 199.8

Read the two lines: the first gives the simulated means, the second the simulated variances.

For $T_1$: mean 100.0 against a true 100, so the bias is 0, confirming unbiasedness. Variance 666.4.

$\mathrm{MSE}(T_1) \approx 666.4 + (100.0 - 100)^2 = 666.4$

For $T_2$: mean 83.8, so the bias is $83.8 - 100 = -16.2$, negative as the bounded-support argument predicts. Variance 199.8.

$\mathrm{MSE}(T_2) \approx 199.8 + (-16.2)^2 = 199.8 + 262.4 = 462.2$

Conclusion: prefer $T_2$. It carries a substantial negative bias, and its much smaller variance still gives it the smaller mean squared error. This is the bias-variance tradeoff in action, and it is the case where the unbiased estimator loses.

The mechanism is worth seeing. $T_1$ uses only the average of the five draws, so a sample that happens to be low or high moves it a long way. $T_2$ uses the maximum, which for a sample of five is usually somewhere near the top of the range and rarely far from it, so it is stable even though it is systematically too small.

6. When the biased estimator does not win

The comparison can go the other way, and the qualifier “irrespective of $\theta$” is where it bites.

For $X_i \sim \mathrm{Pois}(\lambda)$ estimating $p_0 = P(X = 0) = e^{-\lambda}$, compare the unbiased proportion of zeros $T_1 = \frac{1}{n}\sum \mathbb{I}(X_i = 0)$ against the biased $T_2 = e^{-\bar{X}_n}$. Simulating at $n = 365$ across $\lambda = 1, \dots, 10$:

• $T_2$ has larger squared bias at small $\lambda$, decaying toward zero as $\lambda$ grows.
• $T_1$ has the larger variance at every $\lambda$.
• The variance difference dominates, so $T_2$ has the smaller MSE throughout.

Here the biased estimator wins again, but only because the variance gap is large enough everywhere. Change the setup and the ordering can flip, which is why the definition of “more efficient” demands the inequality hold for all $\theta$.

7. The tradeoff as a decision

Mean squared error is a choice of criterion, not a law. Squared error weights a miss of 20 as sixteen times worse than a miss of 5, and that weighting is an assumption about what errors cost.

When it is right, accept bias for variance reduction whenever the trade is favourable. When the application genuinely requires an estimator correct on average, for instance when many estimates are aggregated and biases would accumulate, unbiasedness is worth its variance cost.

The instruction is to make the decision consciously rather than defaulting to unbiasedness because it sounds like correctness.

Vocabulary to deploy

• Mean squared error $\mathrm{MSE}(T) = \mathbb{E}[(T-\theta)^2]$.
• The decomposition $\mathrm{MSE}(T) = \mathrm{Var}(T) + [\mathrm{bias}(T)]^2$.
• More efficient: smaller MSE irrespective of $\theta$.
• Bias-variance tradeoff: accepting bias to buy a larger reduction in variance.
• Shrinkage: scaling an estimator toward zero, biasing it while reducing its variance.
• Second-order condition $g''(c) > 0$, required to confirm a minimum.