Judging an estimator

1. The question

You ran maximum likelihood or method of moments and got a formula. Is it any good?

Two things can be wrong with it, and they’re tracked separately:

• Bias. Across all the samples you could have drawn, does it centre on the parameter?
• Variance. How much do those answers scatter?

Both are properties of the recipe, not of the sample you happen to hold. A perfect sample run through a biased recipe still gives a biased answer.

2. Bias

Take the expectation of your estimator, subtract the true parameter, see what’s left. Zero means unbiased. Anything else is the bias, and it’s usually an expression in $\theta$ and $n$ rather than a number.

2.1 How you actually compute it

It’s algebra, not simulation. Nobody ran experiments to find these; they’re derived once on paper and exact for every $n$.

Two tools carry the linear cases:

• Linearity of expectation. $\mathbb{E}[aX + bY] = a\mathbb{E}[X] + b\mathbb{E}[Y]$, independent or not.
• What the family gives you. ”$X_1, \dots, X_n$ iid with mean $\mu$” means $\mathbb{E}[X_i] = \mu$. That’s the assumption, and it’s where $\mu$ enters the algebra.

The sample mean, worked:

Comes out as $\mu$, the same symbol as the truth, so the bias is $\mu - \mu = 0$.

No number was ever computed. The truth stayed a letter and cancelled. That’s what makes the result stronger than knowing $\mu$ would be: it holds for every possible $\mu$ at once.

2.2 Aug 2023 Q2a, worked

$\hat\mu = r\bar{X}_n + (1-r)\bar{Y}_m$. Is $\hat\mu$ unbiased for $\mu$? [2 marks]

Unbiased. Three lines, two marks, and the whole thing is linearity plus $\mathbb{E}[\bar{X}] = \mu$.

2.3 When the estimator isn’t linear

$\max_i x_i$ and $1/\bar{x}$ can’t be handled by pushing expectation through, because expectation doesn’t commute with max or with reciprocal. Then you need the distribution of the estimator itself and an integral.

Papers know this is harder and supply what you need. Apr 2024 Q3b hands you the Gamma density explicitly so you can compute $\mathbb{E}[1/\bar{Y}]$.

3. Why maximum likelihood can be biased

The apparent contradiction: MLE optimizes over the parameter, so how can it systematically miss?

Because MLE and bias answer different questions.

• MLE asks: given the one sample I have, which $\theta$ makes this data most probable?
• Bias asks: if I ran this recipe on every possible sample, where would the answers centre?

MLE never looks at the samples you didn’t draw. It can’t, it only has one. So there’s no contradiction in it being biased; it’s answering its own question correctly, and that question isn’t about averages.

3.1 Where the bias comes from

Three shapes of parameter, and only one is free:

• Parameter is a centre → the MLE lands on the sample’s centre → unbiased. A sample’s centre is an unbiased estimate of the population’s centre.
• Parameter is an edge → the MLE lands on the sample’s edge → biased inward. The largest of $n$ draws is always inside the true endpoint, because you’d have to draw the endpoint exactly to reach it.
• Parameter is a spread → biased low, because the spread gets measured around $\bar{x}$ rather than around $\mu$.

3.2 The Uniform maximum, and where n/(n+1) comes from

$\text{Unif}(0,\theta)$, MLE is $\max_i x_i$. No draw can exceed $\theta$, so the max can’t either, and it almost certainly falls short.

By how much: $n$ draws mark $n$ points on the interval, and $n$ points cut it into $n+1$ segments. Since the draws are uniform, no position is special, so every segment has the same expected length by symmetry. The gap above your largest draw is one of those $n+1$ segments.

With $n = 5$ the max reads about 83% of $\theta$. Bias is $-\theta/(n+1)$.

The bias shrinks with $n$ and is never zero at finite $n$. That’s asymptotic unbiasedness, which maximum likelihood has in general.

3.3 Sample size helps, sample quality doesn’t

Worth separating, because they’re different claims.

A perfectly representative sample doesn’t help. Five evenly spaced draws from $\text{Unif}(0,100)$ land at 16.7, 33.3, 50, 66.7, 83.3. The max is 83.3 and the truth is 100. Nothing about that sample being good rescues it. The recipe reads low.

A bigger sample does help. $n = 500$ gives 99.8% of $\theta$ instead of 83%. More draws crowd closer to the endpoint.

So bias isn’t bad luck in the draw. It’s the recipe, and its size depends on $n$.

3.4 Why the MLE for variance is biased

$\hat\sigma^2 = \frac{1}{n}\sum_i (x_i - \bar{x})^2$ reads low, every sample, in the same direction.

The reason is one substitution: you measured spread around $\bar{x}$ instead of around $\mu$. And $\bar{x}$ is by definition the point that minimizes the sum of squared distances to your data. So measuring from anywhere else, including from the true $\mu$, gives a bigger number.

You’re comparing against the minimum, so you can only come out low.

Numbers. Truth is $\mu = 170$; your five heights are 165, 168, 172, 174, 176, so $\bar{x} = 171$:

Every alternative centre gives more. Not because those numbers are special, because 171 is the minimizer for that data.

And it reads low whichever way the sample is shifted. A sample of tall people has $\bar{x}$ too high, a sample of short people has it too low, and both produce a spread that is too small. Errors in $\mu$ have signs and cancel; the resulting errors in $\sigma^2$ do not.

The correction is $(n-1)$ in the denominator instead of $n$, which is why every stats package uses it.

4. The per-family table

Bias is a fact about a pair: the family and the recipe. Method of moments on the same family has a different bias than maximum likelihood does. So it’s a lookup, the same way the density and the moment formula are lookups.

The corrected Uniform estimator $\hat\theta = \frac{n+1}{n} m_n$ is the one on Aug 2023 Q4.

Method of moments is unbiased when the parameter is a linear function of the moment, and biased otherwise. $\text{Unif}(0,\theta)$ has $\theta = 2\mathbb{E}[X]$, linear, so $2\bar{x}$ is unbiased. $\text{Exp}(\lambda)$ has $\lambda = 1/\mathbb{E}[X]$, curved, so $1/\bar{x}$ is biased. Same Jensen reason that $\mathbb{E}[\sqrt{T}] \ne \sqrt{\mathbb{E}[T]}$.

5. The workflow

The course teaches estimation and bias as two units, which makes them look disconnected. They’re one loop:

1. Estimate. Run maximum likelihood or method of moments, get a formula.
2. Check. Compute $\mathbb{E}[\hat\theta]$ and see whether $\theta$ cancels.
3. Correct, if it doesn’t and correcting is worth it.

Nobody ships the raw Uniform max or the $1/n$ variance. The corrected versions are what get used, and step 2 is how you find them.

Step 3 says “if it’s worth it” because removing bias can raise variance. The raw max is biased and tight; $2\bar{x}$ is unbiased and noisy. Which is better depends on both, which is what mean squared error is for.

6. The sampling distribution

Before you draw the sample, $\hat\theta$ is random, because the sample is. So it has a distribution of its own: a centre, a spread, a shape. That’s the sampling distribution, and bias and variance are just descriptions of it.

The cleanest way to hold it: an estimator is a transformation from the family to a new distribution.

Same centre, narrower by a factor of $n$. Which is exactly why more data helps and why the centre never moves. Similarly $\max_i X_i$ turns $\text{Unif}(0,\theta)$ into something centred at $\theta \cdot \frac{n}{n+1}$, which is the bias.

Two levels, and confusing them is a real error:

• the data has a distribution, the family, with variance $\sigma^2$
• the estimator has a distribution, narrower, with variance $\sigma^2/n$

Sometimes the output distribution has a name. The sum of $n$ exponentials is $\text{Gamma}(n,\lambda)$, which is what Apr 2024 Q3b hands you so you can integrate against it. Sometimes it doesn’t, and you only compute its mean and variance. For large $n$ the central limit theorem says the sample mean’s distribution is approximately normal whatever the family, which is what makes confidence intervals possible.

7. Variance

The expected squared distance from the estimator’s own mean. Squared so deviations in both directions count as spread instead of cancelling. The second form is the one you compute with, since $\mathbb{E}[\hat\theta^2]$ is usually easier to get than the definition.

$\theta$ does not appear. Variance is scatter around wherever the estimator centres, right or wrong. Putting $\theta$ in place of $\mathbb{E}[\hat\theta]$ gives you mean squared error instead, which is the same shape measured from the truth. The three, side by side:

7.1 Every estimator has its own variance

Exactly parallel to bias: it’s a per-estimator fact, derived once and then looked up.

$\text{Var}(\bar{x}) = \sigma^2/n$ is the one worth memorising, because the sample mean is what confidence-interval questions use. The formula is the same for every family; what changes is the number you substitute for $\sigma^2$, and that’s on the formula sheet: $1/\lambda^2$ for $\text{Exp}(\lambda)$, $(b-a)^2/12$ for $\text{Unif}(a,b)$, $\lambda$ for $\text{Pois}(\lambda)$.

Two different variances, and they get confused. $\sigma^2$ is how spread out the data is, a fact about the world that doesn’t change with $n$. $\text{Var}(\bar{x})$ is how spread out your answer is, a fact about your procedure that shrinks with $n$:

Its square root, $\sigma/\sqrt{n}$, is the standard error, and it’s what appears in every confidence interval.

The one that carries most questions:

Two rules for the algebra:

Constants come out squared. And the sum rule requires $X$ and $Y$ to be independent, unlike linearity of expectation which never needs it. So bias calculations survive dependence and variance calculations don’t, which is why the questions state “independent samples” explicitly.

7.1 Unbiased and useless

An estimator can aim perfectly and still be worthless. “Use only the first observation” is unbiased for $\mu$, since a single draw has expectation $\mu$. Its variance is $\sigma^2$ rather than $\sigma^2/n$, so it scatters $n$ times as widely as the sample mean.

Right aim, terrible precision. Which is why neither number decides alone.

8. Efficiency and mean squared error

Efficiency compares two unbiased estimators: the one with smaller variance is more efficient. When a question gives you a family of unbiased estimators indexed by some constant and asks for the best one, it’s asking you to minimize the variance over that constant.

Aug 2023 Q2b is exactly this: $\hat\mu = r\bar{X}_n + (1-r)\bar{Y}_m$, all unbiased for any $r$ by part (a), and “for what value of $r$ is $\hat\mu$ most efficient?” is 8 marks of writing down $\text{Var}(\hat\mu)$ in terms of $r$ and differentiating.

8.1 The Cramér-Rao bound

Efficiency compares the estimators you happen to have. The Cramér-Rao bound says how good any unbiased estimator could possibly be: for a given family and $n$, there’s a floor below which no unbiased estimator’s variance can go.

where $I(\theta)$ is the Fisher information, a measure of how sharply the likelihood distinguishes nearby values of $\theta$. A sharply peaked likelihood means the data pins the parameter down well, so the floor is low.

An unbiased estimator that achieves the bound is efficient, and nothing unbiased beats it.

Not on any of the three past finals. Know what it claims and don’t spend an hour on it.

Mean squared error is the referee when bias and variance trade against each other:

One number for total badness. It’s what lets you compare a biased tight estimator against an unbiased noisy one. On $\text{Unif}(0,\theta)$: $\max_i x_i$ is biased with low variance, $2\bar{x}$ is unbiased with high variance, and MSE decides.

Which is also why step 3 of the workflow says “correct it if it’s worth it.” Removing bias can raise variance enough to make the total worse.

9. Consistency

The same question asked in the limit: as the sample size grows, do the bias and the variance both shrink to zero? Equivalently, does $\text{MSE} \to 0$ as $n \to \infty$.

So consistency isn’t a seventh idea. It’s MSE again, evaluated at infinity instead of at your actual $n$.

Biased and consistent is the common case, and it’s where most maximum likelihood estimators live. The Uniform max has bias $-\theta/(n+1)$:

Nonzero at every finite $n$, approaching zero. The same way $1/n$ is never zero and its limit is.

All four combinations exist:

The bottom right is the case worth noticing. Nothing requires $n$ to appear in a bias term, so an estimator can be biased by a fixed amount forever and no quantity of data helps.

Two things consistency does not tell you. It’s a statement about the limit, so it says nothing about how bad things are at the $n$ you have. An estimator that only becomes reasonable at $n = 10^9$ is consistent and useless with 25 observations. And nearly everything sensible is consistent, so passing the test rules out disasters rather than identifying good estimators.

The one direction that is airtight: if the bias grows with $n$, it cannot converge to zero, so the estimator is not consistent.