Bias

An estimator is a random variable, so it has an expectation. Bias asks whether that expectation is the parameter you were trying to estimate.

Definition. Let $T$ be an estimator for a parameter $\theta$. The bias of $T$ is $\mathrm{bias}(T) = \mathbb{E}[T] - \theta$ and $T$ is unbiased if $\mathbb{E}[T] = \theta$ for every value of $\theta$.

Unbiased means correct on average across all possible samples. It does not mean close on any particular sample, and an unbiased estimator can be badly wrong every single time as long as the errors cancel. That is why bias is never the only criterion, and why Mean Squared Error exists.

Two things bias is not:

• Not a property of an estimate. A number cannot be unbiased. Bias is a property of the rule.
• Not required to hold at one $\theta$. Unbiasedness must hold for every $\theta$ in the parameter space, which is what makes it a property of the estimator rather than a coincidence.

1. Proving unbiasedness directly

The default method is to compute $\mathbb{E}[T]$ and compare it to $\theta$. Two facts carry almost every such computation.

Linearity of expectation. $\mathbb{E}[aX + bY] = a\mathbb{E}[X] + b\mathbb{E}[Y]$, whether or not $X$ and $Y$ are independent. This is the workhorse, and its independence-free status matters when the sample is dependent.

Identical distribution. All $X_i$ share one expectation, so $\mathbb{E}[X_i] = \mathbb{E}[X_1]$ and the sum collapses.

Worked example: the sample mean. For a random sample with $\mathbb{E}[X_i] = \mu$,

$\mathbb{E}[\bar{X}_n] = \mathbb{E}\left[\frac{1}{n}\sum_{i=1}^n X_i\right] = \frac{1}{n}\sum_{i=1}^n \mathbb{E}[X_i] = \frac{1}{n} \cdot n\mu = \mu$

The sample mean is always unbiased for the population mean, whatever the distribution.

Worked example: measurement error. With $M_i = c + U_i$, $\mathbb{E}[U_i] = 0$:

$\mathbb{E}[\bar{M}_n] = \frac{1}{n}\sum_{i=1}^n \mathbb{E}[M_i] = \frac{1}{n}\sum_{i=1}^n \left(c + \mathbb{E}[U_i]\right) = \frac{1}{n} \cdot nc = c$

so $\bar{M}_n$ is unbiased for $c$ for every $n$.

Worked example: rescaling to remove bias. For the discrete uniform on $\{1, \dots, N\}$, $\mathbb{E}[X_i] = \frac{N+1}{2}$, so the sample mean estimates $\frac{N+1}{2}$ rather than $N$. Invert the relationship: if $\mathbb{E}[\bar{X}_n] = \frac{N+1}{2}$ then $2\mathbb{E}[\bar{X}_n] - 1 = N$, and by linearity

$\mathbb{E}\left[2\bar{X}_n - 1\right] = 2 \cdot \frac{N+1}{2} - 1 = N$

so $T_1 = 2\bar{X}_n - 1$ is unbiased for $N$. This is the general repair. If $\mathbb{E}[T] = g(\theta)$ for an invertible linear $g$, applying $g^{-1}$ to $T$ produces an unbiased estimator. It works because expectation commutes with linear functions, and it fails for non-linear ones, which is the next section.

Worked example: a family indexed by a constant. For $X_i \overset{\text{i.i.d.}}{\sim} \mathrm{Exp}(\lambda)$ with $\mu = 1/\lambda$, consider $T_c = c\sum_{i=1}^n X_i$.

$\mathbb{E}[T_c] = c \cdot n \cdot \mathbb{E}[X_1] = \frac{cn}{\lambda} = cn\mu$

so $T_c$ is unbiased exactly when $cn = 1$, giving $c = 1/n$ and recovering the sample mean.

2. Establishing bias without computing an expectation

Some expectations are hard or impossible to compute in closed form. Three arguments settle the question anyway, and each appears on exams precisely because it avoids the calculation.

Jensen’s inequality: bias from non-linearity

Jensen’s inequality. If $\varphi$ is convex, then $\mathbb{E}[\varphi(Y)] \ge \varphi(\mathbb{E}[Y])$. If $\varphi$ is strictly convex and $Y$ is not constant, the inequality is strict. For concave $\varphi$ the inequality reverses.

The consequence for estimation is the general principle behind most bias. If $T$ is unbiased for $\theta$, then $\varphi(T)$ is biased for $\varphi(\theta)$ whenever $\varphi$ is strictly convex or strictly concave. Expectation passes through linear functions and nothing else.

Worked example. For $X_i \overset{\text{i.i.d.}}{\sim} \mathrm{Pois}(\lambda)$, the probability of a zero count is $p_0 = P(X_1 = 0) = e^{-\lambda}$. A natural estimator is $\hat{p}_0 = e^{-\bar{X}_n}$. Is it unbiased?

Take $\varphi(y) = e^{-y}$, which is strictly convex since $\varphi''(y) = e^{-y} > 0$. Jensen gives

$\mathbb{E}\left[e^{-\bar{X}_n}\right] \ge e^{-\mathbb{E}[\bar{X}_n]} = e^{-\lambda} = p_0$

and since $\varphi$ is strictly convex and $\bar{X}_n$ is not constant, the inequality is strict. So $\mathbb{E}[\hat{p}_0] > p_0$: the estimator has positive bias and is not unbiased.

Notice what this argument did not require. $\mathbb{E}[e^{-\bar{X}_n}]$ was never computed. Convexity plus unbiasedness of the inner estimator settles the direction of the bias on its own.

The bias here shrinks as $n$ grows, because $\bar{X}_n$ concentrates on $\lambda$ and a nearly-constant random variable makes Jensen’s gap nearly zero. At $n = 10$ with $\lambda = 1$ the bias is roughly 0.02.

Bounded support: bias from a hard limit

If an estimator can never exceed the parameter, and sometimes falls short, its expectation must be below the parameter.

Worked example. Raffle tickets numbered $1, \dots, N$, sampled with replacement, estimator $T_2 = \max\{X_1, \dots, X_n\}$. Is it unbiased?

Two observations. First, $T_2 \le N$ always, since every ticket number is at most $N$. Second, $P(T_2 < N) > 0$, since all $n$ draws can avoid ticket $N$, which happens with probability $\left(\frac{N-1}{N}\right)^n > 0$.

A random variable bounded above by $N$ that is strictly below $N$ with positive probability has $\mathbb{E}[T_2] < N$. So the maximum has negative bias: it systematically underestimates.

The structure generalizes. Any estimator confined to one side of the parameter is biased in that direction, and the argument is two lines with no integration.

The maximum can be repaired. For sampling without replacement from $\{1,\dots,\theta\}$ the exact expectation is computable:

$\mathbb{E}\left[X_{(n)}\right] = \frac{n(\theta+1)}{n+1} \implies T_2' = \frac{n+1}{n}X_{(n)} - 1$

is unbiased. The correction inflates the maximum to compensate for the gap between the largest observed value and the true ceiling.

The method of moments variance

The method of moments produces estimators that are often biased, and the standard case explains the $n-1$ in the sample variance.

For $X_1, \dots, X_n \overset{\text{i.i.d.}}{\sim} \mathcal{N}(\mu, \theta)$ with $\theta = \sigma^2$ unknown, matching the first two moments gives

$\hat{\theta} = \frac{1}{n}\sum_{i=1}^n X_i^2 - \left(\frac{1}{n}\sum_{i=1}^n X_i\right)^2 = \frac{1}{n}\sum_{i=1}^n \left(X_i - \bar{X}_n\right)^2$

which is the sample variance with divisor $n$ rather than $n-1$. It is biased downward, and

$\mathbb{E}\left[\frac{1}{n}\sum_{i=1}^n (X_i - \bar{X}_n)^2\right] = \frac{n-1}{n}\sigma^2$

The reason is that $\bar{X}_n$ is fitted to the same data. The deviations are measured from the sample mean, and $\bar{X}_n$ is by construction the value minimizing $\sum (x_i - a)^2$. So the sum of squared deviations from $\bar{X}_n$ is smaller than the sum from the true $\mu$, and dividing by $n-1$ instead of $n$ scales it back up by exactly the right factor:

$\mathbb{E}\left[S_n^2\right] = \mathbb{E}\left[\frac{1}{n-1}\sum_{i=1}^n (X_i - \bar{X}_n)^2\right] = \frac{n}{n-1} \cdot \frac{n-1}{n}\sigma^2 = \sigma^2$

If the true $\mu$ is known, no correction is needed and $\tilde{s}_n^2 = \frac{1}{n}\sum (X_i - \mu)^2$ is unbiased with divisor $n$, because no degree of freedom was spent.

3. Estimating bias by simulation

When the expectation is intractable, simulate: fix a true parameter value, generate many samples, compute the estimator each time, and average.

$\widehat{\mathbb{E}[T]} = \frac{1}{m}\sum_{j=1}^m t_j, \qquad \widehat{\mathrm{bias}}(T) = \widehat{\mathbb{E}[T]} - \theta$

m <- 5000
n <- 10
lambda0 <- 1
t1 <- numeric(m)
for (i in seq(m)) {
  x <- rpois(n, lambda0)
  t1[i] <- exp(-mean(x))
}
mean(t1)      # estimate of E[T1]
exp(-lambda0) # the true p0

With $n = 10$ and $\lambda_0 = 1$ this returns roughly 0.387 against a true $p_0 = e^{-1} \approx 0.368$, so the estimated bias is about $+0.02$, positive as Jensen predicts. Repeating across a grid of $n$ shows the bias shrinking toward zero, which is consistency.

A single run tells you nothing about bias. One draw at $n = 10$ might give $t_1 = 0.449$ against a true 0.368, and that gap is sampling variation, not evidence of bias. Bias is a statement about the average of many replications.

4. The limits of unbiasedness as a criterion

Unbiasedness is one property, and on its own it is a weak recommendation.

Two unbiased estimators can differ enormously in spread, which is efficiency. And a biased estimator can beat an unbiased one on total error, which is mean squared error. The raffle problem is the clean case: the unbiased $2\bar{X}_n - 1$ has a much larger variance than the biased maximum, and by mean squared error the biased estimator wins.

Vocabulary to deploy

• Bias $\mathrm{bias}(T) = \mathbb{E}[T] - \theta$; unbiased if $\mathbb{E}[T] = \theta$ for all $\theta$.
• Positive bias (overestimates on average), negative bias (underestimates).
• Linearity of expectation, which holds without independence.
• Jensen’s inequality: $\mathbb{E}[\varphi(Y)] \ge \varphi(\mathbb{E}[Y])$ for convex $\varphi$, strict when $\varphi$ is strictly convex and $Y$ is non-degenerate.
• The invariance-of-linearity principle: expectation commutes with linear functions and with nothing else.
• Bounded-support argument: $T \le \theta$ always and $T < \theta$ with positive probability implies negative bias.
• Degrees of freedom as the reason for the $n-1$ divisor.