Estimators and Sampling Distributions

An estimator is a rule for turning data into a guess about an unknown parameter. Since the data are random, the guess is random too, and it has a distribution. Everything in the rest of the course is a claim about that distribution.

1. Statistic, estimator, estimate

Definition. Given a random sample $X_1, \dots, X_n$, a sample statistic is any function of the sample, $T = h(X_1, X_2, \dots, X_n)$.

A statistic is a random variable, because it is a function of random variables. $\bar{X}_n$ is a statistic, $S_n^2$ is a statistic, and so is $X_1$ on its own, and so is the maximum $X_{(n)}$.

An estimator is a statistic used to estimate a parameter. Nothing distinguishes it mathematically from any other statistic; the difference is intent.

An estimate is the number the estimator produces once data arrive.

$\underbrace{T = h(X_1, \dots, X_n)}_{\text{estimator, a random variable}} \qquad \underbrace{t = h(x_1, \dots, x_n)}_{\text{estimate, a number}}$

Both halves of this distinction earn marks and the phrasing matters. The estimator is a rule applied to the random sample before the measurements are observed, and it has a distribution. The estimate is the realized value obtained once the actual measurements are recorded, and it is a fixed number with no distribution.

Concretely, with measurements $M_i = c + U_i$ of an unknown concentration $c$, the estimator is $\bar{M}_n = \frac{1}{n}\sum_{i=1}^n M_i$ and the estimate is $\bar{m}_n = \frac{1}{n}\sum_{i=1}^n m_i$ computed from the recorded numbers.

2. The sampling distribution

Definition. The sampling distribution of a statistic is its probability distribution: the distribution of the values it would take across all possible samples of size $n$.

This is the object the course is about, and it is worth being concrete about what it means, because it describes an experiment nobody performs.

Draw a sample of size $n$ and compute $T$. Throw the sample away, draw another, compute $T$ again. The values differ, because the samples differ. Repeat forever and the histogram of $T$ values is its sampling distribution.

You only ever see one sample and one value. The sampling distribution is what lets you say something about how far that one value is likely to be from the truth.

Worked example. Deliveries take $W = X + Y$ minutes, where $X$ is preparation and $Y$ is travel time. The app refunds any order over 45 minutes, so the parameter of interest is $p = P(W > 45)$.

Define the indicator $I_i = \mathbb{I}(W_i > 45)$, which is 1 if order $i$ was refunded and 0 otherwise. Then $I_i \sim \mathrm{Bernoulli}(p)$, so $\mathbb{E}[I_i] = p$ and $\mathrm{Var}(I_i) = p(1-p)$.

The estimator is the sample proportion, which is a sample mean of indicators:

$\hat{p}_n = \frac{1}{n}\sum_{i=1}^n I_i = \bar{I}_n$

Its mean and variance follow immediately:

$\mathbb{E}[\hat{p}_n] = \frac{1}{n}\sum_{i=1}^n \mathbb{E}[I_i] = \frac{1}{n}\cdot np = p$

$\mathrm{Var}(\hat{p}_n) = \frac{1}{n^2}\sum_{i=1}^n \mathrm{Var}(I_i) = \frac{1}{n^2}\cdot np(1-p) = \frac{p(1-p)}{n}$

The variance step uses independence: the variance of a sum of independent random variables is the sum of the variances.

A detail that reveals whether the sampling distribution is understood. With $n = 10$, can $\hat{p}_n$ equal 0.15? No. The numerator is a count between 0 and 10, so the estimator can only take the values $0, 0.1, 0.2, \dots, 1$. The sampling distribution is discrete, and it becomes finer as $n$ grows.

3. The mean and variance of the sample mean

Two facts used constantly. For a random sample with $\mathbb{E}[X_i] = \mu$ and $\mathrm{Var}(X_i) = \sigma^2$:

$\mathbb{E}[\bar{X}_n] = \mu, \qquad \mathrm{Var}(\bar{X}_n) = \frac{\sigma^2}{n}, \qquad \mathrm{sd}(\bar{X}_n) = \frac{\sigma}{\sqrt{n}}$

The derivations are one line each. For the mean, using linearity:

$\mathbb{E}[\bar{X}_n] = \mathbb{E}\left[\frac{1}{n}\sum_{i=1}^n X_i\right] = \frac{1}{n}\sum_{i=1}^n \mathbb{E}[X_i] = \frac{1}{n}\cdot n\mu = \mu$

For the variance, using independence and the fact that constants come out squared:

$\mathrm{Var}(\bar{X}_n) = \mathrm{Var}\left(\frac{1}{n}\sum_{i=1}^n X_i\right) = \frac{1}{n^2}\sum_{i=1}^n \mathrm{Var}(X_i) = \frac{1}{n^2}\cdot n\sigma^2 = \frac{\sigma^2}{n}$

Which fact needs which assumption is examinable. The mean calculation uses only linearity of expectation, which holds whether or not the $X_i$ are independent. The variance calculation needs independence, because otherwise covariance terms survive:

$\mathrm{Var}(X_i + X_j) = \mathrm{Var}(X_i) + \mathrm{Var}(X_j) + 2\mathrm{Cov}(X_i, X_j)$

This matters in the German tank problem, where serial numbers are sampled without replacement and are therefore dependent. The expectation argument still goes through; a variance argument would not.

The consequence of $\mathrm{Var}(\bar{X}_n) = \sigma^2/n$ is the reason more data helps. The spread of the estimator shrinks as $n$ grows, so the estimate concentrates around the truth. Note that the standard deviation falls as $1/\sqrt{n}$, so halving the spread requires four times the data.

4. Chebyshev’s inequality

Theorem. For a random variable $Y$ with $\mathbb{E}[Y] < \infty$ and $\mathrm{Var}(Y) < \infty$, and any $\varepsilon > 0$, $P\left(|Y - \mathbb{E}[Y]| \ge \varepsilon\right) \le \frac{\mathrm{Var}(Y)}{\varepsilon^2}$

In words: the probability of landing more than $\varepsilon$ from the mean is bounded by the variance divided by $\varepsilon^2$. Small variance forces the random variable to stay near its mean.

The bound needs no assumption about the shape of the distribution, which makes it universally applicable and usually loose. That combination is exactly its role: it proves things in general, and gives conservative numbers in particular.

Proof. Start from the definition of variance and use the law of the unconscious statistician:

$\mathrm{Var}(Y) = \mathbb{E}\left[(Y - \mathbb{E}[Y])^2\right] = \int_{-\infty}^{\infty} (y - \mathbb{E}[Y])^2 f_Y(y)\, dy$

The integrand is non-negative, so restricting the region of integration can only decrease the value:

$\mathrm{Var}(Y) \ge \int_{|y - \mathbb{E}[Y]| \ge \varepsilon} (y - \mathbb{E}[Y])^2 f_Y(y)\, dy$

On that region $(y - \mathbb{E}[Y])^2 \ge \varepsilon^2$, so replacing the integrand by $\varepsilon^2$ again only decreases it:

$\mathrm{Var}(Y) \ge \varepsilon^2 \int_{|y - \mathbb{E}[Y]| \ge \varepsilon} f_Y(y)\, dy = \varepsilon^2 P\left(|Y - \mathbb{E}[Y]| \ge \varepsilon\right)$

Rearranging gives the result. $\blacksquare$

Worked example: how many measurements? A sensor measures a true concentration $c$ with independent errors of mean 0 and variance 4. Using $\bar{M}_n$, how large must $n$ be to guarantee $P(|\bar{M}_n - c| < 0.5) \ge 0.95$?

Here $\mathbb{E}[\bar{M}_n] = c$ and $\mathrm{Var}(\bar{M}_n) = 4/n$. Chebyshev with $\varepsilon = 0.5$:

$P\left(|\bar{M}_n - c| \ge 0.5\right) \le \frac{4/n}{0.5^2} = \frac{4/n}{1/4} = \frac{16}{n}$

so

$P\left(|\bar{M}_n - c| < 0.5\right) \ge 1 - \frac{16}{n}$

Requiring this to be at least 0.95:

$1 - \frac{16}{n} \ge 0.95 \iff \frac{16}{n} \le 0.05 \iff n \ge \frac{16}{0.05} = 320$

The smallest such $n$ is $320$.

5. The law of large numbers

Theorem (weak law of large numbers). Let $X_1, X_2, \dots$ be independent with common mean $\mu < \infty$ and common variance $\sigma^2 < \infty$. Then for every $\varepsilon > 0$, $\lim_{n \to \infty} P\left(|\bar{X}_n - \mu| \ge \varepsilon\right) = 0$ written $\bar{X}_n \xrightarrow{p} \mu$, and read as converges in probability.

The sample mean concentrates on the true mean as the sample grows. This is the theorem that licenses estimating an expectation by an average.

Proof. Apply Chebyshev to $\bar{X}_n$, which has mean $\mu$ and variance $\sigma^2/n$:

$P\left(|\bar{X}_n - \mu| \ge \varepsilon\right) \le \frac{\mathrm{Var}(\bar{X}_n)}{\varepsilon^2} = \frac{\sigma^2}{n\varepsilon^2}$

Since $\sigma^2 < \infty$ and $\varepsilon$ is fixed, the right side tends to 0 as $n \to \infty$. $\blacksquare$

The proof shows why both finiteness conditions appear in the statement: an infinite variance breaks the bound.

Applying the law to something other than the sample mean

The law is stated for a sample mean, and its reach comes from noticing that many statistics are sample means of something.

The recipe. To show $T_n \xrightarrow{p} c$, write $T_n$ as an average of i.i.d. random variables $Y_i$, check that $\mathbb{E}[Y_i]$ and $\mathrm{Var}(Y_i)$ are finite, and conclude $T_n \xrightarrow{p} \mathbb{E}[Y_i]$.

Worked example: a variance estimator. With $\mu$ known, consider

$\tilde{s}_n^2 = \frac{1}{n}\sum_{i=1}^n (X_i - \mu)^2$

Set $Y_i = (X_i - \mu)^2$. If the $X_i$ are i.i.d. then so are the $Y_i$, and $\tilde{s}_n^2 = \bar{Y}_n$ is their sample mean. Now

$\mathbb{E}[Y_i] = \mathbb{E}\left[(X_i - \mu)^2\right] = \mathrm{Var}(X_i) = \sigma^2 < \infty$

and $\mathrm{Var}(Y_i) < \infty$ is the extra condition required, which is a statement about the fourth moment of $X_i$. Given both, the law gives $\tilde{s}_n^2 \xrightarrow{p} \sigma^2$.

Worked example: the empirical CDF. Fix a constant $a$ and let $G_{n,a} = F_n(a)$ be the empirical CDF evaluated at $a$. Writing $I_i = \mathbb{I}(Y_i \le a)$,

$F_n(a) = \frac{1}{n}\sum_{i=1}^n I_i$

which is a sample mean of Bernoulli variables. Their mean is $\mathbb{E}[I_i] = P(Y_i \le a) = F(a)$, and an indicator has variance $F(a)(1-F(a)) \le 1/4 < \infty$. So $F_n(a) \xrightarrow{p} F(a)$: the empirical CDF converges to the true CDF at every point.

Worked example: a second moment. For $X_i \sim \mathrm{Pois}(\lambda)$, find $c$ with $Y_n = \frac{1}{n}\sum X_i^2 \xrightarrow{p} c$.

$Y_n$ is the sample mean of $X_i^2$, and all moments of a Poisson are finite, so the law gives $Y_n \xrightarrow{p} \mathbb{E}[X_1^2]$. Using the variance identity $\mathrm{Var}(X) = \mathbb{E}[X^2] - (\mathbb{E}[X])^2$ rearranged,

$\mathbb{E}[X_1^2] = \mathrm{Var}(X_1) + (\mathbb{E}[X_1])^2 = \lambda + \lambda^2$

so $c = \lambda + \lambda^2$.

A warning about transformations

Convergence does not automatically pass through a function. Knowing $\bar{X}_n \xrightarrow{p} 1/\lambda$ does not by itself establish $1/\bar{X}_n \xrightarrow{p} \lambda$, and the corresponding statement about expectations is false in general: $\mathbb{E}[1/\bar{X}_n] \ne 1/\mathbb{E}[\bar{X}_n]$. This is the same phenomenon that produces bias in Bias.

6. The central limit theorem

The law of large numbers says where $\bar{X}_n$ ends up. The central limit theorem says what its distribution looks like on the way, and that answer is the same for every starting distribution.

Theorem (central limit theorem). Let $X_1, \dots, X_n \overset{\text{i.i.d.}}{\sim} F$ for any $F$ with finite mean $\mu$ and finite variance $\sigma^2 > 0$. Then $Z_n = \frac{\bar{X}_n - \mu}{\sigma/\sqrt{n}} \xrightarrow{d} Z \sim \mathcal{N}(0,1)$ equivalently, for large $n$, $\bar{X}_n$ is approximately $\mathcal{N}\!\left(\mu, \frac{\sigma^2}{n}\right)$.

The phrase carrying the weight is “any $F$.” The sample mean of exponentials, Poissons, or Bernoullis is approximately normal for large $n$, which is why normal-based procedures apply to data that are not remotely normal.

Standardization

Standardizing subtracts the mean and divides by the standard deviation, producing a variable with mean 0 and variance 1.

$Z = \frac{X - \mu}{\sigma} \qquad \text{and for a sample mean} \qquad Z_n = \frac{\bar{X}_n - \mu}{\sigma/\sqrt{n}} = \sqrt{n}\,\frac{\bar{X}_n - \mu}{\sigma}$

The two forms of $Z_n$ are the same expression, since dividing by $\sigma/\sqrt{n}$ is multiplying by $\sqrt{n}/\sigma$. The denominator is the standard deviation of $\bar{X}_n$, not of $X_i$, which is the step most often got wrong.

The sample proportion

Applying the theorem to indicators gives the result used constantly.

Let $p = P(A)$ and let $\hat{p}_n$ be the proportion of $n$ independent trials in which $A$ occurs. Then $\mathbb{E}[\hat{p}_n] = p$, $\mathrm{Var}(\hat{p}_n) = \frac{p(1-p)}{n}$, and for large $n$, $\hat{p}_n$ is approximately $\mathcal{N}\!\left(p, \frac{p(1-p)}{n}\right)$.

Worked example. Each customer leaves a review independently with probability $p = 0.3$. What is the probability that more than 40 of the next 100 do?

Let $G_i$ indicate that customer $i$ leaves a review, so $G_i \sim \mathrm{Bernoulli}(0.3)$ and $\sum_{i=1}^{100} G_i \sim \mathrm{Binomial}(100, 0.3)$.

Exact. $P\left(\sum G_i > 40\right) = 1 - P\left(\sum G_i \le 40\right)$, which is 1 - pbinom(40, 100, .3) $= 0.01249841$.

By the central limit theorem. Convert the count to a proportion by dividing both sides by $n$:

$P\left(\sum_{i=1}^{100} G_i > 40\right) = P\left(\bar{G}_n > 0.4\right)$

with $\bar{G}_n$ approximately $\mathcal{N}\!\left(0.3, \frac{0.3 \times 0.7}{100}\right)$. Standardizing,

$P\left(\bar{G}_n > 0.4\right) \approx P\left(Z > \frac{0.4 - 0.3}{\sqrt{0.3 \times 0.7/100}}\right)$

which is 1 - pnorm(.4, mean = .3, sd = sqrt(.3 * .7 / 100)) $= 0.01454817$.

The two differ because $n = 100$ is not infinite. The approximation improves with $n$: at $n = 10$ the sampling distribution of a proportion is visibly discrete and lumpy, by $n = 500$ it is indistinguishable from a normal curve.

Chebyshev against the central limit theorem

Both bound the probability that $\bar{X}_n$ is far from $\mu$, and they answer different questions.

• Chebyshev gives a guaranteed bound with no distributional assumption, valid at any $n$, and conservative.
• The central limit theorem gives an approximation, requires $n$ to be large, and is far tighter.

Use Chebyshev when the question says “guarantee” or gives no distributional information; use the central limit theorem when it says “approximately” or $n$ is large.

7. Distributions built from the normal

Three distributions arise from normal samples. They matter because the sample variance is itself random, and accounting for that randomness produces these shapes.

Chi-squared

Definition. The $\chi^2(n)$ distribution is the distribution of $X = \sum_{i=1}^n Z_i^2$ where $Z_1, \dots, Z_n$ are independent standard normals. Its mean is $\mathbb{E}[X] = n$.

The parameter $n$ is the degrees of freedom.

Result. For $Y_1, \dots, Y_n \overset{\text{i.i.d.}}{\sim} \mathcal{N}(\mu, \sigma^2)$, $\frac{(n-1)S_n^2}{\sigma^2} \sim \chi^2(n-1)$

The degrees of freedom is $n-1$ rather than $n$ because the deviations are taken from $\bar{Y}_n$ rather than $\mu$. Once $n-1$ of the deviations are known the last is determined, since they sum to zero, so one degree of freedom has been spent estimating the mean.

The t distribution

Definition. The $t(n)$ distribution is the distribution of $Y = \frac{Z}{\sqrt{X_n/n}}$ where $Z \sim \mathcal{N}(0,1)$, $X_n \sim \chi^2(n)$, and the two are independent.

Result. For $Y_1, \dots, Y_n \overset{\text{i.i.d.}}{\sim} \mathcal{N}(\mu, \sigma^2)$, $\frac{\bar{Y}_n - \mu}{S_n/\sqrt{n}} \sim t(n-1)$

This is the reason the t distribution exists. Standardizing with the true $\sigma$ gives a standard normal. Replacing $\sigma$ by the estimate $S_n$ puts a random quantity in the denominator, and that extra randomness makes the ratio more spread out than a normal. The $t$ density is bell-shaped and symmetric but with heavier tails.

As $n \to \infty$ the $t(n)$ distribution converges to the standard normal, because $S_n$ converges to $\sigma$ and the denominator stops being random. At $n = 1$ the $t$ is visibly heavy-tailed; by $n = 20$ it is close to normal.

The F distribution

Definition. The $F(m,n)$ distribution is the distribution of $W = \frac{X_m/m}{Y_n/n}$ where $X_m \sim \chi^2(m)$, $Y_n \sim \chi^2(n)$, independent.

A ratio of scaled chi-squared variables, used to compare two variances.

Vocabulary to deploy

• Sample statistic $h(X_1, \dots, X_n)$; estimator against estimate.
• Sampling distribution: the distribution of a statistic across repeated samples.
• Converges in probability, $\bar{X}_n \xrightarrow{p} \mu$; converges in distribution, $Z_n \xrightarrow{d} Z$.
• Chebyshev’s inequality, distribution-free and conservative.
• Weak law of large numbers, proved from Chebyshev.
• Central limit theorem, valid for any distribution with finite mean and variance.
• Standardization $Z_n = \frac{\bar{X}_n - \mu}{\sigma/\sqrt{n}}$.
• Degrees of freedom; chi-squared, t and F distributions.
• Linearity of expectation, which needs no independence, against the variance-of-a-sum rule, which does.