Confidence Intervals

A single number hides its own uncertainty. Reporting that the mean failure time is 649 CPU seconds says nothing about whether the truth is plausibly 600 or plausibly 400. A confidence interval reports a range together with a statement about how often that procedure captures the parameter.

Definition. Let $\alpha \in (0,1)$. The interval $(\ell_n, u_n)$, where $\ell_n = g(x_1, \dots, x_n)$ and $u_n = h(x_1, \dots, x_n)$, is a $100(1-\alpha)\%$ confidence interval for $\theta$ if the corresponding random variables $L_n = g(X_1, \dots, X_n)$ and $U_n = h(X_1, \dots, X_n)$ satisfy $P(L_n < \theta < U_n) = 1 - \alpha$ for every value of $\theta$. If $P(L_n < \theta < U_n) \ge 1-\alpha$, the interval is conservative.

$1-\alpha$ is the confidence level, $\alpha$ is the significance level.

Which symbols are random is the whole subject and it is examined directly.

• Random: $L_n$ and $U_n$, because they are functions of the random sample.
• Not random: $\theta$, which is a fixed unknown constant; $\alpha$, which you chose; and $\ell_n, u_n$, which are numbers computed from data already in hand.

1. The pivot

Constructing an interval means finding a quantity whose distribution is known even though $\theta$ is not.

Definition. A pivot is a random variable that (1) depends only on the random sample and the parameter of interest $\theta$, and not on any other parameter, and (2) has a distribution that does not depend on $\theta$ or on any other unknown parameter.

The two conditions do different work. The first makes $\theta$ the only unknown in the expression, so it can be isolated. The second makes the probability statement usable, since a distribution depending on the unknown could not supply numerical quantiles.

The standard pivot for a normal mean with known variance is

$\frac{\bar{X}_n - \theta}{\sigma/\sqrt{n}} \sim \mathcal{N}(0,1)$

It contains $\bar{X}_n$ and $\theta$ and nothing else unknown, and its distribution is standard normal regardless of what $\theta$ is.

Critical values

Definition. $z_p$ is the critical value of the standard normal satisfying $P(Z > z_p) = p$. Equivalently $z_p$ is the $(1-p)$th quantile. By symmetry, $z_{1-p} = -z_p$.

For a 95% interval, $\alpha = 0.05$ and $z_{\alpha/2} = z_{0.025}$ is the 0.975 quantile, which is qnorm(0.975) $= 1.959964$, usually written 1.96.

2. Inversion: turning a probability statement into an interval

The derivation below is the one to be able to reproduce. Every interval in the course is this argument with a different pivot.

Start from the pivot and its known distribution. By construction, the middle $1-\alpha$ of a standard normal lies between $-z_{\alpha/2}$ and $z_{\alpha/2}$:

$P\left(-z_{\alpha/2} < \frac{\bar{X}_n - \theta}{\sigma/\sqrt{n}} < z_{\alpha/2}\right) = 1-\alpha$

Now isolate $\theta$ inside the inequality, one operation at a time.

Multiply through by $\sigma/\sqrt{n}$, which is positive so the inequalities are unchanged:

$P\left(-z_{\alpha/2}\frac{\sigma}{\sqrt{n}} < \bar{X}_n - \theta < z_{\alpha/2}\frac{\sigma}{\sqrt{n}}\right) = 1-\alpha$

Subtract $\bar{X}_n$ throughout:

$P\left(-\bar{X}_n - z_{\alpha/2}\frac{\sigma}{\sqrt{n}} < -\theta < -\bar{X}_n + z_{\alpha/2}\frac{\sigma}{\sqrt{n}}\right) = 1-\alpha$

Multiply by $-1$, which reverses both inequalities:

$P\left(\bar{X}_n - z_{\alpha/2}\frac{\sigma}{\sqrt{n}} < \theta < \bar{X}_n + z_{\alpha/2}\frac{\sigma}{\sqrt{n}}\right) = 1-\alpha$

Comparing with the definition, $L_n = \bar{X}_n - z_{\alpha/2}\frac{\sigma}{\sqrt{n}}$ and $U_n = \bar{X}_n + z_{\alpha/2}\frac{\sigma}{\sqrt{n}}$. Substituting the observed $\bar{x}_n$ gives the realized interval

$\left(\bar{x}_n - z_{\alpha/2}\frac{\sigma}{\sqrt{n}},\ \bar{x}_n + z_{\alpha/2}\frac{\sigma}{\sqrt{n}}\right)$

Sign-flipping on the multiplication by $-1$ is where this derivation is usually lost. For a symmetric pivot the two errors cancel and the answer still looks right, which is why it survives; for the chi-squared pivot below it does not.

3. The four intervals for a mean

Which construction applies depends on what you know. The decision is a two-question tree: is the data normal, and is $\sigma$ known.

Chebyshev, when nothing is known about the shape

Applicable to any distribution with finite mean and variance. Recall Chebyshev’s inequality: $P(|Y - \mathbb{E}[Y]| \ge \varepsilon) \le \mathrm{Var}(Y)/\varepsilon^2$.

Applied to $\bar{X}_n$, which has mean $\theta$ and variance $\sigma^2/n$:

$P\left(|\bar{X}_n - \theta| < \varepsilon\right) \ge 1 - \frac{\sigma^2}{n\varepsilon^2}$

Set the right side to $1-\alpha$ and solve for $\varepsilon$.

Worked example: a conservative 80% interval. Set $1 - \frac{\sigma^2}{n\varepsilon^2} = 0.8$, so $\frac{\sigma^2}{n\varepsilon^2} = 0.2$ and

$\varepsilon^2 = \frac{\sigma^2}{0.2n} = \frac{5\sigma^2}{n}, \qquad \varepsilon = \sqrt{\frac{5\sigma^2}{n}}$

giving the interval

$\left(\bar{x}_n - \sqrt{\frac{5\sigma^2}{n}},\ \bar{x}_n + \sqrt{\frac{5\sigma^2}{n}}\right)$

The statement is that we are at least 80% confident, since Chebyshev gives an inequality rather than an equality. That is what makes it conservative: the true coverage is at least the stated level and usually much more, so the interval is wider than necessary.

Normal data, known variance

The construction derived above:

$\left(\bar{x}_n - z_{\alpha/2}\frac{\sigma}{\sqrt{n}},\ \bar{x}_n + z_{\alpha/2}\frac{\sigma}{\sqrt{n}}\right)$

Normal data, unknown variance

$\sigma$ is almost never known. Replacing it with the sample standard deviation $S_n$ puts a random quantity in the denominator, so the pivot is no longer standard normal:

$\frac{\bar{X}_n - \theta}{S_n/\sqrt{n}} \sim t(n-1)$

This is the studentized mean, as against the standardized mean which divides by the known $\sigma$. The extra randomness makes the distribution more spread out than a normal, so the correct critical values come from $t(n-1)$:

$\left(\bar{x}_n - t_{n-1,\alpha/2}\frac{s_n}{\sqrt{n}},\ \bar{x}_n + t_{n-1,\alpha/2}\frac{s_n}{\sqrt{n}}\right)$

Because $t$ has heavier tails than the normal, $t_{n-1,\alpha/2} > z_{\alpha/2}$, so the interval is wider. That extra width is the price of not knowing $\sigma$, and the effect vanishes as $n$ grows.

The interval depends on neither the mean nor the variance of the data-generating distribution, which is what makes it computable.

Large sample, unknown distribution

If the data are not normal but $n$ is large, the central limit theorem gives

$\frac{\bar{X}_n - \theta}{S_n/\sqrt{n}} \ \dot\sim\ \mathcal{N}(0,1)$

approximately, so

$\left(\bar{x}_n - z_{\alpha/2}\frac{s_n}{\sqrt{n}},\ \bar{x}_n + z_{\alpha/2}\frac{s_n}{\sqrt{n}}\right)$

is an approximate interval. Note the combination: $s_n$ in place of $\sigma$, but the normal critical value rather than the $t$ one, because the justification here is the central limit theorem rather than an exact distributional result.

The fifth case, a small sample from an unknown distribution, is the bootstrap.

The comparison, on one data set

100 simulated observations from $\mathcal{N}(0,1)$, with $\bar{x}_n = -0.206$:

Every interval that estimates the variance is wider than the one that knows it. Uncertainty about $\sigma$ propagates into uncertainty about $\theta$.

4. Interpretation

This is the most-tested conceptual point in the course, and the tempting reading is wrong.

The correct statement. Before the data are seen, $L_n$ and $U_n$ are random and $\theta$ is fixed, so

$P(L_n < \theta < U_n) = 1-\alpha$

is a genuine probability statement about a random interval. Its meaning is a frequency across repeated sampling: 95% of all possible samples of size $n$ yield an interval containing the true $\theta$, and 5% yield one that does not.

Why the realized interval is different. Once you compute $\ell_n = 2$ and $u_n = 5$, nothing in the statement "$2 < \theta < 5$" is random. $\theta$ is a fixed constant and so are 2 and 5. The statement is either true or false, and no probability attaches to it. You cannot know which, but that is ignorance, not randomness.

So these are all wrong for a realized interval $(2,5)$:

1. “95% of all samples will yield a sample mean between 2 and 5.”
2. “The true mean is between 2 and 5 with 95% probability.”
3. “There is a 95% chance the true mean is between 2 and 5.”
4. “There is a 95% chance that the interval 2 to 5 includes the true mean.”

Statement 1 confuses the distribution of $\theta$ with the distribution of $\bar{X}_n$. Statements 2 and 3 assign a probability to a fixed constant. Statement 4 is the subtle one: moving the “chance” onto the interval does not help, because the realized interval is also fixed.

The approved form uses the word confidence rather than probability: we are 95% confident that the true value of $\theta$ lies between $\ell_n$ and $u_n$. The confidence attaches to the procedure, not to the particular numbers it produced.

Fifty simulated 95% intervals from $\mathcal{N}(0,1)$ data illustrate it: each interval either contains 0 or does not, and roughly two or three of the fifty miss.

5. Sample size and precision

$n$ appears in every interval through $\sqrt{n}$ in the denominator, so wider samples give narrower intervals. The width shrinks like $1/\sqrt{n}$, so quartering the width takes sixteen times the data.

What more data does not buy is coverage. The probability that the interval misses $\theta$ is $\alpha$ regardless of $n$. A 95% interval computed from a million observations is extremely narrow and still misses 5% of the time. Precision and coverage are separate knobs, and only precision responds to sample size.

6. Worked example: an interval for a standard deviation

The machinery is not restricted to means. Any pivot supports the same inversion.

Speed of light measurements, $n = 100$, modelled as $\mathcal{N}(\mu, \sigma^2)$ with both unknown, and we want a 95% interval for $\sigma$.

There is no simple distribution for $S_n$ itself, but there is one for a function of it:

$\frac{(n-1)S_n^2}{\sigma^2} \sim \chi^2(n-1)$

which is a pivot: it contains only the sample and $\sigma$, and its distribution depends on neither.

$P\left(c_l < \frac{(n-1)S_n^2}{\sigma^2} < c_u\right) = 0.95$

The chi-squared distribution is not symmetric, so the two critical values must be looked up separately rather than differing by a sign:

qchisq(0.025, n - 1)   # 73.36108
qchisq(0.975, n - 1)   # 128.422

Inverting for $\sigma$ requires taking reciprocals, which reverses the inequalities and swaps which critical value goes where:

$\sqrt{\frac{(n-1)S_n^2}{c_u}} < \sigma < \sqrt{\frac{(n-1)S_n^2}{c_l}}$

With $s_n = 79.01$:

sqrt((n - 1) / c(c_u, c_l)) * s_n   # 69.3718 91.7846

The interval is $(69.37, 91.78)$, and $s_n = 79.01$ is not at its centre, because the chi-squared distribution is skewed. Any interval built on a non-symmetric pivot is asymmetric about the estimate.

Vocabulary to deploy

• Confidence interval, confidence level $1-\alpha$, significance level $\alpha$.
• Conservative interval: coverage at least $1-\alpha$.
• Pivot: depends only on the sample and $\theta$, with a distribution free of unknowns.
• Inversion: rearranging a probability statement about a pivot to isolate $\theta$.
• Critical value $z_p$, $t_{m,p}$; standard error, the standard deviation of an estimator.
• Standardization (divide by $\sigma$) against studentization (divide by $S_n$).
• Coverage against precision: $n$ buys precision only.