Hypothesis Testing

A confidence interval reports a range of plausible values. A hypothesis test answers a different question: the data are consistent with a specific claim, or they are not.

Definition. A statistical test is a test between two contradictory claims, based on a statistical analysis of observed data. The null hypothesis $H_0$ is the claim initially assumed true. The alternative hypothesis $H_1$ is a contradictory claim.

The null represents a default state: the status quo, no effect, zero difference. The alternative is usually the theory you would accept if the null is rejected.

The logic is a jury’s: innocent until proven guilty. The null is presumed and only abandoned if the evidence is strong enough. A test can conclude that there is enough evidence to reject $H_0$, or that there is not. It never concludes that $H_0$ is true. Failing to convict is not a finding of innocence, and the standard phrasing is “fail to reject.”

An example. For evaluation scores in a ward, testing whether the mean exceeds 80:

$H_0: \mu = 80 \qquad H_1: \mu > 80$

1. The test statistic

Definition. A test statistic is any sample statistic $T = h(X_1, \dots, X_n)$ whose numerical value is used to decide whether to reject $H_0$.

The realized value assesses how likely the data are if the null were true. The standard statistic for a mean is

$T = \frac{\bar{X}_n - \mu_0}{S_n/\sqrt{n}}$

where $\mu_0$ is the value the null asserts. Under $H_0$ the numerator has expectation zero, so $\mathbb{E}[T] = 0$: the statistic is built so that the null predicts a value near zero, and a large value is evidence against.

Comparing two wards on the same test: Ward 11 gives $t = 1.615$ and Ward 13 gives $t = 13.09$. Both are positive, and 13.09 is far out in the tail of anything the null permits.

2. The p-value

Definition. The p-value is the probability of obtaining a result equal to or more extreme than what was observed, under the null hypothesis.

For a right-tailed alternative $H_1: \mu > \mu_0$, the p-value is $P(T \ge t \mid H_0)$. The smaller the p-value, the stronger the evidence against $H_0$, because a small p-value means data like these are rare under the null.

The tail used is determined by the alternative:

• $H_1: \mu > \mu_0$ gives the right tail, $P(T \ge t)$.
• $H_1: \mu < \mu_0$ gives the left tail, $P(T \le t)$.
• $H_1: \mu \ne \mu_0$ gives both tails, $P(|T| \ge |t|)$.

The first two are one-tailed, the third is two-tailed. Picking the wrong one changes the answer by roughly a factor of two.

Worked example. $n = 50$ observations, testing $H_0: \mu = 0$ against $H_1: \mu \ne 0$ with $\sigma = 1$ known, sample mean $-0.0641$. The statistic is

$t = \frac{\bar{x}_n - \mu_0}{\sigma/\sqrt{n}} = \frac{-0.0641 - 0}{1/\sqrt{50}} = -0.453$

Two-tailed, so the p-value is $P(|Z| \ge 0.453) = 0.650$. Data like these are entirely ordinary under the null, so there is no evidence against it.

The p-value is not the probability that the null is true. It is a probability computed assuming the null, about data. The null is a fixed claim and has no probability attached, for the same reason a realized confidence interval has none.

3. Significance level and the critical region

A p-value is a continuous measure of evidence. Turning it into a decision requires a threshold, fixed in advance.

Definition. The significance level $\alpha$ is the largest acceptable probability of rejecting the null hypothesis when the null hypothesis is true, with $0 < \alpha < 1$.

Reject $H_0$ when the observed p-value is smaller than $\alpha$.

The same decision can be described in terms of the statistic rather than the p-value.

Definition. The critical region $K \subset \mathbb{R}$ is the set of values of $T$ for which $H_0$ is rejected in favour of $H_1$. Values on the boundary are critical values.

Worked example. Testing $H_0: \mu = 0$ against $H_1: \mu \ne 0$ at $\alpha = 0.1$, with $\sigma$ known. The critical values are $\pm z_{\alpha/2} = \pm 1.645$, so

$K = (-\infty, -1.645) \cup (1.645, \infty)$

The observed $t = -0.453$ is not in $K$, so do not reject. This agrees with the p-value of 0.650, which exceeds 0.1, as it must: the two routes are the same comparison read from different ends.

With $\sigma$ unknown the critical values come from the $t$ distribution: $\pm t_{n-1,\alpha/2} = \pm 1.677$ at $n = 50$, slightly wider than the normal values.

4. Choosing the test statistic

Which statistic and which reference distribution depends on two questions: is the underlying distribution normal, and is $\sigma^2$ known.

All four distributions hold under the assumption that the null hypothesis is true, which is what makes the p-value computable.

The pattern matches the confidence interval constructions exactly, because they are the same machinery. Exact results in the normal column, central limit theorem approximations in the unknown column, and the $t$ distribution appearing precisely when the variance is estimated from normal data.

5. Tests and confidence intervals are equivalent

A test using the p-value at significance level $\alpha$ and a test using the confidence interval at level $1-\alpha$ give the same decision.

Both reject when $\mu_0$ is implausible, and “implausible” means the same thing in both.

The test rejects when

$t = \frac{\bar{x}_n - \mu_0}{s_n/\sqrt{n}} \in (-\infty, -t_{n-1,\alpha/2}) \cup (t_{n-1,\alpha/2}, \infty)$

The interval rejects when

$\mu_0 \notin \left(\bar{x}_n - t_{n-1,\alpha/2}\frac{s_n}{\sqrt{n}},\ \bar{x}_n + t_{n-1,\alpha/2}\frac{s_n}{\sqrt{n}}\right)$

These are the same condition rearranged. Take the interval’s lower bound exceeding $\mu_0$:

$\bar{x}_n - t_{n-1,\alpha/2}\frac{s_n}{\sqrt{n}} > \mu_0 \iff \frac{\bar{x}_n - \mu_0}{s_n/\sqrt{n}} > t_{n-1,\alpha/2}$

which is exactly the right half of the critical region, and the upper bound gives the left half the same way.

The practical consequence is that a confidence interval answers hypothesis tests for free. Given a 90% interval of $(79.49, 85.12)$ for a ward’s mean score, the value 80 lies inside, so $H_0: \mu = 80$ is not rejected at $\alpha = 0.1$. Any $\mu_0$ inside the interval fails to be rejected, and any outside is rejected.

6. Type I and Type II errors

Two ways to be wrong, and they are not symmetric.

A Type I error is rejecting a true null: a false positive. Its probability is $\alpha$, by construction, since that is what the significance level was defined to control.

A Type II error is failing to reject a false null: a false negative. Its probability is $\beta$, and it is not chosen directly. It depends on how far the truth is from $\mu_0$, on $n$, and on $\alpha$.

The connection to confidence intervals is direct. A $(1-\alpha)$ interval misses the true parameter with probability $\alpha$, so when the null is true it excludes $\mu_0$ with probability $\alpha$, and excluding means rejecting.

The two errors trade against each other. Lowering $\alpha$ makes rejection harder, cutting Type I errors and raising Type II. The only way to reduce both is more data.

7. Cautions on p-values

A small p-value does not indicate an important departure. Statistical significance and practical importance are different. A sample mean of 80.5 with $p = 0.04$ rejects at the 5% level; a sample mean of 90 with $p = 0.051$ does not. The second is a far larger departure from $\mu_0 = 80$, and only the first clears the threshold, because the p-value confounds effect size with sample size.

A p-value gives evidence against $H_0$ and no indication of direction or magnitude. It does not tell you how wrong the null is, only how surprising the data would be if it were right.

Binary thinking hides detail. Reject against fail-to-reject collapses a continuous measure into one bit. Reporting the p-value itself, and the estimate, and an interval, carries more than the verdict.

A rough reading of the strength of evidence:

8. Worked example, end to end

Ward 13 private housing, testing whether mean evaluation score exceeds 80 at $\alpha = 0.05$.

Hypotheses. $H_0: \mu = 80$ against $H_1: \mu > 80$, one-tailed because the question asks whether quality is good rather than whether it differs.

Test statistic. Sample mean 91.36, sample standard deviation 7.56, so with $\sigma$ unknown

$t = \frac{91.36 - 80}{7.56/\sqrt{n}} = 13.09$

p-value. Right tail only, $P(T \ge 13.09)$, which is 0 to reported precision.

Decision. $0 < 0.05$, so reject $H_0$. There is very strong evidence that the mean score exceeds 80.

The same data give a 90% confidence interval that lies entirely above 80, which is the equivalence again.

Ward 11, by contrast, gives $t = 1.615$ and $p = 0.076$. At $\alpha = 0.05$ this does not reject, and its confidence interval $(79.49, 85.12)$ contains 80. The correct statement is that the data provide weak evidence against $H_0$ and do not establish either claim, not that the mean equals 80.

Vocabulary to deploy

• Null hypothesis $H_0$, alternative hypothesis $H_1$; reject or fail to reject, never accept.
• Test statistic $T$; p-value, the probability of a result as or more extreme under $H_0$.
• One-tailed and two-tailed p-values.
• Significance level $\alpha$; critical region $K$; critical values.
• Type I error (reject a true null, probability $\alpha$); Type II error (fail to reject a false null, probability $\beta$).
• Equivalence: reject at level $\alpha$ exactly when the $(1-\alpha)$ confidence interval excludes $\mu_0$.
• Statistical significance against practical importance.