7: Hypothesis Testing

Hypothesis testing provides a framework for making binary decisions from data: is the observed effect real, or could it have arisen by chance? This framework underpins A/B testing, model comparison, and scientific inference.

Framework

A hypothesis test has four components:

1. Null hypothesis $H_0$: the default assumption (no effect, no difference)
2. Alternative hypothesis $H_1$: the claim we want to support
3. Test statistic $T(\mathbf{X})$: a function of the data that measures evidence against $H_0$
4. Decision rule: reject $H_0$ if $T$ falls in the rejection region

Example Setup

Testing whether a model improves over a baseline:

• $H_0$: $\mu_{\text{model}} - \mu_{\text{baseline}} = 0$ (no improvement)
• $H_1$: $\mu_{\text{model}} - \mu_{\text{baseline}} < 0$ (model has lower error)
• Test statistic: $T = \frac{\bar{X}_{\text{model}} - \bar{X}_{\text{baseline}}}{SE}$
• Reject $H_0$ if $T < -z_\alpha$

Error Types

Type I error rate $\alpha$: probability of rejecting $H_0$ when it is true (false positive). Conventionally set to 0.05.

Type II error rate $\beta$: probability of failing to reject $H_0$ when it is false (false negative).

Power $= 1 - \beta$: probability of correctly rejecting a false $H_0$. Depends on:

• Effect size: larger effects are easier to detect
• Sample size: more data increases power
• Significance level: higher $\alpha$ increases power (at the cost of more false positives)
• Variance: lower noise increases power

P-Values

The p-value is the probability of observing a test statistic as extreme as or more extreme than the observed value, assuming $H_0$ is true:

for a one-sided test (or $P(|T| \geq |t_{\text{obs}}|)$ for two-sided).

Decision rule. Reject $H_0$ if $p \leq \alpha$.

What a p-value is: The probability of the data (or more extreme) under $H_0$.

What a p-value is not:

• The probability that $H_0$ is true
• The probability that the result is due to chance
• A measure of effect size

A small p-value with a large sample can correspond to a trivially small effect. Always report effect sizes alongside p-values.

Common Tests

One-Sample t-Test

Test $H_0: \mu = \mu_0$ with unknown variance:

Two-Sample t-Test

Test $H_0: \mu_1 = \mu_2$ (independent groups):

with Welch’s approximation for degrees of freedom (does not assume equal variances).

Paired t-Test

For paired observations $(X_i, Y_i)$, test $H_0: \mu_D = 0$ where $D_i = X_i - Y_i$:

Paired tests are more powerful than two-sample tests when pairs are correlated (e.g., two models evaluated on the same test set).

Chi-Squared Test

Test independence in a contingency table or goodness-of-fit:

where $O_i$ are observed counts and $E_i$ are expected counts under $H_0$.

Permutation Test

A nonparametric test that computes the test statistic under all (or many random) permutations of group labels:

1. Compute the observed test statistic $t_{\text{obs}}$
2. Randomly permute group labels $B$ times
3. Compute $t^*_b$ for each permutation
4. $p = \frac{1}{B}\sum_{b=1}^B \mathbb{I}[t^*_b \geq t_{\text{obs}}]$

No distributional assumptions required. Exact when all permutations are enumerated.

Multiple Testing

Testing $m$ hypotheses simultaneously at level $\alpha$ each gives a family-wise error rate of $1 - (1-\alpha)^m \approx m\alpha$ for small $\alpha$. Testing 20 metrics at $\alpha = 0.05$ gives a 64% chance of at least one false positive.

Bonferroni Correction

Test each hypothesis at $\alpha/m$. Controls the family-wise error rate (FWER): $P(\text{any false rejection}) \leq \alpha$. Conservative: may miss real effects.

Benjamini-Hochberg (BH)

Controls the false discovery rate (FDR): $E[\text{false rejections}/\text{total rejections}] \leq q$.

Algorithm:

1. Sort p-values: $p_{(1)} \leq p_{(2)} \leq \cdots \leq p_{(m)}$
2. Find the largest $k$ such that $p_{(k)} \leq \frac{k}{m} q$
3. Reject hypotheses $1, \ldots, k$

BH is less conservative than Bonferroni and is preferred when controlling the proportion of false discoveries (rather than any false discovery) is acceptable.

Application to A/B Testing

A/B testing is hypothesis testing applied to product metrics:

Sample size calculation. For a two-sample z-test detecting effect size $\delta$ at significance $\alpha$ and power $1-\beta$:

Sequential testing. Standard tests assume a fixed sample size. “Peeking” (checking results before the planned sample size) inflates the false positive rate. Group sequential methods and always-valid p-values allow early stopping while maintaining error guarantees.

The Kolmogorov-Smirnov Test

The KS test compares two empirical distributions (or one empirical vs a reference):

The tracker model uses KS tests to diagnose feature-space disjointness between labeled and unlabeled domain populations, finding $p < 0.001$ on every feature. This confirmed that the domain-level formulation was structurally inappropriate.

Summary

Hypothesis testing provides the decision framework; confidence intervals provide the uncertainty quantification. Report both: the CI conveys the range of plausible effects, the p-value conveys whether the effect is distinguishable from chance.