From STA238 Probability Statistics and Data Analysis II

Estimating a parameter

1. The question

You’re given a distribution and a set of data drawn from it, and one of the distribution’s parameters is unknown. Estimate it.

Two methods, and the question always names which one. Across all three past finals there isn’t a single case where you choose. Same input and output either way, different machinery inside, and they can give different answers on the same problem.

Paper	Question	Method named	Marks
Apr 2023 Q1	”Find the maximum likelihood estimator of θ”	MLE	3 + 3 + 4
Aug 2023 Q3a	”Derive the maximum likelihood estimate for σ”	MLE	8
Aug 2023 Q3b	”State the maximum likelihood estimates for α and β”	MLE, support case	2
Apr 2024 Q1a	”Derive the maximum likelihood estimator for θ”	MLE	4
Apr 2024 Q3a	”What is the method of moments estimator for λ?”	MoM	2

Two things that fall out of that table. Apr 2023 Q1 splits the four steps into separately marked parts, which confirms the marks are per step. And method of moments is worth far fewer marks than maximum likelihood, because it’s one line of algebra rather than four steps.

Aug 2023 Q3b says “state” rather than “derive,” which is the tell that it’s the support case where there’s nothing to differentiate.

2. Maximum likelihood

The idea: try every possible value of the parameter, and pick the one that makes the data you actually saw most probable.

Not the most probable parameter. The parameter under which your data was least surprising. The parameter itself is a fixed unknown, not something with a distribution.

The likelihood of a parameter is the probability of seeing all your data if that parameter were true. The observations are independent, so that’s the product of their individual densities:

L(\theta) = \prod_{i=1}^n f_\theta(x_i)

2.1 The four steps

Write the likelihood. Substitute the density and take the product over $i$ .
Take the log. The product becomes a sum, which is what makes it differentiable.
Differentiate with respect to the parameter and set to zero.
Solve for the parameter.

Only the algebra inside changes with the distribution. The steps never do.

Step 2 is legal because $\log$ is increasing, so it doesn’t move where the maximum is. It’s necessary because differentiating a product of $n$ terms is unpleasant and differentiating a sum is not.

Step 3 finds the peak: the log-likelihood is a curve in $\theta$ , and at its top the slope is zero.

There’s a fifth step some questions want, checking the second derivative is negative to confirm it’s a maximum. Papers often say you may assume it.

2.2 Worked: exponential

$X_1, \dots, X_n \overset{iid}{\sim} \text{Exp}(\lambda)$ , with $f(x) = \lambda e^{-\lambda x}$ .

L(\lambda) = \prod_{i=1}^n \lambda e^{-\lambda x_i} = \lambda^n e^{-\lambda \sum_i x_i}

$n$ copies of $\lambda$ multiplied give $\lambda^n$ , and multiplying exponentials adds the exponents.

\ell(\lambda) = n \log \lambda - \lambda \sum_i x_i

\frac{d\ell}{d\lambda} = \frac{n}{\lambda} - \sum_i x_i = 0 \quad \Longrightarrow \quad \hat\lambda = \frac{n}{\sum_i x_i} = \frac{1}{\bar{x}}

Note this estimator is biased. Maximum likelihood promises nothing about bias.

3. When the parameter is in the support

The support is the range of values the distribution can produce. Usually the parameter only appears in the formula and the range is fixed, and then the four steps work.

Sometimes the parameter is one of the bounds. $\text{Unif}(0, \theta)$ has range $0 \le x \le \theta$ , so changing $\theta$ changes which values are possible at all.

Then there’s a cliff rather than a peak. If $\theta$ were below your largest observation, that observation would have been impossible, so the likelihood is exactly zero there:

L(\theta) = \begin{cases} 1/\theta^n & \theta \ge \max_i x_i \\ 0 & \text{otherwise}\end{cases}

Differentiating gets you nothing, because $1/\theta^n$ is strictly decreasing and its derivative is never zero. So reason about it instead:

Write the likelihood with its case split.
State which direction it moves. Smaller $\theta$ gives a larger likelihood, because a narrower uniform is a taller one.
Name the constraint. $\theta$ cannot go below $\max_i x_i$ .
Push $\theta$ to the boundary: $\hat\theta = \max_i x_i$ .

Say on the page that the likelihood isn’t differentiable at the boundary and the maximum is at the edge of the feasible region. That’s where the marks are.

The recognition rule: if the parameter appears in the limits of the support, do not differentiate.

For $\text{Unif}(\alpha, \beta)$ with both unknown, the same argument twice gives $\hat\alpha = \min_i x_i$ and $\hat\beta = \max_i x_i$ . That’s Aug 2023 Q3b, 2 marks.

4. Method of moments

The other recipe, and the more intuitive one. Same input and output as maximum likelihood, different machinery, and in a sense the opposite direction:

Maximum likelihood searches. Consider every candidate parameter, score each by how probable it makes the data, take the winner.
Method of moments inverts. Compute a summary of the data, and read the parameter off it using a formula you already have.

The name is jargon rather than description. “Moment” is the technical term for $\mathbb{E}[X^k]$ , and the method matches the theoretical moments to the sample ones.

4.1 What a moment is

A moment is an expected value of a power of the random variable. The first moment is $\mathbb{E}[X]$ , which is the mean. The second is $\mathbb{E}[X^2]$ . The $k$ -th is $\mathbb{E}[X^k]$ . That’s the whole definition.

Moments are not parameters. They’re a bridge between the two things you have. For any family, the moments are known formulas in the parameter, and those formulas are on the formula sheet:

Family	$\mathbb{E}[X]$
$\text{Exp}(\lambda)$	$1/\lambda$
$\text{Unif}(0,\theta)$	$\theta/2$
$\text{Pois}(\lambda)$	$\lambda$
$\text{Binom}(n,p)$	$np$

So a moment is something you can estimate from the sample and something with a known relationship to the parameter. That’s what makes it usable.

4.2 The method

You want the parameter and can’t observe it.
You can observe the sample mean, which estimates the true mean.
The true mean is a known formula in the parameter.
Substitute and solve backwards.

For $\text{Exp}(\lambda)$ : theoretical mean is $1/\lambda$ , sample mean is $\bar{x}$ , so $1/\lambda = \bar{x}$ and $\hat\lambda = 1/\bar{x}$ . One line.

For $\text{Pois}(\lambda)$ the mean is $\lambda$ , so $\hat\lambda = \bar{x}$ directly. That’s a coincidence of that family, not the general pattern.

4.3 More than one unknown

One unknown parameter needs one equation, so the first moment is enough. Two unknowns need two equations, so you use the first and second moments together:

\mathbb{E}[X] = \bar{x} \qquad \text{and} \qquad \mathbb{E}[X^2] = \frac{1}{n}\sum_i x_i^2

and solve the pair. Normal with both $\mu$ and $\sigma^2$ unknown is the standard case.

That’s why it’s moments plural: you use as many as you have unknowns.

4.4 It can disagree with maximum likelihood

Same problem, two legitimate estimators, different answers. $\text{Unif}(0,\theta)$ has mean $\theta/2$ , so method of moments gives $\hat\theta = 2\bar{x}$ , while maximum likelihood gives $\max_i x_i$ .

The MoM one is unbiased and noisy. The MLE one is biased low and much tighter. Neither dominates, which is what judging estimators is for.

Use maximum likelihood by default. Use method of moments when the question asks for it by name, or when the likelihood is intractable.

5. Invariance

If a question hands you the MLE of one thing and asks for the MLE of a function of it, apply the function. The MLE of $g(\theta)$ is $g(\hat\theta)$ , for invertible $g$ . Given $\hat\sigma^2$ , the MLE of $\sigma$ is $\sqrt{\hat\sigma^2}$ .

Why it works. Square root is monotonic on positives, so it doesn’t move where the maximum is. Whatever value of $\sigma^2$ maximizes the likelihood, its square root maximizes the same likelihood written in terms of $\sigma$ . Nothing was recomputed, the axis was relabelled.

Say “by invariance” and apply the function. Free marks.

5.1 Why it needs a name

Because the same move fails for expectation, and the two get conflated.

$\mathbb{E}[T] = \sigma^2$ does not give $\mathbb{E}[\sqrt{T}] = \sigma$ . Concretely, suppose $T$ is 4 half the time and 16 half the time:

\mathbb{E}[T] = 10, \qquad \sqrt{\mathbb{E}[T]} \approx 3.16

\sqrt{4} = 2, \quad \sqrt{16} = 4, \qquad \mathbb{E}[\sqrt{T}] = 3

3 and 3.16 are different. Square root is curved: it pulls large values down harder than small ones, so shrinking first and then averaging gives less than averaging first.

For a straight line it’s fine. $\mathbb{E}[2T + 5] = 2\mathbb{E}[T] + 5$ always, because linear functions preserve averages. The general statement is Jensen’s inequality: for a concave $g$ like $\sqrt{\cdot}$ , $\mathbb{E}[g(T)] \le g(\mathbb{E}[T])$ , and the inequality flips for a convex $g$ like squaring.

So both of these are true of the same estimator at once: $\sqrt{\hat\sigma^2}$ is the maximum likelihood estimator of $\sigma$ , and it is biased for $\sigma$ . Invariance is about maxima, bias is about averages, and they don’t constrain each other.

6. The algebra

The whole calculus surface for these questions.

Logs

$\log(ab) = \log a + \log b$ , which is what turns the product into a sum
$\log(a^n) = n \log a$
$\log(e^k) = k$ , which kills the exponential in every exponential-family density

Derivatives

$\frac{d}{d\theta} \log \theta = 1/\theta$
$\frac{d}{d\theta}(c\theta) = c$
$\frac{d}{d\theta} \theta^n = n\theta^{n-1}$
Parameter in a denominator: rewrite first. $\frac{x^2}{2\sigma^2} = \frac{x^2}{2}\sigma^{-2}$ , then the power rule gives $-x^2\sigma^{-3}$ . This is the hardest move that appears.

Method marks are awarded per step, so writing the likelihood, logging it, differentiating and solving carries most of the credit even if the algebra slips.