Maximum Likelihood

Every estimator so far has been produced by inspection: notice that the sample mean estimates the population mean, or that a rescaling removes a bias. Maximum likelihood is a procedure that manufactures an estimator from the model, mechanically, for any parametric family.

The idea is one sentence. Among all values the parameter could take, choose the one that makes the data you actually observed most probable.

1. The likelihood function

For a discrete model with parameter $\theta$ and observations $x_1, \dots, x_n$ from a random sample, the probability of seeing exactly this data is the product of the individual probabilities, by independence:

$L(\theta) = \prod_{i=1}^n p_\theta(x_i)$

For a continuous model the probability mass function is replaced by the density:

$L(\theta) = \prod_{i=1}^n f_\theta(x_i)$

Definition. $L(\theta)$ is the likelihood function. The maximum likelihood estimator of $\theta$ is the value maximizing $L(\theta)$.

The reversal is what makes this a statistical object rather than a probabilistic one. The same expression $\prod p_\theta(x_i)$ read as a function of $x$ with $\theta$ fixed is a probability. Read as a function of $\theta$ with $x$ fixed at the observed data, it is a likelihood. The data are known and the parameter is the variable.

The likelihood is not a probability distribution over $\theta$. It does not integrate to 1 over the parameter space, and nothing here treats $\theta$ as random.

Why the logarithm

Maximizing a product of $n$ terms is unpleasant, since differentiating requires the product rule $n$ times. Take logarithms:

$\ell(\theta) = \log L(\theta) = \sum_{i=1}^n \log p_\theta(x_i)$

The log-likelihood turns the product into a sum. This is legitimate because $\log$ is strictly increasing, so it preserves the location of the maximum: whatever maximizes $L$ maximizes $\ell$, and conversely.

2. The recipe

Four steps, and each earns marks separately on an exam.

1. Write the likelihood. Substitute the pmf or density and take the product over $i$.
2. Take the log and simplify. Products become sums, exponents come down as multipliers.
3. Differentiate with respect to $\theta$, set to zero, solve. The resulting equation is the score equation, and its solution is the candidate.
4. Verify it is a maximum with the second derivative.

Step 4 is not optional and is the step most often dropped. A vanishing first derivative locates a stationary point, which could be a minimum or an inflection. Showing $\ell''(\hat\theta) < 0$ is what makes the candidate a maximum, and it is separately marked.

Worked example: the Poisson

$X_1, \dots, X_n \overset{\text{i.i.d.}}{\sim} \mathrm{Pois}(\lambda)$, with pmf $p_\lambda(x) = \frac{e^{-\lambda}\lambda^x}{x!}$. Find the MLE of $\lambda$.

Likelihood.

$L(\lambda) = \prod_{i=1}^n \frac{e^{-\lambda}\lambda^{x_i}}{x_i!} = \frac{e^{-n\lambda}\,\lambda^{\sum_i x_i}}{\prod_i x_i!}$

Log-likelihood.

$\ell(\lambda) = -n\lambda + \left(\sum_{i=1}^n x_i\right)\log\lambda - \log\left(\prod_{i=1}^n x_i!\right)$

The last term contains no $\lambda$, so it will vanish under differentiation. Carrying it costs nothing and dropping it early is fine as long as you say why.

Differentiate and solve.

$\ell'(\lambda) = -n + \frac{\sum_i x_i}{\lambda} = 0 \implies \hat\lambda = \frac{1}{n}\sum_{i=1}^n x_i = \bar{x}_n$

Second-order check.

$\ell''(\lambda) = -\frac{\sum_i x_i}{\lambda^2} < 0$

for all $\lambda > 0$, provided $\sum_i x_i > 0$. So the critical point is a maximum.

The estimator is $T = \bar{X}_n$.

Computing the estimate. With counts $0$ through $7$ occurring $5, 17, 22, 17, 11, 5, 2, 1$ times over $n = 80$ hours:

$\bar{x}_{80} = \frac{0(5) + 1(17) + 2(22) + 3(17) + 4(11) + 5(5) + 6(2) + 7(1)}{80} = \frac{200}{80} = 2.5$

Worked example: the exponential

$X_1, \dots, X_n \overset{\text{i.i.d.}}{\sim} \mathrm{Exp}(\lambda)$, density $f_\lambda(x) = \lambda e^{-\lambda x}$ for $x \ge 0$.

$L(\lambda) = \prod_{i=1}^n \lambda e^{-\lambda x_i} = \lambda^n e^{-\lambda \sum_i x_i}$

$\ell(\lambda) = n\log\lambda - \lambda\sum_{i=1}^n x_i$

$\ell'(\lambda) = \frac{n}{\lambda} - \sum_{i=1}^n x_i = 0 \implies \hat\lambda = \frac{n}{\sum_i x_i} = \frac{1}{\bar{x}_n}$

$\ell''(\lambda) = -\frac{n}{\lambda^2} < 0$

so it is a maximum. The MLE of the rate is the reciprocal of the sample mean, which matches the fact that $\mathbb{E}[X] = 1/\lambda$.

Worked example: the Bernoulli

$X_1, \dots, X_n \overset{\text{i.i.d.}}{\sim} \mathrm{Bernoulli}(p)$, pmf $p^x(1-p)^{1-x}$ for $x \in \{0,1\}$. Writing $s = \sum_i x_i$ for the number of successes:

$L(p) = \prod_{i=1}^n p^{x_i}(1-p)^{1-x_i} = p^{s}(1-p)^{n-s}$

$\ell(p) = s\log p + (n-s)\log(1-p)$

$\ell'(p) = \frac{s}{p} - \frac{n-s}{1-p} = 0$

Multiply through by $p(1-p)$:

$s(1-p) - (n-s)p = 0 \implies s - sp - np + sp = 0 \implies \hat p = \frac{s}{n} = \bar{x}_n$

$\ell''(p) = -\frac{s}{p^2} - \frac{n-s}{(1-p)^2} < 0$

so the sample proportion is the MLE.

3. The invariance principle

Often the quantity of interest is not $\theta$ itself but a function of it.

Invariance principle. If $\hat\theta$ is the MLE of $\theta$ and $g$ is an invertible function, then $g(\hat\theta)$ is the MLE of $g(\theta)$.

You do not redo the maximization. Find the MLE of the parameter, then apply the function.

Worked example. For $X_i \sim \mathrm{Pois}(\lambda)$, estimate $p_0 = P(X_1 = 0)$.

Under a Poisson, $p_0 = e^{-\lambda}$. The map $g(\lambda) = e^{-\lambda}$ is strictly decreasing on $\lambda > 0$ and therefore invertible. Since the MLE of $\lambda$ is $\bar{X}_n$, invariance gives

$\hat{p}_0 = e^{-\bar{X}_n}$

At $\bar{x}_{80} = 2.5$ the estimate is $e^{-2.5} \approx 0.0821$.

Invariance is about maximization, not about expectation, and confusing the two is the standard error. The MLE of $g(\theta)$ is $g$ of the MLE. The expectation of $g(T)$ is not $g$ of the expectation, unless $g$ is linear. That is why $e^{-\bar{X}_n}$ is the maximum likelihood estimator of $p_0$ and is nonetheless biased, as Jensen’s inequality shows.

4. When calculus fails

The recipe assumes the maximum is interior and the likelihood is differentiable there. When the parameter appears in the support rather than only in the functional form, neither holds.

Worked example: the uniform. $X_1, \dots, X_n \overset{\text{i.i.d.}}{\sim} \mathrm{Unif}(0,\theta)$, with density

$f_\theta(x) = \begin{cases} \frac{1}{\theta} & 0 \le x \le \theta \\ 0 & \text{otherwise}\end{cases}$

The likelihood is

$L(\theta) = \prod_{i=1}^n f_\theta(x_i) = \begin{cases} \frac{1}{\theta^n} & \theta \ge \max_i x_i \\ 0 & \text{otherwise}\end{cases}$

The case split is the whole content. If $\theta$ is smaller than some observation, that observation was impossible under the model, so the density is zero there and the whole product collapses to zero.

Differentiating $1/\theta^n$ gives $-n/\theta^{n+1}$, which is never zero. There is no stationary point, so step 3 produces nothing.

Argue directly instead. On the region where the likelihood is non-zero, $1/\theta^n$ is strictly decreasing in $\theta$, so the likelihood is maximized by taking $\theta$ as small as it is allowed to be. The constraint is $\theta \ge \max_i x_i$. Therefore

$\hat\theta = X_{(n)} = \max\{X_1, \dots, X_n\}$

The recognition rule: if the parameter appears in the limits of the support, do not differentiate. Write the likelihood with its case split, see which direction it moves in, and push $\theta$ to the boundary the constraint allows.

The same reasoning gives the MLE of $N$ for a discrete uniform on $\{1, \dots, N\}$: the likelihood is $1/N^n$ for $N \ge \max_i x_i$, decreasing, so $\hat N = \max_i X_i$. From observed tickets $58, 23, 41, 7, 66$ the estimate is 66.

This estimator is biased, by the bounded-support argument: it can never exceed $N$ and is strictly below it with positive probability.

5. The method of moments

The second general recipe. Where maximum likelihood asks which parameter makes the data most probable, the method of moments asks which parameter makes the model’s moments match the data’s.

Definition. The $k$th moment of $X$ is $\mu_k' = \mathbb{E}[X^k]$. The $k$th sample moment is $m_k = \frac{1}{n}\sum_{i=1}^n X_i^k$. If $\theta = g(\mu_1', \dots, \mu_K')$, the method of moments estimator is $\hat\theta = g(m_1, \dots, m_K)$.

The recipe: express the parameter in terms of the population moments, then substitute sample moments for population moments.

Worked example: discrete uniform. For $X_i$ uniform on $\{1, \dots, N\}$, $\mathbb{E}[X_i] = \frac{N+1}{2}$. Solving for $N$ gives $N = 2\mathbb{E}[X_i] - 1$, so

$\hat N = 2\bar{X}_n - 1$

From tickets $58, 23, 41, 7, 66$: $\bar{x} = 195/5 = 39$, so $\hat N = 2(39) - 1 = 77$.

Worked example: continuous uniform. For $X_i \sim \mathrm{Unif}(0,\theta)$, $\mathbb{E}[X_i] = \theta/2$, so $\hat\theta = 2\bar{X}_n$.

Note this differs from the MLE, which is $X_{(n)}$. Two recipes applied to one model give two different estimators, and choosing between them is the mean squared error question. Here the method-of-moments estimator can be absurd: if the sample is $\{1, 9\}$ then $2\bar{x} = 10$, but if the sample is $\{1, 2, 3, 40\}$ then $2\bar{x} = 23$ while an observation of 40 was seen, so the estimate is impossible.

Worked example: two moments. For $X_i \sim \mathcal{N}(\mu, \theta)$ with $\theta = \sigma^2$,

$\mu_2' = \mathbb{E}[X^2] = \mathrm{Var}(X) + (\mathbb{E}[X])^2 = \theta + \mu^2 = \theta + (\mu_1')^2$

so $\theta = \mu_2' - (\mu_1')^2$ and

$\hat\theta = \frac{1}{n}\sum_{i=1}^n X_i^2 - \bar{X}_n^2 = \frac{1}{n}\sum_{i=1}^n (X_i - \bar{X}_n)^2$

which is the sample variance with divisor $n$, and is biased. Method of moments estimators are not guaranteed unbiased, and this is the standard counterexample.

6. Choosing between the two recipes

Use maximum likelihood by default. It uses the whole distributional form rather than a few summaries, and it satisfies invariance.

Use the method of moments when the likelihood is intractable or when the question asks for it. It is usually easier: one expectation and some algebra.

The two often coincide. For the Poisson and the Bernoulli both give $\bar{X}_n$. They diverge exactly where the distributional shape carries information the moments do not, which is the uniform case: the maximum knows about the boundary and the mean does not.

Vocabulary to deploy

• Likelihood function $L(\theta) = \prod_i f_\theta(x_i)$; log-likelihood $\ell(\theta) = \log L(\theta)$.
• Maximum likelihood estimator: the maximizer of the likelihood.
• Score equation $\ell'(\theta) = 0$; second-order condition $\ell''(\hat\theta) < 0$.
• Invariance principle: the MLE of $g(\theta)$ is $g(\hat\theta)$ for invertible $g$.
• Moment $\mu_k' = \mathbb{E}[X^k]$; sample moment $m_k$; method of moments.
• Boundary maximum: the case where the parameter enters the support and the likelihood is maximized at a constraint rather than a stationary point.