Worked Problems

Problems in the format the course uses, with solutions written the way marks are actually awarded.

Three observations about how this material is graded, drawn from the structure of the course’s own solutions.

1. Naming the principle earns the mark. Writing “by the invariance principle,” “by Jensen’s inequality,” “by the law of large numbers” is a scoring event, not decoration.
2. A bare answer earns nothing. Where a question says to justify, an unjustified correct answer scores zero.
3. The second-order check is separately marked. Solving $\ell'(\theta) = 0$ is not a complete maximum likelihood derivation.

Problem 1: modelling and maximum likelihood

An emergency room records the number of patients arriving in each one-hour window. Over $n = 80$ windows:

(a) A colleague proposes a Normal model. Explain why this is not appropriate and propose a better family.

(b) Modelling the counts as a random sample from $\mathrm{Pois}(\lambda)$, derive the maximum likelihood estimator $T_1$ of $\lambda$ and verify it is a local maximum.

(c) Compute the maximum likelihood estimate from the data.

(d) Let $p_0 = P(X_1 = 0)$. Give the MLE of $p_0$ and compute the estimate.

(e) Is the estimator in (d) unbiased for $p_0$? Justify.

Solution

(a) The counts are non-negative integers, while a Normal random variable is continuous and places probability on negative and non-integer values. A suitable family for counts of arrivals in a fixed window is the Poisson family $\{\mathrm{Pois}(\lambda) : \lambda > 0\}$.

(b) The Poisson pmf is $p_\lambda(x) = \frac{e^{-\lambda}\lambda^x}{x!}$, so

$L(\lambda) = \prod_{i=1}^n \frac{e^{-\lambda}\lambda^{x_i}}{x_i!} = \frac{e^{-n\lambda}\lambda^{\sum_i x_i}}{\prod_i x_i!}$

$\ell(\lambda) = -n\lambda + \left(\sum_{i=1}^n x_i\right)\log\lambda - \log\left(\prod_{i=1}^n x_i!\right)$

Setting the derivative to zero:

$\ell'(\lambda) = -n + \frac{\sum_i x_i}{\lambda} = 0 \implies \hat\lambda = \frac{1}{n}\sum_{i=1}^n x_i = \bar{x}_n$

Second-order check:

$\ell''(\lambda) = -\frac{\sum_i x_i}{\lambda^2} < 0$

for all $\lambda > 0$, since $\sum_i x_i > 0$ here. So the critical point is a local maximum, and $T_1 = \bar{X}_n$.

(c)

$\bar{x}_{80} = \frac{0(5) + 1(17) + 2(22) + 3(17) + 4(11) + 5(5) + 6(2) + 7(1)}{80} = \frac{200}{80} = 2.5$

(d) Under a Poisson, $p_0 = P(X_1 = 0) = e^{-\lambda}$. The map $g(\lambda) = e^{-\lambda}$ is strictly decreasing and hence invertible, so by the invariance principle

$\hat{p}_0 = e^{-T_1} = e^{-\bar{X}_n}$

The estimate is $e^{-2.5} \approx 0.0821$.

(e) No. Take $\varphi(y) = e^{-y}$, which is strictly convex. By Jensen’s inequality,

$\mathbb{E}\left[e^{-\bar{X}_n}\right] \ge e^{-\mathbb{E}[\bar{X}_n]} = e^{-\lambda} = p_0$

and since $\varphi$ is strictly convex and $\bar{X}_n$ is not constant, the inequality is strict. So $\mathbb{E}[\hat{p}_0] > p_0$: positive bias, not unbiased.

Problem 2: mean squared error over a family

Sensor battery lifetimes $X_1, \dots, X_n \overset{\text{i.i.d.}}{\sim} \mathrm{Exp}(\lambda)$, with the parameter of interest $\mu = 1/\lambda$. Consider $T_c = c(X_1 + \cdots + X_n)$ for $c > 0$.

(a) For which $c$ is $T_c$ unbiased for $\mu$, and which familiar estimator does that give?

(b) Show that $\mathrm{MSE}(T_c) = \dfrac{c^2n + (cn-1)^2}{\lambda^2}$.

(c) Find the $c$ minimizing the MSE, verify it is a minimum, and explain why a biased estimator might still be preferred.

(d) Using the Cramér-Rao bound with $f_\mu(x) = \frac{1}{\mu}e^{-x/\mu}$, show that the estimator from (a) is an MVUE of $\mu$.

Solution

(a) $\mathbb{E}(T_c) = cn\mathbb{E}(X_1) = cn/\lambda = cn\mu$, so $T_c$ is unbiased exactly when $cn = 1$, giving $c = 1/n$. This is the sample mean $\bar{X}_n$.

(b) By independence, $\mathrm{Var}(T_c) = c^2n\mathrm{Var}(X_1) = c^2n/\lambda^2$. The bias is

$\mathbb{E}(T_c) - \mu = \frac{cn}{\lambda} - \frac{1}{\lambda} = \frac{cn-1}{\lambda}$

Using $\mathrm{MSE} = \mathrm{Var} + \mathrm{bias}^2$:

$\mathrm{MSE}(T_c) = \frac{c^2n}{\lambda^2} + \frac{(cn-1)^2}{\lambda^2} = \frac{c^2n + (cn-1)^2}{\lambda^2}$

(c) Let $g(c) = c^2n + (cn-1)^2$. Then

$g'(c) = 2cn + 2n(cn-1) = 2n\left[c(n+1) - 1\right] = 0 \implies c^* = \frac{1}{n+1}$

and $g''(c) = 2n + 2n^2 > 0$, so $c^*$ is a minimum.

The resulting $T_{c^*} = \frac{1}{n+1}\sum_i X_i$ is biased low, but the small bias buys a variance reduction large enough that its MSE falls below that of $\bar{X}_n$. Measuring quality by MSE, trading a little bias for less variance is worthwhile.

(d) With $\log f_\mu(x) = -\log\mu - x/\mu$,

$\frac{\partial}{\partial\mu}\log f_\mu(x) = -\frac{1}{\mu} + \frac{x}{\mu^2} = \frac{x - \mu}{\mu^2}$

$\mathbb{E}\left[\left(\frac{X-\mu}{\mu^2}\right)^2\right] = \frac{\mathrm{Var}(X)}{\mu^4} = \frac{\mu^2}{\mu^4} = \frac{1}{\mu^2}$

using $\mathrm{Var}(X) = 1/\lambda^2 = \mu^2$. The Cramér-Rao lower bound is therefore $\mu^2/n$. Since

$\mathrm{Var}(\bar{X}_n) = \frac{\mathrm{Var}(X_1)}{n} = \frac{\mu^2}{n}$

the sample mean attains the bound and is an MVUE of $\mu$.

Problem 3: two estimators compared

A raffle sells tickets numbered $1, \dots, N$, with $N$ unknown. Each drawn ticket is returned before the next draw. Observed: $58, 23, 41, 7, 66$. Model as a random sample from the discrete uniform on $\{1, \dots, N\}$, with $\mathbb{E}(X_i) = \frac{N+1}{2}$ and $\mathrm{Var}(X_i) = \frac{N^2-1}{12}$.

(a) Why does returning each ticket matter for modelling the draws as a random sample?

(b) Use the method of moments to construct $T_1$, show it is unbiased, and compute the estimate.

(c) The MLE is $T_2 = \max\{X_1, \dots, X_5\}$. Compute the estimate. Is $T_2$ unbiased? Justify without computing $\mathbb{E}(T_2)$.

(d) A simulation at $N = 100$ prints:

[1] 100.0  83.8
[1] 666.4 199.8

being the means and variances of $T_1$ and $T_2$ over 10,000 replications. Estimate both MSEs and state which estimator you prefer.

Solution

(a) With replacement, every draw has the same distribution and the draws do not influence one another, so $X_1, \dots, X_5$ are independent and identically distributed. Without replacement they would still be identically distributed but not independent, since each draw changes what remains in the drum, and the random-sample model would fail.

(b) Matching the first moment: set $\bar{X}_n = \frac{N+1}{2}$ and solve, giving $T_1 = 2\bar{X}_n - 1$. Unbiasedness:

$\mathbb{E}(T_1) = 2\mathbb{E}(\bar{X}_n) - 1 = 2 \cdot \frac{N+1}{2} - 1 = N$

The estimate: $\bar{x} = (58+23+41+7+66)/5 = 195/5 = 39$, so $t_1 = 2(39) - 1 = 77$.

(c) $t_2 = \max\{58, 23, 41, 7, 66\} = 66$.

$T_2$ is not unbiased. $T_2 \le N$ always, and $P(T_2 < N) > 0$, since all five draws can avoid ticket $N$ with positive probability. Therefore $\mathbb{E}(T_2) < N$: the maximum systematically underestimates and has negative bias.

(d) Using $\mathrm{MSE} = \mathrm{Var} + \mathrm{bias}^2$ with the simulated values:

$\mathrm{MSE}(T_1) \approx 666.4 + (100.0-100)^2 = 666.4$

$\mathrm{MSE}(T_2) \approx 199.8 + (83.8-100)^2 = 199.8 + 262.4 = 462.2$

$T_2$ has a substantial negative bias, yet its much smaller variance gives it the smaller MSE, so by the MSE criterion prefer $T_2$. This is the bias-variance tradeoff.

Problem 4: model assumptions and the law of large numbers

A café records customers arriving in each 15-minute window over $n = 20$ windows:

(a) Propose a family and identify its parameter. State both random-sample assumptions and describe one concrete way one could fail here.

(b) Compute $\bar{x}_{20}$. Which feature of the model distribution does it estimate, and by what result?

(c) With $X_i \sim \mathrm{Pois}(\lambda)$, let $Y_n = \frac{1}{n}\sum_{i=1}^n X_i^2$. Find $c$ such that $\lim_{n\to\infty} P(|Y_n - c| > \epsilon) = 0$ for every $\epsilon > 0$.

(d) At a second location, mean(y) is 3.95 and var(y) is 9.638. Is a Poisson model appropriate?

Solution

(a) The Poisson family $\{\mathrm{Pois}(\lambda) : \lambda > 0\}$, with parameter $\lambda$, the expected number of customers arriving in a 15-minute window.

The two assumptions:

• Independence of $X_1, \dots, X_{20}$. This could fail if customers arrive in groups spilling across neighbouring windows, or if a long queue in one window discourages arrivals in the next.
• Identically distributed. This could fail if the arrival rate changes over the afternoon, so a rush after classes end gives some windows a systematically higher rate than quiet mid-afternoon ones.

(b)

$\bar{x}_{20} = \frac{0(2) + 1(5) + 2(6) + 3(4) + 4(2) + 5(1)}{20} = \frac{42}{20} = 2.1$

It estimates the expectation $\mathbb{E}[X_1]$ of the model distribution, here $\lambda$. By the law of large numbers $\bar{X}_n \xrightarrow{p} \mathbb{E}[X_1]$, so for a reasonably large sample the observed average is close to the model mean.

(c) The variables $X_1^2, X_2^2, \dots$ are i.i.d. with finite mean and variance, since all moments of a Poisson are finite. $Y_n$ is their sample mean, so by the law of large numbers $Y_n \xrightarrow{p} \mathbb{E}[X_1^2]$. Using the variance identity,

$\mathbb{E}[X_1^2] = \mathrm{Var}(X_1) + (\mathbb{E}[X_1])^2 = \lambda + \lambda^2$

so $c = \lambda + \lambda^2$.

(d) No. Under a Poisson the mean and variance are both $\lambda$, so the sample mean and sample variance should be close. Here the variance is about 2.4 times the mean, far more spread than a Poisson allows. This is overdispersion, plausibly because the arrival rate varies across windows downtown, violating the identically-distributed assumption.

Problem 5: Chebyshev and sample size

A sensor measures a true CO₂ concentration $c$, with $M_i = c + U_i$, where the errors are independent with $\mathbb{E}[U_i] = 0$ and $\mathrm{Var}(U_i) = 4$. The estimator is $\bar{M}_n$.

(a) Explain the difference between an estimator and an estimate of $c$, then show $\bar{M}_n$ is unbiased.

(b) Find $\mathrm{Var}(\bar{M}_n)$.

(c) Using Chebyshev’s inequality, find the smallest $n$ with $P(|\bar{M}_n - c| < 0.5) \ge 0.95$.

Solution

(a) The estimator $\bar{M}_n = \frac{1}{n}\sum_i M_i$ is a random variable: a rule applied to the random sample before the measurements are observed. An estimate is the realized number $\bar{m}_n = \frac{1}{n}\sum_i m_i$ obtained once the actual measurements are recorded.

$\mathbb{E}[\bar{M}_n] = \frac{1}{n}\sum_{i=1}^n \mathbb{E}[M_i] = \frac{1}{n}\sum_{i=1}^n (c + \mathbb{E}[U_i]) = \frac{1}{n} \cdot nc = c$

so $\bar{M}_n$ is unbiased for every $n$.

(b) $c$ is a constant, so $\mathrm{Var}(M_i) = \mathrm{Var}(U_i) = 4$. By independence,

$\mathrm{Var}(\bar{M}_n) = \frac{1}{n^2}\sum_{i=1}^n \mathrm{Var}(M_i) = \frac{4n}{n^2} = \frac{4}{n}$

(c) Applying Chebyshev with $\varepsilon = 0.5$:

$P\left(|\bar{M}_n - c| \ge 0.5\right) \le \frac{\mathrm{Var}(\bar{M}_n)}{0.5^2} = \frac{4/n}{1/4} = \frac{16}{n}$

so $P(|\bar{M}_n - c| < 0.5) \ge 1 - \frac{16}{n}$, and

$1 - \frac{16}{n} \ge 0.95 \iff \frac{16}{n} \le 0.05 \iff n \ge 320$

The smallest such $n$ is $320$.

Problem 6: reading plots

(a) A boxplot of $V_1$ has a longer upper whisker, all outlier points above the box, and the median in the lower half of the box. Is the distribution symmetric, left-skewed, or right-skewed?

(b) The five-number summary of $V_1$ is $0.14, \ 1.45, \ 2.06, \ 3.26, \ 9.70$. Using the $1.5 \times \mathrm{IQR}$ rule, is the maximum drawn as an outlier?

(c) Two data sets have the same median and IQR, so near-identical boxplots, but one histogram shows a feature the boxplot cannot. What feature?

(d) From the shape in (a), is the sample mean of $V_1$ greater or less than its median?

Solution

(a) Right-skewed. The upper whisker is longer than the lower one, all the outlier points sit above the box, and the median lies closer to the lower quartile.

(b)

$\mathrm{IQR} = 3.26 - 1.45 = 1.81$

$\text{upper limit} = 3.26 + 1.5 \times 1.81 = 3.26 + 2.715 = 5.975$

Since $9.70 > 5.975$, the maximum lies beyond the upper whisker limit and is drawn as an outlier point.

(c) Bimodality. A boxplot reports only the five-number summary, so it cannot display the number of modes; a histogram with two clear peaks shows a feature the box cannot represent.

(d) Greater. The long upper tail and the high outliers pull the sample mean above the median.

Problem 7: confidence intervals

$X_1, \dots, X_n$ is a random sample from a distribution with mean $\mu < \infty$ and variance $\sigma^2 < \infty$, and $\hat\mu = \bar{X}_n$.

(a) Construct a conservative 80% confidence interval for $\mu$ in terms of $\hat\mu$, $n$, and $\sigma^2$.

(b) Explain why it is called conservative.

(c) A 95% interval computed from data is $(2, 5)$. Which of these are correct? (i) There is a 95% chance the true mean is between 2 and 5. (ii) 95% of all samples yield an interval containing the true mean. (iii) We are 95% confident the true mean lies in $(2,5)$.

Solution

(a) $\mathbb{E}[\hat\mu] = \mu$ and $\mathrm{Var}(\hat\mu) = \sigma^2/n$. By Chebyshev,

$P(|\hat\mu - \mu| < \varepsilon) \ge 1 - \frac{1}{\varepsilon^2}\cdot\frac{\sigma^2}{n}$

Setting the bound to 0.8 gives $\frac{\sigma^2}{n\varepsilon^2} = 0.2$, so

$\varepsilon^2 = \frac{\sigma^2}{0.2n} = \frac{5\sigma^2}{n}, \qquad \varepsilon = \sqrt{\frac{5\sigma^2}{n}}$

The interval is

$\left(\hat\mu - \sqrt{\frac{5\sigma^2}{n}},\ \hat\mu + \sqrt{\frac{5\sigma^2}{n}}\right)$

(b) Chebyshev’s inequality gives a bound rather than an equality, so the coverage satisfies $P(L_n < \mu < U_n) \ge 0.8$ rather than $= 0.8$. The true coverage is at least the stated level and generally higher, so the interval is wider than necessary.

(c) Only (ii) is correct.

(i) is wrong because $\theta$ is a fixed constant and 2 and 5 are fixed numbers, so nothing in the statement is random and no probability attaches to it.

(ii) is correct, and it is the repeated-sampling statement the confidence level actually makes.

(iii) applies “confident” to a realized numeric interval. The approved form states confidence about the procedure, using the symbolic bounds $\ell_n$ and $u_n$; once the numbers are named, the interval either contains $\mu$ or does not.

Problem 8: hypothesis testing

Building evaluation scores in a ward, $n$ buildings, sample mean 91.36, sample standard deviation 7.56. Test whether the mean score exceeds 80 at $\alpha = 0.05$.

(a) State the hypotheses.

(b) Which test statistic, and what is its distribution under $H_0$?

(c) The p-value is approximately 0. State the conclusion.

(d) A 90% confidence interval for a second ward is $(79.49, 85.12)$. What does it say about $H_0: \mu = 80$ at $\alpha = 0.1$?

Solution

(a) $H_0: \mu = 80$ against $H_1: \mu > 80$. One-tailed, because the question asks whether the mean exceeds 80 rather than whether it differs from it.

(b) The population variance is unknown and estimated by $s_n$, so

$T = \frac{\bar{X}_n - \mu_0}{S_n/\sqrt{n}} \sim t(n-1)$

under $H_0$, assuming the scores are a random sample from a normal distribution. Observed:

$t = \frac{91.36 - 80}{7.56/\sqrt{n}} = 13.09$

(c) Since $p \approx 0 < 0.05$, reject $H_0$. There is very strong evidence that the mean evaluation score exceeds 80.

(d) The value 80 lies inside $(79.49, 85.12)$, so $H_0: \mu = 80$ is not rejected at $\alpha = 0.1$. A test at level $\alpha$ and a $(1-\alpha)$ confidence interval give the same decision: reject exactly when $\mu_0$ falls outside the interval. The correct statement is that the data do not provide sufficient evidence against $H_0$, not that the mean equals 80.

Problem 9: maximum likelihood where calculus fails

$X_1, \dots, X_n \overset{\text{i.i.d.}}{\sim} \mathrm{Unif}(0,\theta)$.

(a) Write the likelihood function, being careful about its support.

(b) Show the usual differentiation approach fails, and find the MLE.

(c) The method of moments estimator is $T_1 = 2\bar{X}_n$. Give one reason it can produce an unreasonable estimate.

Solution

(a) The density is $f_\theta(x) = 1/\theta$ for $0 \le x \le \theta$ and 0 otherwise. So

$L(\theta) = \prod_{i=1}^n f_\theta(x_i) = \begin{cases} \dfrac{1}{\theta^n} & \theta \ge \max_i x_i \\[4pt] 0 & \text{otherwise}\end{cases}$

If $\theta$ is smaller than any observation, that observation was impossible under the model, its density is 0, and the whole product is 0.

(b) On the region where the likelihood is non-zero, $\ell(\theta) = -n\log\theta$ and

$\ell'(\theta) = -\frac{n}{\theta}$

which is never zero for $\theta > 0$. There is no stationary point.

Argue directly instead: $1/\theta^n$ is strictly decreasing in $\theta$, so the likelihood is maximized by making $\theta$ as small as the constraint allows. The constraint is $\theta \ge \max_i x_i$, so

$\hat\theta = X_{(n)} = \max\{X_1, \dots, X_n\}$

(c) $T_1$ uses only the sample mean and ignores the observed values themselves, so it can return an estimate smaller than an observation already seen. With data $\{1, 2, 3, 40\}$, $2\bar{x} = 23$, but a value of 40 was observed, so $\theta \ge 40$ and an estimate of 23 is impossible under the model.