The Statistical Model

A hospital is deciding whether to put another nurse on the overnight shift. Someone counts the patients arriving each hour for eighty hours and writes the counts down:

$3, \quad 1, \quad 0, \quad 4, \quad 2, \quad \dots$

Eighty numbers. And nobody wants to know anything about them.

The hospital doesn’t care what happened during those eighty hours. Those hours are finished. The nurse would be working next month, on hours nobody has counted, and the question is what those hours will be like. So the eighty numbers are only worth having as evidence about something they aren’t: the emergency room itself, and how patients turn up at it.

Here’s the part I want you to sit with. Count a different eighty hours at the same hospital and you get different numbers. Nothing about the hospital changed. The numbers changed because which hours you happened to pick changed.

So two separate things exist, and almost everything in this course depends on not mixing them up.

1. The process. The emergency room and whatever governs how patients arrive at it. This is the thing you want to know about. You never get to see it.
2. The data. Eighty numbers the process threw out this time. This is all you ever hold.

A statistical model is a guess about what the process is. I write the guess using probability, and the reason is specific: the process doesn’t give the same answer twice, and probability is the mathematics built for describing things that come out differently on different occasions.

The guess is never exactly right. A real emergency room isn’t a tidy mathematical object and never will be. George Box’s line is the one everyone quotes: all models are wrong, some are useful. What makes one useful is that it’s close enough to the real process that answers you get out of the model are answers you can trust about the hospital.

1. Random variables

To write any of this down I need a way to talk about a number the process might produce, before it produces it.

A random variable is a quantity whose value depends on how something turns out. Before it happens you don’t know the value, but you do know which values are possible and how likely each one is.

The count of patients between 2am and 3am tonight is a random variable. You don’t know it yet. You do know it’ll be a whole number, it won’t be negative, and 2 is likelier than 40.

A die roll is a random variable. Before the roll it’s one of $1$ through $6$, each with probability $1/6$. After the roll it’s a 4, and it’s a 4 permanently.

You’ll also see the formal version, which is worth being able to read: a random variable $X$ is a function from a sample space to the real numbers, $X : \Omega \to \mathbb{R}$. The sample space $\Omega$ is the set of things that could happen, and $X$ attaches a number to each one. If $\Omega$ is the set of birth months, $X$ might send January to 1 and February to 2. All the definition says is that a random variable is a rule for turning an outcome into a number.

Capitals and lowercase

The notation carries an idea, which is why it’s worth more attention than notation usually gets.

$X_i$, capital, is the random variable: the quantity while it still has a range of possible values attached to it.

$x_i$, lowercase, is what it turned out to be.

$X_1$ is “however many patients arrive in the first hour.” $x_1 = 3$ is the 3 that got written down. Before the night, $X_1$ had a whole distribution of possibilities. Afterwards, $x_1$ is 3 and stays 3.

Every interesting question in this course is about the capitals. When you ask whether a method is reliable, or how much its answers bounce around, or whether it’s systematically off, you’re asking how it behaves across all the data sets it could have been handed. The lowercase numbers have no bounce left in them. They already happened.

Expectation

The expectation $\mathbb{E}[X]$ is the long-run average value of a random variable. Roll a die a thousand times, average the results, and you’ll be near 3.5. That’s the expectation.

You compute it by weighting each possible value by how likely it is:

$\mathbb{E}[X] = \sum_x x \cdot P(X = x)$

For the die, every value has probability $1/6$, so

$\mathbb{E}[X] = \tfrac{1}{6}(1) + \tfrac{1}{6}(2) + \tfrac{1}{6}(3) + \tfrac{1}{6}(4) + \tfrac{1}{6}(5) + \tfrac{1}{6}(6) = \tfrac{21}{6} = 3.5$

A die never shows 3.5. The expectation isn’t a value you expect to see, despite the name. It’s the balance point of the distribution: put the probabilities on a seesaw at their values, and 3.5 is where it balances.

Two facts about expectation get used constantly.

Adding a constant shifts it. $\mathbb{E}[X + c] = \mathbb{E}[X] + c$. If every outcome moves up by 5, the average moves up by 5.

It’s linear. $\mathbb{E}[aX + bY] = a\mathbb{E}[X] + b\mathbb{E}[Y]$. The average of a sum is the sum of the averages, and constants pull out front. This holds no matter how $X$ and $Y$ are related to each other, which will matter later in a way that’s easy to miss, so it’s worth noticing now that nothing was assumed about independence here.

Variance

Expectation says where a random variable sits. It says nothing about how far it strays.

Compare two games. In the first you win $5 every time. In the second you win $0 or $10 on a coin flip. Both have expectation $5, and they’re obviously not the same game.

Variance measures the straying. The natural thing is to average the distance from the expectation, but raw distances cancel: values above and below are equally far away with opposite signs, and they sum to zero for every random variable, which measures nothing. Squaring fixes it, since a squared distance is positive whichever side you’re on.

$\mathrm{Var}(X) = \mathbb{E}\left[(X - \mathbb{E}[X])^2\right]$

The average squared distance from the expectation. Small variance means the variable stays near its expectation; large variance means it swings.

The squaring leaves you in the wrong units. Square the dollars in that game and you get squared dollars, which is meaningless. Taking the square root undoes it, and the result is the standard deviation, back in dollars.

Adding a constant doesn’t change the variance. $\mathrm{Var}(X + c) = \mathrm{Var}(X)$. Sliding the whole distribution five to the right moves the balance point and leaves the spread exactly as it was.

2. The random sample

Back to the hospital. Saying “each hourly count is a random variable” is true and gets me nowhere, because I have eighty random variables and no statement about how they relate.

Think about what would have to be true for the eighty counts to be worth averaging.

They’d have to be describing the same thing. If the first forty came from a Tuesday afternoon and the last forty from a Saturday night, averaging them gives a number describing neither.

And they’d have to be telling me eighty separate things. If every hour’s count were a perfect copy of the one before, I’d have one observation written out eighty times, and I’d be a lot less informed than the count of numbers suggests.

Those two requirements have a name.

Definition. A random sample is a collection of random variables $X_1, X_2, \dots, X_n$ that have the same probability distribution and are mutually independent.

Written short: $X_1, \dots, X_n \overset{\text{i.i.d.}}{\sim} F$, read as “independent and identically distributed with distribution $F$.” $F$ is the cumulative distribution function, one standard way of pinning a distribution down: $F(x) = P(X \le x)$, the probability the variable lands at $x$ or below.

The two assumptions inside are separate and you’ll be asked to name them separately.

Identically distributed. Every $X_i$ has the same distribution. Whatever produced observation 1 produced observation 40.

Independent. Knowing one observation tells you nothing about any other. Formally $P(X_i = x \mid X_j = y) = P(X_i = x)$: learning $X_j$ leaves the probabilities for $X_i$ untouched.

Neither of these is something you can check by looking at the data. There’s no column of numbers that proves independence. You argue for them from how the data were collected, and the useful skill is being able to say how they’d break in a particular setting.

How they break

A drifting rate breaks identical distribution. Count customers entering a café in each 15-minute window through an afternoon. Classes let out at 3pm and there’s a rush. Those windows have a genuinely higher arrival rate than the quiet ones at 2pm. Every window was measured identically and they still aren’t draws from one distribution, because the thing generating them changed partway through.

Interaction breaks independence. Same café. Customers come in groups, and a group arriving at 2

Here’s a case where the two assumptions come apart cleanly, which is why it gets used.

An army captures tanks and reads the serial numbers off them. Serial numbers run $1$ to $\theta$, where $\theta$ is the total number built and is what you want to know. Each captured tank is a draw from $\{1, \dots, \theta\}$, and you can’t capture the same tank twice.

Are the draws identically distributed? Yes. Every tank is equally likely to be the next one captured, so every draw faces the same set of possibilities with the same probabilities.

Are they independent? No, and one line shows it. Say the first tank captured was number 5. Then the second one can’t be number 5:

$P(X_2 = 5 \mid X_1 = 5) = 0, \qquad \text{while} \qquad P(X_2 = 5) = \frac{1}{\theta}$

Knowing the first draw changed the probabilities for the second, which is exactly what independence rules out.

Sampling with replacement is the fix. Draw a raffle ticket, write the number down, put it back in the drum, draw again. Each draw meets the same full set of tickets, so the draws really are independent. Keeping the ticket out breaks independence the way the tanks do.

3. Committing to a family

Definition. A data set of repeated measurements $x_1, x_2, \dots, x_n$ of the same quantity is modelled as the realization of a random sample $X_1, X_2, \dots, X_n$. The model may include a partial specification of the probability distribution of each $X_i$.

“Partial” is the word doing the work there.

You’ll almost never claim to know the distribution exactly. What you claim is that it belongs to a family: a set of distributions that all share a shape and differ only in one or two numbers.

Take the Poisson. There isn’t a Poisson distribution, there’s one for every positive number $\lambda$. $\mathrm{Pois}(0.5)$ describes something that almost never happens. $\mathrm{Pois}(50)$ describes something happening constantly. Both are Poisson, and they look nothing like each other. Saying “arrivals are Poisson” fixes the shape and leaves $\lambda$ open.

Definition. Specifying the model parameter $\theta$ fully specifies the model. The partially specified model $\{F_\theta : \theta \in \Theta\}$ is called a parametric model.

That notation is dense, so here it is piece by piece. $\theta$ is the unknown number, the parameter. $F_\theta$ is the specific distribution you get once $\theta$ is fixed at a value. $\Theta$ is the set of values $\theta$ is allowed to be, the parameter space. The braces are set notation. Read the whole thing as: the collection of all the distributions $F_\theta$, as $\theta$ ranges over what’s allowed.

For the exponential, $\theta = \lambda$ and $\Theta = (0, \infty)$, since a rate can be any positive number. For the normal, $\theta$ is a pair, $\theta = (\mu, \sigma^2)$.

Why commit to a family? Because the problem is unanswerable otherwise.

“Find the distribution that produced this data” searches over every distribution there is. That space is unimaginably large, and eighty numbers cannot single out a member of it. “The distribution is Poisson, find $\lambda$” is a question about one unknown number, and eighty numbers say a lot about one number. Every technique in the rest of this course answers the second kind of question. None of them is available before you commit.

The commitment costs you something real: get the family wrong and everything built on top of it is wrong, and nothing in the machinery will tell you.

4. Picking the family

You’ll be handed a description of some data and asked to propose a family and defend it. Three steps.

Match the support

The support of a distribution is the set of values it gives any probability to. Check it first, because it eliminates candidates immediately.

Patient counts are whole numbers and can’t be negative, so the support has to be $0, 1, 2, \dots$. A normal distribution is continuous and puts probability on every real number, $-1.4$ and $2.73$ included. There’s no such thing as $-1.4$ patients arriving, so the normal is the wrong shape for counts. You can say that before looking at a single value.

Match the mechanism

Families encode different stories about how numbers get produced. Recognizing the story is most of the work.

• Counting events in a fixed window (arrivals per hour, sales per day): $\mathrm{Pois}(\lambda)$, with $\lambda$ the expected number of events in that window.
• Waiting for the next event (battery lifetime, time between failures): $\mathrm{Exp}(\lambda)$. The Poisson’s partner: when events happen at a steady rate with no memory of each other, counts in a window are Poisson and the gaps between events are exponential.
• Successes out of a fixed number of tries (wins in 82 games): $\mathrm{Binomial}(n, p)$.
• A single yes or no (did this delivery take over 45 minutes): $\mathrm{Bernoulli}(p)$.
• A measurement scattered symmetrically around a true value: $\mathcal{N}(\mu, \sigma^2)$.
• Every value in a range equally likely (a serial number from $1$ to $N$): discrete uniform, or $\mathrm{Unif}(a,b)$ if the values are continuous.

Check the family against itself

Some families force relationships between their own mean and variance. If the data violate one, that family is dead on its own terms.

The Poisson is the cleanest case:

$\mathbb{E}[X] = \mathrm{Var}(X) = \lambda$

Mean and variance are the same number. That’s a property of the shape, not a coincidence.

So if someone proposes a Poisson model and the data have sample mean $3.95$ and sample variance $9.64$, those should be close, and instead the variance is about $2.4$ times the mean. The data are far more spread out than any Poisson permits. One line of arithmetic kills the model.

That situation has a name. Overdispersion is variance exceeding what the family allows, and the usual culprit is a rate that varies across observations, which is identical distribution failing.

A question in the shape you’ll be asked

Over 20 windows of 15 minutes each, a café records the number of customers arriving:

Customers	0	1	2	3	4	5
Windows	2	5	6	4	2	1

Propose a family for these data and identify its parameter. State both assumptions the random sample requires, and describe one concrete way one of them could fail here.

Support first: these are counts, so whole numbers, none negative. That kills every continuous family.

Mechanism: counting events in a fixed window. That’s Poisson. So the family is $\{\mathrm{Pois}(\lambda) : \lambda > 0\}$, with parameter $\lambda$, the expected number of customers arriving in a 15-minute window.

Both assumptions, named:

Independence of $X_1, \dots, X_{20}$, and identical distribution across the twenty windows.

A concrete failure, and the question wants it specific to a café rather than a definition of dependence: identical distribution fails if the arrival rate shifts during the afternoon, so that windows right after classes end have a systematically higher rate than mid-afternoon ones. Independence fails if customers arrive in groups that straddle the boundary between two windows.

Either one earns the mark as long as it names a mechanism in this setting.

5. Where the model pays off

Take a sensor measuring something with a fixed true value: the concentration of CO₂ in a room, say, which we’ll call $c$. Measure it repeatedly and you get different answers every time. The room didn’t change. The measuring did.

Write each measurement as the truth plus whatever the instrument got wrong that time:

$M_i = c + U_i$

$c$ is a fixed number nobody knows. $U_i$ is the error on measurement $i$, and it’s the random part.

Now suppose the instrument isn’t biased, meaning it’s as likely to read high as low and doesn’t systematically drift one way. That assumption is exactly $\mathbb{E}[U_i] = 0$.

Take the expectation of a measurement:

$\mathbb{E}[M_i] = \mathbb{E}[c + U_i] = c + \mathbb{E}[U_i] = c + 0 = c$

using linearity, and using the fact that $c$ is a constant so its expectation is itself.

And the variance:

$\mathrm{Var}(M_i) = \mathrm{Var}(c + U_i) = \mathrm{Var}(U_i)$

since adding a constant doesn’t change spread. The measurements vary exactly as much as the errors do, which makes sense: $c$ contributes nothing to the variation because it never moves.

Look at what $\mathbb{E}[M_i] = c$ says. The expectation of a measurement is the quantity you’re trying to find. So if I can estimate the expectation of this distribution, I’ve estimated the CO₂ concentration. A question about a room has turned into a question about a number in a model, and the rest of the course is machinery for answering that second kind of question well.

Are the random-sample assumptions defensible here? Identically distributed: yes, every measurement is of the same unchanging quantity with the same instrument. Independent: yes, provided one reading doesn’t affect the next, which is a claim about the apparatus rather than about the numbers.

6. A second worked family

Software runs until it crashes, the elapsed time is recorded, it restarts, and this repeats. The times between failures, in CPU seconds, start:

$30, \quad 26, \quad 65, \quad 233, \quad 379, \quad 983, \quad 0, \quad 75, \quad \dots$

and continue for 136 observations. Which family?

Support. Times, so non-negative, and measured continuously rather than counted. Every discrete family is out, and so is anything putting probability on negative numbers.

Mechanism. These are waiting times. A crash can happen at any moment, the chance of one doesn’t depend on how long you’ve already been running, and one crash doesn’t make the next likelier. That’s a Poisson process, and waiting times between events in a Poisson process are exponential.

So:

$X_1, \dots, X_n \overset{\text{i.i.d.}}{\sim} \mathrm{Exp}(\lambda)$

with $\theta = \lambda$ and $\Theta = (0, \infty)$, and cumulative distribution function $F(x) = 1 - e^{-\lambda x}$ for $x \ge 0$.

Does the data agree? A histogram of these times is heavily right-skewed: lots of short gaps, a handful of very long ones, a long thin tail. That’s the shape of an exponential density. Nothing in the data argues against the choice, which is the most you get from this kind of check.

7. When the model is wrong

Annual global surface temperatures from 1970 to 2022 sit close to a straight line. Fit one and it describes those five decades well.

Take the same line back to 1880 and it fails badly. The real record is roughly flat through the early part of the century and then climbs, and no single straight line matches both stretches.

Two things worth taking from that.

A model is fitted over a range and defensible over that range. The line says something about 1970 to 2022 and nothing at all about 1880. Extrapolating past your data asserts that the process behaved the same way during a period you never observed, which is a claim your data cannot support.

A relationship in the data isn’t a cause. The year doesn’t heat the planet. Other things do, and those things happened to move steadily across the window observed, which makes year look explanatory when it’s only correlated. Fitting a model whose form is wrong for the process is misspecification, and what makes it dangerous is that it still produces confident-looking answers.

Vocabulary to deploy

• Random variable: a quantity whose value depends on how something turns out; formally a function $X : \Omega \to \mathbb{R}$ from a sample space to the reals.
• Realization: the observed value $x_i$ of the random variable $X_i$. Capital for the variable, lowercase for the number.
• Distribution: which values a random variable takes and with what probability. Cumulative distribution function $F(x) = P(X \le x)$.
• Expectation $\mathbb{E}[X]$, the long-run average and the balance point of the distribution.
• Variance $\mathrm{Var}(X) = \mathbb{E}[(X - \mathbb{E}[X])^2]$, the average squared distance from the expectation; standard deviation, its square root, back in the original units.
• Linearity of expectation: $\mathbb{E}[aX + bY] = a\mathbb{E}[X] + b\mathbb{E}[Y]$, with no assumption about how the variables are related.
• Statistical model: a guess about the data-generating process, written as a probability distribution.
• Random sample: $X_1, \dots, X_n$ independent and identically distributed, $X_1, \dots, X_n \overset{\text{i.i.d.}}{\sim} F$.
• Parametric model: a family $\{F_\theta : \theta \in \Theta\}$; model parameter $\theta$; parameter space $\Theta$.
• Support: the set of values a distribution assigns positive probability to.
• Overdispersion: sample variance materially exceeding the sample mean, refuting a Poisson model.
• Misspecification: fitting a model whose form is wrong for the process.