The Linear Model

Everything up to here estimates a single unknown parameter from a sample of one measured quantity. The linear model estimates a relationship: how one measured quantity moves with another.

The data are now bivariate, pairs $(x_1, y_1), (x_2, y_2), \dots, (x_n, y_n)$. Each observation carries two numbers, and the question is what connects them.

1. Reading a scatter plot

Plot the pairs with $x$ horizontal and $y$ vertical. Three features are worth naming, because they are what the model has to capture or fail to.

1. Shape. Linear, exponential, U-shaped, or no discernible form. A linear model is only defensible when the shape is linear.
2. Direction. Positive (y rises with x), negative, or none.
3. Strength. How tightly the points cluster around the trend. Strong, weak, or absent.

A scatter plot of Gentoo penguin bill depth against bill length shows a positive, roughly linear, moderately strong relationship. That reading is what licenses the model below.

2. Deterministic against statistical

A deterministic model says $y = g(x)$, and for a straight line

$g(x) = \alpha + \beta x$

Give it an $x$ and it returns one $y$, exactly. That is a claim no real data satisfies: two penguins with the same bill length have different bill depths.

The statistical model keeps the line and adds a term for everything the line does not explain:

$Y_i = g(x_i) + U_i$

What changes is what $x$ maps to. In the deterministic model $x_i$ determines $y_i$. In the statistical model $x_i$ determines $\mathbb{E}[Y_i]$, the expected value of the response, and the realized $Y_i$ scatters around it. That shift, from a value to an expectation, is the entire content of adding the error term.

3. The simple linear regression model

Definition. Consider bivariate data $(x_1, y_1), \dots, (x_n, y_n)$. A simple linear regression model assumes that $x_1, \dots, x_n$ are nonrandom and that $y_1, \dots, y_n$ are realizations of random variables $Y_1, \dots, Y_n$ such that $Y_i = \alpha + \beta x_i + U_i$ for $i = 1, \dots, n$, where $U_1, \dots, U_n$ are independent random variables with $\mathbb{E}(U_i) = 0$ and $\mathrm{Var}(U_i) = \sigma^2$.

The vocabulary, and each term has a standard alternative that appears interchangeably:

• The $x$-variable is the explanatory variable or independent variable.
• The $y$-variable is the response variable or dependent variable.
• The line $y = \alpha + \beta x$ is the regression line, with $\alpha$ the intercept and $\beta$ the slope.

Four assumptions are doing work here, and an exam question asking you to state the model wants them named.

1. The $x_i$ are nonrandom. Only the response is modelled as random. The explanatory variable is treated as fixed and known.
2. The errors have mean zero. $\mathbb{E}(U_i) = 0$. Without this the line is systematically offset and $\alpha$ absorbs the difference.
3. The errors have constant variance. $\mathrm{Var}(U_i) = \sigma^2$, the same $\sigma^2$ for every $i$. The scatter around the line does not widen or narrow along $x$.
4. The errors are independent. One observation’s deviation from the line says nothing about the next one’s.

4. Why the response variables are not identically distributed

This is the point most worth understanding, and it is the one place the linear model breaks the pattern every earlier page established.

Every model so far assumed a random sample: $X_1, \dots, X_n$ independent and identically distributed. Here that second half fails.

$Y_1, \dots, Y_n$ are independent, because they are independent errors added to constants.

$Y_1, \dots, Y_n$ are not identically distributed, and the one-line proof is the means:

$\mathbb{E}[Y_i] = \alpha + \beta x_i \qquad \ne \qquad \mathbb{E}[Y_j] = \alpha + \beta x_j$

whenever $x_i \ne x_j$. Different explanatory values produce different expected responses, which is the whole reason to fit a line. A model in which the $Y_i$ were identically distributed would be a model in which $x$ carried no information.

They do share a variance: $\mathrm{Var}(Y_i) = \mathrm{Var}(U_i) = \sigma^2$ for every $i$, since $\alpha + \beta x_i$ is a constant. So the responses have a common spread and different centres.

The consequence for the rest of the course: results proved for a random sample do not transfer here without checking. The law of large numbers and the central limit theorem as stated on Estimators and Sampling Distributions assume identical distribution.

5. The regression line is a conditional mean

Taking expectations of the model equation, and using $\mathbb{E}[U_i] = 0$:

$\mathbb{E}[Y_i] = \mathbb{E}[\alpha + \beta x_i + U_i] = \alpha + \beta x_i + \mathbb{E}[U_i] = \alpha + \beta x_i$

The first two terms come out unchanged because $x_i$ is nonrandom, so $\alpha + \beta x_i$ is a constant.

So the fitted line does not predict an observation, it predicts an average. A point on the regression line is the expected response at that $x$, not the response you would see. Individual observations scatter around it with variance $\sigma^2$.

Worked example: orange trees

The Orange data set holds 35 measurements of tree circumference in millimetres and tree age in days. Modelling circumference $W_i$ against age $t_i$:

$W_i = \alpha + \beta t_i + U_i, \qquad i = 1, \dots, 35$

with $W_i$ the circumference and $t_i$ the age of the $i$th measurement. Here $n = 35$, circumference is the dependent variable, age the independent one.

The fitted line is $y = 17.4 + 0.107x$. What is the expected circumference of a tree 300 days old?

$\mathbb{E}[W] = 17.4 + 0.107(300) + \underbrace{\mathbb{E}[U]}_{=\,0} = 17.4 + 32.1 = 49.5 \text{ mm}$

In R this is 17.4 + 0.107 * 300, returning 49.5.

Say expected circumference rather than circumference. A particular 300-day-old tree will not measure 49.5 mm; the model claims that is the average over trees of that age.

6. Fitting the line in R

Orange |>
  ggplot(aes(x = age, y = circumference)) +
  theme_classic() +
  geom_point() +
  geom_smooth(method = "lm", se = FALSE, colour = "maroon")

geom_point() draws the scatter. geom_smooth(method = "lm") fits and draws the least-squares line, with lm standing for linear model. se = FALSE suppresses the confidence band around the fit, which is the confidence interval machinery applied to the line rather than to a single parameter.

7. Misspecification

A model fitted over one range of $x$ is defensible over that range and nowhere else.

Annual global surface temperature from 1970 to 2022, expressed as change from 1950, fits a straight line well:

$t_i = \alpha + \beta y_i, \qquad i = 1, \dots, 53$

with $y_i$ the year and $t_i$ the recorded average temperature.

Extend that line back to 1880 and it fails completely. The actual series is roughly flat and then rising; the straight line extrapolated backwards diverges from it badly.

Two conclusions, both examinable as short-answer reasoning.

The fit describes the window it was fitted to. The line may describe the rate of change across those five decades and says nothing about the century before.

A fitted relationship is not a causal one. Year does not explain temperature. Other factors drive the change and happen to be correlated with time over the observed window. Misspecified models lead to misleading results.

Vocabulary to deploy

• Bivariate data: paired observations $(x_i, y_i)$.
• Explanatory or independent variable $x$; response or dependent variable $y$.
• Regression line $y = \alpha + \beta x$; intercept $\alpha$; slope $\beta$.
• Simple linear regression model $Y_i = \alpha + \beta x_i + U_i$, with $x_i$ nonrandom, $\mathbb{E}(U_i) = 0$, $\mathrm{Var}(U_i) = \sigma^2$, and the $U_i$ independent.
• Independent but not identically distributed: the $Y_i$ share a variance and differ in mean.
• Deterministic model $y = g(x)$ against statistical model $Y_i = g(x_i) + U_i$.
• Misspecification, and the distinction between a fitted relationship and a causal one.