Summarizing a Sample

Someone gives you a hundred measurements and asks what the data look like. You cannot answer by reading them out. A hundred numbers is more than anyone can hold at once, so before doing anything else you replace them with something small enough to take in: a few numbers, or a picture.

That replacement is called summarizing, and it always costs you something. Turning a hundred numbers into one number means ninety-nine numbers’ worth of detail is now unrecoverable from what you kept.

Here is how that bites. These two data sets have the same mean:

$\{4, 5, 5, 6\} \qquad \text{and} \qquad \{0, 0, 10, 10\}$

Both average to 5. One is four values huddled together, the other is two values at each end of a range and nothing in the middle. Report only the mean and those two are indistinguishable, even though almost nothing else about them is alike.

So the question to keep asking about any summary is: what did this one throw away, and does that matter for what I am about to say? Different summaries discard different things, which is why there are several and why you use more than one.

There are two kinds.

Numerical summaries replace the data with a few numbers, describing where the data sit and how spread out they are.

Graphical summaries draw a picture of the whole shape.

One more thing links this page to the rest of the course. Each numerical summary is an estimate of a corresponding feature of the underlying distribution: the sample mean estimates the expectation, the sample variance estimates the variance, and so on. That correspondence is what makes summaries more than description, and it is set out in section 6.

1. Where the data sit

The sample mean

Add the numbers and divide by how many there are.

$\bar{x}_n = \frac{x_1 + x_2 + \cdots + x_n}{n}$

The bar on top means “average of,” and the subscript $n$ records how many values went into it.

When the data arrive as a frequency table rather than a list, do not write the list out. Multiply each value by the number of times it occurred, add those products, and divide by the total count:

$\bar{x}_n = \frac{\sum_j v_j f_j}{\sum_j f_j}$

where $v_j$ is a value and $f_j$ is how many times it appeared.

Worked example. Counts of customers in a 15-minute window, over 20 windows:

$\bar{x}_{20} = \frac{0(2) + 1(5) + 2(6) + 3(4) + 4(2) + 5(1)}{20} = \frac{0 + 5 + 12 + 12 + 8 + 5}{20} = \frac{42}{20} = 2.1$

On an exam this is worth two marks: one for setting up the weighted sum correctly, one for the arithmetic.

The sample median

Sort the numbers from smallest to largest and take the middle one. If there is an even count there is no single middle value, so average the two closest to the middle.

To write that down properly we need notation for sorted data. The order statistics are the data arranged in increasing order, written $x_{(1)} \le x_{(2)} \le \cdots \le x_{(n)}$. The parentheses in the subscript are what distinguish them: $x_1$ is the first number you happened to record, and $x_{(1)}$ is the smallest number in the set. So $x_{(1)}$ is the minimum and $x_{(n)}$ is the maximum.

With that,

$\mathrm{Med}_n = \begin{cases} x_{\left(\frac{n+1}{2}\right)} & n \text{ odd} \\[6pt] \dfrac{x_{\left(\frac{n}{2}\right)} + x_{\left(\frac{n}{2}+1\right)}}{2} & n \text{ even}\end{cases}$

For $\{1, 2, 3, 1, 2\}$: sort to get $\{1, 1, 2, 2, 3\}$, five values, so the median is the third one, 2. For $\{1, 1, 2, 3, 3, 3\}$: six values, so average the third and fourth, $(2+3)/2 = 2.5$.

Why there are two of them

Mean and median are both trying to say where the data sit, and they respond very differently to extreme values.

Take $\{1, 3, 8\}$. The mean is 4 and the median is 3.

Now change the 8 to 888, giving $\{1, 3, 888\}$. The mean jumps to about 297. The median is still 3.

The reason is structural. The mean uses the magnitude of every value, so one huge number drags it as far as you like. The median uses only rank, so replacing the largest value with something even larger does not move it at all: it was the biggest before and it is still the biggest.

A summary that resists being dragged around by extreme values is called robust. The median is robust and the mean is not.

On an exam this usually runs backwards. You are shown the shape of a distribution and asked which is larger.

• Right-skewed means a long tail stretching to the right. Those far-out values pull the mean up while leaving the median where it is, so mean > median.
• Left-skewed is the mirror image: mean < median.
• Symmetric means the two are roughly equal.

2. Quantiles

The median splits the data in half. There is nothing special about halves, and the general version is worth having.

Definition. The $p$th empirical quantile $q_n(p)$ is the value with a proportion $p$ of the data below it.

So $q_n(0.5)$ is the median, $q_n(0.25)$ is the value a quarter of the way up, and $q_n(0.75)$ is three-quarters of the way up. Those last two are common enough to have names: the lower quartile and the upper quartile.

Percentiles are the same thing on a different scale: the $p$th quantile is the $100p$th percentile, so $q_n(0.25)$ is the 25th percentile.

There is a wrinkle. If you want the value a quarter of the way through six numbers, there is no such data point, since a quarter of the way lands between the first and second. So you interpolate:

$q_n(p) = x_{(k)} + \alpha\left(x_{(k+1)} - x_{(k)}\right), \qquad k = \lfloor p(n+1) \rfloor, \quad \alpha = p(n+1) - k$

Reading that: $p(n+1)$ is the position you want, $k$ is the whole-number part of it (the $\lfloor \cdot \rfloor$ brackets mean round down), and $\alpha$ is the leftover fraction. You take the value at position $k$ and move $\alpha$ of the way toward the next one.

Worked example. $Y = \{1, 3, 4, 6, 7, 8\}$, so $n = 6$. Find $q_n(0.25)$.

$p(n+1) = 0.25 \times 7 = 1.75$

So $k = \lfloor 1.75 \rfloor = 1$ and $\alpha = 1.75 - 1 = 0.75$. The answer is three-quarters of the way from the first value to the second:

$q_n(0.25) = x_{(1)} + 0.75\left(x_{(2)} - x_{(1)}\right) = 1 + 0.75(3-1) = 1 + 1.5 = 2.5$

The five-number summary

Five numbers that between them describe centre, spread, and extent: the minimum, the lower quartile, the median, the upper quartile, and the maximum. This is what a box plot draws.

3. How spread out the data are

Knowing the centre is not enough. Two data sets can have the same mean and look nothing alike, one tightly clustered and one scattered across a wide range.

Sample variance and standard deviation

The idea: measure how far each value is from the mean, and average those distances.

Raw distances would cancel, since values above and below the mean have opposite signs. So square them first.

$s_n^2 = \frac{1}{n-1}\sum_{i=1}^{n}\left(x_i - \bar{x}_n\right)^2$

The sample standard deviation $s_n$ is the square root of that, which undoes the squaring and returns the answer to the units of the original data. If the data are in seconds, the variance is in seconds squared and the standard deviation is in seconds.

Why $n-1$ and not $n$? You are averaging $n$ squared distances, so dividing by $n$ looks right, and it is wrong. Here is the reason.

You want to know how far the data spread from the true centre of the distribution. You do not know the true centre, so you use $\bar{x}_n$ instead. And $\bar{x}_n$ is not just any value: it is the one number that makes $\sum (x_i - a)^2$ as small as it can possibly be. That is a property of the mean.

So measuring distances from $\bar{x}_n$ gives you the smallest total you could have got from these data, which is systematically smaller than the total you would have got from the true centre. Dividing by $n-1$ instead of $n$ makes the result slightly bigger, and it turns out to correct for this exactly. The Bias page proves it.

For a frequency table, weight each squared deviation by its count and subtract one from the total:

$s_n^2 = \frac{\sum_j (v_j - \bar{x}_n)^2 f_j}{\left(\sum_j f_j\right) - 1}$

Median absolute deviation

The variance has the same weakness as the mean: it squares distances, so one far-out value dominates it.

The robust alternative applies the median idea twice. Find the median. Find how far each value is from it. Take the median of those distances.

$\mathrm{MAD}(x_1, \dots, x_n) = \mathrm{Med}\left(|x_1 - \mathrm{Med}_n|, \dots, |x_n - \mathrm{Med}_n|\right)$

Worked example. $\{5, 10, 8, 7\}$.

Sort: $\{5, 7, 8, 10\}$. Four values, so the median is $(7+8)/2 = 7.5$.

Distances from 7.5: $|5 - 7.5| = 2.5$, $|10 - 7.5| = 2.5$, $|8 - 7.5| = 0.5$, $|7 - 7.5| = 0.5$.

Sort those: $\{0.5, 0.5, 2.5, 2.5\}$. Median is $(0.5 + 2.5)/2 = 1.5$.

So $\mathrm{MAD} = 1.5$.

A warning for R. Calling mad(x) returns 2.2239 rather than 1.5, because R multiplies by a constant chosen to make the MAD comparable to the standard deviation for normal data. To get the definition above, use mad(x, constant = 1).

4. Pictures

The histogram

Split the range into intervals of equal width and count how many observations fall in each. Draw a bar over each interval.

The intervals are called bins. If the bins have width $b$ and start at $x_0$, then bin $i$ covers $[x_0 + (i-1)b, \; x_0 + ib)$. The square bracket means the left endpoint is included and the round bracket means the right endpoint is not, so every observation lands in exactly one bin.

For the bar heights there is a choice. Plain counts work, but it is often more useful to make the area of each bar represent the proportion of data in that bin, which requires height

$\text{height of bin } i = \frac{\text{count in bin } i}{n \cdot b}$

Dividing by $n$ turns a count into a proportion, and dividing by $b$ turns a proportion into a height whose bar has the right area. The payoff is that the whole histogram then has total area 1, which is the same property a probability density has, so you can draw a candidate density on top and compare them directly.

Histograms are not unique, and this matters. The same data with a different bin width, or the same width starting at a different place, produces a visibly different picture. Bins too narrow and you see noise; too wide and real features get smoothed away. Neither the width nor the starting point is determined by the data, so both are choices you are making, and a feature that appears under one choice and vanishes under another was never solidly there.

Kernel density estimates

A histogram is a staircase, and its steps are artifacts of where you happened to put the bin edges. A kernel density estimate smooths that out.

The idea: instead of dropping each observation into a bin, put a small smooth bump centred on it, then add all the bumps together.

$f_{n,h}(t) = \frac{1}{n}\sum_{i=1}^{n} \frac{1}{h}K\!\left(\frac{t - x_i}{h}\right)$

Read the formula from the inside out. $K$ is the kernel, a small symmetric bump shape (a probability density in its own right). The bandwidth $h$ controls how wide each bump is. Writing $K$ of $(t - x_i)/h$ shifts the bump so it sits over the data point $x_i$ and scales it to width $h$. The $1/h$ out front keeps each bump’s area at 1 as it gets narrower. Summing stacks all $n$ bumps, and the $1/n$ makes the total area 1 again.

The bandwidth does for a KDE what bin width does for a histogram: small $h$ gives a spiky estimate, large $h$ oversmooths, and the choice is yours rather than the data’s.

The empirical cumulative distribution function

Definition. The empirical CDF $F_n(x)$ is the proportion of observations less than or equal to $x$.

$F_n(x) = \frac{\#\{i : x_i \le x\}}{n}$

It starts at 0 to the left of all your data, jumps up by $1/n$ at each observation, and reaches 1 after the largest one. It is a staircase with $n$ steps.

For $Y = \{-2, -0.5, 0, 1, 1.5\}$ with $n = 5$, $F_5$ is 0 below $-2$, then 0.2, 0.4, 0.6, 0.8, and 1 above 1.5.

Why bother, when it looks less informative than a histogram? Because it involves no choices. No bin width, no bandwidth, no starting point. It uses every observation exactly and throws nothing away, which makes it the one picture on this page that cannot mislead you through a choice you made.

The box plot

A compact drawing of the five-number summary.

The box runs from the lower quartile to the upper quartile, so its length is the interquartile range:

$\mathrm{IQR} = q_n(0.75) - q_n(0.25)$

which is the width of the middle half of the data. A line inside the box marks the median.

The whiskers extend outward from the box, but only so far. The rule is that a whisker reaches the most extreme observation still within $1.5 \times \mathrm{IQR}$ of the box. Anything beyond that gets drawn as an individual dot and is flagged as a possible outlier.

That rule is a calculation, and exams ask you to do it.

Worked example. A data set has five-number summary $0.14, \ 1.45, \ 2.06, \ 3.26, \ 9.70$. Is the maximum drawn as an outlier?

First the IQR:

$\mathrm{IQR} = 3.26 - 1.45 = 1.81$

Then the furthest the upper whisker is allowed to reach:

$3.26 + 1.5 \times 1.81 = 3.26 + 2.715 = 5.975$

The maximum is 9.70, which is beyond 5.975, so it lies past the whisker and is drawn as a separate point.

Reading skew from a box plot

Three signals, and a strong answer names more than one.

1. Whisker lengths. A longer upper whisker means a longer upper tail.
2. Where the outliers are. Dots above the box and none below points to right skew.
3. Where the median sits in the box. A median closer to the lower quartile means the upper half of the middle 50% is more stretched out.

All three pointing upward means right-skewed, which by section 1 means the mean exceeds the median.

5. What each picture cannot show

The useful exam question is not what a display shows, it is what it hides.

A box plot shows five numbers and nothing else. Everything about the shape between those five points is gone. Two data sets can have the same five-number summary and produce near-identical box plots while one is bimodal, meaning its histogram has two separate peaks, and the other has a single peak. A box plot has no way to show the number of peaks. If a question gives you two near-identical box plots and different histograms, this is what it is testing.

A histogram shows shape, and partly shows your choices. Bin width and origin are yours, and features can appear or disappear when you change them.

A single number hides group structure. Female Gentoo penguins have a mean bill depth of 14.2 mm and males 15.7 mm. Pool them and the overall mean describes neither group, and nothing in that one number reveals there were two groups.

The general shape: the more a summary compresses, the more it hides. The mean is one number and hides nearly everything. The empirical CDF keeps every observation and hides almost nothing.

6. What each summary is estimating

The summaries on this page are not just descriptions of the numbers in front of you. Each one is an estimate of a corresponding feature of the distribution the data came from, and that correspondence is what the rest of the course is built on.

Read the table as a set of promises: compute the thing on the left from your data, and you have a guess at the thing on the right about the world. Whether those guesses are any good, and what “good” even means for a guess, is the subject of Estimators and Sampling Distributions and everything after it.

Vocabulary to deploy

• Order statistics $x_{(1)} \le \cdots \le x_{(n)}$, the data sorted; empirical quantile $q_n(p)$.
• Five-number summary: minimum, lower quartile, median, upper quartile, maximum.
• Interquartile range $\mathrm{IQR} = q_n(0.75) - q_n(0.25)$, the width of the middle half.
• The $1.5 \times \mathrm{IQR}$ rule for whisker extent and outlier identification.
• Sample variance $s_n^2$ with divisor $n-1$; sample standard deviation; median absolute deviation.
• Robustness: resistance of a summary to extreme values. Median and MAD are robust; mean and variance are not.
• Density histogram, with bar area proportional to the proportion of data and height $\frac{\text{count}}{nb}$; bin, bin width.
• Kernel density estimate, kernel $K$, bandwidth $h$.
• Empirical cumulative distribution function $F_n$.
• Bimodal, right-skewed, left-skewed.