Sampling and bias

A biased sampling method does not improve when you ask more people. It gets more precisely wrong: the estimates cluster more tightly around a value that was never the right one. That single sentence is most of the design unit, and it is the one thing a student can carry into every question about how data were collected.

The framework puts it exactly this way. Bias is “a systematic error in the sampling procedure that results in a statistic being consistently larger or consistently smaller than the parameter.” Consistently. Not sometimes, and not fixably by working harder.

Three ways to choose thirty students

Below are 400 students at one school. Each is plotted by where on campus they spend their time and by how many hours they slept last night, and the two are related: the students who cluster near the library sleep less. The true mean for all 400 is 7.10 hours, drawn as the vertical line in the lower panels. In real life nobody gets to see it.

Run 300 of each and the three rows separate immediately. The simple random sample sits on the line. The other two sit to its left, because both of them systematically over-recruit students who sleep less: convenience because those students are the ones standing nearby, voluntary response because those are the students who feel strongly enough to answer.

Now drag the sample size from 5 to 60 and watch what changes. All three distributions narrow, by about three-quarters. Not one of the three moves sideways. At \(n = 10\) the convenience method centres near 6.68 and the voluntary method near 6.40; at \(n = 60\) they centre near 6.71 and 6.49, still short of 7.10 by about four-tenths and six-tenths of an hour. The simple random sample centres on 7.10 at both.

Bias is about centre, variability about spread

Those are two different failures with two different remedies, and the interactive separates them cleanly.

Variability is the spread of the estimates around wherever they centre. It is a nuisance rather than an error, and a bigger sample fixes it, which is why the spread shrank when you raised \(n\).

Bias is the distance from that centre to the truth. No sample size touches it. The only repair is a different sampling method, because the fault is in how individuals were chosen and not in how many.

A student who has these straight can answer the standard exam question — will increasing the sample size correct this problem? — without hesitating. For variability, yes. For bias, never.

The vocabulary, and the four methods

The population is every individual you want to describe, and the sample is who you actually reach. The gap between the two — everyone the method could never have selected — is where undercoverage lives. A parameter describes the population and a statistic describes the sample, a distinction developed in writing parameters in AP Statistics.

Four random methods are named in the course, and the framework defines each by its mechanism:

• Simple random sample. Every sample of size \(n\) has the same chance of being selected.
• Stratified. Split the population into non-overlapping strata that are internally similar, take a simple random sample within each, and combine. Because each stratum is homogeneous, the estimate carries less variability than a simple random sample of the same size.
• Cluster. Split into clusters, each ideally mirroring the whole population, take a simple random sample of clusters, and use everyone inside the chosen ones. Its advantage is logistical rather than statistical.
• Systematic. A random start, then every \(k\)th individual.

Strata are built to be similar inside and different from each other; clusters are meant to be miniature copies of the population. That is the whole distinction, and it is what makes stratified sampling reduce variability while cluster sampling mostly saves effort.

The named biases

The framework names four, and each has a different point of failure. Undercoverage happens when the method leaves part of the population out or makes it less likely to be chosen. Nonresponse happens when people selected for the sample do not answer, and the ones who did answer differ from the ones who did not. Response bias is when the answers themselves lean one direction, from confusing or leading questions — question wording bias — or from self-reporting. Voluntary response bias is what happens when the sample consists entirely of volunteers, the second row of the interactive.

They are worth telling apart because the exam asks which one a described study suffers from, not whether bias exists.

What each kind of randomness buys

Two different randomisations do two different jobs, and mixing them up is the most common way to lose a conclusion question.

Random selection is how individuals get into the study. It is what licenses generalising to the population. Random assignment is how selected individuals get sorted into treatment groups. It is what licenses a cause-and-effect claim, because it makes the groups alike in every extraneous variable.

The four combinations give four different sentences:

	Random assignment	No random assignment
Random selection	Cause and effect, generalisable to the population	Association only, generalisable to the population
No random selection	Cause and effect, but only for individuals like those studied	Association only, and only for individuals like those studied

An observational study is one where treatments are not imposed; the researcher records what is already there. That is why it cannot establish causation on its own: a confounding variable, associated with both the explanatory and the response variable, offers a rival explanation that the design has not ruled out. The reversal in Simpson’s paradox is that rival explanation doing its work in public.