Increasing a sample size reduces sampling variability. It does not correct systematic bias, and a biased method can therefore become more precise without becoming more accurate.

Three ways to sample

The population below contains 400 students, and their campus location is associated with their hours of sleep. The population mean is 7.10 hours.

Top: the whole population, plotted by campus position and hours of sleep. Choosing a method and drawing one sample inks the students it selected, and the shape of the ink is the argument. Convenience takes a vertical slice near the surveyor. Voluntary response takes a horizontal band along the bottom, because the students sleeping least are the ones with something to say. A simple random sample scatters. Bottom: three hundred studies by each method, with the true mean marked. Raise the sample size and watch every distribution narrow while two of them stay exactly as far from the line as they were.

A convenience sample selects students who are easy to reach. A voluntary response sample overrepresents people who choose to respond, often because they have stronger opinions or experiences. A simple random sample gives every possible sample of the specified size an equal chance of selection.

In the simulation, the simple random samples center around the population mean. The convenience and voluntary-response methods systematically overrepresent students who sleep less, and their sampling distributions center below the truth.

Bias and variability are different

Bias concerns the center of a sampling distribution, and a method is biased when it systematically overestimates or underestimates the population parameter. Variability concerns the spread of estimates from sample to sample. Increasing \(n\) reduces that spread, but it does not move the center of a biased sampling distribution toward the truth.

In the simulation, raising the sample size makes all three distributions narrower. Only the random-sampling distribution is centered correctly, and the other two become more tightly concentrated around biased values.

Population, sample, parameter, statistic

The population is the full group we want to describe, and the sample is the subset actually observed. A parameter is a numerical characteristic of the population, and a statistic is calculated from the sample. Inference uses statistics to learn about parameters, and the sampling method determines whether that inference can reasonably generalize to the population of interest.

Four probability sampling methods

Simple random sample

Every possible sample of size \(n\) has the same chance of being selected.

Stratified random sample

Divide the population into nonoverlapping strata and take a random sample within each. Strata are usually chosen so individuals are relatively similar within each group and meaningfully different across groups, and stratification can reduce sampling variability when the grouping variable is informative.

Cluster sample

Divide the population into clusters, randomly select clusters, and sample everyone or many individuals within the selected clusters. Good clusters resemble small versions of the population, and cluster sampling is often attractive for logistical reasons.

Systematic sample

Choose a random starting position, then select every \(k\)-th individual. The method still depends on having a sampling frame without a problematic periodic pattern.

Common sources of bias

Undercoverage occurs when some parts of the population are excluded or systematically less likely to be sampled. Nonresponse occurs when selected individuals do not respond and the respondents differ meaningfully from the nonrespondents. Response bias occurs when recorded answers systematically differ from the truth because of wording, interviewer effects, memory, social desirability, or other features of measurement. Voluntary response bias occurs when participation is self-selected.

These are different mechanisms, and a strong response names the one that matches the design.

Random selection and random assignment

Random selection and random assignment solve different problems. Random selection supports generalization from the sample to the population, and random assignment supports causal comparison by balancing other variables across treatment groups, apart from chance. The conclusions depend on which forms of randomization were used.

  Random assignment No random assignment
Random selection Causal conclusion may be supported, with generalization to the sampled population Association only, with generalization to the sampled population
No random selection Causal conclusion may be supported for individuals like those studied Association only, with limited generalization

An observational study does not impose treatments. Without random assignment, confounding remains a possible explanation for an association.

A useful final distinction is precision versus accuracy. A biased estimate can have a very small standard error, and a narrow confidence interval centered on the wrong value is still wrong.