Bias in Sampling and Surveys

The poll that got it spectacularly wrong

In 1936, a popular American magazine, the Literary Digest, mailed about 10 million straw-vote ballots asking people how they would vote in the presidential election. More than 2 million came back, one of the largest polls ever taken. The magazine confidently predicted that Alf Landon would defeat President Franklin Roosevelt.

Roosevelt won in a landslide, carrying every state except two. A much smaller poll, run by George Gallup with careful sampling methods, predicted the result correctly.

How can over two million responses be so wrong? The answer is bias, and it is the subject of this lesson.

What is bias?

Bias is a systematic error: something in the way data are collected pushes the results consistently in one direction.

Compare it with sampling variability (from the lesson Variability in Data and Sampling Variability):

• Sampling variability is random. Results scatter on both sides of the truth. A bigger random sample makes the scatter smaller.
• Bias is systematic. Results lean the same way every time. A bigger sample does not remove it; you just get the wrong answer more precisely.

Think of a dartboard where the bullseye is the true population value and each dart is the result from one sample.

Five kinds of bias

1. Sampling bias. The way the sample is chosen makes some members of the population more (or less) likely to be included. It includes undercoverage, when the sampling frame misses part of the population, and the bias that comes from non-random samples (convenience and voluntary-response samples).

Example: a survey about public transit conducted by phoning landlines during working hours reaches mostly older people and people who work from home, and few daily commuters.

2. Non-response bias. Some people chosen for the sample do not respond, and they differ from those who do.

Example: a survey about workload sent to all teachers is answered mostly by teachers with lighter workloads, because the busiest teachers have no time to fill it in.

3. Response bias. People answer, but not accurately. Causes include leading or loaded questions, wanting to look good (social desirability: people over-report exercise and voting and under-report junk food), fear of consequences, an interviewer's presence, or simply not remembering.

Example: "Most doctors recommend daily exercise. How many days a week do you exercise?"

4. Measurement bias. The instrument or method consistently over- or underestimates.

Example: a bathroom scale that reads 1.5 kg too heavy; measuring reaction time with a phone app that has a built-in delay; estimating screen time from memory when phones record it exactly.

5. Cultural bias. The questions or methods reflect one cultural group's language, experiences or assumptions.

Example: "What did you get for Christmas?" assumes everyone celebrates Christmas. A reading test full of references to cottage life assumes experiences many students have not had. A survey offered only in English in a neighbourhood where many residents speak other languages leaves people out. "Family" questions that assume only two parents and children may not fit families that include grandparents, aunts, uncles or other relatives.

Textbooks sometimes sort examples differently (wording problems are often called response bias, sometimes measurement bias). What matters most is that you can explain how the flaw pushes the results and how to fix it.