D. Statistical Analysis · One-Variable Data AnalysisMargin of Error and Confidence Level
D1.4About 75 minutes
Learning goals- I can interpret a statistic reported with a margin of error and confidence level, such as "accurate to within 3 percentage points, 19 times out of 20".
- I can write and interpret a confidence interval for a population proportion.
- I can calculate a margin of error for a proportion at the 90%, 95% and 99% confidence levels.
- I can explain and calculate how sample size, margin of error and confidence level are related, including finding the sample size needed for a given margin of error.
Words to know
margin of error (ME)
The largest likely difference, at a stated confidence level, between a sample statistic and the population parameter, caused by random sampling.
confidence interval
A range of values, statistic ± margin of error, that is likely to contain the parameter.
confidence level
The percent of all possible samples whose confidence intervals would contain the true parameter, such as 95% ("19 times out of 20").
critical value z*
The z-score that matches a confidence level: 1.645 for 90%, 1.96 for 95%, 2.576 for 99%.
percentage point
A difference between two percents: from 35% to 38% is 3 percentage points (but about 8.6 percent of 35%).
"Accurate to within 3 percentage points, 19 times out of 20"
Canadian news reports about polls often end with a sentence like: "A random sample of 1 067 Ontario adults was surveyed. The results are considered accurate to within ±3 percentage points, 19 times out of 20." Most people skip over it. By the end of this lesson you will know exactly what it promises, and what it does not promise.
In the last lesson you saw that a sample proportion p̂ wobbles around the true proportion p with standard error √(p(1 − p) ÷ n), and that for large samples its distribution is approximately normal. Put those two facts together with the 68–95–99.7 rule and you get the margin of error.
From standard error to margin of error
About 95% of a normal distribution lies within 1.96 standard deviations of its centre (more precisely than "2" from the 68–95–99.7 rule: P(Z < 1.96) = 0.975). So for about 95% of random samples, p̂ lands within 1.96 standard errors of p. Turned around:
95% confidence interval for p: p̂ ± 1.96 √(p̂(1 − p̂) ÷ n)
(We use p̂ in the formula because p is unknown.)
- The margin of error is ME = z* √(p̂(1 − p̂) ÷ n).
- "19 times out of 20" is just 95%: 19 ÷ 20 = 0.95.
- Other confidence levels use other critical values: **90%: z* = 1.645; 95%: z* = 1.96; 99%: z* = 2.576**.
Worked exampleExample 1: A poll's confidence interval
In a random sample of 1 000 Ontario adults, 400 support a proposed change to the school year. Find the 95% confidence interval for the proportion of all Ontario adults who support it, and interpret it.
- p̂ = 400 ÷ 1 000 = 0.40.
- SE = √(0.40 × 0.60 ÷ 1 000) = √0.00024 ≈ 0.01549.
- ME = 1.96 × 0.01549 ≈ 0.030, or 3.0 percentage points.
- Interval: 0.40 ± 0.030 → 0.370 to 0.430, or 37.0% to 43.0%.
"We are 95% confident that between about 37% and 43% of Ontario adults support the change." In poll language: 40%, accurate to within 3 percentage points, 19 times out of 20.
What "95% confident" really means
The true proportion p is a fixed number; it does not move. What changes from sample to sample is the interval. The 95% describes the method: if many random samples were taken and an interval built from each, about 95% of those intervals would contain p, and about 5% (1 in 20) would miss it. For any one poll, we never know whether it is one of the lucky 19 or the unlucky 1.
How n, ME and confidence level are related
The curriculum asks you to investigate these relationships. From ME = z* √(p̂(1 − p̂) ÷ n):
- Bigger sample → smaller margin of error. ME is proportional to 1 ÷ √n: four times the sample size halves the ME.
- Higher confidence → bigger margin of error. To be more sure of catching p, you need a wider net: z* grows from 1.645 to 1.96 to 2.576.
- p̂ near 0.5 → biggest ME. p̂(1 − p̂) is largest at 0.5, so pollsters often quote the ME for p̂ = 0.5 as a safe maximum for every question in the poll.
You cannot have everything: for a fixed n, a narrower interval means less confidence.
| n | ME at 90% (z* = 1.645) | ME at 95% (z* = 1.96) | ME at 99% (z* = 2.576) |
|---|
| 100 | ±8.2 points | ±9.8 points | ±12.9 points |
| 400 | ±4.1 | ±4.9 | ±6.4 |
| 1 068 | ±2.5 | ±3.0 | ±3.9 |
| 2 401 | ±1.7 | ±2.0 | ±2.6 |
Choosing a sample size
Solve ME = z* √(p(1 − p) ÷ n) for n:
**n = p(1 − p) × (z* ÷ ME)²**
If you have no estimate of p, use p = 0.5, which gives the largest (safest) n. Always round up.
Worked exampleExample 2: How many people should a pollster ask?
A pollster wants a margin of error of 3 percentage points at the 95% confidence level. (a) How large a sample is needed if p is unknown? (b) What if she wants ±2 points? (c) What if a previous poll suggests p is about 0.20 and she wants ±3 points?
- (a) n = 0.25 × (1.96 ÷ 0.03)² = 0.25 × 4 268.4 ≈ 1 067.1. Round up: n = 1 068.
- (b) n = 0.25 × (1.96 ÷ 0.02)² = 0.25 × 9 604 = 2 401. Going from ±3 to ±2 more than doubles the cost.
- (c) n = 0.20 × 0.80 × (1.96 ÷ 0.03)² = 0.16 × 4 268.4 ≈ 682.9. Round up: n = 683.
(a) 1 068, (b) 2 401, (c) 683. This is why so many Canadian polls survey about 1 000 to 1 100 people: it is the size that gives about ±3 points, 19 times out of 20.
Is one candidate really ahead?
A poll reports Party A 38% and Party B 35%, accurate within ±3 points, 19 times out of 20. Party A's interval is 35% to 41% and Party B's is 32% to 38%. The intervals overlap a lot, and the 3-point gap is no bigger than the margin of error, so the poll cannot confidently say who is ahead. News reports often call this a "statistical tie". (A formal test for the difference between two proportions, which you may meet in university, is more precise than comparing the intervals, but the rough rule is a good first check.)
What the margin of error does NOT cover
The margin of error measures only random sampling error in a random sample. It says nothing about:
- biased samples (volunteers, online opt-in panels, people who refuse to answer being different from those who answer);
- leading or confusing question wording;
- people changing their minds after the poll.
Many pollsters who use opt-in online panels avoid quoting a classical margin of error for this reason, because their samples are not true random samples; some quote a different measure instead. Also, results for a subgroup (for example, only the 150 respondents aged 18 to 24) have a much larger margin of error than the whole poll, because n is much smaller.
Worked exampleExample 3: A margin of error for a mean
The same idea works for means: ME = z* × σ ÷ √n. A random sample of 64 Grade 12 students reports a mean of 2.4 hours of homework per week night. Assume σ = 0.8 h. Find a 95% confidence interval for the mean of all Grade 12 students in the board.
- SE = 0.8 ÷ √64 = 0.8 ÷ 8 = 0.1 h.
- ME = 1.96 × 0.1 = 0.196 h (about 12 minutes).
- Interval: 2.4 ± 0.196 → 2.204 h to 2.596 h.
We are 95% confident that the board's mean is between about 2.2 and 2.6 hours per week night.
Just for funA pollster's favourite bedtime story ends: "...and they lived happily ever after, plus or minus 3 percentage points, 19 times out of 20."
Hands-on activitySpreadsheet investigation: how big must a sample be?
You need: a computer or tablet with a free spreadsheet (Google Sheets or similar).
- In cell A1 type the true proportion 0.40 (pretend you know the population).
- In B2:B1001 type
=IF(RAND()<$A$1,1,0). Each row is one randomly chosen person (1 = supports, 0 = does not). - In D2 type
=AVERAGE(B2:B101) (a sample of 100), in D3 =AVERAGE(B2:B401) (400), and in D4 =AVERAGE(B2:B1001) (1 000). - Make the spreadsheet recalculate 20 times: type any number in an empty cell and press Enter (every edit makes RAND() pick new values; in Excel you can also press F9). Each time, record whether each sample proportion is within 0.03 of 0.40.
- For each sample size, what fraction of your 20 trials landed within 3 points? Which sample size gives about 19 out of 20?
What to look for: samples of 100 land within 0.03 of 0.40 only about half the time, samples of 400 about 78% of the time, and samples of 1 000 about 95% of the time (the theory: z = 0.03 ÷ SE gives about 0.61, 1.22 and 1.94 standard errors). With only 20 trials your fractions will bounce around, but the pattern should be clear: you need roughly 1 000 people for ±3 points, 19 times out of 20.
Your turn
1A poll says 52% support a plan, accurate within ±4 percentage points, 19 times out of 20. Write the confidence interval and the confidence level.
48% to 56%, at the 95% confidence level.
2In a random sample of 500 people, 150 say yes. Find the 95% margin of error.
p̂ = 0.30. SE = √(0.30 × 0.70 ÷ 500) = √0.00042 ≈ 0.0205. ME = 1.96 × 0.0205 ≈ 0.040 (about 4.0 points).
3For the same sample, find the 99% margin of error.
ME = 2.576 × 0.0205 ≈ 0.053 (about 5.3 points).
4If a pollster quadruples her sample size, what happens to the margin of error?
It is halved (ME is proportional to 1 ÷ √n).
5How many people are needed for ±5 percentage points at 95% confidence, with p unknown?
n = 0.25 × (1.96 ÷ 0.05)² = 0.25 × 1 536.64 = 384.16 → 385.
6A friend says, "There is a 95% chance the true value is in this poll's interval." Give a more careful statement.
If the poll were repeated many times with new random samples, about 95% of the intervals built this way would contain the true proportion. This particular interval either contains it or it does not.
7Why do pollsters often report the margin of error for p̂ = 0.5?
p̂(1 − p̂) is largest at 0.5, so that ME is the maximum for any question in the poll; it is a safe, conservative figure.
More practice: warm-up
1"19 times out of 20" is what confidence level?
2What is z* for 90% confidence?
3A result of 61% ± 2 points gives what interval?
4Does a higher confidence level make the margin of error larger or smaller?
5Does the margin of error account for biased question wording?
No. It covers only random sampling error.
More practice: on your own
1A random sample of 1 200 Canadian adults finds that 63% have taken a winter road trip of more than 100 km. Find the 95% confidence interval.
SE = √(0.63 × 0.37 ÷ 1 200) = √0.00019425 ≈ 0.01394. ME = 1.96 × 0.01394 ≈ 0.0273. Interval ≈ 60.3% to 65.7%.
2A poll of 1 000 voters has a ±3.1-point margin of error. Only 160 of the respondents are aged 18 to 24. Estimate the maximum margin of error for that subgroup.
ME ≈ 1.96 × √(0.25 ÷ 160) = 1.96 × 0.0395 ≈ 0.077, about ±7.7 points. Subgroup results are much less precise.
3Poll results: Candidate X 44%, Candidate Y 39%, ±3 points, 19 times out of 20. Can the pollster say X is ahead?
Intervals: X 41% to 47%, Y 36% to 42%. They overlap (41% to 42%), and the 5-point gap is less than twice the margin of error, so by the rough rule the poll cannot say with confidence that X is ahead, although it leans that way. (A formal test of the difference might reach a different conclusion.)
4A school board wants to estimate the proportion of students who bike to school, within ±4 points at 90% confidence. Last year's estimate was 15%. How many students should it survey?
n = 0.15 × 0.85 × (1.645 ÷ 0.04)² = 0.1275 × 1 691.3 ≈ 215.6 → 216 students.
5A news site runs an online poll: "Should the speed limit be raised? 72% say yes (12 400 votes)." A reader computes a margin of error of ±0.8 points. What is wrong?
The voters chose themselves (a self-selected sample), so it is not a random sample, and anyone could vote more than once. The margin of error formula does not apply; the result could be badly biased no matter how many votes it has.
Challenge
1Show that, for p̂ = 0.5 and 95% confidence, ME ≈ 1 ÷ √n is a good shortcut. Use it to estimate the ME for n = 2 500.
ME = 1.96 × √(0.25 ÷ n) = 1.96 × 0.5 ÷ √n = 0.98 ÷ √n ≈ 1 ÷ √n. For n = 2 500: 1 ÷ 50 = 0.02, about ±2 points (exact: 0.98 ÷ 50 = 0.0196).
2A pollster has budget for 1 500 interviews. Find her maximum ME at 95% and at 99%. Which would you report to the public, and why?
95%: 1.96 × √(0.25 ÷ 1 500) ≈ 1.96 × 0.01291 ≈ ±2.5 points. 99%: 2.576 × 0.01291 ≈ ±3.3 points. Most Canadian polls report the 95% level ("19 times out of 20"); it is the convention readers understand, and either is honest as long as the level is stated.
3Using the normal model, find the confidence level of an interval p̂ ± 2.5 points from a random sample of 1 000 with p̂ = 0.5. (Use the table in lesson 11 or technology.)
SE = √(0.25 ÷ 1 000) ≈ 0.01581. z = 0.025 ÷ 0.01581 ≈ 1.58. P(−1.58 < Z < 1.58) ≈ 2(0.9429) − 1 ≈ 0.886, so about an 89% confidence level (between 1.5 and 2 standard errors, as the table values 0.9332 and 0.9772 suggest).
Quiz
1. A poll reports 45%, accurate to within 3 percentage points, 19 times out of 20. What is the confidence interval?
2. To cut the margin of error in half, a pollster should
3. Which change makes a margin of error larger?
4. What is the 95% margin of error for p̂ = 0.5 and n = 400?
5. Which statement correctly interprets a 95% confidence level?
6. A large online opt-in poll quotes a tiny margin of error. Why should you be cautious?
7. How many people are needed for a 95% margin of error of ±2 percentage points, with p unknown?
Big idea (D1.4): a poll's margin of error, ME = z* √(p̂(1 − p̂) ÷ n), and its confidence level go together. Bigger samples shrink the ME (4 times the people halves it); higher confidence widens it. The ME covers random sampling error only, not bias.
How to help: Next time a poll appears in the news, read the fine print together. Ask: What is the interval? Is the lead bigger than the margin of error? Was the sample random? How many people were in the subgroup being discussed?
Common difficulties:
- Thinking "95% confident" means 95% of people agreed, or that the poll is "95% correct".
- Doubling n and expecting the ME to halve.
- Forgetting to round the sample size up.
- Applying margins of error to self-selected online polls.
- Mixing up percentage points and percent.