D. Statistical Analysis · Two-Variable Data Analysis and Validity

Evaluating the Validity of Statistical Claims

D3.2 D3.1About 70 minutes

Learning goals

I can assess the validity of a claim in the media by examining the source, sample, method and possible bias.
I can identify unstated cause-and-effect assumptions in a statistical claim.
I can explain why a headline about a graph may or may not be true, using the data.
I can write a short, balanced critique of a statistical claim and suggest what further information is needed.

Words to know

validity
How well a conclusion is supported by the data and the way the data were collected.

source
The person or organization that collected or published the data. Ask whether they have an interest in the result.

sampling bias
A systematic error caused by a sample that does not represent the population, such as a voluntary-response online poll.

response bias
A systematic error caused by the way questions are asked or answered, such as a leading question.

margin of error
For a random sample, the amount by which a reported percentage could reasonably differ from the true population value, stated with a confidence level such as 19 times out of 20.

peer review
Checking of a study by independent experts before it is published in a scientific journal.

"BIG INCREASE IN PROFITS!"

This is based on the curriculum's own sample problem. Suppose a company newsletter shows a line graph of its profits for the four quarters of the year under the headline "Big Increase in Profits!" Should investors cheer?

In the last lesson you built a toolkit of tricks to watch for. In this lesson you will use it systematically to judge whether a conclusion is valid: whether the data, and the way they were collected and presented, really support it.

Left: the company's graph, with the axis starting at $4.0 million, looks like a rocket. Right: the same quarterly profits on an axis from 0 show steady, modest growth.

Worked example

Example 1: Could the headline be true? Could it be false?

Quarterly profits were $4.1, $4.2, $4.3 and $4.6 million. Give reasons the headline "Big Increase in Profits" may or may not be justified.

Size of the increase: from Q1 to Q4, (4.6 − 4.1) ÷ 4.1 ≈ 12.2%. From Q3 to Q4 it is (4.6 − 4.3) ÷ 4.3 ≈ 7.0%. Whether that is "big" depends on the industry and on past years.
The graph: the vertical axis starts at 4.0, so the rise looks far steeper than it is.
Seasonality: many businesses earn more in Q4 (holiday shopping). The fair comparison is Q4 this year vs Q4 last year, which is not shown.
Profit vs sales: profit can jump because of a one-time event (selling a building, a tax refund) even when the business is not growing.
Inflation: if prices rose about 2% to 3% over the year, part of the growth in dollars is not real growth.
Source: a company newsletter has an interest in good news. Audited financial statements are more trustworthy.

The headline may be true (12% growth in a year could be strong) but is not yet justified: the truncated axis exaggerates, and we need last year's figures, the reasons for the Q4 jump, and an independent source before accepting "big".

A checklist for any claim

The curriculum lists the questions to ask: the source of the data, the method of collection, possible sources of bias, the sample size, and any unstated cause-and-effect assumptions. Work through them in order, then give a verdict.

Seven questions to ask before accepting a statistical claim. The names are a memory aid for this course, not an official list.

Notes on each question

Source. Who collected and who published the data? A national statistics agency such as Statistics Canada, a peer-reviewed journal or an independent polling firm usually publishes its methods. An advertiser or advocacy group may still be accurate, but has a stake in the result.
Sample. How many, and how chosen? A random sample of 1000 Canadians gives a percentage that is typically within about 3 percentage points of the true value, 19 times out of 20 (you work with margins of error in the unit One-Variable Data Analysis). A self-selected online poll of 50 000 can be much worse, because size does not fix bias.
Method. Survey, administrative records, sensors, experiment? Only a randomized experiment supports cause directly.
Measure. What exactly was counted? "Unemployed", "poverty" and "crime" have technical definitions that may differ from everyday meanings.
Bias. Leading questions, non-response, groups left out of the sample (people without internet, people in remote communities), or a tiny subgroup.
Cause. Is a correlation being presented as cause? Words like "leads to", "boosts", "makes" signal a causal claim.
Context. Compared with what? Over what time period? Per person or total? Percent or percentage points?

Worked example

Example 2: An advocacy poll

A group campaigning for a new arena posts: "78% of residents want a new arena! (Online poll, 2400 votes.)" Assess the claim.

Source: an advocacy group that wants the arena: possible bias in how it is reported.
Sample/method: an online poll that anyone could answer, probably shared mainly with supporters: voluntary response, not random. People may vote more than once.
Measure: we do not know the question. "Do you support a new arena?" differs from "Do you support a new arena if property taxes rise by 3%?"
Size: 2400 votes sounds large, but a large biased sample is still biased.
Missing information: who voted (residents only?), the exact wording, and how the poll was advertised.

The claim is not valid as a measure of residents' opinion. A fair estimate would need a random sample of residents, a neutral question, and a reported margin of error.

Worked example

Example 3: An unstated cause

A news story reads: "Students who take music lessons score 10% higher in math, so schools should add music to raise math marks." Identify the unstated assumption and evaluate it.

The data show a correlation between music lessons and math marks.
The recommendation assumes music lessons cause higher math marks.
Possible common causes: family income (lessons cost money), parental involvement, or time available for practice and homework.
Possible reverse cause: students who are already strong in patterns and focus may be more likely to continue lessons.
To test the claim, you would need an experiment: randomly assign some students to music lessons and compare math growth.

The conclusion depends on an unstated cause-and-effect assumption that the data do not support. The correlation is interesting, but adding music to raise math marks needs experimental evidence.

Writing a critique

A good critique is balanced. Say what the claim does well, what is missing or doubtful, and what information would settle the question. Use the language of the course: sample, bias, correlation, cause, percentage points, margin of error. Avoid "it's fake" or "it's true" without reasons.

Just for fun

Breaking news: a survey finds that 100% of the people who answered it enjoy answering surveys. The numbers are accurate. The conclusion about "people" is useless. That is the difference between a correct statistic and a valid claim.

Hands-on activity

Critique a real news statistic

You need: a recent news article (newspaper, CBC, a local news site, or a Statistics Canada The Daily release) that uses a statistic or a graph, and paper.

Copy down the claim and where it came from.
Answer the seven checklist questions. Write "not stated" when the article does not say.
If possible, find the original source (the study, poll or StatCan table the article quotes). Does the article report it accurately?
Write a 150-word critique ending with a verdict: valid, doubtful or not valid, and what extra information you would want.
Share it with someone at home and see whether they agree.

What to look for: many news articles leave out the sample size, the exact question, or the margin of error; a link to the original source is a good sign. Statistics Canada releases usually describe their methods and data quality in notes at the end.

Your turn

A sunscreen ad says "Clinically proven to protect" with no further details. What information would you want?

A radio station asks listeners to call in and says 65% of callers oppose a new bike lane. Identify the main problem.

A poll of 1000 randomly selected Ontarians, accurate within about 3 percentage points 19 times out of 20, finds 52% support a policy. Can the newspaper say "a majority supports" it?

A headline says "Coffee drinkers live longer." What unstated assumption might readers make?

Why is a statistic from Statistics Canada usually more trustworthy than one from a company ad?

A survey question asks, "Don't you agree that our parks need more funding?" Name the problem.

More practice: warm-up

Name three things to check about a statistical claim.

Does a large sample guarantee a valid result?

What does "19 times out of 20" refer to in a poll?

What kind of sample is an online poll anyone can answer?

What design is needed to support a cause-and-effect claim directly?

More practice: on your own

An energy drink company funds a study finding that its drink "improves focus". The study had 24 participants and no control group. Evaluate the claim.

A headline reads "Crime up 50% in small town!" The town had 4 break-ins last year and 6 this year. Evaluate.

A graph in a news story shows housing prices for one city from 2020 to 2022 only, rising steeply, under the headline "Prices will keep soaring." Identify two problems.

A study reports that people who sleep 7 to 8 hours earn more than those who sleep less. A website says "Sleep more to get a raise!" Identify the unstated assumption and suggest two alternative explanations.

Explain why a report on "Canadian teens' internet use" based only on a survey of students in Toronto high schools might be biased.

Challenge

Company profits were $4.1, $4.2, $4.3 and $4.6 million in four quarters this year. Last year the same quarters were $3.9, $4.0, $4.4 and $4.8 million. Re-evaluate the headline "Big Increase in Profits".

A researcher claims a new reading program works because students' scores rose from September to June. Explain why this is not enough evidence, and design a better study.

A poll reports 40% support with a margin of error of ±3 percentage points, 19 times out of 20. Another poll a month later reports 42%. A headline says "Support surges!" Evaluate.

Check your understanding

Quiz

A company graph with its vertical axis starting at $4.0 million is titled "Big Increase in Profits". Which is the best first response?

Which sample is most likely to give a valid estimate of Ontario opinion?

"Kids who eat breakfast get better marks, so breakfast boosts marks." The main flaw is:

A poll finds 51% support, margin of error ±3 percentage points 19 times out of 20. Which statement is valid?

Which is an example of response bias?

Why does a large self-selected sample not guarantee accuracy?

For parents and teachers

Big idea (D3.2): to judge a statistical claim, check the source, sample, method, measure, bias, cause assumptions and context, then give a balanced verdict. The curriculum's example, a graph headlined "Big Increase in Profits", shows how a true number can still support a doubtful conclusion.

How to help: Read a news story with a statistic together and ask your teen to walk you through the seven questions. Ask them to find the original source; it is a skill they will use constantly in university.

Common difficulties:

Treating a big sample as automatically reliable.
Calling a claim "fake" without giving reasons, or accepting it without checking.
Missing hidden causal language ("leads to", "boosts").
Forgetting seasonal or year-over-year comparisons.