C. Organization of Data for Analysis · Organization of Data for Analysis

Designing Effective Surveys and Questionnaires

C2.4 C2.1About 70 minutes

Learning goals

I can describe the parts of an effective questionnaire and the ways a survey can be given.
I can write clear, neutral open and closed questions, including rating scales, with categories that do not overlap and cover every answer.
I can identify and fix leading, loaded, double-barrelled, vague and assumption-based questions.
I can explain how anonymity, confidentiality and randomized response encourage honest answers.
I can design and pilot-test a short questionnaire for a research question.

Words to know

questionnaire
The written set of questions used in a survey.

open question
A question people answer in their own words. Rich information, but hard to tally.

closed question
A question with a fixed set of answer choices (yes/no, multiple choice, rating scale). Easy to tally and compare.

Likert (rating) scale
A closed question that asks how much a person agrees or disagrees with a statement, usually on a 5-point or 7-point scale.

leading question
A question worded to push the respondent toward a particular answer.

double-barrelled question
A question that asks about two things at once but allows only one answer.

anonymity
Nobody, not even the researcher, can link answers to the person who gave them.

confidentiality
The researcher knows who gave which answers but promises not to reveal it.

pilot test
A small trial run of a survey to find confusing questions before the real survey.

One question, two answers

Imagine two versions of a survey question:

"Should the city waste more money on a new downtown arena?"
"Should the city build a new downtown arena?"

The first will get far fewer "yes" answers, yet both claim to measure the same opinion. Survey results are only as good as the questions. The curriculum asks you to design surveys that collect reliable data while respecting ethics, privacy and the need for honest responses, and that avoid bias, including cultural bias.

Ways to give a survey

Personal interview: high response and can clarify questions, but expensive, and the interviewer's presence can affect answers on sensitive topics.
Telephone: faster and cheaper than in person; many people do not answer unknown numbers.
Mail or paper (self-administered): private, so good for sensitive questions; response rates are often low.
Online: cheap and fast; may leave out people without Internet access, and open links invite voluntary-response bias.

Structure of a good questionnaire

Introduction: who is doing the survey, its purpose, how long it takes, that taking part is voluntary, and how answers will be kept private (anonymous or confidential).
Easy, interesting questions first to build trust.
Group related questions and move logically from topic to topic.
Sensitive questions later, once trust is established.
Demographic questions (age group, grade) near the end, and only those you actually need.
Thank the respondent.

Keep it short. Every extra question lowers the response rate.

Kinds of questions

Open: "What is the biggest challenge of Grade 12?" Rich, unexpected answers; hard to summarize; people may skip them.
Closed: yes/no (dichotomous): "Do you have a part-time job?"
Closed: multiple choice: one answer from a list. Include "Other (please specify)" when you cannot list every possibility.
Closed: checklist: "Check all that apply."
Closed: ranking: "Rank these five activities from most to least favourite."
Closed: rating scale (Likert scale): level of agreement or satisfaction on a balanced scale.
Closed: numerical: "How many hours did you sleep last night?" (with units).

Answer choices must be mutually exclusive (no answer fits two boxes) and exhaustive (every possible answer has a box).

A balanced 5-point Likert scale: the same number of negative and positive options, with a neutral middle. Responses are ordinal data.

Answer categories must not overlap and must cover every possible answer. Using "under 2" and "2 to under 4" gives every value exactly one box.

Writing good questions: avoid these problems

Leading: "Don't you agree that...?" or "Most experts think X. Do you?"
Loaded words: "waste", "beautiful", "dangerous", "so-called".
Double-barrelled: "Do you like the food and the prices in the cafeteria?" (What if you like the food but not the prices?)
Vague words or time frames: "Do you exercise regularly?" Regularly means different things to different people. Better: "In the past 7 days, on how many days did you exercise for at least 30 minutes?"
Double negatives: "Do you disagree that students should not have cellphones in class?"
Jargon or acronyms respondents may not know.
Assumptions: "How many hours a week do you spend at your part-time job?" assumes everyone has one. Ask a filter question first: "Do you have a part-time job?" If yes, then ask the hours.
Cultural assumptions: about holidays, family structure, foods, sports, or first language. Have people from different backgrounds read your questions.
Too personal or unnecessary: if you do not need a person's exact age, ask for an age group.

Getting honest answers

People may not answer honestly when a question is embarrassing, when they want to look good, or when they fear consequences. Ways to help:

Anonymity: no names or identifying details are collected, so nobody can link an answer to a person.
Confidentiality: the researcher knows who answered but promises not to reveal it, and stores data securely.
Self-administered surveys (paper or online) rather than face-to-face interviews for sensitive topics.
Neutral, non-judgmental wording.
Randomized response: a clever technique that uses a coin so no one, not even the researcher, can know what any individual's true answer was, yet the overall proportion can still be estimated.

Randomized response: each respondent privately flips a coin. Only the respondent knows whether a "Yes" is real or came from the coin, but the researcher can still estimate p.

Worked example

Example 1: Fixing flawed questions

Identify the problem with each question and rewrite it. (a) "Do you agree that our hard-working teachers deserve a raise?" (b) "How satisfied are you with the gym and the library?" (c) "How much time do you spend on social media?" (d) "What did you do with your family over March Break?" (e) "How old are you? 13–15, 15–17, 17–19"

(a) Leading and loaded ("agree", "hard-working"). Rewrite: "Should teachers' salaries be increased, decreased or kept the same?"
(b) Double-barrelled. Ask two separate questions: one about the gym and one about the library, each on the same 5-point satisfaction scale.
(c) Vague time frame and units. Rewrite: "Yesterday, about how many hours did you spend on social media? (under 1, 1 to under 2, 2 to under 3, 3 or more)"
(d) Assumption that every student spent the break with family and did something notable. Rewrite: "During March Break, which of these did you do? (check all that apply) ... Other (please specify)"
(e) Overlapping and not exhaustive: 15 and 17 fit two boxes; ages under 13 and over 19 have no box. Rewrite: "under 13, 13 to 14, 15 to 16, 17 to 18, 19 or older".

Each rewrite is neutral, asks one thing, uses a clear time frame, makes no assumptions, and has non-overlapping, complete answer choices.

Worked example

Example 2: Randomized response

A school wants to estimate the proportion p of students who have ever used a phone during a test. Each of 200 students privately flips a coin. Heads: answer truthfully. Tails: flip again and say "Yes" for heads, "No" for tails. The school receives 70 "Yes" answers. Estimate p.

Observed proportion of Yes: 70 ÷ 200 = 0.35.
Theory (from the tree): P(Yes) = (1/2)p + (1/2)(1/2) = p/2 + 1/4.
Set them equal: p/2 + 0.25 = 0.35, so p/2 = 0.10.
p = 0.20.

About 20% of students have used a phone during a test. No individual's answer reveals anything, because any "Yes" might have come from the coin.

Worked example

Example 3: Designing questions for a research question

The curriculum suggests the question: is there a relationship between a person's age and their hours per week of Internet use? Write three survey questions to collect the data and classify each one's data type.

Age: "What is your age in years? ____" gives numerical discrete data (or use age groups for more privacy, giving ordinal data).
Internet use: a vague question like "How much do you use the Internet?" would give unreliable answers. Better: "In the past 7 days, about how many hours in total did you spend online, not counting school or work? ____ hours" gives numerical continuous data (recorded to the nearest hour).
Context for analysis: "What is your main way of going online? (phone / computer / tablet / other)" gives categorical nominal data that could help explain differences.
Plan the sample: people of all ages, not just students, chosen randomly, or the relationship cannot be studied.

Age (discrete or ordinal), weekly hours online (continuous), main device (nominal). The two key questions give two-variable data for a scatter plot in Two-Variable Data Analysis and Validity.

Just for fun

Survey designer's nightmare: "Do you never not avoid double negatives? (yes / no)". Nobody knows what "yes" means, including the person who wrote it.

Hands-on activity

Design and pilot a survey

You need: paper or a free online form tool, and 2 or 3 family members or friends to pilot-test it.

Choose a research question, for example "Is there a relationship between screen time before bed and hours of sleep for Grade 12 students?"
Write an introduction (purpose, about how long, voluntary, anonymous).
Write 6 to 8 questions: at least one numerical question with units and a clear time frame, one Likert-scale question, one multiple-choice question with an "Other" option, and one open question.
Check every question against the list of problems in this lesson (leading, double-barrelled, vague, overlapping categories, assumptions, cultural assumptions).
Pilot test: ask 2 or 3 people to fill it in out loud, saying what they are thinking. Note any question they hesitate over, misread or cannot answer.
Revise the survey and write a short note listing what you changed and why.

What to look for: pilot testers almost always find something: a missing answer option, an unclear time frame, or a question that is answered in two different ways. Finding these before the real survey is the point. Keep your final survey; it can become part of your Culminating Data Management Investigation.

Your turn

What is wrong with: "How often do you eat fast food? (rarely / sometimes / often)"? Rewrite it.

Rewrite this double-barrelled question: "Is the bus service fast and reliable?"

What is the difference between anonymity and confidentiality?

Give one advantage and one disadvantage of open questions.

A randomized response survey (same coin method as Example 2) of 400 people gets 130 "Yes" answers. Estimate p.

Why should demographic questions usually come near the end?

What is a pilot test and why is it done?

More practice: warm-up

Is "Do you have a pet? (yes / no)" open or closed?

What does "exhaustive" mean for answer choices?

Name one loaded word that should be avoided in survey questions.

What type of data do Likert-scale responses produce?

Which survey method is usually best for sensitive questions: personal interview or anonymous self-administered survey?

More practice: on your own

Fix the assumption: "Which streaming service do you use most?"

A question offers income choices: under $40 000; $40 000–$60 000; $60 000–$80 000; over $80 000. Find two problems and fix them.

Write a balanced 5-point satisfaction scale for a question about a community centre's programs.

With the coin method, what proportion of "Yes" answers would you expect if nobody had the sensitive trait? If everybody did?

Explain why the order of questions can affect answers. Give an example.

Challenge

A randomized response design uses a die instead of a coin: roll 1 to 4, answer the real question truthfully; roll 5, say "Yes"; roll 6, say "No". Find P(Yes) in terms of p, and estimate p if 41% of 500 respondents say Yes.

Explain why the randomized response technique protects privacy but gives a less precise estimate than a direct question answered honestly.

Design a short questionnaire (5 questions) to study the curriculum's question "Is there a relationship between marks and hours of study?" for Grade 12 students. Include an introduction sentence, and state how you would encourage honest answers.

Check your understanding

Quiz

"Do you like the cafeteria's food and its prices?" is an example of a:

Which set of answer choices for "hours of homework last night" is best?

A survey records no names, IP addresses or other identifying details. The survey is:

Using the coin method of randomized response, 38% of respondents say "Yes". The estimated proportion with the trait is:

Which question contains an assumption that should be fixed with a filter question?

What is the main purpose of a pilot test?

For parents and teachers

Big idea (C2.4, C2.1): reliable survey data depend on well-designed questions (clear, neutral, one idea at a time, with complete non-overlapping choices), a sensible structure, and conditions that encourage honest answers, such as anonymity or the randomized response technique.

How to help: Be a pilot tester! When your teen designs a survey, answer it out loud and say where you hesitate or what you are unsure about. Also look at real surveys you receive (store receipts, apps) and spot leading or double-barrelled questions together.

Common difficulties:

Overlapping categories (0–2, 2–4) and categories that leave answers out.
Vague words such as "often" or "regularly" without a time frame.
Confusing anonymity (no one can identify you) with confidentiality (identities are known but protected).
Algebra in the randomized response formula: P(Yes) = p/2 + 1/4, so p = 2 × (P(Yes) − 0.25).