Grade by Grade
B. Software Development · Software Development: Problem Solving, Design, Testing and the Life Cycle

Testing with a Test Plan: Expected versus Actual

B4.4About 70 minutes
Learning goals
  • I can identify the test scenarios a program needs from its specification.
  • I can choose suitable test data: normal, boundary and invalid values.
  • I can calculate expected outcomes before running a program, record actual outcomes, and conclude pass or fail.
  • I can write a simple automated test that compares expected and actual results.

Words to know

test plan
A table that lists test scenarios, input data, expected outcomes, actual outcomes and a pass/fail conclusion for each test.
test scenario (test case)
One situation to check, such as "a senior buys a ticket".
expected outcome
The correct result, worked out from the specification before the program is run.
actual outcome
What the program really did when the test was run.
normal data
Typical, valid input in the middle of an allowed range.
boundary data
Valid or invalid input right at the edges of a range, such as 12 and 13 for a "under 13" rule. Many bugs live here.
invalid (erroneous) data
Input the program should reject, such as letters for an age, or −1.
regression testing
Running all the old tests again after a change, to make sure the fix did not break anything else.

"It works on my computer"

You typed 40 into your ticket program, saw $14, and called it done. But what about 13? 65? −1? "ten"? A program is not tested until it has been tested on purpose, with a plan.

The Ontario curriculum describes testing very precisely. A test plan must:

  1. identify test scenarios,
  2. identify suitable input data,
  3. calculate expected outcomes,
  4. record actual outcomes, and
  5. conclude pass or fail by comparing expected with actual.

In Programming Basics you met the idea of testing with a full range of cases. This lesson turns it into a formal plan.

1. Scenariowhat situationis tested2. Input datanormal, boundary,invalid3. Expectedwork it outBEFORE running4. Actualrun and recordwhat happened5. Concludepass ifactual = expectedA FAIL is good news for the tester: a bug found before the user finds it.
The five steps of a test plan. The expected outcome is calculated from the specification before the program runs, so the program cannot "convince" you it is right.

Choosing test data

You cannot test every possible input, so choose inputs that are most likely to reveal bugs. Divide the inputs into groups (classes) that the program should treat the same way, then pick:

  • Normal data: one or two typical values from the middle of each group.
  • Boundary data: the values on both sides of every edge between groups. Programmers often write < when they meant <=, so boundaries catch off-by-one errors.
  • Invalid data: values the program must reject safely: out of range, wrong type, empty.

For the cinema rule from the flow chart lesson (under 13: $9; 65 and over: $10; otherwise $14), the groups and boundaries look like this:

child $9adult $14senior $10020406080100boundary: 12 | 13boundary: 64 | 65normal: 5normal: 40normal: 80invalid: −1 and "ten" are not ages at all, so they must be rejected
Ages split into three groups. Normal tests come from the middle of each group; boundary tests sit on both sides of each edge (12 and 13, 64 and 65); invalid tests lie outside the allowed values.
Worked example

Example 1: Writing a test plan

Write a test plan for ticket_price(age), then run it on a version of the program that a student wrote.

  1. Scenarios: child, adult and senior prices; the two boundaries; invalid ages.
  2. Input data and expected outcomes come from the specification, calculated before running: 5 → $9; 12 → $9; 13 → $14; 40 → $14; 64 → $14; 65 → $10; 80 → $10.
  3. Run each test and record the actual result.
  4. Conclude: 65 expected $10 but got $14: FAIL. Looking at the code, the student wrote age > 65 instead of age >= 65. Every other test passes.
  5. After fixing it, run all the tests again (regression testing), not just the one that failed.

The test plan below found a boundary bug that the normal tests (5, 40, 80) all missed.

#ScenarioInputExpectedActualPass/Fail
1child, normal5$9$9pass
2child, boundary12$9$9pass
3adult, boundary13$14$14pass
4adult, normal40$14$14pass
5adult, boundary64$14$14pass
6senior, boundary65$10$14FAIL
7senior, normal80$10$10pass
8invalid: negative−1error message, ask again
9invalid: texttenerror message, ask again
Test plan for the ticket price. Tests 8 and 9 check the input routine (see the exceptions lesson); their Actual column is filled in when that part is tested.

Automating the tests

Typing each test by hand is slow, and slow tests do not get re-run. Because ticket_price is a function that returns a value (modular design pays off!), a short loop can run the whole plan in a fraction of a second:

def ticket_price(age):
if age < 13:
return 9
elif age > 65:
return 10
else:
return 14
ages = [5, 12, 13, 40, 64, 65, 80]
expected = [9, 9, 14, 14, 14, 10, 10]
for i in range(len(ages)):
actual = ticket_price(ages[i])
if actual == expected[i]:
result = "pass"
else:
result = "FAIL"
print(ages[i], expected[i], actual, result)

Output: one line per test (age, expected, actual, result). Every line says pass except 65 10 14 FAIL. Change > to >= on line 4 and run it again: all seven pass.

More kinds of testing

  • Black-box testing chooses tests from the specification only, without looking at the code. The plan above is black-box.
  • White-box testing looks at the code and makes sure every path runs at least once: every branch of every if, loops that run zero times, once and many times.
  • Regression testing reruns all the tests after every change.
  • User (acceptance) testing: real users try the program to confirm it meets the specification, often at the end of the testing phase.
Worked example

Example 2: White-box test design

Design tests that make every branch of this function run: larger(a, b) returns a if a > b, otherwise b.

  1. There are two paths: the if is true, or it is false.
  2. True path: (9, 3) → expected 9.
  3. False path: (3, 9) → expected 9.
  4. The boundary between the paths is a = b: (5, 5) → expected 5 (the false path runs and returns b, which equals a).
  5. Negative numbers are an extreme case worth one test: (−2, −7) → expected −2.

Four tests: (9, 3), (3, 9), (5, 5), (−2, −7) with expected results 9, 9, 5, −2.

Just for fun

A software tester walks into a café. Orders 1 coffee. Orders 0 coffees. Orders −1 coffees. Orders 99 999 999 coffees. Orders a lizard.

Then the first real customer walks in and asks where the washroom is. The whole café crashes.

Hands-on activity

Test a family member's program (or your own)

You need: a computer with Python, paper, and a program from an earlier lesson (the HST calculator, the temperature helper or the bill splitter).

  1. Read the program's specification (or write one in three sentences).
  2. Draw a test plan table with the columns #, Scenario, Input, Expected, Actual, Pass/Fail.
  3. Write at least 8 tests: 2 normal, 4 boundary and 2 invalid. Fill in Expected using a calculator before you run anything.
  4. Run each test and fill in Actual and Pass/Fail.
  5. For every fail, write a one-line bug report: test number, what happened, what should have happened. Fix the bug and rerun all tests.
  6. Bonus: automate the plan with a loop like the one in this lesson.

What to look for: expected values written before running; boundaries tested on both sides; at least one invalid input; every failed test rerun after the fix.

Your turn

1

List the five parts of a test plan the curriculum describes.

2

Why should expected outcomes be calculated before running the program?

3

A program gives free entry to children under 6. Which ages are the boundary tests?

4

Classify each test value for a mark from 0 to 100: 73, 0, 100, 101, −5, "A".

5

In the automated test output, which test failed, and what bug does it reveal?

6

What is regression testing, and why do it after fixing the 65 bug?

7

Write expected outcomes for hst_total(price) (price × 1.13, rounded to the cent) for 100, 0 and 19.99.

More practice: warm-up

1

What does it mean if the actual outcome equals the expected outcome?

2

Is 50 a normal or a boundary value for "pass if mark >= 50"?

3

Give one example of invalid data for a program that asks for a number of people.

4

Which kind of testing chooses tests from the specification without looking at the code?

5

What should you write when a test fails?

More practice: on your own

1

A grading function returns A (80-100), B (70-79), C (60-69), D (50-59) and R (below 50) for whole-number marks. List a full set of boundary test inputs with expected outputs.

2

A test plan for the bill splitter has these tests: 120 split 4 ways, 100 split 5 ways, 60 split 3 ways. What is wrong with it? Add three better tests.

3

Explain the difference between black-box and white-box testing, and give one advantage of each.

4

Design white-box tests for: if n % 2 == 0: print("even") / else: print("odd"). Include a negative and zero.

5

A loop should add up the numbers in a list. Which three list sizes does a white-box tester choose, and why?

Challenge

1

For the leap-year rule (divisible by 400 → leap; else divisible by 100 → not leap; else divisible by 4 → leap; else not leap), design a minimal set of tests that runs every branch, with expected results.

2

Your automated tests all pass, but a user still finds a bug. Give two reasons this can happen and how to improve the test plan.

3

Write an automated test (Python) for hst_total(price), which should return the price plus 13% HST rounded to 2 decimal places. Use the inputs 100, 0 and 19.99.

Check your understanding

Quiz

1.

Which comes first in a test plan?

2.

For the rule "discount if age is 65 or over", which pair are the boundary tests?

3.

A test's expected outcome is $10 and the actual is $14. The test:

4.

Which is invalid data for a program that needs a mark from 0 to 100?

5.

What is regression testing?

6.

Why are boundary tests so important?

For parents and teachers

Big idea: B4.4 asks students to use a test plan: identify test scenarios and suitable input data, calculate expected outcomes, record actual outcomes, and conclude pass or fail by comparing them.

How to help: Before your teen runs a test, ask "What should it print?" and have them write the answer down first. Then check together. Ask "What about the edge?" for every rule with a number in it (under 13, at least 50...).

Common difficulties:

  • Filling in the expected column after running the program (copying the actual result).
  • Testing only normal values; forgetting boundaries on both sides, and invalid input.
  • Fixing a bug and rerunning only the failed test, not all of them.
  • Vague test plans ("try some numbers"). Every test needs a specific input and a specific expected result.