B. Software Development · Software Development: Managing a Team Software Project

A Formal Test Plan, Part 2: Integration, Regression, System and Acceptance Testing

A4.2 B1.3About 75 minutes

Learning goals

I can explain why modules that pass their own unit tests can still fail together, and write integration tests that check the seams.
I can use temporary files in tests so that tests never touch real data.
I can run regression tests after every change and add a test for every bug that is fixed.
I can test non-functional requirements such as speed, and plan system and acceptance tests with the client.
I can write a clear bug report and a test summary report.

Words to know

integration testing
Testing modules together to check the interfaces (seams) between them: the data they pass and the assumptions they share.

stub and driver
A stub is a simple stand-in for a module that is not written yet; a driver is test code that calls a module that has no caller yet. They allow incremental integration.

system testing
Testing the complete program as a whole, the way users will use it, against all the requirements, including non-functional ones.

acceptance testing
Testing by (or with) the client, using the acceptance criteria, to decide whether the software is accepted.

regression
A bug in something that used to work, caused by a later change.

continuous integration (CI)
Automatically building and running all the tests every time someone pushes or opens a pull request.

bug report
A record of a defect: title, steps to reproduce, expected and actual results, severity, environment and status.

test summary report
A short report at the end of a test cycle: what was tested, how many tests passed and failed, open bugs, and a recommendation.

The parts work. Does the whole?

By the end of sprint 2, every LendIt module passed its unit tests. Then Amara tried a simple story: lend three items to one member, save, quit, restart, and lend a fourth. LendIt allowed it. The 3-item limit, a Must requirement that had passed its unit tests, did not work in real use.

Unit tests check pieces in isolation. Bugs also live in the seams between pieces: one module returns text where another expects a number, or a date in a different format, or a list in a different order. This lesson finishes the formal test plan from A4.2 with integration and regression testing, and adds system and acceptance testing, which complete the code, test, deploy cycle of B1.3.

The bug at the seam. loans.py compares member numbers as integers, but the buggy load_loans returned them as strings read from the file. 1042 == "1042" is False in Python, so loans were never counted.

Integration testing

An integration test uses two or more real modules together. For LendIt the most important seam is between storage.py and the rest: everything saved must come back exactly as it was, with the same types.

There are two broad approaches:

Big bang: wait until all modules are finished and test everything together. Simple, but when it fails it is hard to tell which module is at fault, and it happens late.
Incremental: add modules one at a time. Top-down starts from main.py with stubs for unfinished modules; bottom-up starts from low-level modules like storage.py with drivers (test code) calling them. Each step adds one seam, so a failure points to that seam.

LendIt integrates bottom-up: loans with storage first, then items, then the menu.

Here is the storage module. (Reading and writing CSV files is taught in the lesson Reading and Writing Text and CSV Files in the unit Data Types, Files and Expressions.)

"""storage.py: load and save LendIt loans (CSV files with ISO dates)."""

import csv

from datetime import date

FIELDS = ["item_id", "member_no", "borrowed", "due", "returned"]

def save_loans(path: str, loans: list) -> None:

"""Write all loans to a CSV file (R5)."""

with open(path, "w", newline="") as f:

writer = csv.DictWriter(f, fieldnames=FIELDS)

writer.writeheader()

for loan in loans:

row = dict(loan)

for key in ("borrowed", "due", "returned"):

row[key] = loan[key].isoformat() if loan[key] else ""

writer.writerow(row)

def load_loans(path: str) -> list:

"""Read loans from a CSV file; return [] if the file does not exist (R5)."""

try:

with open(path, newline="") as f:

rows = list(csv.DictReader(f))

except FileNotFoundError:

return []

loans = []

for row in rows:

loans.append({

"item_id": row["item_id"],

"member_no": int(row["member_no"]),

"borrowed": date.fromisoformat(row["borrowed"]),

"due": date.fromisoformat(row["due"]),

"returned": date.fromisoformat(row["returned"]) if row["returned"] else None,

})

return loans

storage.py. Dates are written as ISO text and turned back into date objects when loaded; member numbers are turned back into int. A missing file means "no loans yet", not a crash.

The integration tests. Notice setUp and tearDown: each test gets a fresh temporary folder that is deleted afterwards, so tests never touch real data and never affect each other.

import os

import tempfile

import unittest

from datetime import date

from loans import due_date, late_fee, can_borrow

from storage import save_loans, load_loans

def make_loan(item, member, borrowed, returned=None):

return {"item_id": item, "member_no": member, "borrowed": borrowed,

"due": due_date(borrowed), "returned": returned}

class TestLoansWithStorage(unittest.TestCase):

def setUp(self):

self.folder = tempfile.TemporaryDirectory() # a fresh, empty folder

self.path = os.path.join(self.folder.name, "loans.csv")

def tearDown(self):

self.folder.cleanup() # delete it afterwards

def test_round_trip_keeps_every_value(self):

loans = [make_loan("SS-01", 1042, date(2027, 1, 4)),

make_loan("SK-07", 7, date(2027, 1, 2), date(2027, 1, 9))]

save_loans(self.path, loans)

self.assertEqual(load_loans(self.path), loans)

def test_limit_still_applies_after_reload(self):

loans = [make_loan(i, 1042, date(2027, 1, 4)) for i in ("SS-01", "SK-07", "GM-12")]

save_loans(self.path, loans)

self.assertFalse(can_borrow(1042, load_loans(self.path)))

def test_fee_from_reloaded_loan(self):

save_loans(self.path, [make_loan("SS-01", 1042, date(2027, 1, 4), date(2027, 1, 21))])

loan = load_loans(self.path)[0]

self.assertAlmostEqual(late_fee(loan["due"], loan["returned"]), 0.75)

def test_missing_file_gives_empty_list(self):

self.assertEqual(load_loans(self.path), [])

if __name__ == "__main__":

unittest.main(verbosity=2)

test_integration.py. Every test saves with storage.py and then uses the loaded data with loans.py: they test the seam, not the modules alone.

With the correct storage.py, all four tests pass. With the buggy version (which forgot int()), the report shows two failures:

$ python3 -m unittest test_integration

.F.F

======================================================================

FAIL: test_limit_still_applies_after_reload (test_integration.TestLoansWithStorage...)

...

AssertionError: True is not false

======================================================================

FAIL: test_round_trip_keeps_every_value (test_integration.TestLoansWithStorage...)

...

- 'member_no': '1042',

+ 'member_no': 1042,

...

Ran 4 tests in 0.004s

FAILED (failures=2)

Output shortened with ... (terminal, not Python). The round-trip test's diff shows the problem directly: '1042' (a string) where 1042 (an int) was saved.

Worked example

Example 1: Why the unit tests missed it

The storage module had a unit test that saved a list of loans and checked that load_loans returned a list of the same length. The loans module's tests built their loans directly with member_no as an int. Explain why both passed, and what the integration test does differently.

The storage unit test checked only the length of the loaded list, not the values and types, so the string member numbers went unnoticed.
The loans unit tests never used loaded data; they built loans in the test with ints, which is what can_borrow expects.
Each module was correct by its own assumptions, and the assumptions disagreed about the type of member_no.
The integration test feeds the output of one real module into the other, so a disagreement shows up. The round-trip test (assertEqual(load_loans(path), loans)) compares every value and type.
Lesson for the unit tests too: the storage test should have compared the whole loaded list with what was saved.

Unit tests check each module against its own idea of the interface. Integration tests check that the ideas match. The interface specification should have said member_no: int (and now it does).

Regression testing

A regression is a bug in something that used to work, caused by a later change. Every change is a risk: the tidy-up that broke can_borrow in the last lesson was a regression. The defence is simple: re-run every test after every change, and keep main green.

Two habits make regression testing powerful:

Every bug fix comes with a test that would have caught it. Write the test first, watch it fail, fix the bug, watch it pass. The test stays in the suite forever, so that bug can never quietly come back. test_limit_still_applies_after_reload is LendIt's regression test for bug B-07.
Automate it. Many hosting sites can run the whole suite automatically on every push and every pull request, which is called continuous integration (CI). A pull request whose tests fail cannot be merged. Without a hosting site, the team rule is "run python3 -m unittest before every merge".

A bug's life. A tester reports it, someone is assigned, the fix comes with a regression test, the tester verifies it, and the bug is closed. If the test still fails, the bug is reopened.

Model: a bug report

Bug B-07: 3-item limit not applied after restarting LendIt · Reported by Amara, Wed Nov 18, 2026 · Severity: high (breaks Must requirement R7) · Environment: Python 3.13, Windows laptop, build a91c2e0

Steps to reproduce: 1. Start LendIt. 2. Lend SS-01, SK-07 and GM-12 to member 1042. 3. Save and quit. 4. Start LendIt again. 5. Lend SK-02 to member 1042.

Expected: "Member 1042 already has 3 items out." The loan is refused.

Actual: the loan is accepted. Before the restart (step 2 plus one more loan) it was correctly refused.

Notes: happens only after loading from the file, so the storage module is suspected. Assigned to Priya. Status: Fixed in storage.py (member numbers are now converted with int()); regression test test_limit_still_applies_after_reload added. Verified by Amara, Thu Nov 19.

System testing, including non-functional requirements

System testing runs the complete program the way users will, against all the requirements. Functional scenarios come straight from the user stories ("lend, return late, see the fee"). They can be automated by feeding a file of typed answers to the program, for example python3 main.py < scenario1.txt, and comparing what it prints with an expected-output file.

Non-functional requirements need tests too. N2 says any menu choice must respond within 1 second with 500 items and 2000 loans. A simple timing test uses time.perf_counter():

import time

from loans import can_borrow

# N2: 2000 loans spread over 400 members

loans = [{"item_id": f"X-{i}", "member_no": i % 400, "returned": None}

for i in range(2000)]

start = time.perf_counter()

for member in range(400): # 400 borrow checks, far more than one menu choice

can_borrow(member, loans)

elapsed = time.perf_counter() - start

print("Loans:", len(loans), "Checks: 400")

print("N2", "PASS" if elapsed < 1.0 else "FAIL")

perf.py: a timing test for requirement N2, saved next to loans.py. It times 400 borrow checks against 2000 loans, a far heavier job than one menu choice, and compares the total with 1 second.

On a typical computer this prints Loans: 2000 Checks: 400 and N2 PASS; the measured time itself varies from run to run and computer to computer, which is why the program prints only PASS or FAIL against the requirement. The test must also be run on the client's laptop, the slowest computer that matters (requirement N1). Other non-functional tests for LendIt: N3 (usability) is tested by timing a new volunteer after 15 minutes of training; N4 (privacy) by inspecting the saved files for anything besides member numbers and names.

Acceptance testing

Acceptance testing is done with the client, using the acceptance criteria from the user stories. It answers the question that matters most: does this solve the client's problem? For LendIt, Priya books a 45-minute session with Denise (booked in week 1, as risk K4 required). Amara prepares a script: each Must and Should requirement has a short scenario with sample data, the steps to perform, and the expected result. Denise performs each step herself; Amara records pass or fail. Anything Denise rejects becomes a bug report or a change request.

The outcome is recorded in the test summary report, the last document of the testing phase and an input to closing the project.

Worked example

Example 2: A test summary report

At the end of week 5, Amara writes the test summary. Unit: 46 tests, all passing. Integration: 9 tests, all passing. System: 12 scenarios, 11 passed, 1 failed (the French menu, N5, a Could requirement that was not built). Performance (N2): passed on the centre's laptop. Acceptance: 11 Must/Should scenarios (one per requirement), all passed with Denise. Open bugs: one low-severity bug (a help screen typo). Write the summary's conclusion.

Total automated tests: 46 + 9 = 55, all passing.
System scenarios: 11 of 12 passed. The failure is N5 (French menus), a Could requirement deliberately left out of version 1.0 and listed as out of scope by agreement; it is not a defect.
Every Must and Should requirement passed acceptance testing with the client.
Open defects: 1, low severity (help screen typo), to be fixed before delivery.
Exit criteria from the test plan (all Must tests pass, no critical bugs open) are met.

Recommendation: ready for release once the typo is fixed. 55 of 55 automated tests pass; all Must and Should requirements are accepted by the client; the only failed scenario (N5) is out of scope for version 1.0.

Just for fun

A tester walks into a snack bar. Orders 1 poutine. Orders 0 poutines. Orders 999999 poutines. Orders −1 poutines. Orders a moose. Everything works perfectly. Then the first real customer asks where the washroom is, and the snack bar crashes. (This is why system and acceptance tests use real user stories.)

Hands-on activity

Project step 9: Integrate, regress and accept

You need: your project, your unit tests from project step 8, and your client for 30 to 45 minutes.

Find your project's most important seam (usually saving and loading, or the menu calling a module). Write at least three integration tests for it, using tempfile so no real data is touched. Include a full round-trip test.
Make a habit of regression testing: run python3 -m unittest before every commit to main. Write the rule in your README.
When you find a bug, write a bug report like B-07, then a test that fails because of it, then fix it.
Test one non-functional requirement (speed, privacy or usability) and record the result.
Write an acceptance test script from your user stories: scenario, steps, sample data, expected result. Run it with your client and record pass or fail for each.
Write a one-page test summary report with a recommendation.

What to look for: integration tests that use real modules together; no test that writes to your real data files; every fixed bug has a test; the client, not you, decides whether each acceptance scenario passes.

Your turn

Why can two modules that each pass their unit tests still fail together?

What do setUp and tearDown do in the integration tests?

In Python, what is the value of 1042 == "1042"?

What is a regression, and what is the best defence against one?

Name the parts of a good bug report.

Who decides whether an acceptance test passes?

What is the difference between system testing and acceptance testing?

More practice: warm-up

What does CI stand for in software testing?

What is a stub?

Which Python function gives a high-resolution timer for timing tests?

What does a round-trip test check?

Give one non-functional requirement and how to test it.

More practice: on your own

Write an integration test method that saves an empty list of loans and checks that loading gives an empty list.

Compare big-bang and incremental integration: one advantage of each.

Rewrite this as a useful bug report title and steps: "Returns don't work."

Why is the performance test run on the centre's laptop and not only on the developers' computers?

The test summary in Example 2 shows one failed system scenario, yet recommends release. Explain why that is reasonable.

Challenge

Design an integration test that would catch this bug: save_loans writes dates as 04/01/2027 (day/month/year) but load_loans reads ISO format. Which existing test catches it, and what error do you expect?

The team wants a regression test for bug B-07 that does not depend on the order of rows in the file. Explain how test_limit_still_applies_after_reload already achieves this.

Explain why a test that writes to the real loans.csv could be dangerous, and how continuous integration makes good test isolation even more important.

Check your understanding

Quiz

What do integration tests check that unit tests do not?

Why do LendIt's integration tests use tempfile.TemporaryDirectory()?

A bug is fixed. What should be added to the test suite?

Which is acceptance testing?

What is continuous integration (CI)?

Which detail makes a bug report most useful to the developer?

For parents and teachers

Big idea: This lesson completes A4.2 (a formal testing plan including unit, integration and regression testing) and B1.3 (code, test, deploy). Students learn why parts that work alone can fail together, how to write integration tests that use temporary files, why every bug fix needs a regression test, how to test speed and other non-functional requirements, and how to run acceptance tests with the client.

How to help: Be the client for the acceptance test: follow the script yourself and honestly mark each step pass or fail. Try one thing that is not in the script (a real user always does). Ask to read a bug report they wrote: could you repeat the bug from the steps alone?

Common difficulties:

Assuming unit tests are enough.
Tests that read or write the real data files.
Fixing bugs without adding a test, so the same bug returns later.
Bug reports without steps to reproduce.