A. Programming Concepts and Skills · Data Types, Files and Expressions

Limits of Finite Representation: Floating-Point Precision and Rounding Errors

A1.4 A1.2About 75 minutes

Learning goals

I can describe how a 64-bit float stores a number and what limits its precision and range.
I can explain why values such as 0.1 are stored inexactly and how rounding errors accumulate.
I can compare floats safely with a tolerance and avoid float-controlled loops.
I can choose integer cents, Decimal or Fraction when exact results are required.

Words to know

floating-point number
A number stored as a sign, significant digits and an exponent, like scientific notation in base 2.

IEEE 754 double
The standard 64-bit floating-point format (1 sign bit, 11 exponent bits, 52 fraction bits) used by Python's float and Java's double.

precision
How many significant digits a format keeps: 53 bits, about 15 to 17 decimal digits, for a double.

rounding error
The difference between a true value and the nearest value the format can store.

machine epsilon
The gap between 1.0 and the next larger double, about 2.2 × 10⁻¹⁶ (sys.float_info.epsilon).

catastrophic cancellation
Losing most significant digits when subtracting two nearly equal numbers.

tolerance
An allowed difference used when comparing floats, e.g. math.isclose(a, b) or abs(a - b) < 1e-9.

0.1 + 0.2 = 0.30000000000000004?

In ICS3U you saw that 0.1 + 0.2 does not print 0.3, because some decimal fractions have no exact binary form. In Grade 12, that fact becomes a design question. When you write a physics simulation, a banking system or a game engine, you must know exactly what floats can and cannot do, and design your algorithms so that tiny errors do not grow into big ones.

This lesson opens up the float: how it is stored, where its gaps are, which operations lose accuracy, and what to use instead when you need exact answers.

A Python float is a 64-bit IEEE 754 double: value = ± (1.fraction in binary) × 2 raised to the exponent. The fraction field limits precision; the exponent field limits range (up to about 1.8 × 10³⁰⁸). Field widths are roughly to scale (the 1-bit sign is drawn wider so it can be labelled).

How a float is stored

A float is binary scientific notation. Just as 6.02 × 10²³ has significant digits (6.02) and an exponent (23), a double stores:

a sign bit,
an exponent (11 bits), which sets the size,
a fraction (52 bits, plus one hidden leading 1), which sets the significant digits.

Two limits follow:

Precision: only 53 significant bits, about 15 to 17 decimal digits. Python's sys.float_info.dig reports 15, the number of decimal digits that are always kept safely.
Range: the largest double is about 1.8 × 10³⁰⁸. Larger results become inf. Tiny results eventually become 0.0 (underflow).

A fraction such as 1/10 repeats forever in binary (0.0001100110011…₂), so it is cut off after 53 bits. The stored value is extremely close to 0.1, but not equal.

from decimal import Decimal

print(0.1 + 0.2, 0.1 + 0.2 == 0.3)

print(Decimal(0.1)) # the exact value stored for 0.1

total = 0.0

for _ in range(10):

total += 0.1

print(total, total == 1.0)

0.1 has no exact binary form, so the nearest double is stored. Decimal(0.1) reveals it. Small errors add up when the value is used again and again. Output, line by line: 0.30000000000000004 False | 0.1000000000000000055511151231257827021181583404541015625 | 0.9999999999999999 False

Floating-point numbers are not evenly spaced. In this toy format (2 fraction bits), the gap between neighbours doubles at every power of 2. Real doubles have 52 fraction bits, so the gaps are tiny, but they still double: near 10¹⁶ the gap is 2, so 1e16 + 1 == 1e16.

Five rules for working with floats

Never test computed floats with ==. Use math.isclose(a, b) (a relative tolerance) or abs(a - b) < tol with a tolerance that suits your problem.
Do not control loops with floats. Count with integers and compute the float from the counter.
Avoid subtracting nearly equal numbers. The leading digits cancel and the remaining digits are mostly rounding error. Rearranging a formula often helps.
Add small numbers before big ones, or use math.fsum. Adding 1 to 10¹⁶ has no effect, because the gap between doubles there is 2.
Round for display only. Keep full precision while calculating, then use an f-string such as f"{x:.2f}" when printing.

import math, sys

print(math.isclose(0.1 + 0.2, 0.3))

print(abs((0.1 + 0.2) - 0.3) < 1e-9)

print(2.0 ** 53 + 1, 1e16 + 1 == 1e16)

print((1 + 1e-15) - 1)

print(1e308 * 10, 1e-400, sys.float_info.epsilon)

nan = float("nan")

print(nan == nan, math.isnan(nan))

The main limits: compare with a tolerance; above 2⁵³ not every integer exists as a float; subtracting nearly equal numbers loses accuracy (the answer should be 1e-15); too-large results become inf and too-small ones become 0.0; and "not a number" is not even equal to itself. Output, line by line: True | True | 9007199254740992.0 True | 1.1102230246251565e-15 | inf 0.0 2.220446049250313e-16 | False True

Worked example

Example 1: Why the loop never stops

This loop is supposed to print 0.0, 0.1, …, 0.9 and stop. Instead it runs forever. Explain and fix it.

x = 0.0
while x != 1.0:
print(x)
x += 0.1

After ten additions, x is 0.9999999999999999, not 1.0 (see the program above). The next value is about 1.0999999999999999. x jumps past 1.0 without ever equalling it, so x != 1.0 stays True.
Rule: never use == or != to stop a loop on a float that has been computed.
Fix 1: count with an integer and compute the float: for i in range(10): x = i / 10. Each value is computed once, so errors cannot accumulate.
Fix 2: use < with a margin: while x < 1.0 - 1e-9:.
Fix 1 is better: integer loop counters are exact.

The float never equals 1.0 exactly. Loop over integers (for i in range(10)) and compute i / 10.

A Python 3.12 change

Since Python 3.12, the built-in sum adds floats with a more careful (compensated) algorithm, so sum([0.1] * 10) gives 1.0 even though the plain loop gives 0.9999999999999999. On older versions both give 0.9999999999999999. math.fsum has given the correctly rounded result for a long time. Never write code that depends on which version a user has.

When you need exact answers

Money: store whole cents as an int. All the arithmetic is exact, and Lesson 1's // and % turn cents into dollars for display. Banks and payment systems work this way.
Decimal fractions: decimal.Decimal stores numbers in base 10, so Decimal("0.1") is exactly one tenth. Create it from a string; Decimal(0.1) copies the float's error.
Rational numbers: fractions.Fraction(1, 3) is exactly one third.

The cost is speed: these are much slower than floats, which use the processor's built-in hardware. For science and graphics, floats with careful algorithms are the right choice.

from decimal import Decimal

from fractions import Fraction

prices_cents = [499, 1299, 350] # store money as whole cents

total = sum(prices_cents)

print(f"${total // 100}.{total % 100:02d}")

print(Decimal("0.1") + Decimal("0.2"))

print(Fraction(1, 3) + Fraction(1, 6))

Three ways around binary rounding: integer cents (with // and % from Lesson 1 to display them), Decimal for exact decimal arithmetic (create it from a string), and Fraction for exact rational numbers. Output, line by line: $21.48 | 0.3 | 1/2

Worked example

Example 2: A stock index that lost half its value

In 1982 the Vancouver Stock Exchange started a new index at 1000.000. After every trade the index was recalculated and truncated (cut off) to three decimal places. By late 1983 it had fallen to about 525, while the correctly rounded value was about 1099. Explain how such tiny cut-offs could cause this, and how a programmer should avoid it.

Truncating to 3 decimal places throws away up to 0.001 each time, and always in the same direction (down). Rounding to nearest would sometimes go up and sometimes down, so errors would mostly cancel.
The index was updated thousands of times a day for about 22 months. As a rough estimate, an average loss of about 0.0005 per update adds up: 0.0005 × 3000 updates a day × about 400 trading days is about 600 points. That is the size of the error that was found.
The fix was simply to recompute the index properly. Lesson for designers: never repeatedly round an accumulated value; keep full precision internally and round only for display.
This is the same idea as Example 1: small errors that accumulate in one direction become large.

Systematic truncation, repeated thousands of times, drifted the index down by roughly 570 points. Keep full precision and round only when displaying.

Just for fun

Why did the floating-point number break up with the integer?

It said, "You're always so exact. I need someone who can tolerate a little error, like math.isclose."

Hands-on activity

Measuring error

You need: Python, a calculator, and a ruler or measuring tape.

Measure the length of five household objects to the nearest millimetre. Record them in metres (for example 0.237).
Add them in Python as floats, as Decimal values made from strings, and as integer millimetres. Compare the three totals.
Add 0.1 to a float one million times in a loop, and print the result and its difference from 100000. Then do the same with math.fsum.
Find the smallest whole number p for which pow(10.0, p) + 1 == pow(10.0, p) is True. Explain your result using the gap between doubles.

What to look for: integer millimetres and Decimal give exact totals; the float total may show a tiny error. The loop total is about 100000.0000013, off by roughly 1.3 × 10⁻⁶, while math.fsum gives 100000.0. In step 4, p = 16, because the gap between doubles near 10¹⁶ is 2.

Your turn

Why is 0.5 + 0.25 == 0.75 True while 0.1 + 0.2 == 0.3 is False?

Rewrite if price * qty == 12.6: safely.

What does print(1e308 * 10) show?

What is float("nan") == float("nan")?

About how many significant decimal digits does a double keep?

Why should you write Decimal("2.675") and not Decimal(2.675)?

A loop runs for i in range(5): print(i / 4). Is this safe from accumulating error? Why?

More practice: warm-up

How many bits does a Python float use?

What type should you use to store an amount of money exactly?

What does math.isclose(0.1 + 0.2, 0.3) return?

Is 1/2 stored exactly as a float?

What is the name for a result too large for a float?

More practice: on your own

Predict: print(1.1 * 1.1).

Explain why (1 + 1e-15) - 1 gives 1.1102230246251565e-15 rather than 1e-15.

A student averages 1 000 000 marks stored as floats with a running total. Suggest two ways to make the result more accurate.

Why does 1e16 + 1 == 1e16 give True, while n + 1 == n gives False when n = 10**16?

Write code that prints a total of [19.99, 5.01, 0.10] correctly as dollars without any float arithmetic.

Challenge

The quadratic formula x = (−b + √(b² − 4ac)) / (2a) can lose accuracy when b is large and positive. For a = 1, b = 10⁸, c = 1, explain the problem and give a better formula for that root.

Show that 2⁵³ + 1 cannot be stored exactly as a double, but 2⁵³ + 2 can.

A game moves a character by adding velocity * dt to its position 60 times a second for hours. Its position slowly drifts from where the physics says it should be. Give two design changes that limit the drift.

Check your understanding

Quiz

Why is 0.1 + 0.2 == 0.3 False in Python?

Which comparison is the best way to test whether two computed floats are equal?

Which is the best way to store a bank balance exactly?

Approximately how many significant decimal digits does a 64-bit double keep?

What does Decimal("0.1") + Decimal("0.2") give?

A loop while x != 2.0: x += 0.2 starting from x = 0.0 may never end. Why?

What is the result of 1e16 + 1 == 1e16?

For parents and teachers

Big idea: A1.4 names "precision of floating-point real numbers" and "rounding errors" explicitly. Students should leave able to explain why 0.1 + 0.2 is not 0.3 and to design code that is not fooled by it.

How to help: money is the perfect conversation starter. Ask: "If a bank used floats for every account, what could go wrong over millions of transactions?" The answer should mention accumulated rounding error and storing cents as integers.

Common difficulties:

Thinking floats are "wrong" or buggy. They are a precise, well-designed approximation with known limits.
Using round() to make comparisons work. Use math.isclose or work in integers.
Creating Decimal from a float instead of a string.
Believing more decimal places on screen means more accuracy.