D. Statistical Analysis · Two-Variable Data Analysis and ValidityThe Line of Best Fit by Least Squares
D2.4About 75 minutes
Learning goals- I can explain what "least squares" means and why the squares of the residuals are used.
- I can calculate the equation of the least-squares line y = ax + b by hand from a table of sums.
- I can find the line from r, the means and the standard deviations.
- I can interpret the slope and y-intercept in context and use the line to predict.
Words to know
line of best fit
A straight line that models the trend in a scatter plot.
residual
Actual y minus predicted y: e = y − ŷ. Positive for points above the line, negative below.
least-squares (regression) line
The one line that makes the sum of the squared residuals, Σ(y − ŷ)², as small as possible.
ŷ ("y-hat")
The value of y predicted by the model for a given x.
slope (a)
The predicted change in y for each increase of 1 unit in x.
y-intercept (b)
The predicted value of y when x = 0. It has a sensible meaning only if x = 0 makes sense in context.
Everyone's line is different
Give five people the rink data and a ruler, and you will get five slightly different lines of best fit. For predictions that everyone can check, we need one line chosen by a clear rule. The rule used in almost every field (economics, biology, sports analytics, machine learning) is least squares.
Remember from Grade 9: you drew lines of best fit by eye, found their equations, used technology to get the regression line, and met residuals: residual = actual y − predicted y. This lesson shows the formulas behind the technology.
Why squares?
For any line, each point has a residual e = y − ŷ, the vertical gap between the point and the line. We want the residuals small overall. But simply adding them does not work: positive and negative residuals cancel. In fact every line through the centre point (x̄, ȳ) has residuals that add to 0, whatever its slope!
So we square each residual (making them all positive, and punishing big misses much more than small ones) and choose the line that makes
Σ(y − ŷ)², the sum of squared residuals,
as small as possible. Picture each residual as the side of a real square: the least-squares line is the one with the smallest total area of squares.
The formulas
Using calculus (which you will meet in university, and in the Grade 12 course Calculus and Vectors), one can show that the least-squares line ŷ = ax + b has
a = [nΣxy − (Σx)(Σy)] ÷ [nΣx² − (Σx)²]
b = ȳ − a·x̄
Notice that the numerator of a is the same as the numerator of r, and the denominator is the first bracket from r. So if you have already calculated r, you have most of the work done.
The formula for b says the line always passes through the centre of the data, (x̄, ȳ).
There is also a neat link to r: a = r · (s_y ÷ s_x). The slope has the same sign as r, and a stronger correlation (bigger |r|) means a steeper line for the same spreads.
Worked exampleExample 1: Study time
Find the least-squares line for the study-time data (n = 6, Σx = 33, Σy = 422, Σxy = 2456, Σx² = 219). Predict the mark for 7 hours of study.
- a = [6(2456) − 33(422)] ÷ [6(219) − 33²] = 810 ÷ 225 = 3.6.
- x̄ = 33 ÷ 6 = 5.5 and ȳ = 422 ÷ 6 ≈ 70.333.
- b = 70.333 − 3.6(5.5) = 70.333 − 19.8 ≈ 50.53.
- The line is ŷ = 3.6x + 50.53.
- For x = 7: ŷ = 3.6(7) + 50.53 = 25.2 + 50.53 ≈ 75.7.
ŷ = 3.6x + 50.53. Each extra hour of study is associated with about 3.6 more percentage points, on average. A student who studies 7 h is predicted to score about 76%. The intercept (50.5% for 0 hours) is outside the data (2 to 9 h), so treat it with caution.
Worked exampleExample 2: Rink data
For the rink data, n = 12, Σx = −66, Σy = 840, Σxy = −5654, Σx² = 800. Find the least-squares line and interpret the slope and intercept.
- a = [12(−5654) − (−66)(840)] ÷ [12(800) − (−66)²] = −12 408 ÷ 5244 ≈ −2.366.
- x̄ = −66 ÷ 12 = −5.5 and ȳ = 840 ÷ 12 = 70.
- b = 70 − (−2.366)(−5.5) = 70 − 13.014 ≈ 56.99.
- ŷ = −2.37x + 56.99.
Slope: for each 1 °C rise in the daily high, the canteen sells about 2.4 fewer cups, on average. Intercept: on a day with a high of 0 °C, the model predicts about 57 cups. Here x = 0 is inside the data range (−16 °C to 4 °C), so the intercept has a real meaning.
The least-squares line passes through the centre point (x̄, ȳ) = (−5.5, 70). The slope triangle shows a fall of about 7.1 cups for every 3 °C warmer.Worked exampleExample 3: From r and the summary statistics
For 50 adults, height x has x̄ = 170 cm and s_x = 8 cm; mass y has ȳ = 68 kg and s_y = 10 kg; r = 0.6. Find the least-squares line and predict the mass of a person 180 cm tall.
- a = r · s_y ÷ s_x = 0.6 × 10 ÷ 8 = 0.75 kg per cm.
- b = ȳ − a·x̄ = 68 − 0.75(170) = 68 − 127.5 = −59.5.
- ŷ = 0.75x − 59.5. For x = 180: ŷ = 135 − 59.5 = 75.5 kg.
ŷ = 0.75x − 59.5; predicted mass about 75.5 kg. The intercept (−59.5 kg at 0 cm) is meaningless: x = 0 is far outside the data. It only positions the line.
Technology and notation
- Desmos:
y_1 ~ a x_1 + b gives a, b and r (and r²). - Spreadsheet:
=SLOPE(B2:B13, A2:A13) and =INTERCEPT(B2:B13, A2:A13). Note the y-values come first. - Some books write the line as y = mx + b or ŷ = b₀ + b₁x. The meaning is the same: slope and intercept.
- Keep at least 3 or 4 significant digits in a and b; rounding the slope too early makes predictions drift.
Just for funThe method of least squares was published by Adrien-Marie Legendre in 1805. Carl Friedrich Gauss later said he had been using it since 1795, which started one of the best-known "I did it first!" arguments in the history of mathematics.
Hands-on activityPredict a height from an arm span
You need: your arm-span and height data from the first lesson (or the sample family: arm span 152, 160, 165, 171, 178, 183 cm; height 155, 158, 168, 170, 180, 181 cm), a calculator, and a spreadsheet or Desmos if you have one.
- Using arm span as x, find a and b by hand with the formulas.
- Write the equation and interpret the slope in a sentence.
- Measure the arm span of someone who was not in your data (a neighbour, a friend). Predict their height, then measure it. How close were you?
- Check your a and b with technology.
What to look for: for the sample family, ŷ ≈ 0.92x + 14.3 (a ≈ 0.918, b ≈ 14.3). A good prediction is usually within a few centimetres. The slope is close to 1 because height and arm span are close to equal for most adults.
Your turn
1Find the least-squares line for (1, 4), (2, 6), (3, 7), (4, 10), (5, 11).
n = 5, Σx = 15, Σy = 38, Σxy = 132, Σx² = 55. a = [5(132) − (15)(38)] ÷ [5(55) − 15²] = 90 ÷ 50 ≈ 1.800. x̄ = 3, ȳ = 7.6, b = ȳ − a·x̄ ≈ 2.20. ŷ ≈ 1.800x + 2.20
2Use your line from the previous question to predict y when x = 3.5.
ŷ = 1.8(3.5) + 2.2 = 8.50.
3Find the least-squares line for (10, 9), (20, 7), (30, 8), (40, 4), (50, 2).
n = 5, Σx = 150, Σy = 30, Σxy = 730, Σx² = 5500. a = [5(730) − (150)(30)] ÷ [5(5500) − 150²] = −850 ÷ 5000 ≈ −0.170. x̄ = 30, ȳ = 6, b = ȳ − a·x̄ ≈ 11.10. ŷ ≈ −0.170x + 11.10
4Show that the line in the previous question passes through (x̄, ȳ).
x̄ = 30, ȳ = 6. ŷ = −0.17(30) + 11.1 = −5.1 + 11.1 = 6 = ȳ ✓.
5A model for a phone plan is ŷ = 0.05x + 25, where x is megabytes over the limit and ŷ is the monthly bill in dollars. Interpret the slope and intercept.
Slope: each extra megabyte adds about $0.05. Intercept: with no extra data, the bill is about $25 (the base price).
6For a data set, r = −0.9, s_x = 2 and s_y = 5. Find the slope of the least-squares line.
a = r · s_y ÷ s_x = −0.9 × 5 ÷ 2 = −2.25.
7Why can't we choose the line that makes the plain sum of residuals Σ(y − ŷ) as small as possible?
Positive and negative residuals cancel. Every line through (x̄, ȳ) has Σ(y − ŷ) = 0, so this rule does not pick out one line.
More practice: warm-up
1A point is (4, 20) and the line predicts ŷ = 17. Find the residual.
2Is a point below the line's residual positive or negative?
3Through which point does every least-squares line pass?
(x̄, ȳ), the centre of the data.
4If r is negative, what is the sign of the slope a?
Negative (a = r · s_y ÷ s_x, and s_x, s_y are positive).
5ŷ = 2x + 3. Predict y for x = 10.
More practice: on your own
1Find the least-squares line for (3, 20), (5, 18), (6, 15), (8, 14), (9, 10), (11, 9).
n = 6, Σx = 42, Σy = 86, Σxy = 541, Σx² = 336. a = [6(541) − (42)(86)] ÷ [6(336) − 42²] = −366 ÷ 252 ≈ −1.452. x̄ = 7, ȳ = 14.333, b = ȳ − a·x̄ ≈ 24.50. ŷ ≈ −1.452x + 24.50
2Find the least-squares line for (1, 2), (2, 3), (3, 5), (4, 6), (5, 9).
n = 5, Σx = 15, Σy = 25, Σxy = 92, Σx² = 55. a = [5(92) − (15)(25)] ÷ [5(55) − 15²] = 85 ÷ 50 ≈ 1.700. x̄ = 3, ȳ = 5, b = ȳ − a·x̄ ≈ −0.10. ŷ ≈ 1.700x − 0.10
3A car's value is modelled by ŷ = −2400x + 31 000, with x = age in years for cars aged 1 to 8 years. Interpret the slope. Should you use the model for a 15-year-old car?
Slope: the value drops about $2400 per year, on average. No for 15 years: it predicts −2400(15) + 31 000 = −$5000, which is impossible. 15 is far outside the data (1 to 8 years).
4For 40 students, x̄ = 72 (Grade 11 math mark), s_x = 9, ȳ = 70 (Grade 12 mark), s_y = 12, r = 0.75. Find the line and predict the Grade 12 mark of a student with 80 in Grade 11.
a = 0.75 × 12 ÷ 9 = 1. b = 70 − 1(72) = −2. ŷ = x − 2. For x = 80: ŷ = 78.
5Explain why the slope of the least-squares line and r always have the same sign.
a and r share the same numerator, nΣxy − ΣxΣy. Their denominators are both positive (a square-root and a positive bracket), so the signs match.
Challenge
1For the study-time data, the line for predicting x from y is x̂ = 0.266y − 13.2 (approximately). Rearrange it to the form y = ... and compare with ŷ = 3.6x + 50.53. Are they the same line? Why not?
Solving x = 0.266y − 13.2 for y gives y ≈ 3.76x + 49.7. This is not the same as ŷ = 3.6x + 50.53 (slopes 3.76 vs 3.6). The first minimizes horizontal squared distances and the second minimizes vertical ones. They agree only when r = ±1.
2Show that for any data set, the sum of the residuals from the least-squares line is 0.
Σ(y − ŷ) = Σ(y − ax − b) = Σy − aΣx − nb. Since b = ȳ − a·x̄, nb = nȳ − a·nx̄ = Σy − aΣx. So Σ(y − ŷ) = Σy − aΣx − Σy + aΣx = 0.
3If every x-value is increased by 10 (for example, temperatures measured from a different starting point), what happens to a and b?
The slope a does not change (all the deviations x − x̄ stay the same). The intercept changes to keep the line through the new centre: b_new = ȳ − a(x̄ + 10) = b − 10a.
Quiz
1. The least-squares line minimizes:
2. nΣxy − ΣxΣy = 300 and nΣx² − (Σx)² = 150. The slope is:
3. x̄ = 8, ȳ = 30 and a = 2.5. What is b?
4. ŷ = −1.8x + 95 models percent battery left (y) after x hours of video. What does −1.8 mean?
5. r = 0.5, s_x = 4 and s_y = 12. The slope of the least-squares line is:
6. A model has x = years of experience (from 2 to 20 in the data). When is the y-intercept meaningful?
7. Which point always lies on the least-squares line?
Big idea (D2.4): the least-squares line is the one line that makes the total of the squared vertical gaps (residuals) as small as possible. Its slope is a = [nΣxy − ΣxΣy] ÷ [nΣx² − (Σx)²] and it always passes through (x̄, ȳ).
How to help: Ask your teen to read the slope of a model aloud as a sentence with units: "for each extra hour, the mark goes up about 3.6 points, on average." Ask whether the intercept makes sense in the situation.
Common difficulties:
- Using r as the slope.
- Rounding a too early, so b and predictions drift.
- Forgetting to use ȳ − a·x̄ (some students write b = ȳ − x̄).
- Interpreting an intercept that lies far outside the data.
- In spreadsheets, putting the x-range first in
SLOPE (it is SLOPE(y-range, x-range)).