Linear Regression & the Least Squares Method

What Is Linear Regression?

Linear regression is a statistical technique that finds the straight line which best fits a set of data points. The "best" line minimises the total squared distance from each data point to the line — this is the least squares criterion.

It answers the question: Given a set of $(x, y)$ data, what linear relationship $y = a + bx$ best describes the trend?

Regression Playground

Add, drag and remove data points to see the least squares line update in real time. Watch how outliers affect the slope, intercept and R² value.

Launch Playground

Learning Goals: By the end of this guide, you should be able to:

Calculate the equation of the least squares regression line from data.

Interpret the slope and intercept in context.

Understand and calculate the correlation coefficient $r$ and $R^2$ .

Use residuals to assess how well the model fits.

The Least Squares Regression Line

For a dataset of $n$ points $(x_1, y_1), (x_2, y_2), \ldots, (x_n, y_n)$ , the line of best fit is:

\hat{y} = a + bx

where the slope $b$ and intercept $a$ are:

b = \frac{n\sum x_i y_i - \sum x_i \sum y_i}{n\sum x_i^2 - \left(\sum x_i\right)^2}

a = \bar{y} - b\bar{x}

Here $\bar{x}$ and $\bar{y}$ are the means of $x$ and $y$ respectively.

Key property: The regression line always passes through the point $(\bar{x}, \bar{y})$ .

Correlation Coefficient

The Pearson correlation coefficient $r$ measures the strength and direction of the linear relationship:

r = \frac{n\sum x_i y_i - \sum x_i \sum y_i}{\sqrt{\left(n\sum x_i^2 - (\sum x_i)^2\right)\left(n\sum y_i^2 - (\sum y_i)^2\right)}}

Value of $r$	Interpretation
$r = 1$	Perfect positive linear relationship
$0.7 < r < 1$	Strong positive correlation
$0 < r < 0.7$	Weak to moderate positive correlation
$r = 0$	No linear correlation
$r < 0$	Negative correlation (analogous ranges)

The coefficient of determination $R^2 = r^2$ tells you the proportion of variation in $y$ explained by the model.

Worked Example

Data: $(1, 2),\ (2, 4),\ (3, 5),\ (4, 4),\ (5, 5)$

Step 1: Compute the sums: $\sum x = 15$ , $\sum y = 20$ , $\sum xy = 67$ , $\sum x^2 = 55$ , $n = 5$ .

Step 2: Slope: $b = \frac{5(67) - 15(20)}{5(55) - 15^2} = \frac{335 - 300}{275 - 225} = \frac{35}{50} = 0.7$

Step 3: Intercept: $\bar{x} = 3$ , $\bar{y} = 4$ , so $a = 4 - 0.7 \times 3 = 1.9$

Answer: $\hat{y} = 1.9 + 0.7x$

Residuals

A residual is the difference between an observed value and the predicted value:

e_i = y_i - \hat{y}_i

Positive residual: the point is above the line
Negative residual: the point is below the line
If the model fits well, residuals should be randomly scattered around zero with no visible pattern.

Common Mistakes

Extrapolating beyond the data range — The regression line is only reliable within the range of your data. Predicting far outside this range is unreliable.
Confusing correlation with causation — A strong $r$ value means the variables are associated linearly, NOT that one causes the other.
Using a linear model for non-linear data — Always plot the data first. If the scatter plot shows curvature, a linear model is inappropriate.

Exam Tips (A-Level / AP / IB)

For a quick sanity check, the slope $b$ should have the same sign as $r$ .
In context questions, always interpret the slope: "For each additional unit of $x$ , $y$ increases/decreases by $b$ units on average."
When asked about the reliability of a prediction, mention whether the predicted $x$ value is within the data range (interpolation) or outside it (extrapolation).

Frequently Asked Questions

When should I use $y$ on $x$ vs $x$ on $y$ regression?

Use $y$ on $x$ ( $\hat{y} = a + bx$ ) when you want to predict $y$ from a known $x$ . Use $x$ on $y$ when predicting $x$ from known $y$ . The two lines are generally different unless $r = \pm 1$ .

What if my data has outliers?

Outliers can dramatically affect the regression line. Consider whether the outlier is a genuine data point or an error. Report results both with and without the outlier for transparency.

Probability Distributions — The normal distribution underlies many regression assumptions.
Sequences & Series — Summation notation is the language of regression formulas.
Derivative Applications — The least squares method is derived by minimising a function using calculus.

Try the interactive simulation for this topic