Residual Sum of Squares: Formula and Example
By Dr. Zubair Khalid, DVM, MS, PhD ·
The residual sum of squares (RSS) is the sum of every squared vertical distance between an observed value and the value a fitted model predicts for it. It is written RSS = Σ(yᵢ - ŷᵢ)², where yᵢ is the observed value, ŷᵢ is the predicted value, and the squared difference (yᵢ - ŷᵢ) is the residual for that observation.
RSS is the number that least-squares regression is built to make as small as possible. It is also the raw material behind R-squared, the standard goodness-of-fit statistic reported in almost every regression table. If you can compute RSS by hand for a handful of points, you understand what a regression line is actually optimizing, why R-squared behaves the way it does, and why RSS on its own is a poor way to compare two studies.
This guide gives the formula, a full five-point worked example with units, the residual table, the distinction between RSS and total sum of squares (TSS), the R² = 1 - RSS/TSS relationship, and the traps that catch students in exams and reviewers in manuscripts.
What the Residual Sum of Squares Measures
A residual is the gap between what you saw and what your model expected. If a model predicts a value of 12 and you observe 15, the residual is 3. Residuals can be positive (the observation sits above the fitted line) or negative (it sits below).
If you simply added residuals, positive and negative values would cancel and a badly fitting model could produce a sum near zero. Squaring removes the sign and also gives large misses more weight than small ones. A residual of 6 contributes 36 to RSS, while a residual of 2 contributes only 4. That asymmetry is deliberate. Least-squares fitting treats one large error as worse than several small ones.
RSS therefore measures unexplained variation. It is the part of the data the model failed to account for. Small RSS means the fitted line tracks the points closely. Large RSS means the points scatter widely around the line.
The same quantity appears under several names in the literature. You will see residual sum of squares, sum of squared residuals, sum of squares residual, ss residual, and squared residuals used interchangeably. The abbreviation SSE (sum of squared errors) is also common, and in simple regression contexts it means the same thing as RSS. When you read a methods section, check whether the authors define their abbreviation, because SSE is sometimes used for the explained sum of squares instead, which is the opposite quantity.
Why It Matters Beyond the Classroom
RSS is not a textbook curiosity. It is used as a model comparison and diagnostic tool in applied research across the life sciences. In a multiscale geographically weighted regression study of prostate cancer incidence across the 50 U.S. states from 2018 to 2022, model performance was compared using R², adjusted R², residual sum of squares, corrected Akaike information criterion, bandwidth diagnostics, and residual spatial autocorrelation [1]. The same pattern appears in a spatial analysis of teenage pregnancy among adolescents aged 15 to 19 in Ethiopia, where spatial regression models were compared using adjusted R², the corrected Akaike information criterion, and the residual sum of squares [2].
RSS also drives variable selection. A feature-screening method for complex response types such as distributional and matrix-valued data uses a marginal general residual sum of squares as the utility score for judging how important each predictor is [3]. In rehabilitation research, investigators compared a single regression equation with four stratified equations by calculating the residual sum of squares and comparing the results [4]. In goodness-of-fit testing for modified Poisson regression, a normalized residual sum of squares test performed well on both type I error probability and power when the link function was wrong [5].
The pattern is consistent: RSS is the workhorse quantity for judging whether one model fits better than another on the same data.
The Formula and Its Parts
The residual sum of squares formula is:
RSS = Σ(yᵢ - ŷᵢ)²
Each symbol has a specific job.
| Symbol | Name | Meaning |
|---|---|---|
| yᵢ | Observed value | The actual measurement for observation i, in its original units |
| ŷᵢ | Predicted value | The value the fitted model produces for observation i, in the same units |
| yᵢ - ŷᵢ | Residual | The signed vertical distance from the point to the fitted line |
| (yᵢ - ŷᵢ)² | Squared residual | The residual multiplied by itself, always zero or positive |
| Σ | Summation | Add the squared residuals across every observation, i = 1 to n |
| n | Sample size | The number of observations contributing to the sum |
Two points about units matter. The residual carries the same unit as the response variable. If you are modeling body mass in kilograms, residuals are in kilograms. Squaring converts them to kilograms squared. RSS is therefore reported in squared units, which is why it is awkward to interpret directly and why R-squared exists.
The order of operations is fixed. Subtract the predicted value from the observed value first, then square, then sum. Squaring before subtracting gives a different and wrong answer.
Worked Example With Five Data Points
Suppose a small pilot study measures five subjects. The response variable is a blood biomarker concentration in milligrams per deciliter (mg/dL), and the predictor is a continuous variable such as time on treatment in weeks. A simple linear regression is fitted, producing the line:
ŷ = 2.0 + 1.5x
This means the model predicts a biomarker value of 2.0 mg/dL at x = 0 and adds 1.5 mg/dL for each additional week. The five observed predictor values are x = 1, 2, 3, 4, and 5 weeks, and the five observed biomarker values are 4, 5, 8, 8, and 10 mg/dL.
For each observation, compute the predicted value from the fitted line, subtract it from the observed value to get the residual, then square the residual.
Step 1: Predicted Values
- x = 1 week: ŷ = 2.0 + 1.5(1) = 3.5 mg/dL
- x = 2 weeks: ŷ = 2.0 + 1.5(2) = 5.0 mg/dL
- x = 3 weeks: ŷ = 2.0 + 1.5(3) = 6.5 mg/dL
- x = 4 weeks: ŷ = 2.0 + 1.5(4) = 8.0 mg/dL
- x = 5 weeks: ŷ = 2.0 + 1.5(5) = 9.5 mg/dL
Step 2: The Residual Table
| Observation | x (weeks) | Observed y (mg/dL) | Predicted ŷ (mg/dL) | Residual y - ŷ (mg/dL) | Squared residual (mg/dL)² |
|---|---|---|---|---|---|
| 1 | 1 | 4 | 3.5 | 0.5 | 0.25 |
| 2 | 2 | 5 | 5.0 | 0.0 | 0.00 |
| 3 | 3 | 8 | 6.5 | 1.5 | 2.25 |
| 4 | 4 | 8 | 8.0 | 0.0 | 0.00 |
| 5 | 5 | 10 | 9.5 | 0.5 | 0.25 |
| Total | 2.5 | 2.75 |
Step 3: Compute RSS
Add the squared residuals:
RSS = 0.25 + 0.00 + 2.25 + 0.00 + 0.25 = 2.75 (mg/dL)²
The sum of the raw residuals is 0.5 + 0.0 + 1.5 + 0.0 + 0.5 = 2.5 mg/dL. Notice that this sum is not zero here, which happens when the fitted line does not pass exactly through the center of the data in the least-squares sense for this particular set of points. In a properly fitted least-squares line, the residuals always sum to zero, so this example line is close to but not identical to the true least-squares solution for these five points. The arithmetic for RSS is identical either way, and the point of the exercise is the squaring step.
Step 4: Interpret the Number
RSS = 2.75 (mg/dL)². That is the total squared vertical distance between the five observed biomarker values and the fitted line. It is not 2.75 mg/dL of error. Squared units are the price of removing sign cancellation.
Observation 3 contributes 2.25 of the 2.75 total, which is 82 percent of the entire RSS. One point dominates the fit. That is normal and it is exactly why a single outlier can pull a regression line toward itself.
RSS Versus Total Sum of Squares
Total sum of squares (TSS) measures the total variation in the response variable around its own mean. Its formula is:
TSS = Σ(yᵢ - ȳ)²
where ȳ is the mean of the observed values. TSS does not involve the model at all. It is a property of the data.
For the five biomarker values 4, 5, 8, 8, and 10, the mean is 7.0 mg/dL. The deviations from the mean are -3, -2, 1, 1, and 3. Squaring and summing gives 9 + 4 + 1 + 1 + 9 = 24.0 (mg/dL)².
Now the comparison becomes clear.
| Quantity | Formula | What it measures | Value in the example |
|---|---|---|---|
| RSS | Σ(yᵢ - ŷᵢ)² | Variation left unexplained by the model | 2.75 (mg/dL)² |
| TSS | Σ(yᵢ - ȳ)² | Total variation in the response around its mean | 24.0 (mg/dL)² |
| Explained sum of squares | TSS - RSS | Variation the model accounts for | 21.25 (mg/dL)² |
The model explains 21.25 of the 24.0 squared units of variation, and leaves 2.75 unexplained. RSS can never exceed TSS for a model that includes an intercept, because a model that predicts the mean for every observation produces residuals identical to the deviations from the mean.
How R-Squared Follows From RSS
R-squared, also called the coefficient of determination, is defined as:
R² = 1 - RSS/TSS
Using the example values:
R² = 1 - 2.75/24.0 = 1 - 0.1146 = 0.885
The fitted line accounts for about 88.5 percent of the variation in the biomarker values around their mean. The remaining 11.5 percent is residual variation.
This formula explains three behaviors that students often memorize without understanding.
R² is bounded between 0 and 1 for an intercept model. When RSS equals TSS, the model has explained nothing and R² = 0. When RSS is 0, the line passes through every point and R² = 1.
R² is unitless. RSS and TSS are both in squared response units, so the units cancel in the ratio. That is why R² can be compared across studies while RSS cannot.
R² is a rescaled RSS. Any change that lowers RSS without changing TSS raises R². Adding predictors to a regression almost always lowers RSS, which is why R² rises mechanically with more variables and why adjusted R² exists.
The relationship also runs the other way. If you know R² and TSS, you can recover RSS as RSS = TSS × (1 - R²). In a study reporting R² = 0.885 and TSS = 24.0 (mg/dL)², RSS must be 2.75 (mg/dL)².
Why Minimizing RSS Defines Least Squares
The least-squares criterion says: choose the intercept and slope that make RSS as small as possible. That single rule generates the standard regression equations for slope and intercept that you find in every statistics text.
The criterion has practical consequences. Because residuals are squared, the fitted line is sensitive to outliers. A point far from the trend produces a large squared residual and pulls the line toward itself. This is why analysts inspect residual plots rather than trusting R² alone.
The criterion also guarantees two properties for a model with an intercept. The residuals sum to zero, and the fitted line passes through the point (x̄, ȳ). These are mathematical consequences of minimizing RSS, not assumptions imposed in advance.
Other fitting criteria exist. Minimizing the sum of absolute residuals produces median regression, which is more robust to outliers. Minimizing the largest absolute residual produces minimax fitting. Least squares remains dominant because it has closed-form solutions, connects directly to maximum likelihood under normal errors, and links cleanly to R² and the F test.
RSS in Real Research Practice
Model comparison is the most common use. When two models are fitted to the same dataset with the same response variable, the model with the smaller RSS fits the observed data more closely. The prostate cancer spatial analysis compared ordinary least squares, geographically weighted regression, and multiscale geographically weighted regression using RSS alongside R², adjusted R², corrected Akaike information criterion, and residual spatial autocorrelation [1]. The teenage pregnancy analysis used the same trio of adjusted R², corrected Akaike information criterion, and RSS to compare spatial regression models [2].
Stratification decisions use RSS as evidence. A rehabilitation study of 1,100 stroke patients compared a single regression equation with four equations derived from decision-tree strata by calculating the residual sum of squares and comparing the results, testing whether subgroup-specific models predicted discharge motor Functional Independence Measure scores better than one pooled model [4].
Variable screening uses RSS as a ranking score. In an Alzheimer's disease neuroimaging application, investigators needed to select relevant single nucleotide polymorphisms from 582,591 candidates for the distribution of voxel-level intensities across 42 brain regions. Their screening procedure used a marginal general residual sum of squares as the utility for evaluating predictor importance, requiring only a distance between data objects [3].
Goodness-of-fit testing uses RSS directly. For modified Poisson regression, which can produce fitted values exceeding one in binary outcome analysis, a normalized residual sum of squares test performed well on type I error probability and power when the link function was misspecified, and the authors recommended it as a goodness-of-fit test for that model class [5].
Regression model selection in materials science uses RSS as a benchmark. A new regression model for stress-strain relations reduced the residual square sum to less than 50 percent of the value obtained with three existing models [6].
Quality control uses residuals directly. Control charts for monitoring multivariate multiple linear regression profiles include designs that monitor the residuals, with statistics built using squared-sum operators [7].
Common Mistakes and Limitations
Treating RSS as comparable across datasets. RSS depends on sample size and on the units of the response variable. A study with 500 observations will almost always have a larger RSS than a study with 50 observations, even if the first model fits better. A response measured in milligrams will produce a different RSS than the same response measured in grams. Never rank studies by RSS unless the sample size, response units, and model structure are identical.
Confusing RSS with TSS or with the explained sum of squares. RSS is unexplained variation. TSS is total variation. Explained sum of squares is TSS minus RSS. Mixing them up reverses the meaning of a result.
Assuming a low RSS means a good model. RSS can be small because the model fits well or because the response variable has a small range. A model with RSS = 0.01 on a response that only varies between 0.00 and 0.02 has explained very little in relative terms. Always pair RSS with TSS or R².
Reading RSS in squared units as if it were in original units. RSS = 2.75 (mg/dL)² is not 2.75 mg/dL of error. The square root of the mean squared residual, called the root mean squared error, returns to original units and is easier to interpret.
Ignoring residual structure. RSS is a single number that hides pattern. Residuals that fan out, curve, or cluster by group indicate assumption violations that RSS cannot detect. Plot residuals against fitted values and against each predictor.
Comparing RSS across different response transformations. A model fitted to log-transformed data and a model fitted to raw data are not on the same scale. Their RSS values are not comparable.
Forgetting that adding predictors always lowers RSS. This is a mathematical certainty for ordinary least squares, not evidence that the new predictor is useful. Use adjusted R², cross-validation, or an information criterion when comparing models of different complexity.
Quick Review
- RSS = Σ(yᵢ - ŷᵢ)², the sum of squared vertical distances from observed points to the fitted line.
- Residuals carry the response unit. Squared residuals and RSS carry squared response units.
- In the five-point example, RSS = 2.75 (mg/dL)² and TSS = 24.0 (mg/dL)², giving R² = 1 - 2.75/24.0 = 0.885.
- TSS measures variation around the mean. RSS measures variation the model leaves unexplained.
- R² = 1 - RSS/TSS is unitless and comparable across studies. RSS is neither.
- Least squares means choosing the line that minimizes RSS.
- RSS grows with sample size and changes with response units, so it is not a cross-study benchmark.
Frequently Asked Questions
What is the difference between RSS and SSE?
In most regression texts they are the same quantity, the sum of squared residuals. Some authors use SSE for the explained sum of squares instead, so always check the definition in the paper you are reading.
Can the residual sum of squares be negative?
No. Every squared residual is zero or positive, so RSS is always zero or greater. A value of zero means the model passes through every observation.
Why is RSS not comparable between two different studies?
RSS scales with sample size and with the units of the response variable. Two studies with different numbers of observations or different measurement scales will produce different RSS values even if their models fit equally well.
What does a high R-squared mean in terms of RSS?
A high R-squared means RSS is small relative to TSS. For example, R² = 0.90 means the model leaves only 10 percent of the total variation unexplained.
Does minimizing RSS always give the best model?
It gives the best fit to the observed data under the least-squares criterion, but not necessarily the best model for prediction or inference. Adding predictors always lowers RSS, so fit alone cannot guide model choice.
How is RSS related to the mean squared error?
Mean squared error divides RSS by a degrees-of-freedom value, usually n - 2 for simple regression with an intercept. Taking the square root returns the value to the original response units.
Related Articles
- Chi-Square Test of Independence in Genetics
- Genetics lab report example chi-square
- How to Calculate Dilution Factor: Formulas, Examples, and Serial Dilutions
- Standard Plate Count: Formula, Calculation, and Example
- How to Calculate the Specific Activity of an Enzyme: Formula and Examples
- How to Calculate Transformation Efficiency: Formula, Examples, and Common Pitfalls
Sources
- Spatial analysis of predictors of prostate cancer incidence in the united states using multiscale geographically weighted regression (MGWR).
- A multiscale geographically weighted regression analysis of teenage pregnancy and associated factors among adolescents aged 15 to 19 in Ethiopia using the 2019 mini-demographic and health survey.
- Feature screening for metric space-valued responses based on Fréchet regression with its applications.
- Predictive accuracy of multiple regression analysis stratified into four groups of admission motor FIM based on decision tree analysis.
- Goodness-of-fit tests for modified Poisson regression possibly producing fitted values exceeding one in binary outcome analysis.
- A New Regression Model for the Prediction of the Stress-Strain Relations of Different Materials.
- Variable parameters memory-type control charts for simultaneous monitoring of the mean and variability of multivariate multiple linear regression profiles.