What Is Test-Retest Reliability (Video)

Written guide: Correlation Analysis in SPSS

This video explains test-retest reliability, its calculation, and importance in experimental consistency.

Watch on YouTube

Chapters

  1. 0:12 Introduction
  2. 0:27 Measuring the same people twice
  3. 1:31 Choosing the retest interval
  4. 2:39 Worked example
  5. 3:55 Why a correlation is not enough
  6. 4:46 The intraclass correlation for agreement
  7. 6:12 Choosing the ICC model in practice
  8. 7:27 Reading ICC output
  9. 8:45 The Bland-Altman plot
  10. 10:09 Standard error of measurement and smallest detectable change
  11. 11:29 Parallel forms
  12. 12:15 Troubleshooting
  13. 13:13 A grip-strength case study
  14. 15:27 Smallest detectable change and the MCID
  15. 16:36 Recap and quiz
Video transcript

0:12 Introduction

Test-retest reliability tells you how consistently a measurement gives the same result when you repeat it on the same people, as long as the quantity you're measuring stays the same between the two tests. First, we'll measure the same people twice.

0:27 Measuring the same people twice

We'll also choose the retest interval, the time between the two measurements. When a nurse measures a patient's blood pressure on Monday and again a week later with the same cuff, or a researcher gives the same depression questionnaire to a group of students twice, the two results for each person differ by some amount. If the person hasn't changed, a reliable instrument returns nearly the same result both times, and the size of the difference between those two results is the measurement error. If you step on a bathroom scale twice within one minute, your weight has not changed, so any difference between the two readings comes from the scale. Test-retest reliability matters most when you use a measure to track change over time, as in a clinical trial or a rehabilitation program.

1:31 Choosing the retest interval

A test-retest study compares two occasions, so the retest interval, the time between the first measurement and the second, decides what that comparison tells you. If the interval is too short, participants remember their answers or improve with practice, so agreement looks better or scores shift for reasons that have nothing to do with what you're measuring. A practice effect means scores change on repeat testing because of experience with the test itself, and a memory effect means participants remember their first answers and give the same answers again. If the interval is too long, the quantity itself can change through maturation, recovery or a change in mood, and disagreement then includes real change. Choose the interval based on how stable the quantity is. For a questionnaire that measures a stable trait, about two weeks is a common choice. For blood pressure, you measure on separate days under the same conditions. Keep the instrument, the time of day, the setting and the instructions the same on both occasions.

2:39 Worked example

Second, we will work through an example and see why a correlation is not enough. To see these ideas in numbers, we'll use an illustrative data set. Eight participants have their systolic blood pressure measured twice, one week apart, with the same automated cuff. Systolic pressure is the higher of the two pressures a cuff reports, in millimeters of mercury. The first measurements run from 118 to 163 millimeters of mercury, and the retest values run from 115 to 162. Both occasions have a mean of 140 millimeters of mercury. You subtract each first value from its retest value and get minus 3, minus 5, 3, minus 5, 3, 5, 3 and minus 1 millimeters of mercury. The mean difference is 0, and the standard deviation of the differences is 4.0 millimeters of mercury. When you plot the retest values against the first values, the points sit close to the line where the retest equals the first value, and Pearson's correlation, which measures how closely two sets of scores follow a straight line, is 0.98.

3:55 Why a correlation is not enough

A correlation of 0.98 looks reassuring, but a correlation measures only whether the two sets of scores rise and fall together. Agreement asks whether the two occasions give the same number for the same person. Add 5 millimeters of mercury to each retest value, as if the second cuff read high by that amount. The ranking of the participants does not change, so Pearson's r stays at 0.98. Each retest value now reads 5 millimeters of mercury higher than before, so the two occasions no longer agree. A correlation cannot detect the shift, because even a correlation of 1 allows a constant offset of any size. The shifted cuff is like two clocks, one set five minutes fast, that tick in perfect step but never show the same time.

4:46 The intraclass correlation for agreement

Third, we will use the intraclass correlation for agreement, choose its model and read its output. To detect a systematic shift, you need a statistic that counts differences between occasions as error. The intraclass correlation coefficient, or ICC, is the proportion of total variance that comes from real differences between people. For a test-retest design you use a two-way model with absolute agreement, so a systematic difference between the two occasions lowers the ICC. In the example the absolute-agreement ICC is 0.975. After the shift of 5 millimeters of mercury it falls to 0.93, while the consistency ICC stays at 0.97 and Pearson's r stays at 0.98. The consistency ICC ignores a constant offset between occasions and counts only the scatter of the points around a line parallel to the line of equality. Report the model, the type of ICC and whether the value refers to a single measurement, with a 95 percent confidence interval. A single-measurement ICC describes the reliability of one reading, and an average-measures ICC describes the mean of several readings. A confidence interval is the range of values that fit your data at a stated level of confidence.

6:12 Choosing the ICC model in practice

The confidence interval from the previous section tells you how precise one ICC value is. The ICC itself has several forms, and four questions about your design decide which form you report. The first question is whether all participants get the same raters or occasions. If they do, as in a test-retest study, you use a two-way model. If different raters see different participants, you use a one-way model. Next, choose between the random and the mixed version. Choose random when you want to generalize to other occasions and other testers. Choose mixed when only these occasions matter. Both versions give the same value under absolute agreement. Then choose between absolute agreement and consistency. Test-retest needs absolute agreement, because a systematic shift between the two occasions counts as error. Last, choose between single and average measures. Report the single measure when you will use one test in practice. Report the average measure only when the mean of both sessions is the score you will use. Averaging three trials within one session gives one score per session, and the ICC stays a single-measures ICC.

7:27 Reading ICC output

Once you have chosen the form, the software prints both the single and the average version, and you report one row of the table. Take a grip-strength study of 30 adults measured twice, one week apart. In the software output, the Intraclass Correlation Coefficient table has two rows, and the single measures row gives 0.976, with a 95 percent confidence interval from 0.952 to 0.989. The average measures row gives 0.988, with an interval from 0.975 to 0.994. The F test columns give a value of 84.7, with 29 degrees of freedom in the numerator and 29 in the denominator, and a significance below 0.001. The footnotes tell you the model and the type, here two-way mixed effects and absolute agreement. You report the single measures row, because you will use one test in practice. The F test only checks whether the ICC exceeds zero, and a value of 84.7 with a significance below 0.001 confirms an ICC above zero. The confidence interval shows how precise the ICC is.

8:45 The Bland-Altman plot

Fourth, we will draw the Bland-Altman plot. The ICC gives one number, and a Bland-Altman plot shows where the disagreements lie and how large they are, in the units of the measurement. For each participant you plot the difference between the two readings against the mean of the two readings. The first participant, with 118 and 115 millimeters of mercury, has a mean of 116.5 and a difference of minus 3. The mean difference, 0 millimeters of mercury, is the bias, and you draw it as a horizontal line. The 95 percent limits of agreement are the bias plus or minus 1.96 times the standard deviation of the differences. Here the limits run from minus 7.8 to plus 7.8 millimeters of mercury. After the shift of 5 millimeters of mercury, the whole cloud of points moves up by the same amount. The bias becomes 5, and the limits run from minus 2.8 to plus 12.8 millimeters of mercury. You judge the limits against what matters clinically. A difference of 5 to 10 millimeters of mercury in systolic pressure can change a diagnosis or a treatment decision, so limits of plus or minus 7.8 millimeters of mercury are wide for this measurement.

10:09 Standard error of measurement and smallest detectable change

Fifth, we will cover the standard error of measurement and the smallest detectable change. Researchers and clinicians often need to know how much one person's score must change before the change is real. The standard error of measurement is the typical size of the measurement error in one person's score, in the units of the measurement. From the differences, the standard error of measurement equals their standard deviation divided by the square root of 2. Four point zero divided by 1.41 gives about 2.8 millimeters of mercury. The smallest detectable change is the smallest change in one person's score that exceeds measurement error with 95 percent confidence, and it equals 1.96 times the square root of 2 times the standard error of measurement, about 7.8 millimeters of mercury. This value equals the half-width of the limits of agreement from the Bland-Altman plot, because both come from the same standard deviation of the differences. If a patient's systolic pressure falls by 5 millimeters of mercury between visits, the change lies within measurement error. A fall of 10 millimeters of mercury exceeds the smallest detectable change.

11:29 Parallel forms

Sixth, we will look at parallel forms and at troubleshooting. If memory or practice would distort a second round of the same test, you can give a different version of the test instead. Parallel forms are two versions of a test built to measure the same construct with equivalent content and difficulty. A construct is the attribute the test intends to measure, such as anxiety or reading ability. You give form A on the first occasion and form B on the second, and the agreement between the forms estimates reliability without repeating the same items. Building two equivalent forms is a lot of work, and any difference between the forms adds to the measured error.

12:15 Troubleshooting

When test-retest agreement is lower than you expected, the pattern of the differences usually points to the cause. If scores are higher or lower on the second occasion for most participants, look for a practice effect, fatigue or a change in the instrument or the procedure. Check the bias on the Bland-Altman plot and standardize the procedure. If the differences grow with the size of the measurement, the points on the plot form a funnel, and you can analyze the differences as percentages or on a log scale instead. The ICC depends on how much the people in the sample differ from each other, so in a sample of very similar people the ICC can be low even when the measurement errors are small. In that case, report the standard error of measurement as well. With eight participants the estimates are imprecise, and reliability studies commonly plan for at least 30 participants.

13:13 A grip-strength case study

Seventh, we will follow a grip-strength case study. A complete study shows you the design choices and the reported numbers together. We'll use the grip-strength study from the output table, which follows the plan from the troubleshooting section. The 30 adults each attend twice. You measure dominant-hand grip strength with a hand dynamometer, with the participant seated, the shoulder adducted and the elbow flexed at 90 degrees. You record the best of three trials. Two sessions follow, one week apart, with the same tester and the same time of day. Session one gives a mean of 28.7 kilograms and a standard deviation of 6.7. Session two gives a mean of 29.0 kilograms and a standard deviation of 7.0. The mean difference is 0.31 kilograms, and a paired t-test gives p = 0.26, so no systematic shift appears between sessions. The standard deviation of the differences is 1.47 kilograms, and the limits of agreement run from minus 2.6 to plus 3.2 kilograms. The ICC for two-way absolute agreement with a single measure is 0.976, with a 95 percent confidence interval from 0.952 to 0.989. The standard error of measurement is 1.47 divided by 1.414, which gives 1.04 kilograms. The smallest detectable change at 95 percent confidence is 1.96 times 1.414 times 1.04, and that product gives 2.9 kilograms, about 10 percent of the mean. In a paper you report the ICC with its interval, the standard error of measurement and the smallest detectable change, for example an ICC of 0.976 with a 95 percent confidence interval from 0.952 to 0.989, a standard error of measurement of 1.04 kilograms and a smallest detectable change of 2.9 kilograms.

15:27 Smallest detectable change and the MCID

In the final section, we'll compare the smallest detectable change with the MCID. The case study tells you what the instrument can detect, and a clinician also needs to know what size of change matters to the patient. The minimal clinically important difference is the smallest change that patients or clinicians regard as important. Published estimates for grip strength in adults lie between about 5.0 and 6.5 kilograms. In the grip-strength study the smallest detectable change is 2.9 kilograms, which falls below the lower estimate of 5.0 kilograms. A change of 5 kilograms, the size that patients notice, is bigger than the measurement error, so the test can track an individual. Now reverse the comparison, and the conclusion changes. If the smallest detectable change exceeded the minimal clinically important difference, a change that matters could hide inside measurement error. The test would then describe a group and could not follow one person over time.

16:36 Recap and quiz

Here is what you learned in this video. Test-retest reliability compares two measurements of the same people, and the retest interval sets what the comparison measures. A correlation shows whether the scores move together, and the absolute-agreement ICC also counts a shift between occasions as error. The Bland-Altman plot gives the bias and the limits of agreement in the units of the measurement, and the standard error of measurement gives the smallest detectable change for one person. The grip-strength case then showed how the design decides the ICC model and how a smallest detectable change of 2.9 kilograms compares with the minimal clinically important difference. Here is a question for you to test what you learned. A reaction-time test is given twice, one week apart, and each participant is about 20 milliseconds faster on the second test while the order of participants stays the same. Which statistic stays high, Pearson's r or the absolute-agreement ICC, and what would the Bland-Altman plot show? Pause the video and work it out. The answer is next. Pearson's r stays high because the order of participants does not change, while the absolute-agreement ICC falls because it counts the shift as error. The Bland-Altman plot shows a mean difference of about 20 milliseconds, a bias away from zero.

Read more

All videos