How to Calculate Inter-Rater Reliability (Video)

0:24 Why rater agreement matters

First, we will look at why rater agreement matters and what a rating is. When two pathologists grade the same set of biopsy slides, or two observers code the same video of animal behavior, their scores sometimes differ even though both work from the same material. Because both raters score the same subject, a difference between their scores cannot come from the subject, and the difference represents measurement error that the raters add. Two judges who give one dive different scores show the same problem, because the dive happened once and only the judges differ. This error weakens a study, because it adds noise to comparisons between groups, and it can bias an estimate when one rater scores one group and a different rater scores the second group.

1:26 What a rating is

To measure how much the raters agree, you first need to be clear about the kind of rating they produce. A rater is the person, or the instrument, that assigns a score or a category to a subject. Rater, observer, coder and judge all name the same role. A rating is the score or the category that one rater assigns to one subject, such as a slide, a patient or a video clip. Nominal ratings are categories with no order. Examples include a positive or negative result and a behavior type such as grooming or feeding. Ordinal ratings are ordered categories, such as tumor grade 1, 2 and 3. The gaps between the categories need not be equal, and a disagreement of one grade is smaller than a disagreement of two grades. Continuous ratings are numbers on a scale, such as a lesion measured in millimeters, or a pain score from 0 to 10.

2:20 Why percent agreement is not enough

Second, we will see why percent agreement is not enough and introduce Cohen's kappa. Before you choose a statistic for each type of rating, you need to see why the simplest statistic, percent agreement, gives too high an estimate of reliability. Percent agreement is the proportion of subjects on which the raters give the same rating. You count the subjects where the two ratings match and divide by the number of subjects. Raters also agree on some subjects by chance. Two people who each guess the outcome of a series of coin tosses match on about half of the tosses, although neither knows anything about the coin, and in the same way two raters who each call half the slides positive at random agree on about half of the slides. Chance agreement grows when one category is common. If both raters call about 90 percent of the slides positive, chance alone produces about 82 percent agreement. Percent agreement therefore overstates reliability, and the size of the overstatement depends on how common each category is.

3:27 Cohen's kappa

To correct percent agreement for chance, you can use Cohen's kappa, introduced in 1960, which applies to two raters who assign nominal categories. The formula uses two proportions. The observed agreement is the proportion of subjects on which the two raters give the same rating, and the expected agreement is the agreement that chance would produce, calculated from each rater's own category frequencies. Subtract the expected agreement from the observed agreement, and the result is the agreement beyond chance. Subtract the expected agreement from 1, and the result is the largest agreement beyond chance the two raters could reach. Kappa is the agreement beyond chance divided by the largest possible agreement beyond chance. Kappa equals 1 for perfect agreement. Kappa equals 0 when the raters agree no more often than chance allows, and a negative kappa means they agree less often than chance.

4:29 Worked example

Third, we will work through an example and interpret kappa values. To apply the formula, consider two pathologists who each classify the same 100 biopsy slides as positive or negative. The numbers in this example are illustrative. Forty slides come back positive from both pathologists, and 45 come back negative from both pathologists. Pathologist A calls 10 slides positive that B calls negative, and B calls 5 slides positive that A calls negative. The four cells add to 100. You start with the observed agreement. Add the two cells where the pathologists agree, 40 plus 45, which gives 85, and divide by 100, so the observed agreement is 0.85. You then need the totals for each rater. A called 50 of the 100 slides positive, so A's positive proportion is 0.50. A's negative proportion is also 0.50. B called 45 slides positive, so B's positive proportion is 0.45, and B's negative proportion is 0.55. By chance alone, the probability that both pathologists call a slide positive is 0.50 times 0.45, which is 0.225, and the probability that both call a slide negative is 0.50 times 0.55, which is 0.275. Add those two products, and the expected agreement is 0.50. You now have both terms of the formula. Subtract 0.50 from 0.85, which gives 0.35. Subtract 0.50 from 1, which gives 0.50. Divide 0.35 by 0.50, and kappa is 0.70. Read kappa as the share of the possible improvement over chance that the raters reached. These two pathologists reached 70 percent of that improvement, and their raw agreement was 85 percent.

6:37 Interpreting kappa

Many researchers interpret a kappa value with a set of labels proposed in 1977, in which a value below 0 is poor, 0 to 0.20 is slight, 0.21 to 0.40 is fair, 0.41 to 0.60 is moderate, 0.61 to 0.80 is substantial, and 0.81 to 1 is almost perfect. The authors of those labels described the divisions as arbitrary, so treat them as a convention. Judge a value against what the decision requires, because a diagnostic grade that decides treatment needs higher agreement than an exploratory behavior code. The worked example, at 0.70, falls in the substantial band. A kappa estimated from 100 slides carries uncertainty. An approximate 95 percent confidence interval for this value runs from about 0.56 to 0.84, which reaches from the moderate band into the almost perfect band.

7:43 The kappa paradox

Fourth, we will look at the kappa paradox and the responses to the paradox. Kappa also depends on how common each category is, and that dependence produces the kappa paradox. Take a second illustrative set of 100 slides. Both pathologists call 85 slides positive and 5 slides negative, and each calls 5 slides positive that the other calls negative. Observed agreement is 0.90, because the two pathologists agree on 90 of the 100 slides. Each pathologist called 90 of the 100 slides positive, so chance agreement is 0.9 times 0.9 plus 0.1 times 0.1, which is 0.82. Kappa is 0.90 minus 0.82, divided by 1 minus 0.82, which gives 0.08 divided by 0.18, or about 0.44. Raw agreement is higher here than in the first set, at 90 percent against 85 percent, and kappa is much lower, at 0.44 against 0.70. The drop comes from the high prevalence of the positive category, because chance agreement is already 0.82 and little room remains above chance. A weather forecaster in a desert, where rain falls on about 5 percent of days, faces a similar problem, because a forecast of no rain for each day is correct about 95 percent of the time without any skill, and a high rate of correct forecasts therefore says little when one outcome dominates. You therefore report the prevalence of each category and the full agreement table beside the kappa value, and you avoid comparing kappa across samples that differ in prevalence.

9:35 Responses to the kappa paradox

Because kappa depends on the prevalence of each category, researchers have proposed indices and statistics that make the paradox visible or reduce its effect. Two indices describe the agreement table directly. The prevalence index is the absolute difference between the cell where both raters say positive and the cell where both say negative, divided by the number of subjects. In the paradox table, 85 minus 5 gives 80, and dividing by 100 gives a prevalence index of 0.80. In the first example, the difference between 40 and 45 is 5, so the prevalence index is 0.05. The bias index is the absolute difference between the two discordant cells, divided by the number of subjects. The paradox table holds 5 in each discordant cell, so its bias index is 0. The first example holds 10 and 5, which gives 0.05. The prevalence-adjusted bias-adjusted kappa, or PABAK, equals twice the observed agreement minus 1. For the first example, twice 0.85 is 1.70, and subtracting 1 gives 0.70. For the paradox table, twice 0.90 is 1.80, and subtracting 1 gives 0.80. PABAK shows what kappa would be if the categories were balanced and the raters showed no bias. Report PABAK beside kappa, and give the agreement table so a reader can recompute the value. Gwet's AC1 replaces the chance term with one that stays small when a single category dominates. For two categories, the chance agreement is twice q times 1 minus q, where q is the average proportion of positive ratings across the two raters. In the paradox table, q is 0.90, so the chance agreement is 2 times 0.90 times 0.10, which is 0.18. AC1 is then 0.90 minus 0.18, divided by 1 minus 0.18, which is 0.72 divided by 0.82, or 0.88. On the first example, AC1 is 0.70. The paradox table therefore gives kappa 0.44, PABAK 0.80 and AC1 0.88, while the balanced first example gives about 0.70 on all three. State which statistic you report and why you chose that statistic.

12:13 Weighted kappa for ordinal ratings

Fifth, we will extend kappa to ordinal ratings and to three or more raters, and look at Krippendorff's alpha. Cohen's kappa treats each disagreement as equally serious, which suits nominal categories, and for ordered grades you use weighted kappa. Weighted kappa gives partial credit for a near miss. A disagreement between grade 1 and grade 2 costs less than a disagreement between grade 1 and grade 3. Linear weights reduce the credit in equal steps as the distance between the two grades grows, and quadratic weights reduce the credit with the square of that distance, so they penalize large disagreements more heavily than small disagreements. With quadratic weights, weighted kappa is closely related to the intraclass correlation. State which weights you used, because the value of weighted kappa changes with the weighting.

13:10 Fleiss' kappa for three or more raters

Cohen's kappa is limited to two raters, so a study that uses three or more raters needs a different statistic. Fleiss' kappa, introduced in 1971, measures chance-corrected agreement for nominal ratings when the same number of raters scores each subject. The raters do not have to be the same people across subjects. One subject can go to raters 1, 2 and 3, and the next subject to raters 3, 4 and 5. You read Fleiss' kappa on the same scale as Cohen's kappa, from below 0 to 1, with the same interpretation bands and the same caution about their limits.

13:49 Krippendorff's alpha

Fleiss' kappa requires the same number of ratings for each subject and nominal categories, and Krippendorff's alpha removes both restrictions. Krippendorff's alpha compares the disagreement you observe with the disagreement expected by chance, and subtracts their ratio from 1. It accepts any number of raters and tolerates missing ratings, and the distance between two ratings follows the level of measurement, whether nominal, ordinal, interval or ratio. For the first example, the observed term multiplies 199, which is one less than the 200 ratings, by the 15 slides on which the raters disagree, and the expected term multiplies the 95 positive ratings by the 105 negative ratings. Their ratio is 0.30, so alpha is 0.70. For the paradox table, alpha is 0.45, close to the kappa value of 0.44. Alpha uses the same scale as kappa, from below 0 to 1. A common convention sets 0.80 as the level for firm conclusions and 0.667 as the lowest value at which you draw tentative conclusions. Treat both figures as conventions, and judge them against the use the ratings will serve.

15:10 The intraclass correlation coefficient

Sixth, we will cover the intraclass correlation coefficient and how to choose the statistic. Kappa works with categories, so for continuous scores, such as a lesion size in millimeters, you use the intraclass correlation coefficient. The intraclass correlation coefficient is the proportion of the total variance in the scores that comes from real differences between subjects. The remaining variance comes from the raters and from error. A Pearson correlation is not enough on its own. Pearson's r measures whether two raters' scores rise and fall together, and a rater who always scores two points higher still gives a correlation near 1. The form of the ICC follows the design. In a one-way random-effects model, a different set of raters scores each subject. In a two-way model, the same raters score all of the subjects. Use random effects when your raters are a sample from a larger pool. Use mixed effects when these specific raters are the only ones of interest. The two-way model offers two definitions of agreement. Consistency asks whether the raters rank the subjects in the same order, and it ignores a constant offset between raters. Absolute agreement asks whether the raters give the same number to the same subject, and it counts the offset as error. A bathroom scale that always reads 2 kilograms too heavy shows the difference between the two definitions. The scale ranks people correctly by weight, so it is consistent, but its readings never match the true weight, so its absolute agreement is poor. A single-measure ICC describes the reliability of one rater. An average-measure ICC describes the mean of several raters, and it is higher for the same data. One widely used guideline rates values below 0.5 as poor, 0.5 to 0.75 as moderate, 0.75 to 0.9 as good, and values above 0.9 as excellent. Judge the ICC by its 95 percent confidence interval as well as by the point estimate.

17:22 Choosing the statistic

Each of these statistics fits a particular study design, so you choose among them by the type of rating and the number of raters. For nominal ratings, you use Cohen's kappa when two raters score the subjects and Fleiss' kappa when three or more raters score each subject, and with either statistic you report the agreement table beside the kappa value. Ordinal grades from two raters call for weighted kappa, and you state which weights you used. With continuous ratings, you use an intraclass correlation, and you state the model, whether you chose consistency or absolute agreement, and whether the value comes from a single measure or an average measure.

18:04 Reading software output

Seventh, we will read software output and troubleshoot low agreement. Once you have chosen a statistic, you usually compute it in software, and the output holds more numbers than the result you report. In SPSS, you run Crosstabs with the Kappa option. For the first example, the Measure of Agreement table prints a kappa value of 0.700, an asymptotic standard error of 0.071, an approximate T of 7.04, a significance below 0.001, and 100 valid cases. The approximate T tests kappa against 0, so it only tells you that agreement exceeds chance. The value and its standard error give the result you report. A 95 percent interval runs from about 0.56 to 0.84. In R, the kappa2 function in the irr package prints the same analysis as Kappa 0.7, z 7.04 and a p-value of 1.92 times 10 to the minus 12. An intraclass correlation output carries more rows. Two radiologists measured the diameter of ten patients' lesions in millimeters, and the two-way model prints a single-measure ICC for absolute agreement of 0.983, with a 95 percent interval from 0.937 to 0.996. The same table gives a single-measure consistency value of 0.985, an average-measure value of 0.992, and an F test with 9 and 9 degrees of freedom equal to 128.5, with p below 0.001. The F test only tells you that the ICC exceeds 0. When one radiologist will measure in practice, you report the single-measure absolute-agreement row with its interval.

20:00 Troubleshooting low agreement

When kappa or the ICC is low, you look for the source of the disagreement among the raters. Unclear category definitions produce disagreement. Write a codebook that gives a definition for each category, decision rules for borderline cases, and example images. Even with a clear codebook, raters apply the definitions differently until they have practiced on shared material. Before the study starts, you therefore have the raters score a calibration set with agreed reference ratings, discuss each disagreement, and repeat the process until agreement reaches the target. Rater drift is a change over time in how a rater applies the categories. To catch drift, you have the raters re-score calibration subjects at intervals during the study. You also have the raters score without seeing each other's ratings, and ideally without knowing the group of each subject. When you report, give the statistic and its form, the number of subjects, the number of raters, and a 95 percent confidence interval, because a point value from a small sample can mislead a reader.

21:11 A case study with two radiologists

The last part follows a case study with two radiologists. The troubleshooting steps and the reporting rules from the last section apply together in a study of two radiologists who want to know whether they can grade wrist radiographs for fracture with enough agreement to use the grades in a trial. They write a codebook that defines a fracture and gives example images for hairline fractures. They grade a calibration set of 20 training images together and settle the disagreements. Each radiologist then grades the same 60 radiographs independently, with the clinical details hidden. The table has 22 radiographs that both call fracture and 30 that both call no fracture. Radiologist A calls fracture where B does not in 5 cases, and B calls fracture where A does not in 3. Observed agreement is 52 divided by 60, or 0.867. A calls 27 fractures, a proportion of 0.45, and B calls 25, a proportion of 0.417. Expected agreement is 0.508, so kappa is 0.867 minus 0.508, divided by 1 minus 0.508, which is 0.73. The 95 percent interval runs from 0.55 to 0.90. PABAK is twice 0.867 minus 1, or 0.73. The prevalence index is 0.13 and the bias index is 0.03. On the usual labels, 0.73 counts as substantial agreement. The interval reaches down to 0.55, which lies in the moderate band, so the report gives the interval beside the point value. The reported sentence reads: two radiologists independently graded 60 radiographs; observed agreement was 86.7 percent and Cohen's kappa was 0.73, with a 95 percent confidence interval from 0.55 to 0.90. The 8 disagreements then go to a consensus review, and the team revises the codebook entry for hairline fractures.

23:30 Recap and quiz

Here is a summary of what you learned. This lesson began with the measurement error that disagreement between raters adds to a study, then described the three types of rating and showed why percent agreement overstates reliability. Cohen's kappa corrects the observed agreement for the agreement expected by chance, and in the worked two-by-two example you calculated a kappa of 0.70 and placed it in the substantial band, before the second table showed how skewed prevalence lowers kappa despite higher raw agreement. Weighted kappa extends kappa to ordered grades, and Fleiss' kappa extends it to three or more raters. For continuous scores, you use the intraclass correlation and state its model and its type. When the paradox makes kappa hard to read, you report PABAK or Gwet's AC1 beside kappa, and Krippendorff's alpha handles missing ratings and any level of measurement. You then matched each type of rating and number of raters to a statistic, read the values that matter in software output, and followed the radiology case from the codebook to a reported kappa with its confidence interval. Now try a question on what you learned. Two raters score pain on a 0 to 10 scale, and rater B always gives exactly two points more than rater A. Which form of the intraclass correlation reports lower reliability for these raters, consistency or absolute agreement? Pause here and try it before the answer appears. Absolute agreement reports lower reliability, because it counts the constant two-point difference as disagreement, while consistency ignores a constant shift and stays close to 1.