Measuring Agreement in Veterinary Diagnostic Tests

By Dr. Zubair Khalid, DVM, MS, PhD · 2026-08-11

Key Takeaways

• Agreement statistics, such as kappa for categorical data and Bland-Altman analysis for continuous data, are distinct from diagnostic accuracy and assess the interchangeability of measurements, repeatability, or reproducibility, not correctness against a true disease state.
• Kappa statistics are essential for categorical outcomes (e.g., positive/negative tests, lesion grades) and correct for chance agreement; Cohen's kappa is for two observers, Fleiss' kappa for three or more, and weighted kappa is used for ordered categories to account for the severity of disagreement.
• Bland-Altman analysis is the standard for continuous measurements (e.g., serum biochemistry, cell counts) and quantifies systematic bias and the limits of agreement, which represent the range within which 95% of differences between two methods are expected to fall.
• Repeatability assesses variation under identical conditions by the same observer, while reproducibility considers variation with changing conditions like different observers, instruments, or laboratories, with metrics like intraclass correlation coefficient and standard error of measurement being relevant.
• Reporting agreement studies requires transparency regarding study population, sampling, observer training, blinding, and the inclusion of confidence intervals for agreement metrics to reflect precision, with sample size calculations informed by expected confidence interval width.
• Common errors include conflating agreement with accuracy, misinterpreting kappa as correctness, using unweighted kappa for ordinal data where category distance matters, and applying Bland-Altman analysis to data with proportional error without appropriate transformation.

Veterinary researchers frequently need to determine whether two diagnostic methods produce interchangeable results, whether repeated measurements on the same subject are stable, and whether different observers reading the same image or sample reach the same conclusion. These questions concern agreement, a family of statistical concepts distinct from diagnostic accuracy. Accuracy asks how close a test comes to a true disease state. Agreement asks how close two measurements, two observers, or two instruments come to each other. A test can be highly repeatable yet inaccurate, and two tests can agree closely while both misclassify patients. This article provides a procedural reference for designing, analyzing, and reporting agreement studies in veterinary medicine, with emphasis on kappa-type statistics for categorical data and Bland-Altman analysis for continuous data.

The intended reader is a veterinary researcher or graduate student planning a method-comparison study, a clinician evaluating whether a new point-of-care assay can replace a laboratory reference, or a reviewer appraising the statistical soundness of a diagnostic manuscript. The methods described apply across species, from companion animal hematology to production animal serology to wildlife disease surveillance. The scope excludes sensitivity, specificity, predictive values, and likelihood ratios, which are covered in companion articles on diagnostic test accuracy. The focus here is strictly on repeatability, reproducibility, and interchangeability of measurements.

Agreement studies answer a practical clinical question: can the results of one test be used in place of another without changing clinical decisions? When a commercial ELISA is compared with an in-house assay, when two observers grade radiographic findings, or when a point-of-care device is checked against a laboratory analyzer, the statistical question is not whether one is "better" but whether they can be used interchangeably. The choice of statistical method depends on the measurement scale, the number of observers or instruments, the study design, and whether the research question concerns absolute agreement or consistency.

At a Glance

The Conceptual Distinction Between Agreement and Accuracy

Agreement and accuracy address different scientific questions and require different study designs. Accuracy studies compare a test result against a reference standard that defines the true disease state. Agreement studies compare two or more measurements without reference to any external truth. The distinction matters clinically. Two serologic tests for paratuberculosis may show high agreement with each other yet both fail to detect infected cattle in early disease, because the tests share the same biological limitation. Conversely, a highly accurate test may show poor agreement with a less accurate test, and the researcher must decide which measurement the clinical workflow should follow.

The consequences of conflating agreement with accuracy are substantial. A method-comparison study that reports a high kappa coefficient between a new test and an established test may be interpreted as validation, but the kappa only demonstrates that the tests classify the same animals similarly. It does not establish that either test correctly identifies the target condition. The consensus recommendations on diagnostic testing for paratuberculosis in cattle explicitly acknowledge this limitation, noting the paucity of large-scale studies that compare multiple tests on samples from the same animals and the need for accuracy data to complement agreement data.

Measurement Error as the Foundation of Agreement

All measurements contain error. The total variation observed in a set of diagnostic results can be partitioned into true biological variation between subjects and measurement error arising from the instrument, the observer, the sample, and the conditions of measurement. Agreement statistics quantify the contribution of measurement error relative to total variation. When measurement error is small relative to between-subject variation, agreement appears high. When measurement error dominates, agreement is poor even if the underlying test is biologically sound.

Measurement error has several components. Repeatability refers to variation when the same observer measures the same subject with the same instrument under identical conditions within a short time. Reproducibility refers to variation when conditions change, such as different observers, different laboratories, different instruments, or different days. A test can be highly repeatable but poorly reproducible if observer interpretation varies. The comparison of cone beam computed tomography systems versus panoramic imaging for impacted maxillary canines illustrates this distinction in a veterinary-relevant dental imaging context, where the investigators evaluated agreement between observers using the standard error of measurement, kappa statistics, and the coefficient of concordance, recognizing that observer variation is a distinct source of error separate from the imaging modality itself.

Categorical Agreement: The Kappa Family

For categorical outcomes, such as positive or negative test results, or ordinal grades of lesion severity, the kappa statistic is the standard measure of agreement. Kappa corrects for the agreement expected by chance alone, which is essential because two observers classifying animals into two categories will agree on a proportion of cases purely by random allocation. The kappa coefficient ranges from negative values, indicating agreement worse than chance, through zero, indicating chance-level agreement, to a maximum of 1.0, indicating perfect agreement.

The interpretation of kappa values follows published benchmarks, but the researcher must recognize that kappa is influenced by the prevalence of the condition in the study population and by the number of categories. A low kappa can occur when the condition is very rare or very common, even when the observers agree on nearly all cases. This paradox arises because chance agreement is high when one category dominates. Reporting the raw proportion of observed agreement alongside kappa is therefore essential for interpretation.

The choice of kappa variant depends on the design. Cohen's kappa applies to two observers. Fleiss' kappa extends the calculation to three or more observers. Weighted kappa is used for ordinal categories, where disagreements between adjacent categories are less serious than disagreements between distant categories. The weights must be specified a priori and justified, because different weighting schemes produce different kappa values. Linear weights penalise disagreements in proportion to the distance between categories, while quadratic weights penalise larger disagreements more heavily.

The serologic test evaluation in Eurasian wild boar provides a practical example of kappa use in veterinary diagnostics. The investigators compared a point-of-care dual-path platform test with a traditional ELISA for detecting antibodies against Mycobacterium bovis, reporting a kappa agreement of 0.80 between the two formats. This value indicates substantial agreement, but the study also reported sensitivity and specificity separately, recognizing that agreement alone does not establish diagnostic performance. The ELISA comparison for Fasciola hepatica similarly reported a kappa of 0.82 between an in-house ELISA and a commercial kit, described as almost perfect agreement, while separately reporting the diagnostic sensitivity and specificity of the new assay.

Continuous Agreement: Bland-Altman Analysis

When diagnostic tests produce continuous measurements, such as serum biochemistry values, cell counts, or imaging measurements, the Bland-Altman method is the preferred approach. The method plots the difference between two measurements against their mean. The mean difference estimates systematic bias between the methods. The limits of agreement, calculated as the mean difference plus or minus 1.96 times the standard deviation of the differences, define the range within which 95% of the differences between the two methods are expected to fall.

The clinical question for Bland-Altman analysis is whether the limits of agreement are narrow enough to be clinically acceptable. This judgment requires a pre-specified acceptable difference, which should be based on clinical reasoning about how much disagreement can be tolerated before patient management would change. The limits of agreement are statistical estimates, and confidence intervals around them should be reported to reflect sampling uncertainty. The method assumes that the differences are approximately normally distributed and that the magnitude of the difference does not vary with the magnitude of the measurement. When proportional bias is present, the data may require logarithmic transformation or a regression-based modification of the Bland-Altman approach.

The Cercopithifilaria sp. survey in dogs illustrates a low-agreement scenario in a parasitological context. The investigators compared microscopical examination of skin sediments with PCR on skin samples for detecting dermal microfilariae, recording a low level of agreement between the two tests at sites where samples were processed in parallel. The discrepancy likely reflects fundamental differences in what each method detects, with microscopy identifying intact microfilariae and PCR detecting parasite DNA that may persist even when organizms are not visible. This example underscores that agreement statistics quantify interchangeability, and when agreement is poor, the researcher must investigate whether the methods measure different biological entities instead of assuming one method is simply erroneous.

Choosing the Metric: A Decision Framework

The selection of an agreement metric is not a matter of preference. It follows directly from the measurement scale, the number of observers or instruments, the study objective, and the consequences of disagreement. Table 1 summarizes the primary decision points.

The kappa statistic is appropriate when both tests operate on the same categorical scale and when the clinical decision is categorical. In a comparison of an in-house ELISA with a commercial kit for bovine fasciolosis, a kappa of 0.82 was reported, indicating almost perfect agreement beyond chance Salimi-Bejestani et al., 2005. The same study illustrates a common pattern: the kappa value is reported alongside diagnostic sensitivity and specificity, but the agreement statistic answers a different question. The kappa describes whether the two tests classify the same animals into the same categories. It does not state which test is correct.

When the measurement is continuous, Bland-Altman analysis is preferred. The mean difference between methods estimates systematic bias. The limits of agreement, calculated as the mean difference plus or minus 1.96 times the standard deviation of the differences, define the range within which 95% of future differences are expected to fall. Clinical acceptability of those limits is a professional judgment, not a statistical one. A veterinary clinician must decide whether a difference of, for example, 0.5 cm in radiographic measurement of canine crown width is tolerable for the intended clinical purpose. In a study comparing panoramic radiography with two cone beam computed tomography systems for localizing impacted maxillary canines, the authors reported highly significant differences between 2D and 3D imaging in crown width and angulation Alqerban et al., 2011. The statistical significance of the difference does not establish clinical relevance. The limits of agreement do.

Repeatability and Reproducibility in Practice

Repeatability refers to the variation observed when the same measurement is performed repeatedly under identical conditions. Reproducibility refers to variation when conditions change, such as different observers, different laboratories, or different instruments. The distinction matters because a test that is repeatable in one laboratory may not be reproducible across laboratories.

For categorical tests, observer agreement is commonly quantified with kappa statistics. In the canine impaction study, agreement between observers was evaluated using the standard error of measurement, kappa statistics, and the coefficient of concordance Alqerban et al., 2011. The use of multiple agreement indices reflects a practical reality: no single statistic captures all relevant aspects of observer performance. The standard error of measurement quantifies the magnitude of observer-related error in the original measurement units. The kappa statistic captures chance-corrected agreement on the categorical scale. The coefficient of concordance addresses the joint distribution of observer pairs.

For continuous measurements, the repeatability coefficient is defined as 1.96 times the standard deviation of the differences between repeated measurements. It expresses the maximum difference expected between two measurements on the same subject 95% of the time. The coefficient of variation is useful when the standard deviation scales with the mean, but it becomes unstable when mean values approach zero.

Worked Example: Comparing Two Serologic Tests in Wild Boar

Consider a researcher evaluating a point-of-care test for bovine tuberculosis in Eurasian wild boar against a laboratory ELISA. The study design involves testing the same 200 serum samples with both assays. The data are binary: positive or negative.

The first step is to construct the 2 by 2 contingency table of test results. The second step is to calculate the observed proportion of agreement, which is the sum of the concordant cells divided by the total. The third step is to calculate the expected proportion of agreement under independence, derived from the marginal totals. Cohen's kappa is then computed as the ratio of observed agreement beyond chance to the maximum possible agreement beyond chance.

In a study of serologic tests for mycobacterial exposure in wild boar, both the dual-path platform test and the bPPD ELISA achieved a kappa agreement of 0.80 Boadella et al., 2011. The interpretation of this value as substantial agreement is standard, but the researcher must also examine the marginal distributions. The bPPD ELISA had a specificity of 100% and a sensitivity of 79.2%, while the dual-path platform had a sensitivity of 89.6% and a specificity of 90.4%. The kappa value of 0.80 for both comparisons conceals different patterns of disagreement. The bPPD ELISA misses more true positives. The dual-path platform produces more false positives. A researcher selecting between these tests must weigh the relative costs of false negatives and false positives in the specific surveillance context.

The kappa statistic is sensitive to the prevalence of the condition in the study population. When prevalence is very low or very high, kappa can be low even when the observed agreement is high. This is a mathematical property of the statistic, not a defect in the tests. Reporting PABAK alongside kappa provides a more complete picture when marginal distributions are asymmetric.

Species and Context Modifications

The correct choice of agreement metric and the interpretation of its value depend on the species, the production system, and the purpose of testing. In cattle herds being screened for paratuberculosis, the consequences of misclassification differ between a herd with no history of disease and a herd enrolled in a test-negative program. Consensus recommendations for paratuberculosis testing in US cattle emphasize that testing strategies must be adapted to herd-level goals and that no single test is appropriate for all situations Collins et al., 2006. The same principle applies to agreement studies. A kappa of 0.70 may be acceptable for a screening test in a low-prevalence population but unacceptable for a confirmatory test in a high-prevalence population.

In wildlife studies, sample collection conditions impose practical constraints. Skin samples from wild boar may be degraded, and the performance of PCR and microscopy can diverge. In a study of Cercopithifilaria sp. in dogs, a low level of agreement between microscopical examination and PCR was recorded in sites where samples were processed in parallel Otranto et al., 2012. The two tests detect different analytes: microscopy detects dermal microfilariae in skin sediments, while PCR detects parasite DNA. Disagreement between the tests may reflect true biological variation, such as intermittent microfilaraemia, or technical factors, such as DNA degradation. The researcher must decide whether the disagreement represents measurement error or a genuine difference in what the tests measure.

Reporting Agreement Studies

Transparent reporting of agreement studies requires specification of the study population, the sampling method, the number of observers, the observer training, the order of measurements, and the blinding procedures. The ARRIVE guidelines specify the minimum information required for transparent and reproducible animal research publications ARRIVE guidelines. The EQUATOR Network maintains a library of reporting guidelines that includes standards for diagnostic accuracy studies EQUATOR Network. While these guidelines are not specific to agreement studies, their principles apply: the reader must be able to judge whether the study design introduced bias and whether the results generalize to other populations.

The confidence interval around a kappa or a limit of agreement should always be reported. A kappa of 0.80 based on 30 subjects has a wide confidence interval and provides little assurance of true agreement. A kappa of 0.80 based on 300 subjects is more informative. The sample size calculation for an agreement study should be based on the expected width of the confidence interval, not on the expected kappa value alone.

Common Failure Modes and Early Detection

Agreement analyzes fail in predictable ways. The most frequent failure is applying a kappa coefficient to data where the underlying assumption of independent observations is violated. Repeated measurements from the same animal, clustered samples from one herd, or serial readings from a single laboratory technician inflate agreement estimates because the observations are not conditionally independent. Early detection requires scrutiny of the sampling unit before analysis. If the study design collected multiple samples per animal, the analysis must account for clustering, often through generalized estimating equations or mixed-effects models that estimate agreement within and between clusters separately.

A second failure mode is the use of unweighted kappa for ordinal scales where the clinical consequence of disagreement varies by category. A one-category discrepancy on a five-point lesion severity scale is not equivalent to a four-category discrepancy, yet unweighted kappa treats both as simple disagreement. Quadratic weighting, which penalises disagreements proportionally to the square of their distance, is preferred when the ordinal scale reflects an underlying continuum. Detection of this error is straightforward: if the scale is ordinal and the analysis reports only unweighted kappa, the result is likely conservative or misleading depending on the distribution of marginal totals.

A third failure involves Bland-Altman analysis applied to data with proportional error. When measurement variability scales with the magnitude of the measurement, the limits of agreement widen as the mean increases, and the conventional fixed limits become meaningless. The discriminating check is a plot of absolute differences against the mean. If the spread of differences fans out with increasing mean, log-transformation or a ratio-based analysis is required.

The table below summarizes these and related failure modes.

Common Errors in Application

Less experienced analysts frequently misinterpret kappa as a measure of correctness. Kappa quantifies agreement beyond chance, not diagnostic accuracy. A test with high kappa against a reference standard can still misclassify disease if both tests share the same systematic error. The corrective action is to remember that agreement studies answer a different question from accuracy studies, and the two designs cannot substitute for one another.

A second recurring error is the selection of an inappropriate reference standard. When two imperfect tests are compared, neither can serve as a gold standard, and the kappa statistic merely reflects their mutual concordance. This was demonstrated in a survey of canine filarioid infestation where microscopy of skin sediments and PCR on skin samples showed low agreement, with the authors attributing the discrepancy to the different biological compartments sampled and the different stages of the parasite detected by each method. The lesson is that disagreement between tests may reflect genuine biological differences instead of measurement error, and the analysis should be interpreted accordingly.

Students also err by reporting only the kappa point estimate without confidence intervals. The precision of kappa depends heavily on sample size and the prevalence of the trait. A kappa of 0.80 based on 30 animals is not comparable to a kappa of 0.80 based on 300. Reporting intervals and the sample size that produced them is mandatory for interpretable results.

Limitations of the Current Evidence

The veterinary literature on agreement is uneven. Many published studies report kappa or Bland-Altman statistics as secondary outcomes without prespecifying hypotheses, sample size calculations, or acceptable agreement thresholds. This is particularly evident in serologic test comparisons, where studies often evaluate a new in-house assay against a commercial kit using convenience samples. One such evaluation of a Fasciola hepatica ELISA reported a kappa of 0.82 against a commercial test, but the subset of sera used for comparison was small and selected from a larger bank, raising questions about spectrum bias.

Expert opinion still differs on acceptable thresholds for kappa. The widely cited Landis and Koch benchmarks are descriptive labels, not clinical standards. A kappa of 0.60 may be entirely acceptable for a screening test in a low-prevalence population but unacceptable for a confirmatory test in a high-stakes trade setting. The consensus recommendations for paratuberculosis testing in cattle explicitly acknowledge that testing recommendations must balance accuracy, cost, and practical constraints, and that no single testing strategy suits all herds.

Evidence is also limited for agreement metrics in wildlife and exotic species, where sample sizes are small and repeated sampling is often impossible. The wild boar serologic study cited earlier achieved a kappa of 0.80 between two tests, but the authors noted that sex-related differences in sensitivity could not be fully explored because of sample size constraints. In such settings, the reporting standards for animal research emphasize transparency about the number of observations and the handling of missing data, but they do not resolve the underlying statistical limitations.

Escalation and Consultation

Referral to a veterinary biostatistician or epidemiologist is warranted when the analysis involves clustered data, complex weighting schemes, or Bayesian approaches to agreement. Laboratory involvement is indicated when reproducibility across sites is poor, because the source of variation may be reagent lot, equipment calibration, or technician technique instead of the test itself. Regulatory reporting obligations arise when agreement failures affect notifiable disease surveillance or trade certification. The WOAH terrestrial animal health standards specify validation requirements for tests used in international trade, including minimum reproducibility standards, and veterinarians certifying animals for export must ensure that the tests used meet these requirements. When in doubt about whether a disagreement constitutes a reportable event, consultation with the relevant national veterinary authority is appropriate before any public disclosure.

Frequently Asked Questions

How much disagreement between two tests is acceptable before I abandon one of them?

There is no universal threshold. The acceptable level depends on the clinical consequence of misclassification. For a screening test where false negatives delay treatment, even a kappa of 0.80 may be insufficient. For a confirmatory test where false positives trigger culling decisions, you may demand near-perfect agreement. Evaluate disagreement in the context of the decision it informs. Calculate the proportion of discordant pairs and examine whether disagreement clusters near the diagnostic threshold. If discordant results fall in the equivocal zone, the tests may be clinically interchangeable even with moderate kappa values. If disagreement occurs across the full range of values, the tests are measuring different biological phenomena.

What can I do when the reference method or a second test is too expensive for routine use?

Use a two-stage testing strategy. Apply the inexpensive test to the full population, then run the more costly test only on samples that fall near the diagnostic threshold or on a random subset for periodic quality assurance. This preserves the diagnostic information of the expensive test while containing cost. When validating a new low-cost test against an expensive reference, you can use a paired sample design with a smaller number of animals than a full validation study, provided you report the precision of your agreement estimates. The consensus recommendations for paratuberculosis testing in cattle illustrate how practical testing strategies balance cost against diagnostic performance in production settings.

How do I assess agreement when my measurements are ordinal scores instead of continuous values?

Weighted kappa is the appropriate metric for ordinal scales such as lesion severity grades or body condition scores. Choose quadratic weights when the distance between categories carries clinical meaning, and linear weights when all misclassifications are equally serious. Report both unweighted and weighted kappa so readers can judge the effect of the weighting scheme. For ordinal data with few categories, also report the percentage of exact agreement and the percentage of agreement within one category. A high weighted kappa with low exact agreement indicates systematic drift, often caused by one observer consistently scoring one category higher. This pattern is detectable by examining the disagreement matrix for asymmetry.

My practice uses point-of-care tests in the field. How do I assess agreement under field conditions?

Field conditions introduce variability that laboratory validation does not capture. Temperature, operator skill, sample quality, and time between collection and testing all affect performance. Conduct an agreement study using field-collected samples instead of banked specimens. Have multiple operators run the same samples and record environmental conditions. The serologic test evaluation in wild boar demonstrates that field-deployable formats can achieve agreement comparable to laboratory assays when properly validated. Include a subset of samples tested under ideal laboratory conditions to quantify the field effect. Repeat the field assessment periodically, particularly after reagent lot changes or operator turnover. Document all protocol deviations, as these often explain apparent disagreement between test formats.

How should I document agreement data in the medical record for an individual patient?

Record the test name, manufacturer, lot number, and the specific result for each test performed. When two tests are run on the same patient, record both results and note any discordance explicitly. If you acted on one result over the other, state the clinical rationale. This documentation supports future interpretation if the patient is retested or referred. For chronic disease monitoring, record the test platform used at each visit, because changing platforms can produce apparent changes in disease status that reflect method differences instead of true clinical change. The MSD Veterinary Manual advises that laboratory results be interpreted in light of the specific assay used, as method-related variation can mislead clinical decision-making.

How do I explain discordant test results to a client without undermining their confidence in veterinary medicine?

Frame discordance as an expected feature of diagnostic testing, not a failure. Explain that no test is perfect and that different tests measure different aspects of the same condition. Use a concrete analogy, such as two different blood pressure cuffs giving slightly different readings. Describe what the discordance means for the specific case and what additional information would resolve the uncertainty. If one test is more accurate for the patient's stage of disease, say so directly. Provide the client with a clear plan: repeat testing, additional diagnostics, or empirical treatment with scheduled reassessment. The AVMA practice resources emphasize transparent communication about diagnostic limitations as part of informed consent and shared decision-making.

Related Clinical & Scientific Guides

• Conducting Systematic Reviews of Veterinary Diagnostic Test Accuracy
• Bias in Veterinary Research: Types, Sources, and Mitigation
• Cluster Randomized Trials in Veterinary Research: Design and Analysis

References and Further Reading

• Comparison of two cone beam computed tomographic systems versus panoramic imaging for localization of impacted maxillary canines and detection of root resorption.. 2011.
• Consensus recommendations on diagnostic testing for the detection of paratuberculosis in cattle in the United States.. 2006.
• On a Cercopithifilaria sp. transmitted by Rhipicephalus sanguineus: a neglected, but widespread filarioid of dogs.. 2012.
• Development of an antibody-detection ELISA for Fasciola hepatica and its evaluation against a commercially available test.. 2005.
• Serologic tests for detecting antibodies against Mycobacterium bovis and Mycobacterium avium subspecies paratuberculosis in Eurasian wild boar (Sus scrofa scrofa).. 2011.
• Consensus guidelines for the identification and treatment of biofilms in chronic nonhealing wounds.. 2017.
• ARRIVE Guidelines 2.0 for Reporting Animal Research. PLOS Biology, 2020.
• EQUATOR Network Reporting Guidelines. EQUATOR Network.
• MSD Veterinary Manual, Professional Edition. MSD Veterinary Manual.

Related Articles

• Meta-Analysis of Veterinary Diagnostic Test Accuracy
• Appraising Diagnostic Accuracy Studies in Veterinary Medicine
• Diagnostic Test Accuracy Studies in Veterinary Medicine: Design and Reporting
• Conducting Systematic Reviews of Veterinary Diagnostic Test Accuracy
• Applying Competing Risks Analysis in Veterinary Research

This article is educational professional reference material for veterinary audiences. It is not a substitute for veterinary diagnosis, individual clinical judgment, current product labeling, or applicable regulatory requirements.