Intercoder Reliability in Research: A Practical Guide

Intercoder reliability measures the extent to which independent coders assign the same codes to the same qualitative data. When two or more researchers analyze interview transcripts, open-ended survey responses, video recordings, or documentary materials, the consistency of their coding decisions determines whether the resulting analysis can be trusted. This article explains what intercoder reliability means, how to calculate common coefficients such as Cohen's kappa and Krippendorff's alpha, and how to report reliability results in a way that supports the credibility of qualitative research.

The intended readers are students, researchers, life-science professionals, and informed general readers who need to design coding studies, assess published qualitative research, or document the trustworthiness of their own analyses. The practical outcome is a step-by-step calculation guide with worked examples and a reporting checklist that can be applied across disciplines.

What Intercoder Reliability Measures

Intercoder reliability quantifies the degree of agreement between two or more coders who independently apply a coding scheme to the same set of data. The concept rests on a simple premise: if a coding scheme is clear and the data are interpretable, different coders should arrive at similar judgments. When coders disagree frequently, the coding scheme may be ambiguous, the coders may lack sufficient training, or the data themselves may resist consistent interpretation.

The term appears across research traditions. In qualitative content analysis, high intercoder reliability is required for assuring quality when more than one coder is involved in data analysis. A study of patients' perspectives on low back pain demonstrated how intercoder reliability assessment can be used to improve codings in qualitative content analysis. The researchers first developed a coding scheme using a comprehensive inductive and deductive approach, then had two researchers independently code 10 transcripts, and calculated intercoder reliability. The resulting kappa value of .67 was regarded as satisfactory to solid. More importantly, varying agreement rates helped to identify problems in the coding scheme. Low agreement rates indicated that respective codes were defined too broadly and would need clarification. The researchers used these results to improve the coding scheme, leading to consistent and high-quality results [9].

Intercoder reliability differs from related concepts that researchers sometimes confuse. Internal consistency reliability concerns whether multiple items on a survey instrument measure the same underlying construct. Convergent validity concerns whether different measures of the same construct produce similar results. Intercoder reliability specifically addresses the consistency of human judgment in the coding process.

Why Intercoder Reliability Matters

The credibility of qualitative research depends on the transparency of the analytical process. When a single researcher codes all data, readers cannot distinguish between patterns that exist in the data and patterns that the researcher imposed on the data. Multiple coders provide a check on this problem, but only if their agreement is measured and reported.

Intercoder reliability serves several distinct purposes. First, it identifies weaknesses in the coding scheme before full-scale analysis begins. The low back pain study showed that low agreement rates on specific codes signaled definitions that were too broad and required clarification [9]. Second, it provides evidence that the findings are reproducible. Third, it allows researchers to calculate the precision of their measurements, which matters when coding results feed into quantitative comparisons.

The consequences of ignoring intercoder reliability can be seen in applied settings. A study of cause-of-death coding in The Netherlands performed a double coding study in which death certificates from May 2005 were coded again in 2007, with each certificate coded manually by four coders. The intercoder agreement of four coders on the underlying cause of death was 78 percent. Agreement was associated with the specificity of the ICD-10 code, the age of the deceased, the number of coders, and the number of diseases reported on the death certificate. Reliability was high, above 90 percent, for major causes of death such as cancers and acute myocardial infarction. For chronic diseases such as diabetes and renal insufficiency, reliability was low, below 70 percent. The researchers concluded that a statistical office should provide coders with additional rules for coding diseases with low reliability and evaluate these rules regularly [12].

This example illustrates a broader principle. Intercoder reliability is not an abstract statistical exercise. It has direct consequences for the quality of data that inform policy decisions, clinical practice, and public health monitoring.

At a Glance: Choosing an Intercoder Reliability Coefficient

The choice of coefficient depends on the number of coders, the level of measurement, and whether the data are nominal, ordinal, or continuous. The table below summarizes common options.

Percent agreement is the simplest measure but has a known weakness. It does not account for the possibility that coders agree by chance. Cohen's kappa addresses this limitation for two coders working with nominal data. Krippendorff's alpha extends the logic to multiple coders and different levels of measurement. The Iota coefficient, a statistical coefficient similar to kappa that is corrected for chance, has been used in studies of Rorschach Comprehensive System protocol-level variables [8].

Core Principles of Intercoder Reliability

Agreement Is Not the Same as Accuracy

Two coders can agree with each other and both be wrong. Intercoder reliability measures consistency between coders, not the correspondence between codes and some external standard. In the dietary assessment study using the eButton, a multisensor device worn on the chest that uses a camera to passively capture images of everything in front of the child throughout the day, two dietitians independently manually reviewed images to identify eating events, foods, and portion sizes. The dietitians agreed on the identity of 60.5 percent of the 1,026 foods but disagreed on 28.6 percent of the foods and on the names for 10.8 percent of the foods. After verification interviews with child-parent dyads, the dietitians agreed with the dyads on the identity of 77.0 percent of the 921 foods [6]. This study shows that coder agreement and accuracy against an external reference are separate questions.

Reliability Is a Property of the Coding Process

Reliability coefficients describe the performance of a specific coding scheme applied by specific coders to specific data. A coding scheme that produces high reliability in one study may produce low reliability in another study with different data or different coders. Researchers should report reliability results in the context of their own data instead of assuming that published reliability coefficients transfer to new settings.

Low Reliability Signals a Problem to Fix

Low intercoder reliability is also a statistical inconvenience. It indicates that the coding scheme needs revision or that coders need additional training. The low back pain study demonstrated this principle directly. The researchers used the results of their reliability assessment to improve the coding scheme, leading to consistent and high-quality results [9].

Common Coefficients Explained

Percent Agreement

Percent agreement is calculated by dividing the number of coding decisions on which coders agree by the total number of coding decisions. It is intuitive and easy to compute. However, it does not correct for chance agreement. If two coders each assign codes randomly, they will agree on some proportion of cases by chance alone. Percent agreement will overstate the true reliability of the coding process.

Cohen's Kappa

Cohen's kappa corrects for chance agreement when two coders assign nominal codes. The coefficient ranges from less than zero to 1.00, with 1.00 indicating perfect agreement and zero indicating agreement no better than chance. A study of the mapping between pharmaceutical dose forms in the German medication plan and EDQM standard terms used Cohen's kappa to calculate the level of agreement between coders. The results showed that less than half of the dose forms could be coded with EDQM standard terms, and kappa was found to be moderate, which the researchers described as rather unconvincing agreement among coders [7].

The interpretation of kappa values depends on the research context. In the low back pain study, a kappa value of .67 was regarded as satisfactory to solid [9]. In a qualitative analysis of narrative comments from emergency medicine attending physicians to residents, double coding of 20 percent of the comments at random was used to assess intercoder reliability, with a kappa of 0.84 [13].

Krippendorff's Alpha

Krippendorff's alpha is a flexible coefficient that can be used with any number of coders and with nominal, ordinal, interval, or ratio data. It also handles missing data. A qualitative analysis of telephonic comprehensive medication reviews calculated intercoder reliability using Krippendorff's alpha and reported reliability above 95 percent [10]. A study of portrayals of depression on TikTok assessed intercoder reliability using Krippendorff's alpha along with percent agreement [16]. A protocol for validating the Psychodynamic Organizational Diagnostic Instrument SyMOA specifies that Krippendorff's alpha will be used to determine intercoder reliability [15].

Iota Coefficient

The Iota coefficient is a statistical coefficient similar to kappa that is corrected for chance. It has been used in studies of Rorschach Comprehensive System protocol-level variables. In an international sample of 489 Rorschach protocols, Iota values for the variables analyzed ranged from .31 to 1.00, with 2 in the poor range of agreement, 4 in the fair range, 25 in the good range, and 116 in the excellent range of agreement [8].

Practical Workflow for Calculating Intercoder Reliability

Step 1: Develop the Coding Scheme

The coding scheme is the foundation of the entire process. Codes must be defined precisely enough that different coders can apply them consistently. The low back pain study used a comprehensive inductive and deductive approach to develop the coding scheme [9]. Inductive development draws codes from the data themselves. Deductive development draws codes from theory or prior research.

A scoping review of modifications to the Roter Interaction Analysis System coding scheme found that most studies reported modifications to the coding framework, with 159 new codes added across modified codebooks. Most newly added codes were task-focused, particularly question-related codes. The main reasons for modification were scope expansion, study setting, and practical considerations. The review noted that the majority of papers did not document their approach to modification or assessment of the modified tool [14]. This finding underscores the importance of documenting coding scheme development and modification.

Step 2: Train Coders

Coders need to understand the coding scheme and practice applying it before formal reliability testing begins. Training typically involves coding practice materials, discussing disagreements, and refining the coding scheme based on feedback.

Step 3: Select a Reliability Sample

Reliability testing does not require coding the entire dataset. A random sample of the data is sufficient. In the emergency medicine study, 20 percent of the comments were double coded at random to assess intercoder reliability [13]. The low back pain study used 10 transcripts for reliability testing [9].

Step 4: Code Independently

Each coder applies the coding scheme to the reliability sample without consulting other coders. Independent coding is essential. If coders discuss their decisions before recording them, the resulting agreement does not reflect the reliability of the coding scheme.

Step 5: Calculate the Coefficient

Choose the coefficient that matches the number of coders and the level of measurement. For two coders with nominal data, Cohen's kappa is appropriate. For more than two coders or for ordinal or continuous data, Krippendorff's alpha is a better choice.

Step 6: Examine Disagreements

The reliability coefficient summarizes the level of agreement, but the pattern of disagreements provides diagnostic information. The low back pain study found that low agreement rates on specific codes indicated that those codes were defined too broadly and needed clarification [9]. Researchers should examine which codes produce the most disagreement and revise those codes accordingly.

Step 7: Revise and Retest

After revising the coding scheme, repeat the reliability testing with a new sample or with the same sample after coders have been retrained. The goal is to achieve acceptable reliability before coding the full dataset.

Step 8: Code the Full Dataset

Once acceptable reliability has been established, coders can proceed to code the full dataset. Some studies use a single coder for the full dataset after reliability has been established on a sample. Other studies maintain double coding throughout, which allows for ongoing reliability monitoring.

Options and Tradeoffs in Reliability Testing

Full Double Coding Versus Sample-Based Testing

Full double coding of the entire dataset provides the most complete reliability information but doubles the coding effort. Sample-based testing is more efficient but assumes that reliability on the sample generalizes to the full dataset. The emergency medicine study used double coding of 20 percent of the comments at random [13]. The low back pain study used 10 transcripts for reliability testing [9].

Independent Coding Versus Consensus Coding

Independent coding requires coders to make decisions without consulting each other. Consensus coding allows coders to discuss disagreements and arrive at a joint decision. Independent coding is necessary for reliability assessment. Consensus coding can be used after reliability has been established to resolve disagreements on the full dataset.

Number of Coders

Studies can use two coders or more. The cause-of-death study in The Netherlands used four coders for each death certificate [12]. The Rorschach study combined a large international sample to obtain intercoder agreement for 489 protocols [8]. More coders provide more information about the reliability of the coding process but require more resources.

Unit of Analysis

Reliability can be calculated at different levels. Coders may agree on the presence or absence of a theme in a document, on the specific codes assigned to a segment of text, or on the values assigned to individual variables. The choice of unit affects the reliability coefficient. The Rorschach study examined protocol-level variables, meaning that agreement was calculated for summary variables derived from the full protocol instead of for individual responses [8].

Observations and Measurements in Applied Studies

Dietary Assessment

The eButton study provides a detailed example of how intercoder reliability is measured in an applied setting. Thirty children aged 9 to 13 years wore the eButton to take 2 full days of dietary images, and the child-parent dyad participated in a following-day interview to verify what dietitians recorded from the images. Two dietitians independently manually reviewed the images to identify eating events, foods in those events, and portion sizes. Descriptive statistics of agreements and disagreements were calculated between dietitians and with children, and t tests and Bland-Altman plots of differences in total kilocalories were calculated between dietitians and between initial dietitian estimates and those finalized after the verification interviews [6].

The results illustrate the complexity of real-world coding. The dietitians agreed on the identity of 60.5 percent of the 1,026 foods but disagreed on 28.6 percent of the foods and on the names for 10.8 percent of the foods. After the verification interviews, the dietitians agreed with the child-parent dyads on the identity of 77.0 percent of the 921 foods. The child-parent dyad identified 12.4 percent of the day's foods when images were not available or not clear, clarified that 5.4 percent of the foods identified were not consumed by the child, and clarified the identity of 5.2 percent of the foods. A software-based approach using three-dimensional wire mesh could be used to estimate portion size on 24 percent of the foods, and professional judgment was required for the remainder [6].

Medication Review

The qualitative analysis of telephonic comprehensive medication reviews analyzed 32 transcripts from 3 organizations in 13 rounds of coding. Intercoder reliability was calculated using Krippendorff's alpha and was above 95 percent. A total of 21 themes were identified across 4 stages: call opening, medication reconciliation, clinical assessments and guidance, and call closing [10].

Advertising Content Analysis

The study of TV advertising and Latino health disparities analyzed 1,593 health-related advertisements from a 3-week composite sample of Telemundo and National Broadcasting Company prime-time TV. Analyses included intercoder reliability, descriptive and bivariate analysis, and rate ratio and rate difference calculations. The study found that Telemundo had significantly more health-adverse and fewer health-beneficial advertisements than National Broadcasting Company [11].

Clinical Communication

The study of gender differences in emergency medicine attending physician comments to residents examined 10,488 narrative comments from 5 EM training programs in the US. Qualitative analysis included deidentification and iterative coding of the dataset using an axial coding approach, with double coding of 20 percent of the comments at random to assess intercoder reliability, with a kappa of 0.84 [13].

Records and Documentation

What to Record

Researchers should document the following elements of the reliability testing process:

• The coding scheme, including code definitions and examples
• The number of coders and their training
• The method for selecting the reliability sample
• The number of coding decisions in the reliability sample
• The coefficient used and its value
• The pattern of disagreements across codes
• The revisions made to the coding scheme based on reliability testing

How to Report

The reporting of intercoder reliability should be detailed enough that readers can evaluate the trustworthiness of the analysis. The scoping review of RIAS modifications found that the majority of papers did not document their approach to modification or assessment of the modified tool, which challenges the reproducibility and interpretation of results [14]. This finding applies broadly to qualitative research. Researchers should report also the final reliability coefficient but also the process by which it was achieved.

Common Failure Patterns

Failure to Correct for Chance Agreement

Percent agreement is easy to compute but can be misleading. Researchers who report only percent agreement without correcting for chance may overstate the reliability of their coding. The solution is to use Cohen's kappa for two coders or Krippendorff's alpha for more than two coders.

Inadequate Coder Training

Coders who do not understand the coding scheme will produce low reliability. Training should include practice coding, discussion of disagreements, and clarification of code definitions before formal reliability testing begins.

Ambiguous Code Definitions

Codes that are defined too broadly produce low agreement because coders interpret them differently. The low back pain study found that low agreement rates indicated that respective codes were defined too broadly and would need clarification [9]. Code definitions should include inclusion and exclusion criteria and examples of typical and borderline cases.

Coding Scheme Modification Without Documentation

The scoping review of RIAS modifications found that most studies reported modifications to the coding framework, but the majority of papers did not document their approach to modification or assessment of the modified tool [14]. Undocumented modifications make it impossible for readers to interpret reliability results or compare findings across studies.

Reliability Testing on a Nonrepresentative Sample

If the reliability sample is not representative of the full dataset, the reliability coefficient may not generalize. The reliability sample should be selected randomly or stratified to reflect the range of data in the full dataset.

Ignoring the Pattern of Disagreements

A single reliability coefficient summarizes the overall level of agreement but hides the pattern of disagreements. Researchers should examine which codes produce the most disagreement and use that information to improve the coding scheme.

Limitations of Intercoder Reliability

Reliability Does Not Guarantee Validity

High intercoder reliability means that coders agree with each other. It does not mean that the codes accurately capture the phenomenon of interest. The eButton study showed that dietitians agreed with each other on many foods but that verification interviews with child-parent dyads changed the final estimates [6]. The verification interviews provided evidence about accuracy that the reliability coefficient could not provide.

Reliability Coefficients Are Context Dependent

A kappa value that is acceptable in one research context may be unacceptable in another. The interpretation of reliability coefficients depends on the complexity of the coding task, the consequences of coding errors, and the standards of the research community.

Reliability Testing Adds Cost

Reliability testing requires additional coding effort, which increases the cost of the research. Researchers must balance the benefits of reliability testing against the resources required.

Small Samples Produce Unstable Estimates

Reliability coefficients calculated on small samples are imprecise. The Rorschach study analyzed 489 protocols, which provided a solid basis for reliability estimation [8]. Studies with smaller reliability samples should interpret their coefficients with caution.

Quality and Welfare Controls

Quality Controls in the Coding Process

Quality controls for coding include standardized training materials, periodic reliability checks during the coding process, and procedures for resolving disagreements. The medication review study used 13 rounds of coding, which allowed the researchers to monitor reliability throughout the analysis [10].

Welfare Considerations in Research Involving Human Participants

When coding involves data from human participants, researchers must follow ethical guidelines for data collection, storage, and reporting. The studies cited in this article involved children wearing the eButton [6], patients receiving medication reviews [10], and residents receiving narrative feedback [13]. Each study required appropriate consent and data protection procedures.

Professional Escalation Criteria

Researchers should escalate concerns about intercoder reliability when:

• Reliability coefficients fall below the level considered acceptable in their research community
• Disagreements cluster on codes that are central to the research questions
• Coders report that the coding scheme is difficult to apply
• Reliability testing reveals systematic differences between coders that training does not resolve

In these situations, researchers should revise the coding scheme, provide additional training, or consult with colleagues who have expertise in the content area.

Safety and Regulatory Context

Data Quality in Regulated Settings

In regulated settings such as clinical trials, pharmacovigilance, and public health surveillance, the reliability of coding has regulatory implications. The cause-of-death study in The Netherlands showed that reliability varies by ICD-10 code and chapter, with low reliability for chronic diseases such as diabetes and renal insufficiency. The researchers recommended that statistical offices provide coders with additional rules for coding diseases with low reliability and evaluate these rules regularly [12].

Semantic Interoperability in Health Information Systems

The study of pharmaceutical dose forms in the German medication plan found that less than half of the dose forms could be coded with EDQM standard terms and that kappa was moderate. The researchers concluded that there is still vast room for improvement in utilization of standardized international vocabulary and unused potential considering cross-border eHealth implementations in the future [7]. This finding has implications for patient safety, because inconsistent coding of dose forms can lead to medication errors.

Research Data Management

The National Institute of Standards and Technology maintains a Research Data Framework that addresses the management of research data throughout the research lifecycle [1]. Researchers who collect coding data should follow data management practices that ensure data integrity, documentation, and accessibility.

Professional Escalation Criteria

Researchers should seek additional expertise or escalate concerns when:

• The coding task involves specialized knowledge that the research team lacks
• Reliability testing reveals persistent disagreement that cannot be resolved through coding scheme revision
• The consequences of coding errors are severe, as in clinical or regulatory settings
• The research involves data from vulnerable populations, requiring additional ethical oversight

In these situations, consultation with content experts, statisticians, or research ethics committees may be necessary.

Frequently Asked Questions

What is the difference between intercoder reliability and internal consistency reliability?

Intercoder reliability measures the agreement between two or more coders who apply a coding scheme to qualitative data. Internal consistency reliability measures whether multiple items on a survey instrument measure the same underlying construct. Intercoder reliability concerns human judgment in the coding process. Internal consistency reliability concerns the statistical coherence of survey items.

What is the difference between intercoder reliability and convergent validity?

Intercoder reliability measures the consistency of coding decisions between coders. Convergent validity measures whether different instruments or methods that purport to measure the same construct produce similar results. Intercoder reliability is a prerequisite for validity in coding-based research, but it does not by itself establish validity.

When should I use Cohen's kappa instead of percent agreement?

Use Cohen's kappa when you have two coders assigning nominal codes and you want to correct for chance agreement. Percent agreement does not correct for chance and will overstate reliability when coders agree by chance. Cohen's kappa is the most common coefficient for two coders on categorical codes.

When should I use Krippendorff's alpha instead of Cohen's kappa?

Use Krippendorff's alpha when you have more than two coders, when your data are ordinal or continuous instead of nominal, or when you have missing data. Krippendorff's alpha handles these situations more flexibly than Cohen's kappa.

What is a good kappa value?

The interpretation of kappa values depends on the research context. In the low back pain study, a kappa value of .67 was regarded as satisfactory to solid [9]. In the emergency medicine study, a kappa of 0.84 was reported for double coding of 20 percent of the comments [13]. In the pharmaceutical dose form study, moderate kappa was described as rather unconvincing agreement among coders [7]. Researchers should interpret kappa values in the context of their specific coding task and research community standards.

How much of the data should be double coded for reliability testing?

The proportion of data double coded varies across studies. The emergency medicine study double coded 20 percent of the comments at random [13]. The low back pain study used 10 transcripts for reliability testing [9]. The choice depends on the size of the dataset, the complexity of the coding task, and the resources available.

What should I do if intercoder reliability is low?

Low intercoder reliability indicates a problem with the coding scheme, the coders, or both. Examine the pattern of disagreements to identify codes that are defined too broadly or ambiguously. Revise those codes, provide additional training to coders, and retest reliability. The low back pain study demonstrated this process, using low agreement rates to identify codes that needed clarification [9].

How should I report intercoder reliability in my research paper?

Report the coefficient used, the number of coders, the size and selection method of the reliability sample, and the value of the coefficient. Describe the pattern of disagreements and any revisions made to the coding scheme based on reliability testing. The scoping review of RIAS modifications found that many papers did not document their approach to modification or assessment of the modified tool, which challenges reproducibility [14]. Detailed reporting supports the credibility of the analysis.

Related Articles

• qPCR Efficiency Calculation And Reporting
• qPCR Efficiency Calculation And Reporting
• qPCR Efficiency Calculation And Reporting
• Bacterial Genome Annotation: A Practical Quality Checklist
• Snakemake for Research Pipelines: A Practical Starting Framework

References and Further Reading

• Research Data Framework. National Institute of Standards and Technology.
• EQUATOR Network. EQUATOR Network.
• Experimental Design Assistant. NC3Rs.
• NCBI Literature Resources. National Center for Biotechnology Information.
• PubMed. National Library of Medicine.
• Dietary Assessment with a Wearable Camera among Children: Feasibility and Intercoder Reliability.. Journal of the Academy of Nutrition and Dietetics, 2018.
• Intercoder Reliability of Mapping Between Pharmaceutical Dose Forms in the German Medication Plan and EDQM Standard Terms.. Studies in health technology and informatics, 2018.
• Rorschach intercoder reliability for protocol-level comprehensive system variables in an international sample.. Journal of personality assessment, 2011.
• From text to codings: intercoder reliability assessment in qualitative content analysis.. Nursing research, 2008.
• Qualitative analysis of telephonic comprehensive medication review content and delivery.. Journal of the American Pharmacists Association : JAPhA, 2023.
• TV Advertising, Corporate Power, and Latino Health Disparities.. American journal of preventive medicine, 2022.
• The reliability of cause-of-death coding in The Netherlands.. European journal of epidemiology, 2010.
• Gender Differences in Emergency Medicine Attending Physician Comments to Residents: A Qualitative Analysis.. JAMA network open, 2022.
• Modifications to the Roter Interaction Analysis System (RIAS) coding scheme: A scoping review.. 2026.
• Validity and Reliability of the Psychodynamic Organizational Diagnostic Instrument SyMOA: Protocol for a Mixed Methods Study.. 2026.
• Portrayals of Depression on TikTok: Content Analysis of Diagnostic Accuracy, Creator Type, and Stylistic Features.. 2026.
• Impact on Families of Upper Extremity Surgical Treatment for Children with Cerebral Palsy.. 2026.
• ReCal: Intercoder Reliability Calculation as a Web Service. 2010.
• An Analysis of Violations of The Theory of Maxims of Quality in Speech at Bocah-Bocah Kosong (BBK) x Indy Barends. Journal of Pragmatics Research, 2025.
• Validation of a scale for the assessment of paravertebral muscle contraction during lumbar puncture. Revista Brasileira De Anestesiologia, 2005.

This article is educational and does not replace institutional policy, professional advice, or applicable safety and regulatory requirements.