Multiple Comparisons: Controlling the Family-Wise Error Rate

When a researcher tests many hypotheses from a single experiment, the chance of at least one false positive result grows with each additional test. This article explains why multiple testing inflates the Type I error rate, describes the main correction methods including Bonferroni, Holm, and false discovery rate approaches, and provides practical guidance for choosing among them. The intended readers are students, researchers, life-science professionals, and informed general readers who need to interpret or design studies involving multiple statistical tests.

The Multiplicity Problem in Statistical Testing

A single hypothesis test carries a predefined probability of rejecting the null hypothesis when it is actually true. This probability, conventionally set at 0.05, is called the Type I error rate or alpha level. When a study conducts only one test, the probability of at least one false positive remains at 0.05. However, when the same study conducts multiple tests, the probability of at least one false positive across the entire set of tests increases substantially.

Consider a simple example. If a researcher performs 20 independent tests, each at alpha equal to 0.05, the probability of observing at least one significant result by chance alone is approximately 1 minus 0.95 raised to the 20th power, which equals about 0.64. This means the researcher has roughly a two in three chance of finding at least one false positive even when no true effects exist. This phenomenon is known as the multiplicity problem or the multiple comparisons problem.

The statistical literature has long recognized this flaw. A common error in articles submitted to biomedical research journals is testing multiple null hypotheses from a single experiment without correcting for the inflated risk of false positive inference that results from this practice. Multiple comparison procedures are designed to minimize this risk, and the most effective procedures control the family-wise Type I error rate so that not more than one hypothesis within a single family is falsely rejected.

The term family refers to the collection of tests that are considered together for the purpose of error control. The family-wise error rate (FWER) is the probability of making at least one false rejection among all tests in the family. Controlling the FWER at 0.05 means the researcher accepts a 5 percent chance that any one of the significant findings is spurious.

At a Glance: Choosing a Multiple Comparison Correction Method

The table below summarizes the main correction methods, what each controls, when each is appropriate, and the tradeoffs involved. This decision table serves as a practical starting point for researchers planning studies with multiple tests.

The choice among these methods depends on the research question, the number of tests, the cost of false positives versus false negatives, and the field-specific conventions. A researcher studying three predefined outcomes in a clinical trial may prefer Bonferroni or Holm. A researcher analyzing thousands of voxels in a brain image or thousands of genetic variants may prefer an FDR-based approach.

The Bonferroni Correction

The Bonferroni correction is the simplest and most widely known method for controlling the family-wise error rate. The procedure divides the desired alpha level by the number of tests performed. If a researcher conducts 10 tests and wants to control the family-wise error rate at 0.05, each individual test must reach a significance threshold of 0.005.

The Bonferroni method is based on the Bonferroni inequality, which states that the probability of at least one event occurring is less than or equal to the sum of the probabilities of the individual events. When applied to hypothesis testing, this inequality guarantees that the family-wise error rate does not exceed the nominal alpha level, regardless of how the tests are correlated.

One-step procedures based on the Bonferroni or Sidák inequalities control the family-wise Type I error rate effectively. For continuous data under normal distribution theory, the Tukey-Kramer procedure handles all possible pairwise contrasts of means, and the Dunnett procedure handles all possible pairwise contrasts of means with a control mean. These procedures are more powerful than the simple Bonferroni correction because they account for the specific structure of the comparisons being made.

The Bonferroni correction has a well-documented drawback. As the number of tests increases, the threshold for significance becomes extremely stringent, and the procedure loses power to detect true effects. This loss of power is particularly problematic in fields such as neuroimaging, where researchers perform a statistical test for every voxel in an image. Standard procedures for multiple hypothesis testing such as Bonferroni tend to not be sensitive enough to be useful in this context.

The application of Bonferroni correction has been constantly controversial in some fields. In forensic population genetics research, it is common to apply Bonferroni correction to Hardy-Weinberg equilibrium tests referring to multiple loci. However, a survey of research published over a 10-year period in two major forensic genetic journals found no uniform standard for whether to apply the correction, and researchers commonly did not provide any explanation for observed deviations from Hardy-Weinberg equilibrium. The authors of that survey argued that Bonferroni correction may not guarantee a prudent result due to an irrelevant null hypothesis, reluctance to reject the null hypothesis, different interpretations of identical p-values, and inflated Type II error. They recommended a two-step approach: first identifying causes of departures from Hardy-Weinberg equilibrium, and second evaluating the biological consequences of those departures. For forensic researchers, they suggested that if a certain degree of deviation occurs, the first step should involve checking the technique and genotyping results carefully instead of recklessly applying Bonferroni correction.

This example illustrates an important principle. Multiple comparison corrections are statistical tools, and their application should be guided by the research context. A correction method that is appropriate for one field or one type of analysis may be unnecessary or even harmful in another.

The Holm Step-Down Procedure

The Holm procedure, also called the Holm-Bonferroni method, is a step-down improvement over the simple Bonferroni correction. It controls the family-wise error rate at the same level as Bonferroni but with greater power, especially when some hypotheses are logically related.

The procedure works as follows. First, the researcher orders all p-values from smallest to largest. The smallest p-value is compared to alpha divided by the total number of tests. If this p-value is significant, the next smallest p-value is compared to alpha divided by the total number of tests minus one. The process continues, with the denominator decreasing by one at each step. The procedure stops at the first non-significant comparison, and all subsequent hypotheses are declared non-significant.

The Holm procedure has several desirable properties. It controls the family-wise Type I error rate as effectively as the one-step Bonferroni or Sidák procedures. It is more powerful than those one-step procedures, especially when hypotheses are logically related. It can be applied also to continuous data but also to ordinal or categorical data. Among the new step-wise multiple comparison procedures, Holm's step-down procedures are commended for their combination of accuracy, power, and versatility, and they have the virtue of simplicity.

For a researcher who wants strict control of the family-wise error rate but is concerned about the conservatism of the simple Bonferroni correction, the Holm procedure is often a better choice. It provides the same protection against false positives while preserving more power to detect true effects.

False Discovery Rate Methods

The false discovery rate (FDR) is a different approach to the multiple comparisons problem. Instead of controlling the probability of making any false rejection, FDR methods control the expected proportion of false positives among all rejected hypotheses. This distinction matters for large-scale testing problems where the number of tests runs into the thousands or millions.

The Benjamini-Hochberg procedure is the most common FDR-controlling method. It orders all p-values from smallest to largest, then finds the largest p-value that is less than or equal to its rank multiplied by the desired FDR level divided by the total number of tests. All hypotheses with p-values smaller than or equal to this threshold are rejected.

The false discovery rate approach was introduced to the neuroscience literature as a way to find objective and effective thresholds for voxelwise statistics derived from neuroimaging data. With at least one test performed for every voxel in an image, some correction of the thresholds is needed to control the error rates, but standard procedures for multiple hypothesis testing such as Bonferroni tend to not be sensitive enough to be useful in this context. FDR-controlling procedures operate simultaneously on all voxelwise test statistics to determine which tests should be considered statistically significant. The innovation of these procedures is that they control the expected proportion of the rejected hypotheses that are falsely rejected.

The theoretical foundation for FDR control under dependency was established in work that showed the Benjamini-Hochberg procedure controls the false discovery rate even when test statistics are dependent. This property is important for applications such as neuroimaging, where neighboring voxels are highly correlated with each other.

The positive false discovery rate is a related concept that conditions on at least one rejection being made. The q-value is a measure that provides a Bayesian interpretation of the positive false discovery rate, allowing researchers to assign a measure of significance to each individual test.

Recent theoretical work has extended the understanding of FDR control. A general principle for multiple testing methods controlling an expected loss asserts that every such method is a special case of a general closed testing procedure based on e-values. This principle generalizes the closure principle, which is known to underlie all methods controlling family-wise error and tail probabilities of false discovery proportions, to a large class of error rates, in particular to the false discovery rate. Methods derived using this e-closure principle generally control their error rate beyond for one rejected set but simultaneously over many, allowing post hoc flexibility for the researcher.

Practical Workflow for Applying Multiple Comparison Corrections

The following steps provide a practical workflow for researchers who need to apply multiple comparison corrections in their own studies.

Step 1: Define the Family of Tests

Before collecting data, the researcher should define which tests belong to the same family. This definition should be based on the research question and the inferential goals. Tests that address the same underlying question and whose results will be interpreted together should be treated as a family. Tests that address distinct questions and will be interpreted independently may not require correction.

Step 2: Determine the Error Rate to Control

The choice between family-wise error rate and false discovery rate depends on the research context. If the cost of a single false positive is high, such as in confirmatory clinical research or regulatory submissions, the family-wise error rate is appropriate. If the research is exploratory and the goal is to identify candidate effects for further study, the false discovery rate may be more suitable.

Step 3: Select the Correction Method

Use the decision table in the At a Glance section to select a method. For small numbers of tests with strict error control requirements, Bonferroni or Holm are appropriate. For large-scale testing problems, FDR methods are preferred.

Step 4: Apply the Correction and Report Results

Apply the chosen correction to the p-values and report both the raw and corrected results. Transparent reporting allows readers to assess the robustness of the findings. The EQUATOR Network provides reporting guidelines for health research that can help researchers present their statistical methods clearly.

Step 5: Document the Analysis Plan

Pre-specification of the analysis plan, including the multiple comparison correction strategy, reduces the risk of bias. The distinction between hypothesis-generating and confirmatory studies is important in this context. Pre-specification of trial endpoints and the correction method to be applied is a key element of confirmatory research design.

Records and Measurements for Multiple Testing Decisions

Researchers should maintain clear records of the decisions made during the design and analysis of studies involving multiple comparisons. These records support transparency and reproducibility.

The following measurements and records are relevant:

• The total number of tests performed in the family
• The raw p-value for each test
• The correction method applied
• The adjusted significance threshold or adjusted p-values
• The number of tests declared significant before and after correction
• The correlation structure among tests, if known
• The sample size and its relationship to statistical power

Sample size has a direct impact on the reliability of results in multiple testing contexts. In resting-state functional magnetic resonance imaging research, small sample sizes minimized power and decreased the likelihood that significant results reflect true effects. The same principle applies across research fields. A study with insufficient sample size may produce significant results after correction that are not reproducible in independent datasets.

The reproducibility of findings depends on both the correction strategy and the sample size. A strict multiple comparison correction strategy reached the best balance between family-wise error rate and test-retest reliability and replicability in one neuroimaging study. This finding suggests that stricter correction methods, while reducing the number of significant results, may produce more reliable results.

Common Failure Patterns in Multiple Comparisons

Several recurring problems appear in research involving multiple comparisons. Recognizing these patterns helps researchers avoid them and helps readers identify flawed analyses.

Failure to Correct at All

The most common failure is conducting multiple tests without any correction for multiplicity. This pattern appears frequently in biomedical research, where a common statistical flaw is testing multiple null hypotheses from a single experiment without correcting for the inflated risk of false positive inference. The result is an overabundance of significant findings that do not replicate in subsequent studies.

Applying Bonferroni Without Considering Context

The opposite failure is applying Bonferroni correction mechanically without considering whether it is appropriate for the research question. The forensic genetics example illustrates this problem. Researchers applied Bonferroni correction to Hardy-Weinberg equilibrium tests without a uniform standard and without explaining deviations from equilibrium. The correction did not rescue the deviations and may have obscured meaningful biological signals.

Using FDR Methods for Confirmatory Research

False discovery rate methods are designed for exploratory research with large numbers of tests. Using FDR methods in confirmatory research where the cost of a single false positive is high may not provide adequate protection. The family-wise error rate is the appropriate target for confirmatory studies.

Ignoring Dependence Among Tests

Many correction methods assume some form of independence or weak dependence among tests. When tests are strongly correlated, as in neuroimaging or genomic data, the performance of correction methods can change. Nonparametric methods that account for the correlation structure may perform better in these settings.

Inconsistent Validation of Error Control

In some fields, the methods used to validate error control are themselves inconsistent. In tandem mass spectrometry proteomics, three different methods for validating false discovery rate control were identified, one of which was invalid, one of which could only provide a lower bound instead of an upper bound, and one of which was valid but underpowered. The result was a poor understanding of how well error control was being achieved, particularly for data-independent acquisition data. None of the data-independent acquisition search tools consistently controlled the false discovery rate at the peptide level, and the tools struggled particularly with analysis of single cell datasets.

Limitations of Multiple Comparison Corrections

Multiple comparison corrections address the problem of inflated Type I error, but they have important limitations that researchers should understand.

Loss of Power

All correction methods reduce statistical power to detect true effects. The Bonferroni correction is the most conservative and loses the most power. Step-down procedures such as Holm preserve more power while maintaining the same family-wise error rate. FDR methods preserve the most power but provide a different type of error control.

The Problem of Defining the Family

The results of multiple comparison corrections depend on how the family of tests is defined. Two researchers analyzing the same data could reach different conclusions if they define the family differently. One might include all tests in a single family, while another might divide the tests into several families based on substantive criteria.

The Null Hypothesis Problem

Some applications of multiple comparison corrections involve null hypotheses that are not meaningful for the research question. The forensic genetics example illustrates this issue. The null hypothesis of Hardy-Weinberg equilibrium may not be the relevant question for forensic identification, and applying a correction to tests of this hypothesis may not produce prudent results.

Type II Error Inflation

Correction methods that are too conservative inflate the Type II error rate, meaning they fail to detect true effects. This problem is particularly acute in fields with large numbers of tests and small effect sizes. The researcher must balance the risk of false positives against the risk of missed true effects.

The Need for Standardized Reporting

Cross-study comparisons require standardized reporting of methods and results. In the context of CRISPR-based gene editing for cystic fibrosis, a systematic review of 27 research articles emphasized the need for standardized reporting of editing efficiency and functional recovery. The same principle applies to multiple comparison corrections. Without standardized reporting, it is difficult to compare results across studies or to assess the adequacy of error control.

Field-Specific Applications of Multiple Comparison Corrections

Multiple comparison corrections are applied across many fields of research. Understanding how different fields handle the multiplicity problem provides context for choosing appropriate methods.

Neuroimaging

Neuroimaging research involves testing a statistical hypothesis at every voxel in the brain, which can mean hundreds of thousands of tests. Voxel-based morphometry involves a voxel-wise comparison of the local concentration of gray matter between two groups of subjects. The procedure involves spatially normalizing high-resolution images from all subjects into the same stereotactic space, segmenting the gray matter from the spatially normalized images, and smoothing the gray-matter segments. Voxel-wise parametric statistical tests compare the smoothed gray-matter images from the two groups, and corrections for multiple comparisons are made using the theory of Gaussian random fields.

The challenge for functional magnetic resonance imaging research is statistically controlling for false positives without omitting true effects. Traditional methods for multiple comparison correction exist, but alternative tools have been developed that do not rely on strict parametric assumptions. These include Statistical non-Parametric Mapping, 3DClustSim, and Threshold Free Cluster Enhancement. Evaluations of these methods examine which produces the most consistent outcomes even when spatially-autocorrelated noise is added to the original images, assessing the false alarm rate and hit rate of each method.

Permutation-based approaches perform adequately for whole-body magnetic resonance imaging and correlation analyses. These approaches can be improved by including information on image structure. Anatomy-based method extensions that incorporate anatomical priors can achieve the same power while keeping the family-wise error rate closer to the desired rate.

Genomics and Genetics

Genetic association studies often test hundreds of thousands or millions of genetic variants for association with a trait or disease. The Bonferroni correction is commonly applied in this context, but it is extremely conservative given the number of tests. FDR methods are often preferred because they provide a balance between false positives and false negatives.

In a case-control study of hypertension in populations affected by the Aral Sea ecological crisis, researchers investigated the correlation between nine polymorphic variants and the incidence of hypertension. Logistic regression under a log-additive genetic model was used to calculate unadjusted and adjusted odds ratios and 95 percent confidence intervals. Stratified analyses by sex, age, and body mass index were performed with Bonferroni correction. The analysis revealed statistically significant associations between essential hypertension and six polymorphic variants after Bonferroni correction.

Clinical Research

Clinical trials often test multiple endpoints or multiple treatment comparisons. The distinction between hypothesis-generating and confirmatory studies is important in this context. Pre-specification of trial endpoints and the correction method to be applied is a key element of confirmatory research design.

In a retrospective multicenter study of postpartum depressive symptoms, researchers used the Edinburgh Postnatal Depression Scale to evaluate 763 women. Comparisons used univariable and multivariable logistic regression with Bonferroni adjustment. After adjustment and Bonferroni correction, history of abortion and preeclampsia remained statistically significant. Working status and history of stillbirth showed nominal associations before correction but did not remain significant after Bonferroni correction. Neonatal intensive care unit admission was significant in unadjusted analysis but did not remain significant after adjustment.

Proteomics

Mass spectrometry proteomics faces a pressing statistical challenge in assessing whether a given software tool provides accurate error control. Each software tool for searching such data uses its own internally implemented methodology for reporting and controlling the error. Many of these software tools are closed source with incompletely documented methodology, and the strategies for validating the error are inconsistent across tools.

Single-Cell Sequencing

Single-cell sequencing studies involve testing many hypotheses across individual cells. The choice of multiple comparison correction method affects the results and their interpretation. Corrections to published articles in this field highlight the importance of accurate statistical reporting.

Quality and Welfare Considerations in Research Design

The choice of multiple comparison correction method is part of the broader research design process. Good research design reduces the need for aggressive corrections by minimizing the number of tests and increasing the precision of measurements.

The NC3Rs Experimental Design Assistant provides a tool for researchers to design experiments that are robust and reliable. The tool helps researchers think through the design of their experiments, including the statistical analysis plan. Using such tools can reduce the risk of statistical errors and improve the quality of research.

The Research Data Framework from the National Institute of Standards and Technology addresses the broader context of research data management. Good data management practices support transparent and reproducible research, including the appropriate application of statistical methods.

The EQUATOR Network provides reporting guidelines for health research. These guidelines help researchers report their methods and results transparently, including the statistical methods used to address multiplicity.

The National Center for Biotechnology Information provides literature resources that researchers can use to stay current with methodological developments. PubMed provides access to the biomedical literature, including methodological papers on multiple comparison corrections.

Professional Escalation Criteria

Researchers should seek professional statistical advice in several situations:

• When the number of tests is very large and the correlation structure among tests is complex
• When the research is confirmatory and the results will inform regulatory decisions
• When different correction methods produce substantially different conclusions
• When the researcher is uncertain about the appropriate definition of the family of tests
• When the study design involves complex dependencies among tests, such as in neuroimaging or genomic studies

A statistician can help with the choice of correction method, the interpretation of results, and the reporting of methods. The cost of statistical consultation is small compared to the cost of publishing results that do not replicate or that are later found to be based on flawed statistical methods.

Frequently Asked Questions

What is the difference between family-wise error rate and false discovery rate?

The family-wise error rate is the probability of making at least one false rejection among all tests in the family. The false discovery rate is the expected proportion of false positives among all rejected hypotheses. Family-wise error rate control is stricter and is appropriate when the cost of any false positive is high. False discovery rate control is more permissive and is appropriate for large-scale exploratory analyses where some false positives are acceptable.

When should I use the Bonferroni correction?

Use the Bonferroni correction when the number of tests is small, the tests are independent or weakly dependent, and false positives are costly. The Bonferroni correction divides the alpha level by the number of tests. It is simple to apply and guarantees that the family-wise error rate does not exceed the nominal level. However, it becomes very conservative as the number of tests grows.

How is the Holm procedure different from the Bonferroni correction?

The Holm procedure is a step-down improvement over the Bonferroni correction. It orders the p-values from smallest to largest and compares each to a progressively less stringent threshold. The Holm procedure controls the family-wise error rate at the same level as Bonferroni but with greater power, especially when hypotheses are logically related. It can be applied to continuous, ordinal, or categorical data.

What is the false discovery rate and when should I use it?

The false discovery rate is the expected proportion of false positives among all rejected hypotheses. Use FDR methods when the number of tests is large, such as in genomics, neuroimaging, or proteomics, and when some false positives are acceptable. The Benjamini-Hochberg procedure is the most common FDR-controlling method. FDR methods preserve more power than family-wise error rate methods.

Does the Bonferroni correction rescue deviations from Hardy-Weinberg equilibrium?

No. The Bonferroni correction does not rescue deviations from Hardy-Weinberg equilibrium. In forensic population genetics, applying Bonferroni correction to Hardy-Weinberg equilibrium tests has been controversial because the correction may not guarantee a prudent result due to an irrelevant null hypothesis, reluctance to reject the null hypothesis, different interpretations of identical p-values, and inflated Type II error. The recommended approach is to identify the causes of departures from Hardy-Weinberg equilibrium and evaluate the biological consequences of those departures.

How does sample size affect multiple comparison corrections?

Sample size has a direct impact on the reliability of results in multiple testing contexts. Small sample sizes minimize power and decrease the likelihood that significant results reflect true effects. In neuroimaging research, reliability, sensitivity, and positive predictive value rose as sample size increased. A strict multiple comparison correction strategy reached the best balance between family-wise error rate and test-retest reliability and replicability.

What is the q-value?

The q-value is a measure that provides a Bayesian interpretation of the positive false discovery rate. It assigns a measure of significance to each individual test, analogous to how the p-value measures significance in traditional hypothesis testing. The q-value is the minimum false discovery rate at which a test would be declared significant.

Should I correct for multiple comparisons in exploratory research?

The decision depends on the goals of the research. In exploratory research, the goal is often to generate hypotheses for further study. FDR methods are often appropriate in this context because they control the expected proportion of false positives while preserving power to detect true effects. In confirmatory research, the goal is to test pre-specified hypotheses, and family-wise error rate control is typically required. The distinction between hypothesis-generating and confirmatory studies is important in this context.

Related Articles

• Statistical Methods
• Statistical Methods
• Statistical Methods
• Statistical Significance
• Statistical Significance

References and Further Reading

• Research Data Framework. National Institute of Standards and Technology.
• EQUATOR Network. EQUATOR Network.
• Experimental Design Assistant. NC3Rs.
• NCBI Literature Resources. National Center for Biotechnology Information.
• PubMed. National Library of Medicine.
• Voxel-based morphometry--the methods.. NeuroImage, 2000.
• Evaluating Alternative Correction Methods for Multiple Comparison in Functional Neuroimaging Research.. Brain sciences, 2019.
• Multiple comparison correction methods for whole-body magnetic resonance imaging.. Journal of medical imaging (Bellingham, Wash.), 2020.
• Multiple comparison procedures updated.. Clinical and experimental pharmacology & physiology, 1998.
• CRISPR for cystic fibrosis: Advances and insights from a systematic review.. Molecular therapy : the journal of the American Society of Gene Therapy, 2025.
• Correction to: Comparison of minimal residual disease detection in multiple myeloma by SRL 8-color single-tube and EuroFlow 8-color 2-tube multiparameter flow cytometry.. International journal of hematology, 2019.
• Reproducibility of R-fMRI metrics on the impact of different strategies for multiple comparison correction and sample sizes.. Human brain mapping, 2018.
• Correction: comparison of multiple displacement amplification (MDA) and multiple annealing and looping-based amplification cycles (MALBAC) in single-cell sequencing.. PloS one, 2015.
• Does Bonferroni correction "rescue" the deviation from Hardy-Weinberg equilibrium?. 2020.
• Hypothesis-generating and confirmatory studies, Bonferroni correction, and pre-specification of trial endpoints.. 2019.
• Etymologia: Bonferroni correction.. 2015.
• Multiple P-values and Bonferroni correction.. 2016.
• Obstetric and Neonatal Factors Associated with Postpartum Depressive Symptoms: A Retrospective Multicenter EPDS-Based Study.. 2026.
• Genetics of hypertension in populations affected by the Aral Sea ecological crisis: A case-control study.. 2026.
• THE CONTROL OF THE FALSE DISCOVERY RATE IN MULTIPLE TESTING UNDER DEPENDENCY. 2001.
• Thresholding of Statistical Maps in Functional Neuroimaging Using the False Discovery Rate. NeuroImage, 2002.
• Controlling the false discovery rate in behavior genetics research.. Behavioural Brain Research, 2001.
• The positive false discovery rate: a Bayesian interpretation and the q-value. 2003.
• Bringing Closure to False Discovery Rate Control: A General Principle for Multiple Testing. 2025.
• Assessment of false discovery rate control in tandem mass spectrometry analysis using entrapment. bioRxiv, 2024.
• Comparisons of ten corrections methods for t-test in multiple comparisons via Monte Carlo study. Electronic Journal of Applied Statistical Analysis, 2018.

This article is educational and does not replace institutional policy, professional advice, or applicable safety and regulatory requirements.