Bonferroni Correction for Multiple Comparisons: When and How to Use It

The Bonferroni correction is a statistical adjustment that tightens the threshold for declaring a result significant when you perform many statistical tests at once. It works by dividing your chosen significance level, typically 0.05, by the number of tests you run. This method directly addresses the increased risk of false positives, known as type I errors, that arises whenever you test multiple hypotheses on the same data set. For students, researchers, and life-science professionals, understanding when to apply this correction and when to choose a different approach is essential for producing credible findings.

The Multiple Comparisons Problem

Every statistical test carries a chance of producing a false positive. When you set your significance level at 0.05, you accept a 5 percent probability of concluding that an effect exists when it does not. This risk applies to each individual test. If you run only one test, your overall chance of a false positive remains 5 percent. But if you run 20 tests, the probability that at least one of them will appear significant by chance alone rises well above 5 percent.

This inflation of error is the core of the multiple comparisons problem. The Bonferroni correction adjusts probability values because of the increased risk of a type I error when making multiple statistical tests [6]. Without some form of adjustment, researchers who run many tests can easily report findings that are statistical artifacts instead of genuine effects.

Consider a common scenario in agricultural research. A nutritionist measures 15 different blood metabolites in cattle fed two different diets. Running 15 separate t-tests at the 0.05 level means the chance of finding at least one significant difference purely by chance is roughly 54 percent. The Bonferroni correction would require each individual test to reach a p-value below 0.0033, which is 0.05 divided by 15, before being declared significant.

The problem appears across many fields. Correlation coefficients between biological measurements and clinical scales are often calculated in psychiatric research, and calculating numerous correlations increases the risk of a type I error [7]. The same logic applies to any study where many variables are tested against many outcomes.

What the Bonferroni Correction Does

The Bonferroni correction is straightforward in its calculation. You divide your desired overall significance level, often called alpha, by the number of comparisons being made. The result becomes your new threshold for statistical significance.

The formula is simple. If you plan to test 10 hypotheses and want to keep your overall error rate at 0.05, each individual test must achieve a p-value below 0.005 to be considered significant. Alternatively, you can multiply each raw p-value by the number of tests and compare that adjusted value against your original alpha.

This approach controls what statisticians call the family-wise error rate. That is the probability of making at least one false positive across the entire set of tests. The correction is conservative, meaning it errs on the side of avoiding false positives even when that means missing some true effects.

The change in threshold values is larger when the number of correlations goes from 1 to 5 than when it goes from 50 to 100 [7]. This means the first few additional tests require the most dramatic adjustments. A correlation coefficient that is statistically significant at 5 percent when calculated alone can fall below the threshold level of significance when calculated even among a few other coefficients [7].

When the Bonferroni Correction Is Appropriate

The decision to use the Bonferroni correction depends on the circumstances of the study [6]. It should not be used routinely. Three specific situations justify its application.

First, use the correction when you need a single test of the universal null hypothesis that all tests are not significant [6]. This situation arises when your research question is whether any effect exists anywhere in your data, instead of which specific effects are present.

Second, use the correction when it is imperative to avoid a type I error [6]. Some decisions carry high costs for false positives. In regulatory settings, drug approval, or food safety testing, declaring that a treatment works when it does not can have serious consequences.

Third, use the correction when a large number of tests are carried out without preplanned hypotheses [6]. Exploratory analyses that scan many variables for potential associations generate many tests. The Bonferroni correction provides a safeguard against the inevitable false positives that such scanning produces.

The correction has been applied successfully in many research contexts. In a study of optic pathway glioma risk in children with neurofibromatosis type 1, researchers applied Bonferroni correction for multiple comparisons when analyzing the association between mutation position and tumor development [9]. In a radiology study of deep learning methods for blood flow measurement, investigators used F tests with Bonferroni correction to compare multiple hemodynamic parameters across vascular locations [10].

Limitations and Criticisms

The routine use of the Bonferroni correction has been criticized as deleterious to sound statistical judgment [6]. The main criticism is that it tests the wrong hypothesis in many situations. When a researcher has specific predictions about which comparisons should differ, adjusting for all possible comparisons, including those never hypothesized, reduces the chance of detecting real effects.

The correction reduces the chance of a type I error but at the expense of a type II error [6]. A type II error occurs when you fail to detect an effect that actually exists. By making the significance threshold very strict, the Bonferroni correction increases the likelihood of missing genuine findings.

This tradeoff is especially problematic for high-dimensional data. With brain imaging data, corrections for multiple comparisons such as Bonferroni adjustments have negative impacts on statistical power [12]. The same issue applies to genomic studies. In whole-genome association studies that consider several thousand to upwards of 500,000 and 1 million SNPs, a Bonferroni correction is not practical because it does not take into account correlations between the tests due to linkage disequilibrium and hence can be too conservative [13].

The effect sizes of susceptibility alleles will rarely reach the required level of significance in whole-genome association studies if a Bonferroni correction is used, and the number of false negatives is likely to be large [13]. This limitation has driven the development of alternative methods that balance false positives and false negatives more effectively.

At a Glance

Alternatives to the Bonferroni Correction

Several alternative methods address the multiple comparisons problem while preserving more statistical power.

False Discovery Rate Control

The false discovery rate approach controls the expected proportion of rejected hypotheses that are falsely rejected [22]. Instead of limiting the chance of any false positive, this method limits the proportion of false positives among the findings you declare significant.

The Benjamini-Hochberg procedure is the most common FDR method. It ranks all p-values from smallest to largest and compares each against a threshold that becomes less strict as the rank increases. This approach is less conservative than Bonferroni while still providing meaningful error control.

Simulation studies of omic experiments have confirmed the tremendous false discovery rate benefit of correction across a range of contexts [11]. The sensitivity costs of Benjamini-Hochberg were generally modest compared to the FDR benefits [11]. This makes FDR methods attractive for large-scale exploratory studies.

Permutation-Based Methods

Permutation procedures allow for the estimation of adjusted significance levels without assuming independence among variables [15]. These methods are valuable when your data have complex correlation structures that violate the assumptions of simpler corrections.

In metabolome-wide association studies, permutation-based approaches can provide adjusted significance levels that are consistent across different outcomes while effectively controlling the type I error rate [15]. However, the complex non-normal structure of metabolic profiles and outcomes may bias permutation results, leading to overly conservative threshold estimates [15].

For brain imaging data, permutation is often the best option for multiple testing correction due to its exactness and flexibility to handle a variety of common imaging situations [17]. The same logic applies to any data set with strong spatial or temporal correlations.

Other Adjusted Thresholds

The Sidak correction provides a slightly less conservative alternative to Bonferroni when tests are independent. It calculates the adjusted threshold as one minus the nth root of one minus alpha, where n is the number of tests.

Tarone's correction improves statistical power by ignoring hypotheses that never become statistically significant [18]. This approach is useful in settings like sequence motif discovery where many possible hypotheses cannot achieve significance regardless of the data.

Practical Workflow for Applying the Bonferroni Correction

Step 1: Define Your Hypothesis Structure

Before collecting data, decide whether your study involves a single universal null hypothesis or multiple specific hypotheses. If you need to know whether any effect exists across all your tests, the Bonferroni correction is appropriate. If you have specific predictions about particular comparisons, consider whether correction is necessary at all.

Step 2: Count Your Tests

Determine the total number of statistical tests you will perform. This count must include every comparison that could produce a significant result. If you run 10 tests but only correct for 5, your error rate remains inflated.

Step 3: Calculate the Adjusted Threshold

Divide your chosen alpha level by the number of tests. For 20 tests at alpha 0.05, the adjusted threshold is 0.0025. Each individual test must produce a p-value below this threshold to be declared significant.

Step 4: Apply the Correction Consistently

Apply the same correction to all tests in the family. Do not correct some comparisons while leaving others uncorrected based on the results. This selective application inflates your error rate and undermines the validity of your findings.

Step 5: Report Both Raw and Adjusted Values

Report the raw p-values alongside the adjusted significance threshold. This transparency allows readers to evaluate your findings and apply their own judgment about the appropriate level of correction.

Step 6: Document Your Rationale

Explain why you chose the Bonferroni correction instead of an alternative method. State whether you needed to control the family-wise error rate, whether avoiding type I errors was critical, and whether your tests were preplanned or exploratory.

Records and Measurements

Maintain a complete record of every test you perform, including tests that produce non-significant results. This record should include the raw p-value, the adjusted threshold, and the decision about significance for each test.

Document the total number of tests considered in your correction. If you exclude any tests from the correction, record the rationale for that exclusion. Reviewers and readers need this information to evaluate the validity of your approach.

Record the version of any statistical software you use and the specific function or procedure applied. Different software implementations may handle edge cases differently, and this documentation supports reproducibility.

Common Failure Patterns

Ignoring the Multiple Comparisons Problem

Many researchers simply ignore the problem of multiple testing [6]. This failure pattern appears across fields. A systematic review of pharmacovigilance studies found that no common procedure was implemented to correct for multiple testing [14]. The review highlighted a lack of interest in multiple testing correction and recommended the establishment of guidelines [14].

Population genetics studies show the same pattern. A meta-analysis of 215 studies published between 2011 and 2013 found scarce use of family-wise error rate corrections across tests of Hardy-Weinberg equilibrium, linkage disequilibrium, and population differentiation [20]. A replication using 205 studies published between 2013 and 2018 showed the same continued disuse and inconsistencies [20].

Applying the Correction Uncritically

Some authors use the Bonferroni method uncritically with no rationale or discussion [6]. They apply the correction because it is familiar or expected, without considering whether it fits their research question. This mechanical application can reduce statistical power unnecessarily.

Quoting Adjusted Values Incorrectly

Some studies quote adjusted p-values incorrectly or give an erroneous rationale [6]. Common errors include adjusting p-values but reporting them as raw values, applying the wrong number of tests in the calculation, or describing the correction as controlling something other than the family-wise error rate.

Correcting When Not Needed

Applying the Bonferroni correction to confirmatory tests that were preplanned and directly test specific hypotheses can obscure real effects. If your study tests a small number of specific predictions derived from theory or prior evidence, correction may not be necessary.

Using Bonferroni for Large-Scale Correlated Data

Applying the Bonferroni correction to data with thousands of highly correlated tests produces thresholds so strict that almost nothing passes. In whole-genome association studies, this approach is not practical because it does not account for correlations between tests [13]. The result is a large number of false negatives.

Welfare and Safety Context

The choice of multiple testing correction has direct implications for animal welfare and human safety. In toxicology studies, failing to correct for multiple comparisons can lead to false claims that a compound is safe when it actually produces adverse effects. Conversely, overcorrection can hide genuine safety signals.

In agricultural research, the consequences of statistical errors extend to production decisions. A false positive might lead a producer to adopt an ineffective treatment, wasting resources and potentially compromising animal health. A false negative might cause a producer to reject a beneficial intervention that improves welfare or productivity.

The stakes are particularly high in regulatory contexts. Studies submitted for regulatory approval must demonstrate appropriate statistical methods. The Research Data Framework from the National Institute of Standards and Technology provides guidance on data management practices that support rigorous statistical analysis [1]. The EQUATOR Network offers reporting guidelines that help researchers document their statistical methods transparently [2].

Professional Escalation Criteria

Consult a professional statistician when any of the following conditions apply to your work.

Seek expert advice when your study involves more than 20 tests and you are unsure whether the Bonferroni correction is appropriate. A statistician can help you determine whether your hypotheses are preplanned or exploratory and whether a less conservative method might be more suitable.

Consult a statistician when your data have complex correlation structures. If your measurements come from the same animals over time, from littermates, or from animals housed together, the assumption of independent tests is violated. Permutation-based methods or mixed models may be more appropriate.

Seek expert help when your study is part of a regulatory submission or when your findings will inform policy decisions. Regulatory agencies have specific expectations about statistical methods, and a statistician can help you meet those expectations.

Consult a statistician when you are analyzing high-dimensional data such as genomics, metabolomics, or imaging data. The multiple testing problem in these settings requires specialized methods that account for correlations among tests.

Comparing Correction Methods in Practice

The choice among correction methods depends on your research goals and data structure. Simulation studies provide guidance for omic experiments. No correction method can guarantee a low false discovery proportion in a single experiment, but the probability of a high false discovery proportion is small when a high number and proportion of corrected p-values are significant [11].

On average, correction decreased sensitivity, but the sensitivity costs of Benjamini-Hochberg and permutation were generally modest compared to the false discovery rate benefits [11]. In a given experiment, observed sensitivity was always maintained or decreased by Benjamini-Hochberg and Bonferroni, whereas it was often increased by permutation [11]. Overall, permutation had better false discovery rate and sensitivity than Benjamini-Hochberg [11].

Increasing sample size, decreasing variability, or increasing effect size can enable the detection of all true changes while still correcting p-values [11]. This finding underscores the importance of experimental design. A well-powered study can afford strict correction because its tests have sufficient sensitivity to detect real effects.

The Experimental Design Assistant from the NC3Rs provides tools for planning experiments that account for statistical power and multiple testing considerations [3]. Using such tools before data collection can prevent many of the problems that arise from inadequate design.

Specialized Applications

Brain Imaging

The multiple testing problem arises also when there are many voxels or vertices in an image representation of the brain, but also when multiple contrasts of parameter estimates are tested in the same general linear model [17]. Correction for this multiplicity must be performed to avoid excess of false positives [17].

Standard procedures for multiple hypothesis testing such as Bonferroni tend to not be sensitive enough to be useful in neuroimaging contexts [22]. False discovery rate controlling procedures have proven more effective for analyzing neuroimaging data [22].

Genomics and Genetics

Traditional solutions involving adjustments such as the Bonferroni correction worked well in settings involving a few tests, such as candidate gene studies with 10 to 20 tests [13]. They even worked reasonably when the number of tests was somewhat larger, such as a few hundred in genome-wide microsatellite scans [13].

However, dense single nucleotide polymorphism studies often consider several thousand to upwards of 500,000 and 1 million SNPs [13]. In these settings, the Bonferroni correction is too conservative because it does not account for correlations between tests due to linkage disequilibrium [13].

Metabolomics

The search for statistically significant relationships between molecular markers and outcomes is challenging when dealing with high-dimensional, noisy and collinear multivariate omics data such as metabolomic profiles [15]. Permutation procedures allow for the estimation of adjusted significance levels without assuming independence among metabolomic variables [15].

Proteomics

The multiple hypothesis testing problem is inherent in large-scale quantitative omic experiments such as mass spectrometry-based proteomics [11]. Tools for comparing the costs and benefits of different p-value correction methods under different experimental conditions are needed [11].

Designing Studies to Minimize the Need for Correction

The most effective way to manage the multiple comparisons problem is to design studies that require fewer tests. Focusing on the most relevant variables or the use of multivariate statistics is advocated as an alternative to correcting many individual tests [7].

Plan your hypotheses before data collection. Preplanned hypotheses derived from theory or prior evidence require less correction than exploratory analyses. If you specify your primary outcomes in advance, you can limit correction to those outcomes instead of applying it to every possible comparison.

Use multivariate methods when appropriate. Techniques such as multivariate analysis of variance, principal component analysis, or mixed models can test multiple outcomes simultaneously while accounting for their correlations. These approaches often provide more interpretable results than many individual tests.

Increase your sample size. A larger sample provides more statistical power for each test, making it possible to detect true effects even with strict correction thresholds. Power analysis before data collection can help you determine the sample size needed to detect meaningful effects after correction.

Reduce measurement variability. Standardized protocols, calibrated equipment, and careful training of personnel all reduce noise in your data. Lower variability increases the sensitivity of your statistical tests.

Reporting Your Methods

Transparent reporting of multiple testing procedures is essential for the credibility of your research. The EQUATOR Network provides reporting guidelines that help researchers document their statistical methods completely [2]. Following these guidelines ensures that readers can evaluate the appropriateness of your correction choices.

Describe the total number of tests performed and the correction method applied. State your alpha level and the adjusted threshold. Explain whether your tests were preplanned or exploratory. Justify your choice of correction method in the context of your research question.

Report both raw and adjusted p-values when space permits. This transparency allows readers to apply alternative correction methods if they disagree with your choice. It also supports meta-analysis and secondary analysis of your findings.

Frequently Asked Questions

What is the difference between family-wise error rate and false discovery rate?

The family-wise error rate is the probability of making at least one false positive across all tests in a family. The false discovery rate is the expected proportion of false positives among the tests you declare significant. Family-wise error rate control is stricter because it aims to prevent any false positive at all. False discovery rate control accepts that some proportion of your significant findings will be false, but limits that proportion.

Does the Bonferroni correction require independent tests?

The Bonferroni correction does not require independent tests, but it becomes increasingly conservative as correlations among tests increase. When tests are positively correlated, the correction overcorrects because it treats each test as providing completely independent information. This is why alternative methods such as permutation are preferred for highly correlated data.

Can I apply the Bonferroni correction after seeing my results?

You can apply the correction after seeing results, but this practice is problematic. Deciding to correct after observing which tests produce significant results introduces bias. The number of tests and the correction method should be specified before data collection or at least before examining results.

How do I choose between Bonferroni and Benjamini-Hochberg?

Choose Bonferroni when you need to avoid any false positives, when you have few tests, or when the cost of a false positive is high. Choose Benjamini-Hochberg when you have many tests, when you are screening for potential effects, or when you can tolerate some false positives among your significant findings.

What does an adjusted p-value mean?

An adjusted p-value represents the smallest family-wise error rate at which the individual test would be declared significant. If your adjusted p-value is 0.03, the test would be significant at a family-wise error rate of 0.03 or higher. You compare adjusted p-values directly against your chosen alpha level.

Is the Bonferroni correction too conservative for exploratory research?

For exploratory research with many tests, the Bonferroni correction is often too conservative. It can hide genuine effects, particularly when tests are correlated. False discovery rate methods or permutation-based approaches may be more appropriate for exploratory analyses where the goal is to generate hypotheses instead of confirm them.

How many tests are too many for the Bonferroni correction?

There is no fixed number that makes the Bonferroni correction impractical. The correction remains valid regardless of the number of tests. However, as the number of tests grows, the adjusted threshold becomes so strict that few results will pass. For thousands of tests, alternative methods are usually more useful.

Should I correct for multiple comparisons in observational studies?

Observational studies often involve many potential associations, making correction important. However, the choice of correction method depends on the purpose of the study. Confirmatory observational studies testing specific hypotheses may require less correction than exploratory studies scanning many variables.

Related Articles

• What Is Molecular Biology? Definition, Methods, and Examples
• Statistical Methods
• Statistical Methods
• Statistical Methods
• Multiple Genome Alignment

References and Further Reading

• Research Data Framework. National Institute of Standards and Technology.
• EQUATOR Network. EQUATOR Network.
• Experimental Design Assistant. NC3Rs.
• NCBI Literature Resources. National Center for Biotechnology Information.
• PubMed. National Library of Medicine.
• When to use the Bonferroni correction.. Ophthalmic & physiological optics : the journal of the British College of Ophthalmic Opticians (Optometrists), 2014.
• Multiple correlations and Bonferroni's correction.. Biological psychiatry, 1998.
• Multiple hypothesis testing and Bonferroni's correction.. BMJ (Clinical research ed.), 2014.
• Risk of Optic Pathway Glioma in Neurofibromatosis Type 1: No Evidence of Genotype-Phenotype Correlations in A Large Independent Cohort.. Cancers, 2019.
• Deep Learning Automated Background Phase Error Correction for Abdominopelvic 4D Flow MRI.. Radiology, 2022.
• Costs and Benefits of Popular P-Value Correction Methods in Three Models of Quantitative Omic Experiments.. Analytical chemistry, 2023.
• Split-test Bonferroni correction for QEEG statistical maps.. Biological cybernetics, 2008.
• Methods for handling multiple testing.. Advances in genetics, 2008.
• Pharmacovigilance studies without a priori hypothesis: systematic review highlights inappropriate multiple testing correction procedures.. 2023.
• Multiple-testing correction in metabolome-wide association studies.. 2021.
• Multiple-testing Correction in Metabolome-wide Association Studies. 2020.
• Multiple testing correction over contrasts for brain imaging.. 2020.
• MotiMul: A significant discriminative sequence motif discovery algorithm with multiple testing correction. 2020.
• Multiple testing correction over contrasts for brain imaging. 2019.
• Continued misuse of multiple testing correction methods in population genetics-A wake-up call?. 2019.
• THE CONTROL OF THE FALSE DISCOVERY RATE IN MULTIPLE TESTING UNDER DEPENDENCY. 2001.
• Thresholding of Statistical Maps in Functional Neuroimaging Using the False Discovery Rate. NeuroImage, 2002.
• Controlling the false discovery rate in behavior genetics research.. Behavioural Brain Research, 2001.
• The positive false discovery rate: a Bayesian interpretation and the q-value. 2003.
• Bringing Closure to False Discovery Rate Control: A General Principle for Multiple Testing. 2025.
• Assessment of false discovery rate control in tandem mass spectrometry analysis using entrapment. bioRxiv, 2024.
• Bonferroni correction hides significant motif combinations. 13th IEEE International Conference on Bioinformatics and Bioengineering IEEE Bibe 2013, 2013.
• Bonferroni correction for seasonal cointegrating ranks. Economics Letters, 2009.
• Bonferroni in biomedical research. Giornale Italiano Di Nefrologia Organo Ufficiale Della Societa Italiana Di Nefrologia, 2002.
• Tutorial on Bonferroni Correction as a Post Hoc Analysis of a Significant Chi-Squared Test: A Methodological Guide in Food Science. Journal of Food Quality and Hazards Control, 2025.

This article is educational and does not replace institutional policy, professional advice, or applicable safety and regulatory requirements.