How to Handle Publication Bias in a Systematic Review

By Dr. Zubair Khalid, DVM, MS, PhD · 2026-08-27

Quick Answer

• Publication bias occurs when studies with positive or statistically significant results are more likely to be published than those with null or negative findings, which can distort the conclusions of a systematic review.
• Assess publication bias using funnel plot asymmetry and statistical methods like trim-and-fill, while also searching grey literature and trial registries to locate unpublished studies.
• No single method fully eliminates publication bias, so you must combine multiple assessment strategies and transparently report your search and screening decisions.

What Publication Bias Means for Systematic Reviews

Publication bias is a systematic distortion of the evidence base that occurs when the likelihood of a study being published depends on the direction or statistical significance of its results. In a systematic review, you aim to identify, appraise, and synthesize all available evidence on a focused question. If the published literature disproportionately contains positive findings, your review will inherit that distortion even when your own search and screening methods are flawless.

The problem is not limited to journals rejecting null results. Researchers may fail to submit negative findings, industry sponsors may suppress unfavorable outcomes, and smaller studies with null results may languish in file drawers. Each of these pathways removes evidence from the public record, and a systematic review that relies only on what is published will overestimate treatment effects or miss important harms.

For researchers in the life sciences, including those working with animal models or agricultural systems, the consequences are concrete. A review that misses unpublished negative trials may recommend an intervention that does not work, or may fail to identify a safety concern that appeared only in studies that were never published. The National Library of Medicine provides access to biomedical books and research method references that describe the mechanisms and consequences of publication bias in detail.

Core Principles for Identifying Publication Bias

The Difference Between Random Error and Systematic Distortion

Random error affects all studies and is addressed through meta-analytic pooling and confidence intervals. Publication bias is different because it systematically removes entire studies from the evidence base. You cannot fix this problem by adding more studies to your meta-analysis if the missing studies are simply not available to you.

The key distinction is that publication bias operates before you begin your review. Your job is to detect its possible presence, estimate its potential impact, and report your findings honestly so that readers can judge the strength of your conclusions.

How Selective Reporting Compounds the Problem

Selective reporting is closely related to publication bias. A study may be published, but the authors may report only the outcomes that showed significant effects while omitting others. This creates a distorted picture even within the published record. When you extract data for your review, you should check whether all outcomes listed in the study protocol or methods section are actually reported in the results.

The Committee on Publication Ethics core practices address transparency in research and publication, including expectations around data integrity and responsible reporting. These practices matter for your review because they shape the quality of the primary studies you are synthesizing.

Practical Workflow for Assessing Publication Bias

Step 1: Register Your Systematic Review Protocol

Registering your protocol before you begin searching creates a public record of your planned methods, including your search strategy, inclusion criteria, and planned analyses. This registration makes it harder for you to unconsciously adjust your methods after seeing the results, and it allows readers to compare your final report against your original plan.

The EQUATOR Network provides access to reporting guidelines that help you structure your systematic review protocol and final report. Following an appropriate reporting guideline ensures that you disclose your search methods, screening decisions, and assessment of bias risk in a transparent and reproducible way.

Step 2: Design a Comprehensive Search Strategy

A search that covers only major databases will miss a substantial portion of relevant evidence. Your search strategy should include multiple databases, reference lists of included studies, citation tracking, and direct contact with researchers working in the field.

Grey literature is particularly important for addressing publication bias. This includes conference abstracts, theses, dissertations, government reports, and other documents that are not published in peer-reviewed journals. Searching clinical trial registries and other study registries can help you identify studies that were conducted but never published.

Step 3: Screen and Select Studies Transparently

Document every stage of your screening process. Record how many records you identified from each source, how many were excluded at each stage, and the reasons for exclusion. This transparency allows readers to assess whether your search was likely to have captured the available evidence.

The National Library of Medicine hosts research methods resources that describe best practices for systematic review conduct, including screening and data extraction procedures. These resources can help you structure your screening process so that it is reproducible and defensible.

Step 4: Assess the Risk of Bias in Included Studies

Before you can interpret publication bias, you need to understand the quality of the studies you have included. Use a validated risk of bias tool appropriate for your study designs. Studies with high risk of bias may contribute distorted results regardless of publication status.

Your risk of bias assessment should be reported clearly in your review, and you should consider whether to exclude high-risk studies or conduct sensitivity analyses that test whether your conclusions change when they are removed.

Step 5: Use Funnel Plots to Visualize Small-Study Effects

A funnel plot is a scatter plot that displays each study's effect estimate against a measure of its precision, usually the standard error or sample size. In the absence of bias, the plot should resemble a symmetrical inverted funnel, with smaller studies scattered more widely at the bottom and larger studies clustered near the pooled estimate at the top.

Asymmetry in a funnel plot can indicate publication bias, but it can also result from other causes, including true heterogeneity between studies, poor methodological quality in smaller studies, or chance. You should interpret funnel plots cautiously and consider whether the asymmetry is consistent with a plausible mechanism.

Step 6: Apply Statistical Tests for Funnel Plot Asymmetry

Several statistical tests can quantify funnel plot asymmetry. These tests evaluate whether the association between effect size and precision is stronger than would be expected by chance. Common approaches include Egger's regression test and Begg's rank correlation test.

These tests have limited power when the number of studies is small, and they can produce false positives when heterogeneity is present. You should treat them as diagnostic tools instead of definitive proof of publication bias, and you should report their results alongside your funnel plot.

Step 7: Estimate the Impact of Missing Studies

Trim-and-fill is a method that estimates how many studies are missing from a meta-analysis and imputes their likely effect sizes. The method trims asymmetric studies from one side of the funnel plot, recalculates the pooled estimate, and then fills in the missing studies to create a more symmetric plot.

The adjusted estimate from trim-and-fill gives you a sense of how your conclusions might change if the missing studies were available. However, the method makes assumptions about the mechanism of missingness, and its results should be interpreted as a sensitivity analysis instead of a correction.

Step 8: Conduct Sensitivity Analyses

Sensitivity analyses test whether your conclusions are robust to different analytical decisions. You might compare results with and without small studies, with and without studies at high risk of bias, or using different statistical methods for pooling.

These analyses help you understand how much your conclusions depend on specific studies or methodological choices. If your conclusions change substantially under plausible alternative assumptions, you should report this uncertainty clearly.

At a Glance

Options and Tradeoffs in Publication Bias Assessment

Choosing Between Statistical Tests

No single statistical test is universally superior. Egger's regression test is widely used and has reasonable statistical properties, but it assumes a linear relationship between effect size and precision. Begg's rank correlation test is less powerful but makes fewer assumptions. You should select tests based on the characteristics of your data and report your rationale.

When you have fewer than ten studies, statistical tests have limited power to detect asymmetry. In this situation, you should rely more heavily on visual inspection of the funnel plot and on qualitative assessment of whether your search was likely to have captured the available evidence.

Interpreting Funnel Plot Asymmetry

Funnel plot asymmetry does not prove publication bias. Small studies may differ from large studies in important ways, including patient populations, intervention protocols, or methodological quality. These differences can produce asymmetry even when all studies are published.

You should consider alternative explanations for asymmetry and report them honestly. If you find asymmetry, you should explore whether it is driven by specific studies or subgroups and whether it is consistent with a plausible mechanism.

Deciding Whether to Adjust Your Estimates

Statistical adjustment methods like trim-and-fill are controversial. Some methodologists argue that adjustment is preferable to ignoring the problem, while others argue that the assumptions required for adjustment are too strong to justify changing your estimates.

A reasonable approach is to report both the unadjusted and adjusted estimates and to describe the difference between them. This transparency allows readers to judge for themselves how much confidence to place in your conclusions.

Records and Measurements for Publication Bias Assessment

What to Document in Your Review Records

Your review records should allow another researcher to reproduce your search and screening decisions. At minimum, document the following:

• Databases and other sources searched, with the date of each search
• Complete search strategies for each database, including all search terms and filters
• Number of records identified from each source
• Number of records excluded at each screening stage, with reasons for exclusion
• Full text articles assessed for eligibility and reasons for exclusion
• Data extraction forms and any decisions made during extraction
• Risk of bias assessments for each included study
• Statistical analyses performed, including any sensitivity analyses

The National Institutes of Health provides guidance on data management and sharing expectations for funded research. While this guidance is primarily directed at NIH grantees, the underlying principles of transparency and reproducibility apply to all systematic reviews.

Measuring the Completeness of Your Search

You can assess the completeness of your search by checking whether you have identified key studies that are known to exist in the field. Citation tracking and consultation with experts can help you identify studies that your database searches may have missed.

You should also consider searching for studies that are ongoing or unpublished. Clinical trial registries and other study registries can help you identify research that has been conducted but not yet published.

Common Failure Patterns in Publication Bias Assessment

Failure to Search Grey Literature

Many reviews rely exclusively on peer-reviewed journal articles indexed in major databases. This approach systematically misses conference abstracts, theses, government reports, and other documents that may contain relevant evidence. The result is a review that overrepresents positive findings and misses important negative or null results.

Misinterpreting Funnel Plot Asymmetry

Researchers sometimes treat funnel plot asymmetry as definitive proof of publication bias. This interpretation is problematic because asymmetry can result from true heterogeneity, differences in study quality, or chance. You should always consider alternative explanations and report your assessment of their plausibility.

Overreliance on Statistical Adjustment

Trim-and-fill and similar methods are sometimes presented as corrections that eliminate publication bias. In reality, these methods make strong assumptions about the mechanism of missingness, and their results depend on those assumptions. You should treat adjusted estimates as sensitivity analyses instead of definitive corrections.

Ignoring Selective Reporting Within Studies

Publication bias assessment focuses on missing studies, but selective reporting within published studies is equally problematic. A study may report only favorable outcomes while omitting others, creating a distorted picture even when the study itself is published. You should check whether all outcomes specified in the study protocol are reported in the results.

Failing to Register the Review Protocol

Without a registered protocol, you have no public record of your planned methods. This makes it difficult for readers to assess whether your methods were determined in advance or adjusted after seeing the results. Protocol registration is a simple step that substantially improves the credibility of your review.

Limitations of Publication Bias Assessment

Statistical Tests Have Limited Power

All statistical tests for publication bias have limited power, particularly when the number of studies is small. A non-significant test does not prove the absence of publication bias, and a significant test does not prove its presence. You should interpret these tests as part of a broader assessment that includes qualitative judgment.

You Cannot Detect What You Cannot Find

Publication bias assessment can only work with the evidence that is available to you. If studies are missing from the public record entirely, you cannot know their results or even their existence. Your assessment is therefore inherently limited by the completeness of your search.

Methods Are Still Evolving

Statistical methods for detecting and adjusting for publication bias continue to evolve. New methods are developed as methodologists better understand the mechanisms of bias and the properties of different approaches. You should stay current with methodological developments and be prepared to update your approach as the field advances.

Safety and Regulatory Context

Transparency Requirements in Funded Research

The National Institutes of Health requires researchers to plan for data management and sharing as part of their funding applications. While this policy is specific to NIH-funded research, it reflects a broader expectation that research data should be shared to support reproducibility and evidence synthesis.

For systematic reviewers, this means that primary studies should be expected to share their data. When data are not available, you should report this limitation and consider its potential impact on your conclusions.

Researcher Identity and Accountability

The ORCID for Researchers resource describes how persistent researcher identifiers support accurate attribution of research outputs. For systematic reviewers, ORCID records can help you identify all publications by a given researcher, which may be useful when you are trying to locate unpublished or hard-to-find studies.

Publication Ethics Expectations

The Committee on Publication Ethics core practices describe expectations for authors, reviewers, and editors regarding responsible research conduct. These practices include expectations around data integrity, conflict of interest disclosure, and handling of misconduct. For systematic reviewers, understanding these expectations helps you assess the reliability of the primary studies you are synthesizing.

Professional Escalation Criteria

When to Consult a Methodologist

You should consider consulting a statistician or systematic review methodologist in the following situations:

• Your funnel plot shows marked asymmetry that you cannot explain
• Statistical tests for publication bias produce conflicting results
• You are considering whether to adjust your pooled estimates
• Your review includes a small number of studies with substantial heterogeneity
• You are uncertain about which statistical methods are appropriate for your data

When to Report Concerns to Editors or Funders

If you identify evidence of possible research misconduct, including data fabrication or selective reporting that appears intentional, you should consider reporting your concerns to the relevant journal editors or institutional officials. The Committee on Publication Ethics provides guidance on handling allegations of misconduct.

When to Broaden Your Search

If your initial search identifies very few studies, or if you suspect that important evidence may be missing, you should broaden your search before concluding that the evidence base is small. This may involve searching additional databases, contacting researchers directly, or searching for unpublished documents.

Frequently Asked Questions

What is the difference between publication bias and reporting bias?

Publication bias refers to the selective publication of studies based on the direction or significance of their results. Reporting bias refers to the selective reporting of outcomes within a published study. Both can distort the evidence base, but they operate at different levels and require different detection strategies.

How many studies do I need before I can assess publication bias?

Statistical tests for publication bias have limited power with fewer than ten studies. With small numbers of studies, you should rely more heavily on visual inspection of funnel plots and qualitative assessment of your search completeness. You should still report your assessment methods even when statistical power is limited.

Can I correct for publication bias in my meta-analysis?

Statistical methods like trim-and-fill can estimate the potential impact of missing studies, but they cannot truly correct for publication bias. These methods make assumptions about the mechanism of missingness that may not hold in practice. You should report both adjusted and unadjusted estimates and describe the difference between them.

What is grey literature and why does it matter?

Grey literature includes documents that are not published in peer-reviewed journals, such as conference abstracts, theses, government reports, and working papers. Searching grey literature is important because it can identify studies that were never published in journals, which may include null or negative findings that would otherwise be missing from your review.

Should I register my systematic review protocol?

Yes. Registering your protocol before you begin searching creates a public record of your planned methods and reduces the risk that you will unconsciously adjust your methods after seeing the results. Protocol registration also allows readers to compare your final report against your original plan.

What should I do if my funnel plot is asymmetric?

First, consider alternative explanations for the asymmetry, including true heterogeneity, differences in study quality, or chance. Then conduct sensitivity analyses to test whether your conclusions change when you exclude small studies or studies at high risk of bias. Report your findings transparently and describe the uncertainty in your conclusions.

How do I find unpublished studies for my review?

Search multiple databases, including those that index conference abstracts and theses. Search clinical trial registries and other study registries to identify ongoing or completed but unpublished studies. Contact researchers working in the field and ask about unpublished work. Check reference lists of included studies for citations to unpublished documents.

What reporting guidelines should I use for my systematic review?

The EQUATOR Network provides a searchable database of reporting guidelines for health research. For systematic reviews, the PRISMA statement is the most widely used guideline. You should select the guideline that is most appropriate for your study design and follow it throughout your review.

Using the Evidence

Related Articles

• Measurement Bias in Research: Types, Detection, and Mitigation
• Meta-Analysis vs Systematic Review: What's the Difference?
• Methodology Limitations in Research: How to Identify and Address Them
• Publication Bias in Meta-Analysis: Prevention, Assessment, and Adjustments
• Scientific Literature Review Format: A Reproducible Structure for Researchers

References and Further Reading

• Research Methods Resources. National Library of Medicine.
• EQUATOR Network. EQUATOR Network.
• Core Practices. Committee on Publication Ethics.
• NIH Grants and Funding. National Institutes of Health.
• ORCID for Researchers. ORCID.
• Data Management and Sharing Policy. National Institutes of Health.
• NCBI Literature Resources. National Center for Biotechnology Information.
• PubMed. National Library of Medicine.

This article is educational and does not replace institutional policy, professional advice, or applicable safety and regulatory requirements.