# Deciding When to Pool Results in a Meta-Analysis ## Quick Answer - Meta-analysis is appropriate when your systematic review includes studies with sufficient clinical and methodological homogeneity to justify statistical pooling of their results. - Before pooling, assess study design, populations, interventions, comparators, and outcomes for comparability, and document your decisions in a pre-specified protocol. - Statistical pooling is not appropriate when studies differ substantially in design, population, or outcome measurement, even if the data appear numerically combinable. ## At a Glance | Decision Point | What to Assess | Action if Criteria Are Met | Action if Criteria Are Not Met | |---|---|---|---| | Clinical homogeneity | Similarity of participants, interventions, comparators, and outcomes across studies | Proceed to statistical pooling with a fixed or random effects model | Present a narrative synthesis instead of a pooled estimate | | Methodological homogeneity | Consistency in study design, risk of bias, and outcome measurement tools | Pool studies with sensitivity analyses to test robustness | Report results separately by design or measurement approach | | Data availability | Sufficient reported data (means, standard deviations, event counts, or effect sizes) | Extract data using a standardized form and verify entries | Contact authors or use imputation only with clear justification | | Reporting quality | Adherence to reporting guidelines such as PRISMA or other EQUATOR-listed checklists | Use the guideline to structure your methods and results | Document deviations and assess their impact on pooled estimates | ## Understanding Meta-Analysis in Context Meta-analysis is a statistical technique used within a systematic review to combine quantitative results from multiple studies addressing the same question. The goal is to produce a single estimate of effect with greater precision than any individual study can provide. However, the validity of that estimate depends entirely on whether the studies are similar enough to be meaningfully combined. The decision to perform a meta-analysis is not primarily statistical. It is a clinical and methodological judgment made before any pooling calculations begin. Researchers often assume that meta-analysis is an automatic step after completing a systematic literature search. In practice, the decision requires careful evaluation of whether the included studies are asking the same question in comparable ways. Systematic reviews and meta-analyses are governed by established standards for transparency and reproducibility. The [EQUATOR Network](https://www.equator-network.org/) maintains a comprehensive library of reporting guidelines that help researchers select the appropriate checklist for their study type. For systematic reviews with meta-analysis, the PRISMA statement is the most commonly used guideline, though other checklists may be more appropriate depending on the review design. The [National Library of Medicine](https://www.ncbi.nlm.nih.gov/books) provides access to authoritative biomedical texts that describe the methodological foundations of systematic review and meta-analysis. These resources emphasize that the quality of a meta-analysis depends on the quality of the systematic review that precedes it. A meta-analysis cannot correct for poor search strategies, biased study selection, or inadequate data extraction. ## Core Principles for Deciding When to Pool ### Clinical Homogeneity Clinical homogeneity refers to the similarity of the research question across studies. This includes the characteristics of the participants, the nature of the interventions being compared, the comparators used, and the outcomes measured. Studies that differ substantially on any of these dimensions may not be appropriate to combine. Consider a review of antibiotic treatment for bovine respiratory disease in feedlot cattle. If one study enrolls high-risk calves recently arrived at the feedlot and another enrolls cattle that have been on feed for 60 days, the populations differ in ways that could affect treatment response. Similarly, if one study measures treatment success as resolution of fever at 72 hours and another measures it as average daily gain over 28 days, the outcomes are not directly comparable. Clinical homogeneity does not require that studies be identical. Some variation in participant characteristics, dosing regimens, or outcome definitions is expected. The question is whether the variation is small enough that a single pooled estimate provides a meaningful summary. When variation is substantial, the pooled estimate may obscure important differences between subgroups. ### Methodological Homogeneity Methodological homogeneity concerns the design and conduct of the studies. Randomized controlled trials and observational studies answer different questions and have different susceptibility to bias. Combining them in a single meta-analysis requires careful justification and is often better addressed through separate analyses. Study design features that affect methodological homogeneity include: - Randomization method and allocation concealment - Blinding of participants, caregivers, and outcome assessors - Sample size and statistical power - Duration of follow-up - Handling of missing data and withdrawals The [Committee on Publication Ethics](https://publicationethics.org/core-practices) core practices emphasize the importance of transparent reporting of research methods. When studies do not report key methodological details, assessing homogeneity becomes difficult. Researchers should attempt to contact study authors for missing information before deciding whether pooling is appropriate. ### Statistical Heterogeneity Statistical heterogeneity measures the variability in treatment effects across studies beyond what would be expected by chance alone. Common metrics include the I-squared statistic and the Q statistic. High statistical heterogeneity suggests that studies may not be estimating the same underlying effect. Statistical heterogeneity is not the primary criterion for deciding whether to pool. Even when statistical heterogeneity is low, pooling studies that are clinically or methodologically different can produce a misleading estimate. Conversely, high statistical heterogeneity does not automatically preclude pooling if the clinical and methodological questions are sufficiently similar and the heterogeneity can be explored through subgroup analysis or meta-regression. The decision to pool should be made before examining statistical heterogeneity. This prevents the common error of letting the data determine whether studies are comparable. The clinical and methodological assessment comes first, and statistical measures are used to describe and explore variability within the pooled analysis. ## Practical Workflow for the Pooling Decision ### Step 1: Define the Research Question with PICO Before searching for studies, specify the Population, Intervention, Comparator, and Outcomes. This framework guides the search strategy and provides the criteria for determining whether studies are clinically homogeneous. For animal farming research, the population might be defined by species, age class, production stage, or health status. The intervention could be a feed additive, vaccine, management practice, or therapeutic agent. The comparator is typically a placebo, no treatment, or an alternative intervention. Outcomes should be defined with specific measurement methods and time points. A well-defined PICO question makes the homogeneity assessment more objective. When studies are selected based on clear inclusion criteria, the likelihood of inappropriate pooling decreases. ### Step 2: Conduct a Systematic Literature Search The search strategy should be comprehensive and reproducible. Multiple databases should be searched, and the search terms should be documented in the final report. The [National Library of Medicine](https://www.ncbi.nlm.nih.gov/books) provides guidance on developing search strategies for biomedical databases. Search results should be screened against the pre-specified inclusion criteria. Screening should be performed by at least two reviewers independently, with disagreements resolved through discussion or a third reviewer. The screening process should be documented using a PRISMA flow diagram. ### Step 3: Assess Risk of Bias in Included Studies Each included study should be assessed for risk of bias using a validated tool appropriate to the study design. The Cochrane RoB 2 instrument evaluates internal validity of randomised trials across five structured domains: randomisation, allocation concealment, blinding, missing data, and selective reporting. For observational studies, tools such as ROBINS-I may be more appropriate. Risk of bias assessment informs the homogeneity decision in two ways. First, studies with high risk of bias may be excluded from the primary analysis or included only in sensitivity analyses. Second, differences in risk of bias across studies can explain statistical heterogeneity and should be explored. ### Step 4: Extract Data Using a Standardized Form Data extraction should capture all information needed for the analysis, including study characteristics, participant demographics, intervention details, outcome data, and funding sources. A standardized extraction form reduces errors and ensures consistency across studies. The [National Institutes of Health](https://sharing.nih.gov/data-management-and-sharing-policy) data management and sharing policy emphasizes the importance of planning for data collection and documentation. For systematic reviews, this includes documenting the data extraction process and storing extraction forms for transparency. ### Step 5: Assess Clinical and Methodological Homogeneity Using the extracted data, compare the studies on the key dimensions of clinical and methodological homogeneity. Create a table summarizing study characteristics to facilitate this comparison. Look for patterns of similarity and difference that would support or argue against pooling. This assessment should be documented in the methods section of the final report. Reviewers should be able to understand why certain studies were pooled and others were not. ### Step 6: Decide Whether to Pool Based on the homogeneity assessment, make an explicit decision about whether to proceed with meta-analysis. This decision should be made before examining the statistical results. If pooling is not appropriate, present a narrative synthesis that describes the findings of individual studies and explores reasons for differences. If pooling is appropriate, proceed with the statistical analysis. Choose the effect measure, the statistical model, and the methods for exploring heterogeneity before running the analysis. ### Step 7: Explore Heterogeneity and Conduct Sensitivity Analyses Even when pooling is justified, explore sources of heterogeneity through subgroup analyses or meta-regression. These analyses can identify characteristics that modify the treatment effect and provide more nuanced conclusions. Sensitivity analyses test the robustness of the pooled estimate to decisions made during the review process. Common sensitivity analyses include excluding studies with high risk of bias, changing the statistical model, or using different effect measures. ## Options and Tradeoffs in the Pooling Decision ### Fixed Effects versus Random Effects Models The choice between fixed effects and random effects models depends on assumptions about the true treatment effect. A fixed effects model assumes that all studies estimate the same underlying effect and that differences are due to sampling error alone. A random effects model assumes that the true effect varies across studies and incorporates between-study variability into the analysis. The random effects model is generally more conservative and produces wider confidence intervals when heterogeneity is present. However, it gives more weight to smaller studies compared to the fixed effects model. The choice should be specified in the protocol and justified based on the expected similarity of the studies. ### Narrative Synthesis as an Alternative When studies are too heterogeneous to pool, narrative synthesis is the appropriate alternative. This approach involves describing the findings of individual studies, organizing them by intervention type, population, or outcome, and exploring patterns and explanations for differences. Narrative synthesis requires the same systematic approach as meta-analysis. The synthesis should be structured, transparent, and grounded in the extracted data. The [EQUATOR Network](https://www.equator-network.org/) provides guidance on reporting systematic reviews that do not include meta-analysis. ### Subgroup Analysis and Meta-Regression Subgroup analysis divides studies into groups based on a characteristic such as age class, production system, or intervention dose. Meta-regression examines the relationship between study-level characteristics and treatment effects. These approaches can be useful when heterogeneity is present but the studies are otherwise sufficiently similar to pool. However, they should be planned in advance and limited to a small number of pre-specified characteristics to avoid false findings from multiple testing. ## Observations and Measurements That Support the Decision ### Documenting Study Characteristics A study characteristics table is essential for the homogeneity assessment. This table should include for each study: - Author and year - Study design - Population and setting - Intervention and comparator details - Outcome measures and time points - Sample size - Risk of bias rating This table serves as the evidence base for the pooling decision and should be included in the final report. ### Recording Heterogeneity Statistics When pooling is performed, record the statistical heterogeneity measures, including the I-squared statistic and the Q statistic with its p-value. These values describe the proportion of variability in effect estimates that is due to between-study differences instead of sampling error. Interpretation of these statistics should be cautious. The I-squared statistic can be imprecise when the number of studies is small. Confidence intervals for I-squared should be reported when possible. ### Tracking Decisions and Rationale Maintain a decision log that documents each step of the review process, including the rationale for including or excluding studies, the homogeneity assessment, and the choice of statistical methods. This log supports transparency and reproducibility. The [Committee on Publication Ethics](https://publicationethics.org/core-practices) core practices emphasize the importance of accurate and complete reporting of research methods. A decision log helps ensure that the methods section of the final report reflects what was actually done. ## Records and Documentation Requirements ### Protocol Registration Registering the systematic review protocol before starting the review reduces the risk of reporting bias and duplication. Protocol registration is expected by many journals and funding agencies. The [National Institutes of Health](https://grants.nih.gov/) provides guidance on the expectations for rigorous research methods in funded projects. The protocol should specify the research question, inclusion criteria, search strategy, risk of bias assessment methods, and the planned approach to synthesis. The protocol should also state the criteria for deciding whether meta-analysis is appropriate. ### Data Management and Sharing The [National Institutes of Health](https://sharing.nih.gov/data-management-and-sharing-policy) data management and sharing policy requires that research data be managed and shared in ways that are consistent with scientific rigor and transparency. For systematic reviews, this includes making the search strategy, screening decisions, extraction forms, and analysis code available. Data sharing supports the reproducibility of the review and allows others to verify the findings. It also facilitates updates to the review when new studies become available. ### Reporting Guidelines Select the appropriate reporting guideline from the [EQUATOR Network](https://www.equator-network.org/) library. For systematic reviews with meta-analysis, PRISMA is the standard. For other review types, alternative guidelines may be more appropriate. The reporting guideline should be followed throughout the writing of the final report. This ensures that all essential elements are included and that the report meets the expectations of journals and readers. ## Common Failure Patterns in the Pooling Decision ### Pooling Without Assessing Homogeneity The most common error is proceeding directly to statistical pooling without a formal assessment of clinical and methodological homogeneity. This can produce a pooled estimate that is precise but meaningless because the studies are not asking the same question. Prevention requires a structured homogeneity assessment as described above. The assessment should be documented and reported, even when the conclusion is that pooling is appropriate. ### Using Statistical Heterogeneity as the Sole Criterion Some researchers decide whether to pool based solely on the I-squared statistic or the Q test. This approach is problematic because statistical heterogeneity is influenced by sample size and can be low even when studies are clinically different. Statistical heterogeneity should be described and explored, but it should not be the primary determinant of the pooling decision. The clinical and methodological assessment comes first. ### Combining Different Study Designs Pooling randomized trials and observational studies in a single meta-analysis is generally inappropriate because these designs answer different questions and have different susceptibility to bias. If both types of studies are included in the review, they should be analyzed separately. The protocol should specify whether the review will include only randomized trials, only observational studies, or both. If both are included, the analysis should keep them separate. ### Ignoring Risk of Bias Differences Studies with high risk of bias can distort the pooled estimate. Including such studies without sensitivity analyses can lead to conclusions that are not supported by the evidence. Risk of bias should be assessed for all included studies, and the results should inform the analysis. Sensitivity analyses excluding high-risk studies should be performed to test the robustness of the findings. ### Overinterpreting Subgroup Analyses Subgroup analyses that were not pre-specified are prone to false findings. When many subgroup analyses are performed, some will be statistically significant by chance alone. Subgroup analyses should be limited to a small number of pre-specified characteristics. Findings from subgroup analyses should be interpreted cautiously and described as hypothesis-generating instead of definitive. ## Limitations and Professional Escalation Criteria ### When to Seek Methodological Expertise Systematic review and meta-analysis methods are complex, and errors in the pooling decision can invalidate the findings. Consider consulting a statistician or systematic review methodologist when: - The included studies are highly diverse in design or population - The outcome measures are heterogeneous or poorly defined - The number of included studies is small - Statistical heterogeneity is substantial and difficult to explain - The review addresses a topic with important policy or clinical implications The [National Library of Medicine](https://www.ncbi.nlm.nih.gov/books) provides access to methodological references that can support the review process. However, professional consultation is advisable when the review is complex or the stakes are high. ### When to Abandon Pooling Abandon the meta-analysis and present a narrative synthesis when: - Studies differ substantially in population, intervention, comparator, or outcome - The number of studies is too small to support meaningful statistical analysis - The data are insufficiently reported to extract comparable effect estimates - Statistical heterogeneity is extreme and cannot be explained by pre-specified subgroup analyses The decision to abandon pooling should be documented and explained in the final report. A narrative synthesis that is honest about limitations is more valuable than a meta-analysis that is statistically inappropriate. ### When to Escalate to a Funder or Institution If the review is part of a funded project, the funder may have specific expectations for the methods and reporting. The [National Institutes of Health](https://grants.nih.gov/) provides guidance on the conduct of rigorous research and the expectations for transparency and reproducibility. If the review findings have implications for animal health, welfare, or production decisions, consider consulting with a veterinarian or other relevant professional before disseminating the results. The [Committee on Publication Ethics](https://publicationethics.org/core-practices) provides guidance on responsible research conduct and the handling of findings that may have practical consequences. ## Welfare and Safety Context ### Animal Welfare Considerations Systematic reviews in animal farming often address questions related to animal health and welfare. The findings of these reviews can inform management decisions that affect the welfare of large numbers of animals. It is important that the review methods are rigorous enough to support the conclusions. When the review addresses a welfare-related question, consider whether the outcome measures capture relevant welfare indicators. Outcomes such as mortality, morbidity, and behavioral measures may be more relevant than production outcomes alone. ### Food Safety and Public Health Context Reviews addressing antimicrobial use, vaccine efficacy, or disease control in food animals have implications for food safety and public health. The conclusions of these reviews should be communicated carefully, with appropriate attention to uncertainty and limitations. The [National Institutes of Health](https://grants.nih.gov/) supports research that addresses public health priorities, including the responsible use of antimicrobials in animal agriculture. Researchers conducting reviews in this area should be aware of the broader context and the potential implications of their findings. ### Responsible Communication of Findings The findings of a systematic review with meta-analysis should be communicated accurately and without overstatement. The limitations of the review should be described clearly, and the conclusions should be consistent with the strength of the evidence. The [Committee on Publication Ethics](https://publicationethics.org/core-practices) core practices emphasize the importance of accurate reporting and the avoidance of misleading claims. This applies to the final report, presentations, and any communication with stakeholders. ## Frequently Asked Questions ### How do systematic literature reviews differ conceptually from quantitative meta-analyses? A systematic review uses pre-registered protocols, comprehensive database searching, and transparent critical appraisal to synthesise all relevant evidence on a defined research question. ### Can I perform a meta-analysis if my studies are clinically different? Clinical differences between studies do not automatically preclude meta-analysis, but they require careful consideration. If the differences are substantial enough that a single pooled estimate would not provide a meaningful summary, a narrative synthesis is more appropriate. The decision should be based on the specific characteristics of the studies and the research question. ### How many studies do I need for a meta-analysis? No absolute threshold governs the minimum number of studies for a meta-analysis; however, syntheses of fewer than five trials typically yield unstable heterogeneity estimates and wide confidence intervals. ### How is the I-squared (I²) metric computed and interpreted when assessing heterogeneity? The I-squared statistic describes the percentage of variability in effect estimates that is due to between-study heterogeneity instead of sampling error. Higher values indicate greater heterogeneity. However, I-squared should not be used as the sole criterion for deciding whether to pool. The clinical and methodological assessment of homogeneity comes first. ### Should I combine randomized trials and observational studies in one meta-analysis? Combining randomized trials and observational studies in a single meta-analysis is generally not recommended because these designs answer different questions and have different susceptibility to bias. If both types of studies are included in the review, they should be analyzed separately and the results presented distinctly. ### What should I do if my studies are too heterogeneous to pool? If the studies are too heterogeneous to pool, present a narrative synthesis that describes the findings of individual studies and explores patterns and explanations for differences. The synthesis should be structured and transparent, and the reasons for not pooling should be documented in the methods section. ### How do I report a systematic review without meta-analysis? Use the appropriate reporting guideline from the [EQUATOR Network](https://www.equator-network.org/) library. For systematic reviews without meta-analysis, the PRISMA statement can still be used, with the synthesis methods described as narrative. The report should explain why meta-analysis was not performed and describe the approach to narrative synthesis. ### Where can I find guidance on the methods for systematic review and meta-analysis? The [National Library of Medicine](https://www.ncbi.nlm.nih.gov/books) provides access to authoritative biomedical texts on research methods. The [EQUATOR Network](https://www.equator-network.org/) provides reporting guidelines, and the [Committee on Publication Ethics](https://publicationethics.org/core-practices) provides guidance on responsible research conduct. ## Using the Evidence | Source | Best use in this topic | Important limitation | |---|---|---| | [Research Methods Resources](https://www.ncbi.nlm.nih.gov/books) | official guidance | Check the linked page for current local requirements | | [EQUATOR Network](https://www.equator-network.org/) | official guidance | Check the linked page for current local requirements | | [Core Practices](https://publicationethics.org/core-practices) | official guidance | Check the linked page for current local requirements | ## Related Articles - [Meta-Analysis vs Systematic Review: What's the Difference?](/blog/guides/meta-analysis-vs-systematic-review-what-s-the-difference) - [Meta-Analysis vs Systematic Review: Choosing the Right Synthesis Method](/blog/guides/meta-analysis-vs-systematic-review-choosing-the-right-synthesis-method) - [Scoping Review vs Meta-Analysis: Clarifying Their Distinct Roles in Evidence Synthesis](/blog/guides/scoping-review-vs-meta-analysis-clarifying-their-distinct-roles-in-evidence-synthesis) - [Statistical Data Analysis](/blog/guides/statistical-data-analysis) - [Meta-Analysis vs Peer Review: Understanding the Differences](/blog/guides/meta-analysis-vs-peer-review-understanding-the-differences) ## References and Further Reading - [Research Methods Resources](https://www.ncbi.nlm.nih.gov/books). National Library of Medicine. - [EQUATOR Network](https://www.equator-network.org/). EQUATOR Network. - [Core Practices](https://publicationethics.org/core-practices). Committee on Publication Ethics. - [NIH Grants and Funding](https://grants.nih.gov/). National Institutes of Health. - [ORCID for Researchers](https://info.orcid.org/researchers). ORCID. - [Data Management and Sharing Policy](https://sharing.nih.gov/data-management-and-sharing-policy). National Institutes of Health. - [NCBI Literature Resources](https://www.ncbi.nlm.nih.gov/). National Center for Biotechnology Information. - [PubMed](https://pubmed.ncbi.nlm.nih.gov/). National Library of Medicine. > This article is educational and does not replace institutional policy, professional advice, or applicable safety and regulatory requirements.