The Unspoken Statistical Concern

By Dr. Zubair Khalid, DVM, MS, PhD · 2026-08-27

Quick Answer

• Reviewers often flag statistical problems indirectly, so authors must read comments for implied concerns about power, multiple testing, and data handling before drafting a response.
• Build a revision checklist that addresses hidden statistical issues proactively, then respond point by point with specific changes made to the manuscript.
• A statistical review comment may signal a deeper problem with study design or reporting that requires consultation with a biostatistician before resubmission.

Understanding the Unspoken Statistical Concern

Peer review comments about statistics rarely arrive as direct commands. A reviewer might write "the analysis seems underpowered" or "please clarify how you handled missing data" without specifying the exact statistical remedy. These indirect signals create a distinct problem for authors: how do you fix a problem that was never fully described?

The unspoken statistical concern sits between what reviewers write and what they expect. Reviewers often assume authors understand standard statistical conventions, reporting expectations, and the logic behind common analytical choices. When those conventions are violated, reviewers may respond with vague phrasing that leaves authors guessing about the actual flaw.

Statistical review comments typically fall into recognizable patterns. Some point to insufficient sample sizes, others question the number of comparisons made, and many ask for clarification about how data were collected or processed. Each pattern requires a different response strategy, and recognizing the pattern is the first step toward a successful revision.

The practical problem is that most researchers receive limited formal training in statistical communication. A National Library of Medicine gateway to research methods references shows that statistical literacy is treated as a specialized skill instead of a core competency. Authors who understand the common hidden concerns can respond more effectively and avoid prolonged revision cycles.

Why Reviewers Leave Statistical Concerns Implicit

Reviewers operate under time constraints and often lack complete information about the study. They see the manuscript as submitted, not the full context of how data were collected, cleaned, and analyzed. This partial view leads to comments that hint at concerns without fully articulating them.

Another reason for implicit comments is that reviewers may not be statistical specialists. A subject-matter expert reviewing a manuscript might sense that something is wrong with the analysis but cannot articulate the precise statistical issue. Their comment becomes a general request for clarification instead of a specific directive.

The Committee on Publication Ethics core practices emphasize that peer review should be fair and constructive. Reviewers are encouraged to provide useful feedback, but the practical reality is that many comments arrive as questions instead of instructions. Authors must interpret these questions within the context of their own study design.

Reviewers also avoid prescribing specific statistical methods because they cannot know the full data structure. A comment asking "did you consider adjusting for multiple comparisons" is not a demand for a particular correction method. It is an invitation for the author to demonstrate that they thought about the issue and made a defensible choice.

Common Hidden Statistical Concerns in Review Comments

Insufficient Statistical Power

Power concerns appear in comments like "the sample size seems small" or "I worry about the ability to detect meaningful effects." These comments point to the relationship between sample size, effect size, and the probability of detecting a true effect.

Authors should respond by reporting the power analysis that was conducted before data collection, or by acknowledging the limitation and discussing its implications. If no power analysis was performed, the response should explain the basis for the sample size and address whether the study was exploratory.

The response should include specific numbers: the assumed effect size, the alpha level, the target power, and the resulting sample size calculation. If the study was underpowered, the author should state this clearly and discuss how it affects interpretation of the results.

Multiple Testing and Multiplicity

Comments about multiple comparisons often appear as "you performed many tests" or "did you correct for multiple testing?" These concerns arise when authors conduct numerous statistical tests without adjusting the significance threshold.

The response should list every test performed, explain the rationale for each, and describe any correction methods applied. If no correction was used, the author should justify this choice or acknowledge the increased risk of false positives.

Authors should also distinguish between confirmatory and exploratory analyses. Confirmatory tests test pre-specified hypotheses and require stricter control of error rates. Exploratory analyses generate hypotheses and can be reported with appropriate caveats about their preliminary nature.

Data Distribution and Model Assumptions

Reviewers may ask "did you check the assumptions" or "why did you use a parametric test?" These comments point to the statistical assumptions underlying the chosen analytical methods.

The response should describe how assumptions were assessed, including any diagnostic tests or graphical checks performed. If assumptions were violated, the author should explain what alternative methods were considered and why the chosen approach was retained.

Common assumptions include normality, homogeneity of variance, independence of observations, and linearity. Each assumption requires a specific diagnostic approach, and the response should demonstrate that the author understands and addressed each one.

Missing Data Handling

Comments about missing data often appear as "how did you handle missing values?" or "were there dropouts?" These concerns address how incomplete data were managed during analysis.

The response should describe the extent of missing data, the mechanism believed to be responsible, and the method used to handle it. Common approaches include complete case analysis, imputation, and model-based methods that accommodate missingness.

Authors should also discuss sensitivity analyses that examine whether results change under different missing data assumptions. This demonstrates that the handling of missing data was thoughtful instead of arbitrary.

Outliers and Influential Points

Reviewers may ask "did you check for outliers?" or "were there any unusual observations?" These comments point to the potential for extreme values to distort statistical results.

The response should describe how outliers were identified, whether they were retained or removed, and the rationale for the decision. If outliers were removed, the author should report results with and without them to show the impact.

Influential points are observations that disproportionately affect regression results. Authors should describe diagnostic measures used to identify them and discuss their influence on the conclusions.

Confounding and Covariate Adjustment

Comments about confounders appear as "did you adjust for other variables?" or "what about potential confounders?" These concerns address whether observed associations might be explained by variables not included in the analysis.

The response should list potential confounders considered, explain how they were measured, and describe the adjustment strategy. Authors should also discuss residual confounding and its potential impact on the conclusions.

Reporting of Effect Sizes and Uncertainty

Reviewers may ask "please report effect sizes" or "what are the confidence intervals?" These comments point to the need for measures that convey the magnitude and precision of effects, beyond p-values.

The response should include effect sizes with confidence intervals for all primary analyses. This allows readers to assess both statistical significance and practical importance.

Reproducibility and Transparency

Comments about reproducibility appear as "please provide your analysis code" or "can you share the data?" These concerns address the ability of others to verify and build upon the reported results.

The response should describe what materials can be shared, how they can be accessed, and any restrictions on sharing. Authors should also describe their analysis workflow to demonstrate that results are reproducible.

The Reviewer Response Letter Structure

A well-structured response letter addresses each comment directly and demonstrates that the author has taken the review seriously. The letter should begin with a summary of the major changes made, followed by point-by-point responses to each comment.

Each response should include the original comment, the author's understanding of the concern, the specific action taken, and where the change appears in the revised manuscript. This structure allows reviewers to quickly verify that their concerns were addressed.

The National Institutes of Health grant review process emphasizes the importance of responding directly to reviewer concerns. The same principle applies to journal peer review: reviewers want to see that their comments were understood and acted upon.

When responding to statistical comments, authors should avoid defensive language and instead focus on what was learned from the review process. Even when the author disagrees with a comment, the response should acknowledge the reviewer's perspective and explain the reasoning behind the original choice.

Building a Statistical Revision Checklist

A systematic checklist helps authors identify hidden statistical concerns before they become review comments. The checklist should be applied during manuscript preparation and again during revision.

Pre-Submission Statistical Checklist

• Confirm that sample size was justified with a power analysis or clear rationale
• List all statistical tests performed and check for multiplicity issues
• Verify that statistical assumptions were assessed and reported
• Document missing data handling procedures and sensitivity analyses
• Check for outliers and influential points in the dataset
• Identify potential confounders and describe adjustment strategies
• Report effect sizes with confidence intervals for primary analyses
• Prepare analysis code and data documentation for sharing

Revision-Specific Statistical Checklist

• Read each reviewer comment for implied statistical concerns
• Identify the underlying statistical issue behind vague phrasing
• Consult with a biostatistician when the concern is unclear
• Perform additional analyses requested or implied by reviewers
• Document all changes made in response to statistical comments
• Verify that revised analyses are described accurately in the methods section
• Ensure that results and conclusions are consistent with revised analyses

At a Glance: Statistical Review Comment Patterns

Practical Steps for Responding to Statistical Review Comments

Step 1: Categorize Each Comment

Read every reviewer comment and categorize it according to the statistical concern it raises. Some comments may address multiple concerns, so authors should identify each distinct issue raised.

Create a table that lists each comment, the category of concern, and the action required. This table becomes the working document for the revision process.

Step 2: Determine the Required Analysis

For each statistical concern, determine what analysis or reporting change is needed. Some concerns require new analyses, while others require better reporting of existing analyses.

Consult the EQUATOR Network for reporting guidelines relevant to the study type. These guidelines specify what statistical information should be reported and help authors identify gaps in their current manuscript.

Step 3: Perform the Necessary Work

Conduct any additional analyses needed to address reviewer concerns. This may involve power calculations, sensitivity analyses, assumption checks, or alternative modeling approaches.

Document all analyses performed, including the software used, the parameters specified, and the date of analysis. This documentation supports reproducibility and demonstrates rigor.

Step 4: Draft the Response

Write the response letter with each comment quoted and followed by the author's response. The response should state what was done, why it was done, and where the change appears in the manuscript.

Use specific language that shows the reviewer's concern was understood. For example, "We agree that the sample size was limited and have added a power analysis to the methods section" is more effective than "We have addressed this concern."

Step 5: Verify Consistency Across the Manuscript

Ensure that changes made in response to statistical comments are reflected consistently throughout the manuscript. The methods section should describe the revised analysis, the results should report the revised findings, and the discussion should interpret the revised results.

Inconsistencies between sections undermine the credibility of the revision and may trigger additional reviewer concerns.

Records and Measurements for Statistical Responses

Maintaining detailed records of the revision process supports both the response letter and the reproducibility of the final manuscript. Authors should document the following items:

• Original reviewer comments with dates received
• Categorization of each comment and the statistical concern identified
• Analyses performed in response to each concern
• Software versions and analysis parameters used
• Dates when analyses were conducted
• Changes made to the manuscript with section locations
• Communications with co-authors and statistical consultants

These records serve multiple purposes. They support the response letter, provide documentation for reproducibility, and create an audit trail if questions arise after publication.

The National Institutes of Health data management and sharing policy emphasizes the importance of documenting research processes. While this policy applies to NIH-funded research, the principle of thorough documentation applies broadly to scientific practice.

Common Failure Patterns in Responding to Statistical Comments

Responding Defensively

Authors sometimes respond to statistical comments by defending their original analysis without considering whether the reviewer raised a valid point. This approach often leads to extended revision cycles and may result in rejection.

A better approach is to consider each comment on its merits, acknowledge valid concerns, and make changes that strengthen the manuscript. Even when the author disagrees, the response should explain the reasoning clearly and respectfully.

Making Changes Without Understanding the Concern

Some authors make changes in response to statistical comments without fully understanding the underlying issue. This can lead to inappropriate analyses that create new problems.

If the statistical concern is unclear, authors should seek help from a biostatistician or statistical consultant before making changes. Guessing at the intended fix is riskier than asking for clarification.

Overcorrecting in Response to Comments

Authors sometimes overcorrect by applying every possible statistical adjustment, even when the reviewer did not request it. This can obscure the primary findings and make the manuscript harder to read.

The response should address the specific concern raised, not every conceivable statistical issue. Additional analyses can be mentioned as sensitivity checks, but the primary analysis should remain focused on the research question.

Ignoring the Connection Between Statistical and Scientific Concerns

Statistical comments often have scientific implications that extend beyond the analysis itself. A concern about power may indicate that the study cannot support its conclusions. A concern about multiple testing may suggest that the findings are less robust than claimed.

Authors should consider these broader implications when responding and adjust their conclusions if necessary. The response letter should reflect any changes to the interpretation of results.

Failing to Document Changes Clearly

Reviewers need to see exactly what changed in the revised manuscript. Authors who describe changes vaguely or fail to indicate where changes appear make the reviewer's job harder and increase the risk of misunderstanding.

Each response should reference the specific section and page where the change appears. If the manuscript uses line numbers, these should be included in the response.

Welfare and Safety Context in Statistical Reporting

Statistical reporting has direct implications for animal welfare and research safety. Studies involving animals must be designed with sufficient power to detect meaningful effects, because underpowered studies waste animal lives and produce unreliable results.

The National Library of Medicine research methods references emphasize the ethical obligation to design studies that can answer their research questions. An underpowered study exposes animals to procedures without the possibility of producing reliable knowledge.

Statistical transparency also supports the responsible use of animals in research. When authors report their methods clearly, other researchers can evaluate the quality of the evidence and avoid repeating studies with similar design flaws.

For laboratory staff and animal owners, understanding statistical reporting helps in evaluating the quality of research that informs animal care decisions. Studies with inadequate statistical methods may produce misleading conclusions that affect practical decisions.

Limitations of Statistical Review Responses

Authors should recognize the limits of what a response letter can accomplish. A response letter cannot fix fundamental problems with study design that should have been addressed before data collection began.

If the reviewer identifies a problem that cannot be fully addressed through reanalysis, the author should acknowledge this limitation honestly. The response should explain what can and cannot be done and discuss the implications for interpreting the results.

Some statistical concerns require access to data that may no longer be available. If data were lost or destroyed, the author should explain this and describe what analyses remain possible.

The response letter should also acknowledge when the author lacks the statistical expertise to fully address a concern. In these cases, consulting with a biostatistician is appropriate, and the response should reflect the consultant's input.

Professional Escalation Criteria

Authors should seek additional statistical expertise when they encounter any of the following situations:

• The reviewer's comment raises a statistical issue the author does not understand
• The required analysis is beyond the author's statistical training
• Multiple reviewers raise similar statistical concerns
• The statistical concern affects the core conclusions of the study
• The author is uncertain whether the proposed response adequately addresses the concern

A biostatistician can provide guidance on appropriate methods, help interpret reviewer comments, and verify that the response is statistically sound. This consultation is particularly important for complex study designs or when the stakes of the publication are high.

The Committee on Publication Ethics core practices emphasize the importance of sound research practices and honest reporting. Seeking appropriate statistical expertise is part of responsible authorship.

Building Long-Term Statistical Competence

Authors who encounter statistical review comments can use the experience to build their statistical competence for future projects. Each revision cycle provides an opportunity to learn about statistical methods, reporting standards, and reviewer expectations.

The EQUATOR Network provides access to reporting guidelines that help authors understand what statistical information should be included in different types of studies. Familiarity with these guidelines reduces the likelihood of receiving statistical review comments in the first place.

Authors should also maintain their statistical skills through continuing education. Workshops, online courses, and consultation with statisticians all contribute to the ability to design and analyze studies effectively.

For researchers who frequently encounter statistical review comments, developing a relationship with a statistical consultant can be valuable. This relationship provides ongoing support for study design, analysis, and manuscript preparation.

The Role of Data Sharing in Statistical Review

Data sharing is increasingly expected in scientific publishing, and it has direct implications for statistical review. When data are available, reviewers can verify analyses and assess the robustness of findings.

The National Institutes of Health data management and sharing policy requires NIH-funded researchers to plan for data sharing. This policy reflects a broader trend toward transparency in research.

Authors should prepare their data and analysis code for sharing as part of the manuscript preparation process. This preparation includes cleaning data, documenting variables, and creating reproducible analysis scripts.

When responding to statistical review comments, authors should indicate what data and code are available and how they can be accessed. This transparency supports the credibility of the revision and facilitates verification by reviewers.

Integrating Statistical Responses with Broader Manuscript Revision

Statistical responses should not be developed in isolation from other manuscript revisions. Changes to the analysis often require changes to the methods, results, and discussion sections.

The revision process should be coordinated so that all sections of the manuscript tell a consistent story. If the analysis changes, the results must be updated, and the discussion must interpret the updated results.

Authors should also consider whether statistical changes affect the study's conclusions. A revised analysis that changes the significance of findings requires corresponding changes to the abstract and the interpretation in the discussion.

The response letter should describe these broader changes so that reviewers understand the full scope of the revision. This helps reviewers evaluate whether the revised manuscript adequately addresses their concerns.

Frequently Asked Questions

How do I respond when a reviewer says my sample size is too small?

Report the power analysis that was conducted before data collection, including the assumed effect size, alpha level, and target power. If no power analysis was performed, acknowledge this limitation and explain the basis for the sample size. Discuss how the sample size affects the interpretation of results and whether the study should be considered exploratory.

What should I do if a reviewer asks about multiple testing but I did not correct for it?

List every statistical test performed and explain the rationale for each. If the analyses were exploratory, state this clearly and describe the increased risk of false positives. Consider adding a correction method or reporting adjusted results as a sensitivity analysis.

How do I respond to a comment about statistical assumptions?

Describe how each assumption was assessed, including any diagnostic tests or graphical checks performed. If assumptions were violated, explain what alternative methods were considered and why the chosen approach was retained. Provide specific evidence from the diagnostic checks.

What if I do not understand the statistical concern in a review comment?

Seek help from a biostatistician or statistical consultant before making changes. Guessing at the intended fix is risky and may create new problems. A consultant can help interpret the comment and determine the appropriate response.

How should I handle a reviewer request for additional analyses?

Determine whether the requested analysis is appropriate for the research question and feasible with the available data. If the analysis is appropriate, perform it and report the results. If it is not appropriate, explain why and describe what was done instead.

Should I share my data and analysis code in response to statistical comments?

Data sharing supports transparency and allows reviewers to verify analyses. Prepare your data and code for sharing and indicate in your response how they can be accessed. Follow any data sharing policies that apply to your research, such as the NIH data management and sharing policy.

How do I respond when a reviewer suggests a different statistical method?

Consider whether the suggested method is appropriate for the research question and data structure. If it is appropriate, perform the analysis and report the results alongside your original analysis. If it is not appropriate, explain the reasoning and describe why the original method was retained.

What should I include in my response letter for statistical comments?

Include the original comment, your understanding of the concern, the specific action taken, and where the change appears in the revised manuscript. Use specific language that shows the concern was understood and addressed.

Using the Evidence

Related Articles

• Statistical Questions: How to Identify and Formulate Them
• Methodology Limitations in Research: How to Identify and Address Them
• Statistical Questions Examples: How to Write Them
• Common Limitations in Research: Examples and How to Address Them
• Peer Review Metrics: What They Are and How to Use Them

References and Further Reading

• Research Methods Resources. National Library of Medicine.
• EQUATOR Network. EQUATOR Network.
• Core Practices. Committee on Publication Ethics.
• NIH Grants and Funding. National Institutes of Health.
• ORCID for Researchers. ORCID.
• Data Management and Sharing Policy. National Institutes of Health.
• NCBI Literature Resources. National Center for Biotechnology Information.
• PubMed. National Library of Medicine.

This article is educational and does not replace institutional policy, professional advice, or applicable safety and regulatory requirements.