Pilot Testing Your Research Question

By Dr. Zubair Khalid, DVM, MS, PhD · 2026-08-27

Quick Answer

• Pilot data should trigger a formal decision on whether to keep, modify, or abandon your research question based on feasibility, variability, and observed effect size.
• Use a structured decision tree with explicit thresholds before collecting pilot data so revision criteria are fixed in advance.
• Pilot results alone rarely prove a hypothesis, they primarily inform whether a full study is worth the investment of time, funding, and animal or human resources.

Understanding the Purpose of Pilot Testing

Pilot testing serves a distinct function in the research lifecycle. It is not a miniature version of the main study designed to produce publishable findings. Instead, it is a small-scale trial that tests whether your research question can be answered with the methods, resources, and population you intend to use. The pilot phase reveals practical problems before you commit substantial resources to a full investigation.

For life-science researchers working with animal models, laboratory assays, or field observations, the pilot phase answers questions such as: Can I recruit or enroll enough subjects? Can I collect the measurements reliably? Does my intervention produce a measurable response? Is the variability in my outcome measure small enough to detect a meaningful effect? These questions matter because a research question that looks promising on paper may prove unanswerable in practice.

The distinction between a pilot study and a full study carries ethical weight as well. Using animals or human participants in research requires justification. A pilot study that is poorly designed wastes resources and exposes subjects to risk without producing useful information. The National Library of Medicine Research Methods Resources gateway provides access to biomedical research-method references that emphasize the importance of rigorous study design from the earliest stages.

Core Principles of Pilot Testing

Feasibility as the First Gate

Feasibility asks whether you can actually conduct the study you propose. This includes recruitment capacity, equipment availability, technical skill, time constraints, and budget. A research question may be scientifically interesting but practically impossible to answer with your available resources.

Feasibility concerns often emerge during pilot testing in ways that are difficult to predict. For example, a laboratory may discover that the assay they planned to use requires more tissue than their animal model can provide. A field researcher may find that the target species is less abundant than the literature suggested. A clinical researcher may learn that the patient population is smaller than expected or that the consent process takes longer than anticipated.

When feasibility fails, the research question may need modification. You might narrow the population, change the outcome measure, or adjust the sampling strategy. In some cases, the question must be abandoned because the resources required to answer it exceed what is available.

Variability and Measurement Reliability

Variability refers to the spread of your measurements. High variability makes it difficult to detect a true effect because the noise in your data can obscure the signal. Pilot data provide an estimate of variability that you can use to calculate the sample size needed for a full study.

Measurement reliability is closely related. If your instruments, assays, or observers produce inconsistent results, the variability in your data will be inflated. Pilot testing allows you to assess whether your measurement procedures are consistent across repeated measurements, across different technicians, or across different days.

For animal research, variability can arise from genetic background, age, sex, housing conditions, diet, and circadian rhythms. Standardizing these factors during the pilot phase can reduce variability and improve the chances that a full study will detect a real effect. The EQUATOR Network provides reporting guidelines that emphasize transparent description of methods, which helps other researchers understand the sources of variability in your work.

Effect Size Estimation

Effect size describes the magnitude of the difference or relationship you expect to observe. Pilot data can provide a preliminary estimate of effect size, but this estimate comes with substantial uncertainty because pilot samples are small.

A common error is to abandon a promising research question because the pilot effect size appears small. Small pilot samples produce imprecise effect size estimates. A pilot study with ten animals per group may show a small effect that would become statistically significant with thirty animals per group. Conversely, a pilot may show a large effect that shrinks when tested in a larger sample.

The appropriate use of pilot effect size data is to inform sample size calculations and to decide whether the observed effect is large enough to be practically meaningful. If the effect size you observe in the pilot is so small that detecting it would require an impractical number of subjects, the research question may need modification.

The Decision Framework for Your Research Question

Establishing Thresholds Before You Start

The most important step in pilot testing is deciding in advance how you will interpret the results. Without pre-specified thresholds, you risk rationalizing whatever outcome you observe. If the pilot goes well, you may convince yourself the question is viable. If it goes poorly, you may abandon a question that could have worked with minor adjustments.

Set thresholds for three dimensions before collecting pilot data:

1. Feasibility thresholds: What recruitment rate, completion rate, or data quality level is acceptable? For example, you might decide that the study is feasible if at least 80 percent of enrolled subjects complete the protocol.

1. Variability thresholds: What coefficient of variation or standard deviation is acceptable for your primary outcome? This threshold depends on the biological or technical variability inherent to your system.

1. Effect size thresholds: What minimum effect size would make the full study worthwhile? This threshold should reflect practical significance, beyond statistical significance.

Document these thresholds in your pilot protocol. The NIH Grants and Funding resources emphasize the importance of clear methodology in grant applications, and pre-specified pilot thresholds strengthen the scientific rationale for a full study.

Decision Tree for Keep, Modify, or Abandon

The decision tree below provides a structured approach to interpreting pilot results. It assumes you have pre-specified thresholds for feasibility, variability, and effect size.

flowchart TD
    A[Collect Pilot Data] --> B{Feasibility Thresholds Met?}
    B -->|No| C{Can Feasibility Be Fixed?}
    C -->|Yes| D[Modify Methods and Research Question]
    C -->|No| E[Abandon Research Question]
    B -->|Yes| F{Variability Acceptable?}
    F -->|No| G{Can Variability Be Reduced?}
    G -->|Yes| H[Modify Measurement or Design]
    G -->|No| I[Abandon or Substantially Revise]
    F -->|Yes| J{Effect Size Meaningful?}
    J -->|Yes| K[Keep Research Question and Plan Full Study]
    J -->|No| L{Is Larger Sample Feasible?}
    L -->|Yes| M[Keep Question with Adjusted Sample Size]
    L -->|No| N[Modify or Abandon Research Question]

Keep the Research Question

You keep the research question when pilot data meet all three thresholds. Feasibility is confirmed, variability is acceptable, and the observed effect size justifies proceeding to a full study. In this case, the pilot data inform the sample size calculation and refine the methods for the main investigation.

Keeping the research question does not mean the pilot was unnecessary. The pilot has confirmed that your approach works, identified any minor adjustments needed, and provided the data required to justify the full study to funders and ethics committees.

Modify the Research Question

Modification is appropriate when one or more thresholds are not met but the problems appear fixable. Common modifications include:

• Narrowing or broadening the population
• Changing the primary outcome measure
• Adjusting the intervention dose or duration
• Improving measurement procedures
• Extending the study timeline
• Adding a control group or additional covariates

When you modify the research question, you should consider whether the modified question requires additional pilot testing. If the modification changes the population, intervention, or outcome measure substantially, a second pilot may be necessary.

Abandon the Research Question

Abandonment is appropriate when feasibility cannot be achieved, variability cannot be reduced to acceptable levels, or the effect size is too small to be practically meaningful even with a feasible sample size.

Abandoning a research question is not a failure. It prevents the waste of resources on a study that cannot produce useful results. The Committee on Publication Ethics Core Practices emphasize honesty and transparency in research, and this includes being honest with yourself about whether a question can be answered.

At a Glance

Practical Implementation Steps

Step 1: Define Your Research Question Explicitly

Write your research question in a format that specifies the population, intervention or exposure, comparison, and outcome. This structure helps you identify what the pilot study must test. A vague research question produces a vague pilot study.

For example, instead of asking "Does diet affect mouse behavior?", specify "Does a high-fat diet compared to a standard diet alter open-field locomotion in adult male C57BL/6 mice over a six-week period?" This specificity tells you exactly what to measure in the pilot.

Step 2: Identify the Critical Uncertainties

List the assumptions your full study depends on. Rank them by how much uncertainty they carry and how much damage a failure would cause. The pilot should focus on the most critical uncertainties.

Critical uncertainties might include: Can I obtain enough animals? Will the intervention produce a measurable response? Can I measure the outcome reliably? Will the animals tolerate the procedures? Will the effect be large enough to detect?

Step 3: Design the Pilot Study

Design the pilot to answer the critical uncertainties with the minimum number of subjects or samples. The pilot does not need to be powered to detect the main effect. It needs enough data to assess feasibility, estimate variability, and provide a rough effect size.

Consider including a control group in the pilot even if the main study will have one. Without a control group, you cannot estimate the effect size or distinguish treatment effects from background variation.

Step 4: Pre-Specify Your Decision Thresholds

Write down the specific criteria you will use to decide whether to keep, modify, or abandon the research question. These thresholds should be based on the scientific literature, your experience, and practical constraints.

For example, you might specify: "The study is feasible if at least 75 percent of enrolled animals complete the protocol. Variability is acceptable if the coefficient of variation for the primary outcome is below 30 percent. The effect size is meaningful if the standardized mean difference is at least 0.5."

Step 5: Collect and Analyze Pilot Data

Collect the pilot data according to your protocol. Analyze the data with attention to feasibility metrics, variability estimates, and effect size calculations. Do not focus on statistical significance, because pilot studies are rarely powered to detect significant effects.

Step 6: Apply the Decision Tree

Use the decision tree to interpret your results. Be honest about whether thresholds were met. If you are tempted to lower a threshold after seeing the data, ask yourself whether the lower threshold is scientifically justified or simply convenient.

Step 7: Document and Report the Pilot

Document the pilot methods, results, and decisions. This documentation is valuable for grant applications, ethics submissions, and future publications. The EQUATOR Network provides reporting guidelines that can help you report pilot studies transparently.

Records and Measurements to Track

Feasibility Metrics

Track the number of subjects or samples you attempted to enroll, the number successfully enrolled, and the number that completed the protocol. Record the reasons for non-enrollment and dropout. These numbers tell you whether your recruitment and retention strategies will work at full scale.

Also track the time required for each procedure, the amount of consumables used, and the personnel hours required. These data inform the budget and timeline for the full study.

Variability Estimates

Record the standard deviation or coefficient of variation for your primary outcome measure. If you have repeated measurements on the same subjects, calculate the intra-class correlation coefficient to assess measurement reliability.

Track variability across batches, technicians, days, and equipment. If variability is high, identify the source. It may be biological variation, technical variation, or a combination.

Effect Size Estimates

Calculate the effect size from your pilot data using the appropriate metric for your outcome. For continuous outcomes, this might be the standardized mean difference. For binary outcomes, this might be the risk difference or odds ratio.

Record the confidence interval around the effect size estimate. The wide confidence interval from a small pilot reminds you that the true effect may be larger or smaller than observed.

Common Failure Patterns in Pilot Testing

Overinterpreting Pilot Results

The most common failure is treating pilot results as if they came from a full study. Researchers may abandon a viable research question because the pilot effect was not statistically significant, or they may proceed to a full study based on a pilot effect that was inflated by chance.

Pilot studies are designed to estimate feasibility and variability, not to test hypotheses. Statistical significance in a pilot is informative but not decisive. The National Library of Medicine Research Methods Resources gateway includes references that explain the limitations of small samples and the importance of appropriate statistical methods.

Underpowering the Pilot

A pilot that is too small provides unreliable estimates of variability and effect size. If you collect data from only three animals per group, the variability estimate will be imprecise and the effect size estimate will be unstable.

A useful pilot typically includes enough subjects to provide a reasonable estimate of variability. The exact number depends on the outcome measure and the population, but a common rule of thumb is to use at least ten to fifteen subjects per group for continuous outcomes.

Ignoring Feasibility Problems

Some researchers focus exclusively on effect size and ignore feasibility problems. A study that shows a large effect but cannot recruit enough subjects will fail. Feasibility problems identified in the pilot should be addressed before proceeding, even if the effect size looks promising.

Changing Thresholds After Seeing Data

Pre-specified thresholds lose their value if you change them after seeing the results. If you set a feasibility threshold of 80 percent completion and observe 60 percent, you should not lower the threshold to 60 percent simply because you want the study to proceed.

If the pilot data suggest that your thresholds were unrealistic, you can revise them, but you should document the revision and justify it based on new information instead of convenience.

Failing to Document the Pilot

Pilot studies that are not documented cannot inform grant applications, ethics submissions, or publications. The NIH Data Management and Sharing Policy emphasizes the importance of planning for data management, and this includes pilot data.

Welfare and Safety Considerations

Animal Welfare in Pilot Studies

Pilot studies involving animals must meet the same welfare standards as full studies. The pilot is not a license to cut corners on animal care, anesthesia, analgesia, or humane endpoints. Animals used in a pilot study are research subjects with the same protections as those in a full study.

The pilot phase is an opportunity to refine procedures that affect animal welfare. You can assess whether the animals tolerate handling, restraint, injections, or surgery. You can evaluate whether the housing conditions are appropriate. You can identify procedures that cause distress and modify them before the full study.

If the pilot reveals that a procedure causes more distress than anticipated, you must address this before proceeding. This may involve modifying the procedure, adding analgesia, or reconsidering whether the research question justifies the animal use.

Human Participant Safety

For research involving human participants, the pilot phase must follow the same ethical and regulatory requirements as a full study. This includes institutional review board approval, informed consent, and data protection.

The pilot can reveal safety concerns that were not anticipated. If participants experience unexpected adverse events, you must report these according to your institutional policies and reconsider whether the research question can be answered safely.

Laboratory Safety

Pilot testing may involve new procedures, new equipment, or new reagents. Ensure that laboratory safety protocols are followed and that personnel are trained before the pilot begins. The pilot is not the time to discover that a procedure is unsafe.

Limitations of Pilot Testing

Pilot Data Cannot Confirm the Hypothesis

The most important limitation is that pilot data cannot confirm or refute your research hypothesis. The sample size is too small, and the purpose is different. A pilot that shows a promising effect does not prove the effect exists. A pilot that shows no effect does not prove the effect is absent.

Effect Size Estimates Are Imprecise

Effect size estimates from pilot studies carry wide confidence intervals. A pilot with ten animals per group might estimate an effect size of 0.8 with a confidence interval from 0.1 to 1.5. The true effect could be small or large.

This imprecision means you should not base the decision to proceed solely on the pilot effect size. Consider whether the effect size is plausible based on the scientific literature and whether a full study with an appropriate sample size is feasible.

Pilot Results May Not Generalize

Pilot results come from a specific population, setting, and time. They may not generalize to other populations or settings. A pilot conducted in one laboratory may not predict results in another laboratory, even with the same protocol.

This limitation is particularly relevant for animal research, where results can vary across strains, vendors, and housing conditions. The EQUATOR Network reporting guidelines emphasize describing the study context so readers can assess generalizability.

Feasibility in the Pilot May Not Predict Feasibility at Scale

A pilot that is feasible at a small scale may encounter problems at a larger scale. Recruitment may slow, equipment may break, personnel may leave, or costs may rise. The pilot provides a lower-bound estimate of feasibility, not a guarantee.

Professional Escalation Criteria

When to Consult a Biostatistician

Consult a biostatistician before the pilot if you are uncertain about the outcome measure, the study design, or the sample size. Consult again after the pilot to interpret the variability and effect size estimates and to calculate the sample size for the full study.

A biostatistician can help you avoid common errors in pilot interpretation, such as overinterpreting non-significant results or using the pilot effect size without accounting for its imprecision.

When to Consult an Animal Welfare Officer or Ethics Committee

Consult your institutional animal welfare officer or ethics committee if the pilot reveals unexpected animal distress, if you need to modify procedures in ways that affect animal welfare, or if you are uncertain whether the research question justifies continued animal use.

The Committee on Publication Ethics Core Practices emphasize the importance of ethical conduct throughout the research process, and this includes seeking appropriate oversight when welfare concerns arise.

When to Consult a Grant Officer or Program Officer

Consult a grant officer or program officer if the pilot results substantially change the scope, timeline, or budget of the proposed full study. The NIH Grants and Funding resources describe the expectations for grant applications and the importance of aligning your proposal with the funding opportunity.

If the pilot suggests that the full study will require more resources than originally planned, you may need to revise the proposal or seek additional funding.

When to Reconsider the Research Question Entirely

Reconsider the research question entirely if the pilot reveals fundamental problems that cannot be fixed with minor modifications. These problems might include: the intervention has no measurable effect at any feasible dose, the outcome measure cannot be collected reliably, or the population is too difficult to access.

Abandoning a research question is a legitimate scientific decision. It is better to abandon a question after a small pilot than after a large, expensive full study that cannot answer it.

Frequently Asked Questions

How much pilot data is enough to make a decision?

The amount of pilot data needed depends on the outcome measure, the variability in your system, and the decisions you need to make. For continuous outcomes, ten to fifteen subjects per group often provides a reasonable estimate of variability. For feasibility questions, you need enough subjects to observe common problems. If recruitment is difficult, even a small pilot can reveal that the full study is not feasible.

Can I use pilot data in my main study analysis?

Generally, pilot data should not be combined with full study data in the main analysis. Pilot data are collected under conditions that may differ from the full study, and including them can introduce bias. The pilot is used to inform the design of the full study, not to contribute to its statistical power.

What if my pilot shows no effect?

A pilot that shows no effect does not necessarily mean the research question should be abandoned. The pilot may be too small to detect a real effect, or the effect may be smaller than expected. Consider whether the effect size you observed is practically meaningful and whether a larger sample would detect it. If the effect is so small that it is not worth detecting, the research question may need modification.

How do I calculate sample size for the full study from pilot data?

Use the variability estimate from the pilot data to calculate the sample size needed to detect your target effect size with the desired power and significance level. A biostatistician can help with this calculation. Be aware that the variability estimate from a small pilot is imprecise, so consider using the upper confidence limit of the variability estimate to be conservative.

Should I publish my pilot study?

Pilot studies can be published, particularly if they provide useful information about feasibility, methods, or variability that would help other researchers. The EQUATOR Network provides reporting guidelines that can help you report a pilot study transparently. Publishing a pilot can also help you establish a track record for grant applications.

What if my pilot reveals a problem I cannot fix?

If the pilot reveals a problem that cannot be fixed with reasonable effort, the appropriate decision is to abandon the research question and redirect your resources. This is not a failure. It is a successful use of the pilot to avoid wasting resources on a study that cannot succeed.

How do I document pilot decisions for my grant application?

Document the pilot methods, results, and decisions in a pilot report. Include the pre-specified thresholds, the observed values, and the rationale for your decision to keep, modify, or abandon the research question. This documentation demonstrates to reviewers that you have tested your assumptions and have a realistic plan for the full study.

Can I conduct a pilot study without funding?

Pilot studies can be conducted with limited resources, but you must ensure that the pilot is adequately designed and that animal or human subjects are protected. Some institutions provide internal pilot funding. The NIH Grants and Funding resources describe funding mechanisms that may support pilot work, and your institution may have similar programs.

Using the Evidence

Related Articles

• Open Science Practices: A Guide for Researchers
• Scientific Data Provenance For Reproducible Research
• What Is a Data Dictionary? A Practical Guide for Research Teams
• Data Lifecycle Management in Research: From Collection to Preservation
• Data Management Solutions for Research Teams: A Buyer's Guide

References and Further Reading

• Research Methods Resources. National Library of Medicine.
• EQUATOR Network. EQUATOR Network.
• Core Practices. Committee on Publication Ethics.
• NIH Grants and Funding. National Institutes of Health.
• ORCID for Researchers. ORCID.
• Data Management and Sharing Policy. National Institutes of Health.
• NCBI Literature Resources. National Center for Biotechnology Information.
• PubMed. National Library of Medicine.

This article is educational and does not replace institutional policy, professional advice, or applicable safety and regulatory requirements.