# The Pre-Submission Reproducibility Review: A Checklist By the end of this guide you will be able to run a complete reproducibility audit on your own manuscript before you submit it anywhere. You will verify that your data, code, computing environment, and [statistical analysis](/blog/guides/statistical-analysis) can be repeated from scratch by a stranger, you will tag the exact version of your project that matches the submitted paper, and you will fix the most common failure points that reviewers and readers discover after publication. You need a few things on hand before you start: the final version of your manuscript, the raw data (or documented instructions for how it can be obtained), the analysis code, a terminal on your computer, and a version control repository such as Git. If you work in R or Python, have RStudio or a code editor open. If you use a container tool such as Docker, have it installed. Total setup time for a typical computational project is about 30 minutes, and the review itself takes one to three hours depending on project size. ## Why Reproducibility Problems Start Before Peer Review Reproducible research means that an independent person, working from your published materials, can arrive at your reported results. Reproducibility is distinct from replication, which means repeating the underlying experiment to see whether the finding holds. A paper can be perfectly reproducible and still fail to replicate. The pre-submission reproducibility review addresses the first problem only, and it addresses it at the stage where fixes are cheapest, which is before any editor or reviewer sees the work. The scale of the problem is well documented. A 2026 review of the clinical trial literature found a landscape of more than one million registered trials marked by significant redundancy, waste, and lack of reproducibility, with transparency hampered by inadequate prospective registration, unreported results, and limited availability of protocols and statistical analysis plans [1]. The same review noted that peer review workflows underuse appraisal at the pre-submission stage, meaning that reproducibility flaws that could have been caught by authors themselves tend to surface late or not at all [1]. That finding generalizes. Most reproducibility failures are not dramatic cases of fraud. They are quiet chain-of-custody problems: a script that only runs on the author's laptop, a result table that was generated by an earlier version of the code, a sample size that changed after the protocol was written without anyone updating the manuscript, a random seed that was never fixed. Each of these is invisible in the finished PDF. Each is easy to catch with a structured review before submission. ## The Pre-Submission Reproducibility Review: Scope and Timing ![Three-step flow showing freeze the version, rebuild every output, and compare line by line in a reproducibility review.](/article-images/illustrations/pre-submission-reproducibility-review-checklist-review-scope.webp) The pre-submission reproducibility review is a structured internal audit with three parts. First, you freeze the manuscript and the analysis so that nothing changes during the audit. Second, you attempt to rebuild every reported number, table, and figure from raw inputs using only the files a reader would receive. Third, you compare the rebuilt outputs against the manuscript line by line and record what you find. Run the review after the manuscript text is essentially final but before you submit. Running it earlier wastes effort because results will still change. Running it after submission defeats the purpose, because corrections then require formal errata or, in the worst case, retraction. The review is best performed by two people: the author, who knows the history, and a colleague who plays the role of a hostile but fair external reviewer and tries to run the analysis using only the shared files. If you work alone, the clean-room rerun described below substitutes for the colleague. ## What to Have on Hand Gather these items into one place before you start the checklist. Every missing item becomes a checklist failure you will need to resolve or document. - The near-final manuscript, with every figure and table numbered - The raw data files, in the format you actually collected them - The complete analysis code, from raw data to final figure - An environment specification file that records every software version - The random seed values used at every stochastic step - The study protocol and, for trials, the statistical analysis plan - Your version control history, ideally with a clean working directory - The reporting checklist required by your target journal, such as CONSORT for randomized trials or PRISMA for systematic reviews The protocol and analysis plan deserve special emphasis. The trial literature shows that limited protocol and statistical analysis plan availability remains a persistent transparency gap even as registration practices improve [1]. If your study has a written protocol, the pre-submission review is your last good chance to confirm that the manuscript describes what the protocol planned, and to disclose and justify any deviations. ## The Reproducibility Checklist Template The checklist below is the template this article builds on. Use the pass criteria as written, or adapt them to your field. A paper passes the review when every item is marked yes or has a documented, honest exception written into the manuscript itself. | # | Section | Item | Pass criterion | |--|-----|---|--------| | 1 | Design | Study registered prospectively where required | Registration number appears in the manuscript | | 2 | Design | Protocol and analysis plan filed before data collection | Dates of registration and data lock are stated | | 3 | Design | Deviations from protocol listed and explained | A deviations section or paragraph exists | | 4 | Data | Raw data included, deposited, or access path documented | A stranger could obtain the data from the manuscript alone | | 5 | Data | Data dictionary defines every variable and unit | A reader can decode every column | | 6 | Data | Exclusions and [preprocessing](/blog/data-analysis/what-is-data-preprocessing) steps are scripted | No exclusion was applied by hand in a spreadsheet | | 7 | Code | Every number, table, and figure is generated by code | Zero hand-edited outputs | | 8 | Code | Code runs top to bottom from a fresh start | A clean run completes without manual steps | | 9 | Code | Random seeds fixed at every stochastic step | Repeated runs give identical outputs | | 10 | Environment | All software versions recorded in a spec file | environment.yml, requirements.txt, or renv.lock exists | | 11 | Environment | Container or session information archived | Dockerfile or sessionInfo output is stored | | 12 | Statistics | Sample size and power analysis reported | The calculation inputs are stated, not just the result | | 13 | Statistics | Every test matches a prespecified analysis | Unplanned analyses are labeled exploratory | | 14 | Statistics | Effect sizes and uncertainty reported, not only p values | [Confidence intervals](/blog/data-analysis/confidence-interval-formula-examples) or intervals of equivalence present | | 15 | Statistics | Missing data handling described | Method for missing values is named | | 16 | Reporting | Target journal checklist completed | Every checklist item is annotated with a page number | | 17 | Reporting | Author contributions and conflict statements complete | All required statements present | | 18 | Reporting | Data and code availability statements accurate | The stated repository actually contains the files | | 19 | Records | Notebook or log links raw data to results | Every reported number traces to a script and input | | 20 | Records | Review results archived with the submission snapshot | The audit record is versioned and dated | Items 1 through 3 follow directly from the transparency requirements documented in the trial literature, where prospective registration, protocol availability, and reporting of results are recurring weak points across more than a million registered studies [1]. Items 7 through 11 are the computational core, and the worked example below shows exactly how to verify them. ## A Worked Example: Running the Review The following walkthrough uses a small fictional project, a [gene expression](/blog/guides/gene-expression) comparison with one dataset, one analysis script, and three figures. Every command shown is standard. Adapt file names to your project. ### Step 1. Freeze the version Open a terminal in your project directory and confirm the repository is clean, then tag the submission snapshot. ```bash cd myproject git status git add -A git commit -m "Freeze for submission" git tag -a v1.0-submission -m "Version matching submitted manuscript" git log --oneline -3 ``` Expected output: ``` On branch main nothing to commit, working tree clean [main a1b2c3d] Freeze for submission tag 'v1.0-submission' created a1b2c3d (HEAD -> main, tag: v1.0-submission) Freeze for submission 9f8e7d6 Update figure 2 caption 7c6b5a4 Final results table ``` If `git status` reports untracked files, decide now whether they belong in the repository. An untracked analysis script at this stage means the manuscript cites work a reader cannot see. ### Step 2. Export the environment Record every software version so the analysis can be rebuilt elsewhere. For a Python project: ```bash conda env export > environment.yml pip freeze > requirements.txt ``` Expected content of a valid `environment.yml`: ```yaml name: myproject dependencies: - python=3.11.5 - numpy=1.26.4 - pandas=2.2.1 - scipy=1.12.0 ``` For an R project, run inside R: ```r renv::snapshot() sessionInfo() ``` Expected output from `sessionInfo()`, truncated: ``` R version 4.3.2 (2023-10-31) Platform: x86_64-pc-linux-gnu (64-bit) attached base packages: stats, graphics, grDevices, utils, datasets, methods, base other attached packages: ggplot2_3.4.4, dplyr_1.1.4 ``` Save the session information output as a text file in the repository. If the environment export fails, that is a checklist failure at item 10, not a technicality. ### Step 3. Fix every random seed Any step that involves randomness, such as [bootstrapping](/blog/data-analysis/what-is-bootstrapping-resampling-examples), cross-validation, or simulation, must record a seed. In Python: ```python import numpy as np import random SEED = 20261111 random.seed(SEED) np.random.seed(SEED) ``` In R: ```r set.seed(20261111) ``` Then verify determinism by running the stochastic portion twice and comparing outputs. If results change between runs with the same seed, some random draw is escaping your seed, commonly through a package with its own parallel generator, and you must find it before submission. ### Step 4. The clean-room rerun This is the heart of the reproducibility review. Reproduce the conditions a stranger faces by starting from a fresh clone and a fresh environment. ```bash cd /tmp git clone /path/to/myproject myproject-test cd myproject-test conda env create -f environment.yml conda activate myproject python run_all.py ``` Expected output: ``` Loading raw data from data/raw/counts.csv ... ok Filtering low-count genes: 15432 -> 8710 genes Fitting model ... done Writing results/summary_table.csv Writing figures/fig1.png, fig2.png, fig3.png All done in 214.3 s ``` The pass criterion is simple: the run completes from raw input to final figure with no manual intervention and no error. If you find yourself opening a spreadsheet, copying a value, or renaming a file partway through, the pipeline is not yet reproducible. Script that step. ### Step 5. Compare rebuilt outputs against the frozen ones Generate a checksum for every result file. A checksum, such as an MD5 hash, is a short fingerprint of a file's contents. Two files with the same fingerprint have identical contents. ```bash md5sum results/summary_table.csv ``` Expected output: ``` d41d8cd98f00b204e9800998ecf8427e results/summary_table.csv ``` Run the same command on the copy in your original directory. If the fingerprints match, the rerun produced byte-identical results. If they differ, open both files and diff them to find the discrepancy before deciding whether the manuscript or the code is out of date. ### Step 6. Trace every reported number Open the manuscript at the results section. For each number, table, and figure, confirm it appears in a rebuilt output file. A practical format is a simple audit log kept alongside the code: ``` Manuscript p. 6, "fold change of 2.3" -> results/summary_table.csv row 14 Manuscript Figure 1 -> figures/fig1.png Manuscript p. 7, "p = 0.012" -> results/stats_output.txt line 88 Manuscript p. 8, "n = 24 per group" -> data/raw (24 rows per group verified) ``` Any number you cannot trace to an output file is a finding that must be resolved. In practice this step catches the single most common reproducibility failure, which is a manuscript written from an earlier run of the analysis that has since changed. ### Step 7. The colleague dry run Hand a colleague the repository, the manuscript, and nothing else. Ask them to regenerate one figure and one reported number without asking you any questions. Time-box the exercise to about 25 minutes. Their first point of confusion is exactly what a journal reviewer will experience. Fix whatever blocked them, then rerun steps 4 through 6, because any fix can introduce a new failure. ## Common Errors and Fixes These are the failure messages reviewers and colleagues encounter most often during a pre-submission reproducibility review, with their exact text and repairs. **`FileNotFoundError: [Errno 2] No such file or directory: 'data/raw/counts.csv'`** in Python. The script depends on a file that is not in the repository and has no documented source. Fix by including the data, adding a download script, or adding a `data/README` that states exactly where the file comes from and how it is obtained. **`Error in eval(expr, envir, enclos): object 'df' not found`** in R. The script assumes objects exist in your interactive session from earlier manual work. Fix by restarting R completely, then running the entire pipeline with `source("run_all.R")` from a fresh session. Anything that breaks under this test was never scripted. **`ResolvePackageNotFound: - r-base=4.3.1`** from Conda. A pinned version in your environment file is unavailable on the configured channel. Fix by adding the conda-forge channel to your environment file or relaxing the version pin to the nearest available version, then confirm the rerun results are unchanged. **`fatal: not a git repository (or any of the parent directories): .git`**. The clean-room test was run outside a cloned repository, or the project was never under version control at all. Fix by cloning properly, or by initializing version control now and committing the frozen state. **A checksum mismatch between the original and rerun outputs.** Something in the pipeline is nondeterministic despite a fixed seed, such as parallel workers processing items in arbitrary order. Fix by fixing the seed at the parallel step, setting the worker count explicitly, or sorting inputs before processing. **`docker: Error response from daemon: ... permission denied`** during a bind mount. The container cannot read the mounted data directory. Fix by adjusting the mount permissions or running the container with the documented user flag from your project README, then record that flag in the README itself. ## Common Mistakes and Limitations **Reviewing the prose instead of the pipeline.** The most common mistake is reading the methods section for completeness while never rerunning the analysis. Reading catches wording problems. Only a rerun catches the fact that figure 3 was made with an older version of the data. **Treating the checklist as a one-time event.** Run the full review again after any post-review change, however small. A one-line edit to an exclusion threshold invalidates every downstream number. **Testing only on your own machine.** A pipeline that runs only where it was written is not reproducible. The clean-room rerun exists precisely to simulate a stranger's machine. If you cannot set one up, at minimum rerun in a fresh container. **Ignoring the protocol.** Authors tend to audit the computation and forget the design commitments. The trial literature identifies high risk of bias, design flaws, and divergence between what was planned and what was reported as recurring quality problems, so compare the manuscript against the registration and protocol explicitly [1]. **Assuming review equals verification of truth.** A reproducibility review confirms that results follow from data and code. It cannot confirm that the experiment was well designed, that the measurement was valid, or that the finding generalizes. Those questions belong to replication and peer review. **Undocumented restrictions on sharing.** Some data cannot be deposited because of consent terms or licensing. That is acceptable, but the availability statement must describe the restriction and the route by which qualified researchers can request access. Silence reads as concealment. **One-person teams.** If no colleague is available, use the clean-room clone as your adversarial reviewer, and consider posting a preprint so the broader community can examine the materials before formal peer review. Preprint and post-publication review are recognized stages of the appraisal workflow, and the evidence suggests the pre-submission stage is where appraisal is currently least used [1]. ## Frequently Asked Questions ### What is the pre-submission reproducibility review? It is a structured internal audit, performed before journal submission, in which you verify that every reported result can be regenerated from raw data and code by someone other than you, using a reproducibility checklist as the working document. ### When should I run it? ![Timeline of the reproducibility review from gathering materials through setup, the audit, and rerunning after any change.](/article-images/illustrations/pre-submission-reproducibility-review-checklist-review-timing.webp) Run it once the manuscript text is nearly final but before you submit, and rerun the checklist after any change made in response to the review. ### How long does the review take? A small computational project takes one to three hours, and the dominant cost is almost always the first clean-room rerun, which tends to expose environment and path problems that then never recur. ### What is the difference between reproducibility and replication? Reproducibility means an independent person obtains your reported numbers from your materials, while replication means repeating the experiment itself to test whether the finding holds in new samples. ### Do I have to share my data publicly? Not always, but you must either deposit the data, document a legitimate access restriction with a request route, or cite a repository that holds it, and your availability statement must be accurate either way. ### Does this checklist apply to clinical trials? Yes, with additional items, because trials carry specific transparency obligations around prospective registration, protocol and statistical analysis plan availability, and results reporting that this checklist builds in at items 1 through 3. ### What tools do I need? A terminal, version control such as Git, an environment specification tool such as Conda, renv, or pip, and optionally a container system such as Docker, all of which are free and documented by their projects. ### Can I use this checklist for work already submitted or published? Yes, and it is worth doing, but treat any mismatch you find as urgent, because discrepancies discovered after publication require correction through the journal rather than a quiet fix. ## Sources 1. [A literature review and reflection on clinical trials: current challenges, future directions, and potential strategies to overcome practical barriers.](https://pubmed.ncbi.nlm.nih.gov/42609948/)