Versioning Raw Sequencing Data: How to Do It Right

By Dr. Zubair Khalid, DVM, MS, PhD · 2026-11-11

Versioning raw sequencing data means keeping every distinct batch of unprocessed reads, together with its checksums and metadata, uniquely identified and retrievable, while everything downstream is free to change. The rule of thumb is simple: raw files are immutable, so what you actually version is the identity and context of each data batch, not the bytes inside the files. If you only ever work with data you generate yourself, an immutable store plus a versioned manifest is usually the best option. If you regularly pull large public datasets that get re-released by their curators, you need a versioned acquisition and archival process instead. Most laboratories end up combining the two.

This article walks through what versioning actually means for raw data, why the evidence says it matters, the main options with their pros and cons, a side-by-side comparison, worked scenarios, and the mistakes that cause real data loss.

What "Version" Means When the Files Never Change

Version control, in the software sense, tracks successive states of files that people edit. Raw sequencing data breaks that model. A raw read file, whether it is a FASTQ (a text format that stores sequenced reads with per-base quality scores), a BCL (the per-cycle base-call output an Illumina instrument writes during a run), or a run directory straight off the sequencer, is a finished artifact. Once the run passes quality review, nobody should ever edit it. Editing a raw file is data fabrication by accident.

So versioning raw sequencing data is really three separate problems bundled together.

First, there is the versioning of the dataset itself. Each sequencing run, each public data release, each re-delivery from a collaborator is a distinct state. You need a stable identifier for each state so that a paper, a pipeline log, or a regulatory submission can point to exactly one of them.

Second, there is the versioning of the context. The same FASTQ file means something different depending on which instrument produced it, which library preparation kit was used, which flow cell, which lane, which sample sheet, and which base-calling software. Context metadata is the part that gets lost, and it is the part that makes data useless a year later.

Third, there is the versioning of everything that touches the raw data. The reference genome, the alignment software, the counting package, the container image. Raw data does not change, but the interpretation of raw data changes with every tool update, and that is precisely why you must be able to trace back which raw state and which tool state produced a given result.

A concrete vocabulary helps here:

• Checksum is a fixed-length fingerprint computed from a file's contents, such as an MD5 or SHA-256 digest. If even one byte changes, the checksum changes. Checksums are how you prove a file is the same version as the one you recorded.
• Manifest is a small text file listing each data file with its checksum and usually its size. It is the cheapest, most durable version record you can make.
• Provenance is the recorded chain of what happened to a dataset: where it came from, which process touched it, when, with which parameters.

Why Versioning Raw Sequencing Data Actually Matters

Two lines of evidence make the case better than any policy memo.

The first is about scale. Large public genomics resources are not static. The Cancer Genome Atlas (TCGA), a National Cancer Institute effort to profile at least 500 cases across 20 tumor types with multiple genomic platforms, surpassed 1.2 petabytes of data spanning whole genome sequence, whole exome sequence, methylation, RNA expression, proteomic, and clinical datasets [1]. Data on that scale is released through public portals in waves, and files get corrected, added, and superseded. A group that downloaded TCGA in 2014 and a group that downloaded it in 2017 were, without deliberate version tracking, working with materially different datasets while believing they used "the TCGA data." This is exactly the problem the TCGA Expedition system was built to solve. It is a set of scripts in Bash, Python, and Java that download, extract, harmonize, version, and store TCGA data and its metadata, so that every analysis can be tied to a specific archived release rather than to whatever happened to be on the portal that week [1].

The second is about analysis sensitivity. You might assume that raw data is raw data and small details cannot matter much. Work on single-cell RNA-seq says otherwise. A detailed comparison of Seurat and Scanpy, the two most widely used packages for standard single-cell workflows, found considerable differences in their outputs, and the size of those differences was approximately equivalent to what you would see if you had sequenced less than 5% of the reads or analyzed less than 20% of the cells in a benchmarking dataset [2]. The same study found that distinct versions of Seurat and Scanpy can produce very different results, especially in differential expression analysis [2].

Read that finding from the data management side. If choosing software version A instead of version B is equivalent to throwing away most of your sequencing depth, then the pairing of a raw data state with a tool state is the real unit of scientific record. Versioning raw data without versioning its processing context, or vice versa, leaves half the equation unrecorded.

The Main Options for Versioning Raw Sequencing Data

There are five practical approaches. They are not mutually exclusive, and mature genomics data storage setups usually layer two or three of them. Each is explained on its own terms first, then compared.

Option 1: Immutable Files Plus a Versioned Manifest

This is the baseline everyone should implement regardless of anything else. You designate a raw data area that no process and no person is allowed to modify, and you record the identity of each batch in a manifest that lives somewhere separately versioned.

The mechanics look like this. After a run finishes and passes review, you copy the run directory or FASTQ files into the raw area once, using a transfer method that verifies bytes:

rsync -av --checksum /sequencer/runs/260101_M00423/ /data/raw/260101_M00423/

Then you compute checksums and write a manifest:

cd /data/raw/260101_M00423/
sha256sum *.fastq.gz > MANIFEST.sha256
md5sum *.fastq.gz > MANIFEST.md5

You include the sample sheet, the run parameters XML, and any QC reports in the manifest, because they are part of the version. The manifest itself goes into a Git repository, or at minimum gets copied to a second location, because a manifest stored only next to the data it describes dies with the data.

Now each batch has a name (the run directory plus a date), a content fingerprint (the manifest), and a permanent home. If anyone asks "exactly which data did the 2025 breast cancer paper use," the answer is a manifest path and a Git commit hash. If a disk starts silently corrupting files, a scheduled checksum comparison against the manifest catches it:

sha256sum -c MANIFEST.sha256

Pros: nearly free, works with any storage system, immediately auditable, catches corruption. Cons: it does nothing to organize old copies, it does not automatically preserve superseded releases, and it depends on someone actually running the commands. It is versioning of identity, not versioning of history.

Option 2: Object Storage with Native Versioning

Object storage systems (S3-compatible services, Backblaze B2, MinIO, and similar) can be configured so that every write to a key creates a new immutable version and overwrites never destroy the old object. You enable it once per bucket:

aws s3api put-bucket-versioning \
  --bucket lab-raw-sequencing-data \
  --versioning-configuration Status=Enabled

After that, if a file called sample_A/reads_R1.fastq.gz is uploaded three times with three different contents, all three versions exist under the same key, each with a unique version ID, and any of them can be retrieved. Deletion is typically implemented as a tombstone marker, with the underlying bytes retained until an explicit lifecycle policy removes them.

This is genuinely versioning the raw files themselves, automatically, with no discipline required from users. It pairs perfectly with Option 1: enable bucket versioning, and also keep checksum manifests in a separate bucket or repository.

Pros: automatic protection against overwrite and accidental deletion, works at petabyte scale, access controls and audit logs are built in, lifecycle rules can tier old versions to cheaper storage. Cons: storage cost grows with every superseded version unless you set lifecycle policies carefully, the versioning is invisible to tools that do not speak the object storage API, and buckets without proper access control can still be emptied by a compromised credential, so replication to a second location remains necessary for anything irreplaceable.

Option 3: A Versioned Acquisition and Archival Pipeline

When your raw data mostly comes from outside, from public portals, collaborating centers, or core facilities, the hardest problem is not storing your own files. It is capturing the states of data you do not control, before a curator silently replaces them. The right pattern here is a scripted, scheduled pipeline that downloads, validates, versions, and archives each release as a unit.

TCGA Expedition demonstrates the full pattern for a large public resource: scripts download the data, extract it, harmonize it into a consistent structure, version it, and store it, with the resulting repository supporting command-line access on high-performance computing facilities and integration with third-party tools [1]. The same group extended the repository to re-associate TCGA data with de-identified electronic health record data for a subset collected at their institution, which shows the other benefit of an owned copy: you can join external data to your own annotations, something the public portal will never do for you [1].

You do not need their code to apply their pattern. The minimum viable version looks like:

1. A script (Bash or Python) that downloads a defined scope of data, for example one TCGA tumor type or one ENCODE release.
2. A checksum validation step against whatever the provider supplies, plus a manifest of your own if they supply none.
3. A version stamp, usually a date plus the provider's release identifier, written into both the directory name and a metadata record.
4. An archive step that snapshots the validated batch to long-term storage and never modifies it afterward.

Pros: you own a copy of every release you ever used, analyses stay reproducible even after upstream corrections, harmonization happens once instead of per-project, and the pipeline is auditable. Cons: it requires scripting and maintenance effort, it consumes serious storage (multi-petabyte resources are only feasible if you subset deliberately, as TCGA Expedition did by scoping what it mirrored [1]), and it needs monitoring, because a downloader that fails silently for three months leaves three months of releases missing.

Option 4: Git-Based Tracking with Large File Extensions

Git itself cannot handle raw sequencing data. It is designed for many small text files, it stores full content of every version, and it will grind to a halt on a directory of multi-gigabyte FASTQ files. But the Git model of tracked states is exactly right, so three tools extend it to large files.

Git LFS (Large File Storage) replaces big files in a repository with pointer text and stores the actual bytes on a separate LFS server. It works well up to moderate dataset sizes and is the natural choice when raw data files are the exception rather than the rule.

DVC (Data Version Control) is built for this domain. You track directories of data:

dvc add data/raw/260101_M00423
git add data/raw/260101_M00423.dvc
git commit -m "Add run 260101_M00423 raw data"
dvc push

The .dvc pointer file (containing a content checksum of the directory) is versioned in Git, while the bytes are pushed to any configured remote: an S3 bucket, SSH storage, a network drive. Switching between dataset states becomes a Git checkout plus dvc checkout, which means an analysis repository can declare exactly which data state and which code state produced a figure.

git-annex tracks file content by checksum and keeps the bytes wherever they live, including plain directories on mounted drives. It is the most flexible about storage locations and the most forgiving of very large collections.

Pros: code and data states travel together, every change is attributed and dated, checkout semantics make "go back to the 2024 dataset" a single command, remotes give you replication as a side effect. Cons: Git history grows with every tracked change, some teammates will find the pointer model confusing, these tools version what you remember to add rather than watching storage automatically, and none of them should ever hold the primary copy of irreplaceable data. They track, they do not back up.

Option 5: Pinning External Versions Without Mirroring

The lightest option is to not copy public data at all and instead record enough information to re-obtain the exact state later: accession numbers, the repository, the download date, the URL of the specific release, and checksums of every downloaded file. For SRA (Sequence Read Archive) data, that means the run accessions plus an md5sum manifest you compute yourself, because the archive is not obliged to keep every file byte-identical forever.

Pros: zero storage cost, fast to implement. Cons: providers correct and reorganize data, entire resources occasionally migrate formats, and a checksum you never recorded is a checksum you cannot verify later. This option is a last resort for labs with no storage budget, and it should never be used for data central to an active project.

Side-by-Side Comparison

Choosing Between the Options: The Criteria That Decide

The "best versioning raw sequencing data" question has no single answer, but it has a short list of deciding questions.

Who controls the data? Data you generate yourself changes state only once, at creation, so identity versioning (manifest, immutable store) is sufficient. Data you download changes state without your permission, so you need historical versioning, either a pipeline that archives each release or object storage that keeps every upload.

What must be reproducible? If the deliverable is a publication, the unit of record is a raw data state plus a tool state, because tool choice moves results by margins comparable to losing most of a dataset [2]. That argues for Option 4 or its equivalent, since it records the pairing, layered over Option 1 or 2 for the bytes.

How large is the data and how often does it change? Whole genome sequencing centers generating terabytes per week need automatic, storage-level versioning (Option 2) because manual discipline fails at volume. A laboratory generating one run per month can run Option 1 by hand and never lose anything.

What does your regulatory or funding context demand? Grant data management plans and institutional policies increasingly require retained, verifiable raw data. Checksum manifests plus an immutable copy satisfy the verification requirement of essentially any such plan. Object storage audit logs satisfy the "who touched it" requirement.

What is your failure mode? The honest pros and cons summary is that Option 1 protects against corruption but not against deletion, Option 2 protects against overwrite and deletion but not against a compromised account, Option 3 protects against upstream change but not against your own pipeline stopping, and Option 4 protects against ambiguity about what was analyzed but not against data loss. Mature genomics data storage combines them so their weaknesses do not overlap.

Worked Scenarios

Scenario A: A Small Academic Laboratory Generating Its Own Runs

A laboratory sequencing human exomes on a shared instrument, roughly two runs per month, produces around a terabyte of raw data monthly. The pragmatic setup:

1. A raw data directory tree organized by run date and instrument, mounted read-only for everyone except a single ingestion account.
2. On ingest, rsync -av, checksum from the instrument export, then sha256sum over all files plus the sample sheet, producing one manifest per run.
3. The manifests, the sample sheets, and a small README per run live in a Git repository (Option 4, but with pointers only, no large files), pushed to a hosted remote.
4. The raw bytes themselves get two copies: the primary server and an object storage bucket with versioning enabled (Option 2), with a monthly scripted sha256sum -c against the manifests.

Total new tooling: one ingestion script and one cron job. Every run is now identifiable forever, corruption is caught monthly, and an overwriting accident in the bucket cannot destroy the original upload.

Scenario B: A Computational Group Consuming a Large Public Resource

A cancer genomics group uses TCGA data across multiple projects. Public portals release data in waves, files are corrected over time, and the resource exceeds 1.2 petabytes, so mirroring everything is out of reach [1]. The group instead applies the TCGA Expedition pattern at subset scale [1]:

1. A Python script downloads only the platforms the group uses (say, whole exome sequence, RNA expression, and clinical data) for the tumor types under study.
2. Each quarterly acquisition is stamped as tcga-<tumor>-<platform>-<YYYYQn>, validated against provider checksums, given its own manifest, and archived to an immutable bucket.
3. A metadata table records, for every release, the download date, file counts, total bytes, and any known corrections announced upstream.
4. Every project README names the exact release directory it used, and analyses are run against a read-only view of that directory.

When a colleague asks why a survival analysis differs from an older paper, the answer is on record: the earlier paper used the 2019 release, in which certain clinical fields had known issues, and the group's 2023 release contains the corrected values.

Scenario C: Versioning the Data-to-Result Pairing for a Methodologically Sensitive Analysis

A single-cell RNA-seq project will compare clustering and differential expression results across packages. Given documented evidence that different packages and different package versions can produce very different outputs, with divergence roughly equivalent to sequencing less than 5% of reads or analyzing less than 20% of cells [2], the group treats the full state as the versioned unit:

1. Raw FASTQ files land in the immutable store with a manifest (Option 1).
2. The count matrix generation is run inside a pinned container image, and the image digest is recorded.
3. The analysis repository uses DVC, so the raw data directory, the container digest, the package versions, and the code are all referenced by one Git commit.
4. Each re-analysis, for example after a Seurat or Scanpy upgrade, is a new tagged commit, and the paper's supplement points reviewers to the tag rather than to a vague description of "the analysis pipeline."

If a reviewer or a rival group reruns from that tag, they get the same numbers. If the group later wants to test whether conclusions survive a package upgrade, the upgrade is a new tag, and both result sets remain on record for direct comparison, which is exactly the comparison the Seurat and Scanpy study showed can matter [2].

Common Mistakes and Limitations

Versioning processed files and calling it raw data versioning. A versioned count matrix or a versioned BAM is useful, but it versions an interpretation. If the raw data is not itself under version discipline, a reprocessing claim cannot be verified from the original bytes.

Storing the manifest next to the data with no other copy. A manifest is a version record. If it lives only in the directory it describes, corruption or deletion takes the evidence and the crime scene at the same time. Manifests belong in at least two places, one of them a version-controlled repository.

Treating Git as a backup. Git, Git LFS, DVC, and git-annex track states. If the bytes live on one disk and the pointers live in one repository, you have zero backups and a false sense of order. The pointers should be replicated and the bytes should live in at least two independent locations.

Assuming public repositories are frozen. Portal data changes. Files are corrected, supplementary files are added, accessions get updated. Any long project that re-downloads instead of using a pinned, checksummed, archived copy is quietly working with a moving target.

Enabling object storage versioning without lifecycle policies. Versioning that keeps every overwritten copy forever will inflate storage bills steadily. Set lifecycle rules that tier or expire superseded versions on a schedule that matches your retention requirements, and review them annually.

Recording tool versions in a free-text methods paragraph. "We used Seurat for clustering" is not versioning. Record exact package versions, exact reference genome build, and exact parameters in a machine-readable file inside the versioned repository. The finding that distinct package versions produce substantially different single-cell results [2] is precisely the kind of surprise that a free-text paragraph cannot explain retroactively.

Letting the ingestion pipeline fail silently. A scheduled downloader that errors out is worse than no downloader, because everyone believes data is flowing. Monitor it with a completion marker file per release and an alert when a scheduled run does not produce one.

Confusing replication with versioning. Two identical copies of the same file are redundancy, not versioning. Versioning answers "which state of the data is this." Redundancy answers "do I still have it." Projects need both, and the distinction should be visible in directory names and metadata.

One general limitation. The options here address data infrastructure, and the evidence cited comes from specific domains, chiefly large public cancer genomics resources and single-cell RNA-seq tooling [1][2]. Data types with different maturity of upstream curation, for example long-read data or clinical pipelines under formal regulatory control, may carry additional requirements that go beyond what this article covers.

Frequently Asked Questions

Do I need to version the raw files themselves, or is a versioned manifest enough?

For data you generate once and never modify, a checksummed manifest plus an immutable copy is sufficient and is the right baseline. For data that arrives in repeated releases from someone else, such as public portal data, you want historical versioning of the files or directories themselves, because the curator can replace content under the same name at any time.

Is Git enough for raw sequencing data?

Plain Git is not, because it stores full content of every version and collapses under multi-gigabyte binary files. Use Git for manifests, sample sheets, and small metadata, and extend it with Git LFS, DVC, or git-annex for the large files, keeping the actual bytes on replicated storage outside the repository history.

How do I version data I download from public repositories?

Archive each download as a dated, named release with its own checksum manifest and a metadata record, then never modify the archived copy. Scripting the download, validation, versioning, and archival steps makes the process repeatable and catches the moment a provider changes their release.

If a sequencing core redelivers a run with corrected base calls, do I replace the old files?

No. Keep the original batch under its original identity, ingest the redelivery as a new, separately named batch with its own manifest, and record in metadata that the new batch supersedes the old one and why. Replacing in place destroys the only record of what the original analyses actually used.

Which checksum should I use for raw sequencing files?

SHA-256 is the current default and is available everywhere via sha256sum. Many legacy manifests and public repositories use MD5, so computing both for ingested batches costs little and keeps you compatible with older records. The important part is that the checksum exists, is written to a manifest, and is verified on a schedule.

How long should old versions of raw data be kept?

Follow the retention period specified in your funding award, institutional policy, or journal data availability policy, and for data backing publications keep the exact versioned state cited in the paper for at least as long as the funder requires. For superseded intermediate deliveries that back no publication, a lifecycle rule can expire them once the successor batch has been validated and archived.

Sources

1. TCGA Expedition: A Data Acquisition and Management System for TCGA Data.
2. The impact of package selection and versioning on single-cell RNA-seq analysis.