File Versioning in Bioinformatics: Best Practices

By Dr. Zubair Khalid, DVM, MS, PhD · 2026-11-10

File versioning in bioinformatics is the practice of tracking every state of every data file, reference sequence, annotation set, and pipeline script so that any result can be traced back to the exact inputs that produced it. The one-sentence rule of thumb: version code and small text metadata with Git, version large data with content-addressed storage or a data-aware tool such as DVC, and never overwrite a raw file.

Reproducibility failures in bioinformatics rarely come from bad statistics. They come from a reference genome that was silently swapped, an annotation GFF that was updated mid-project, a FASTQ that was trimmed in place, or a container that was rebuilt with different package versions. Alessandri and colleagues describe this directly: large and complex biological datasets create "inconsistencies and the lack of standardization in the analysis process," and those inconsistencies undermine the credibility of results [1]. Versioning is the mechanical layer that prevents those inconsistencies from reaching a publication.

This guide covers the concrete options, their pros and cons, the file formats you will actually version, and the mistakes that break traceability even when a team believes it is versioning properly.

Why Versioning Failures Are a Data Problem, Not a Discipline Problem

Most labs do not fail at versioning because people are careless. They fail because the tools they were taught, namely shared folders and filename suffixes, do not scale to terabytes and do not record relationships between files.

A genome center that produces 400 FASTQ files per run, each paired-end, cannot practically track "sample_R1_final_v3_reallyfinal.fastq.gz" across eight analysts. Filename versioning encodes history in a string that no machine can audit. When two analysts both create _v3, the history forks silently.

Formal versioning solves three distinct problems:

1. Identity. A checksum or content hash proves that the file you analyzed today is byte-identical to the file you analyzed in March.
2. Lineage. A record of which inputs produced which outputs, so a reviewer can walk backward from a figure to a FASTQ.
3. Immutability. Raw data is never modified, so downstream errors are always recoverable by re-running from the raw source.

The literature on file formats reinforces all three. Niu and colleagues found that bioinformatics tools frequently lack formal format specifications and are rarely tested for correct handling of input and output, producing interoperability problems that "at worst" lead to "undetected errors in scientific results" [2]. A versioning system that records checksums and tool versions is the practical defense against that class of error, because it lets you detect when a file changed or when a parser behaved differently.

The Core Versioning Options Compared

The table below is the decision core of this article. Each row is a real option used in production bioinformatics, described on its own terms.

The remainder of this article explains each option, then compares them on the criteria that actually decide a choice: file size, collaboration model, audit requirements, and whether the data is human or controlled-access.

Option by Option: What Each Approach Actually Does

Filename Suffixes

Filename versioning is the default in most labs. It is not a versioning system. It is a labeling convention, and it is worth using as a final export label rather than as history.

The convention that survives contact with real projects is a fixed, sortable schema. A workable pattern is:

{project}_{assay}_{sample}_{read}_{date}_{build}_{state}.{ext}

For example, prj042_rnaseq_S17_R1_2026-03-14_GRCh38.trimmed.fastq.gz. This encodes provenance in the name. It does not encode a checksum, so it cannot prove identity. Pair it with a manifest file.

Git for Scripts and Text

Git is correct for anything that is text and small: shell scripts, Python and R analysis code, Nextflow and Snakemake workflow definitions, YAML configs, sample sheets, GFF3 annotations for small genomes, and result tables under a few megabytes. Git records the full history of every change, supports review through pull requests, and lets you check out the exact code state used for a figure.

Git is wrong for FASTQ, BAM, CRAM, VCF with many samples, mzML, and most binary formats. A 200 GB BAM committed to Git is unrecoverable in practice because every clone downloads it and every history rewrite is expensive. Repository hosting services also impose hard file-size limits.

Use .gitignore aggressively for data directories. A typical rule set excludes data/, .fastq.gz, .bam, .cram, .vcf.gz, results/, and work/.

Git LFS

Git LFS replaces large files with small pointer files and stores the content on a separate server. It keeps the familiar Git workflow while avoiding repository bloat.

Git LFS is reasonable for figures, PDFs, small binary reference files, and test datasets. It is a poor fit for primary sequencing data because LFS storage and bandwidth are metered, and because LFS does not solve the problem of many researchers needing the same terabyte simultaneously without a shared cache.

DVC and Data-Aware Versioning

DVC tracks large files by storing a small .dvc file in Git that contains the content hash and the remote location. The data itself lives in a remote such as S3, Azure Blob, or an institutional filesystem. Checking out an old Git commit and running dvc pull restores exactly the data that commit referenced.

This is the strongest general answer for bioinformatics file versioning best practices in a small-to-medium lab, because it separates the two versioning problems (code history and data history) while linking them. The .dvc files are human-readable YAML, so a reviewer can inspect a hash without special tooling.

Limits are real. DVC assumes a remote exists and is reachable. It assumes analysts run dvc push rather than moving files manually. It adds a command to learn. And it does not lock files, so two people editing the same large output still conflict.

Content-Addressed and Immutable Storage

Object storage with checksums (S3 with SHA-256 metadata, or iRODS with integrity checks) makes identity structural: the address is the hash. If the content changes, the address changes. You cannot accidentally have two different files at the same path.

For institutional archives and anything that must survive staff turnover, this is the most durable option. It requires policy and infrastructure, which is why it is common at genome centers and rare in individual labs.

Workflow Provenance

Workflow managers record what ran, with which inputs, in which container. That is versioning of process, and it is the highest-value layer for pipelines.

Container pinning is the companion practice. Alessandri and colleagues built CREDO specifically because building reproducible Docker images for bioinformatics is hard: a Dockerfile must combine heterogeneous packages and environments such as Bioconductor and Conda, and CREDO stores all package information and dependencies in a GitHub-compatible format so images can be rebuilt from scratch [1]. The practical takeaway is to record the image digest, not the tag. ubuntu:latest is not a version. ubuntu@sha256:... is.

Archive and Encryption Formats

Some versioning problems are really exchange problems. When the deliverable is a publication supplement, a single container with a manifest is better than a directory tree.

The COMBINE Archive and OMEX format is exactly this: a ZIP container with a manifest listing contents, an optional metadata file, and the model and data files needed to reproduce a modeling and simulation experiment [3]. The same principle applies to any supplement: bundle, manifest, hash.

For controlled-access human genomic data, Crypt4GH defines a file container that enables byte-level compatible random access to encrypted data stored in standards such as SAM/BAM/CRAM/VCF/BCF [4]. That matters for versioning because you can keep encrypted copies under policy while still running analyses, instead of maintaining decrypted copies that become an unmanaged versioning liability.

Bioinformatics File Formats and How They Interact With Versioning

You cannot design a versioning scheme without knowing which formats are text, which are binary, which are compressed, and which are containers. The format determines whether Git diffing is meaningful, whether checksums are cheap, and whether partial reads are possible.

FASTQ vs FASTA

This comparison comes up constantly and the distinction is straightforward.

FASTA stores a sequence identifier line beginning with > followed by the sequence. It has no quality information. It is used for reference genomes, transcript sequences, protein sequences, and any context where per-base confidence is not needed.

FASTQ stores four lines per record: an @ header, the sequence, a + separator, and a per-base quality string encoded as ASCII characters. It is the standard for sharing sequencing reads because it combines sequence and quality in one record.

The versioning implications differ sharply:

• FASTA reference files are configuration, not data. A reference genome should be treated as a versioned dependency with a recorded accession, assembly name, and checksum. GRCh38 alone is insufficient because patch releases differ.
• FASTQ files are raw instrument output. They must be immutable. Never trim, deduplicate, or filter in place. Produce a new file with a new name and a new checksum.

FASTQ itself has a history of version drift that is a cautionary tale for format versioning. Cock and colleagues documented that FASTQ "lacks any formal definition to date" and existed in at least three incompatible variants: the original Sanger standard plus the Solexa and Illumina variants, with different quality encodings [5]. A pipeline that assumes one encoding and receives another will produce wrong quality interpretations without necessarily failing. The lesson generalizes: when a format has variants, record the variant, not just the format name.

Alignment and Variant Formats

SAM and BAM are the standard alignment formats, and BAM is the binary, usually compressed form. SamQL was developed because SAM-encoded files are difficult to query efficiently even for experienced bioinformaticians, and because new tools should support existing large datasets without requiring massive file transformations or infrastructure reorganization [6]. That constraint is a versioning constraint: you generally should not rewrite an alignment archive to make it queryable. Keep the archive, index it, and record the index version.

VCF and BCF follow the same text-versus-binary logic. Compress them, keep the index alongside (.tbi or .csi), and version the pair together. An index that does not match its data file is a silent failure mode.

Annotation and Interval Formats

BED is the canonical example of a format whose informal nature causes real bugs. Niu and colleagues created a formal specification from the original documentation, built a verification system called Acidbio, and tested input validation across 80 Bioconda packages that parsed BED, plus fuzzing to catch edge cases [2]. Their finding is that interoperability is not automatic even for a widely used format.

For versioning, that means BED and GFF files need a recorded specification version and a checksum. Two files that look identical can parse differently across tools.

GFF3 integrates with newer annotation formats. The DeNoFo toolkit, developed for de novo gene annotation, provides a standardized annotation format and explicitly integrates with established file formats such as FASTA and GFF to ensure comparability across studies [7][8]. The pattern is worth copying: when your project needs metadata that existing formats cannot hold, define a sidecar format and link it to the standard files rather than overloading the standard.

Specialized Binary and Container Formats

A large family of formats exists because general formats were too slow or too large:

• K-mer sets. The K-mer File Format is a general lossless framework for storing and manipulating k-mer sets, reporting space savings of 3 to 5 times compared with other formats and bringing interoperability across tools [9].
• Mass spectrometry. mzML, an open XML format, is widely adopted but suffers from large file size increases and largely sequential I/O. Toffee is an open random-access HDF5-backed format with lossless compression giving file sizes similar to the original vendor format and reconvertible to mzML without penalty [10]. mzDB uses SQLite with multiple indexing strategies and saves roughly 25 percent of storage versus XML formats, improving access times by a factor of two up to 2000-fold depending on the access pattern [11].
• Macromolecular structures. MMTF is a binary compressed representation reported as several times faster to parse and about a quarter of the file size of PDBx/mmCIF, allowing visualization of structures with millions of atoms in a browser [12].
• Hierarchical scientific data. HDF5 is popular but its complex specification has drawbacks. Exdir uses file system directories for hierarchy, human-readable YAML for metadata, binary NumPy files for datasets, and raw data in subdirectories, which makes it easier for version control systems to track because the data is spread across many files [13].
• Photon counting. Photon-HDF5 is an open format for single-molecule photon-counting data, with phconvert converting proprietary formats and a GUI that adds metadata without requiring YAML editing [14].
• Whole-slide imaging. Scanner-native formats such as Aperio SVS and 3DHistech MRXS differ, and a comparison of the two platforms found that native MRXS and MRXS-to-SVS conversions showed very high similarity in pixel values, color profile, and AI predictions, whereas native Aperio SVS files differed significantly in image size, pixel size, pixel number, and color profile [15]. Format translation and scanner hardware are separable effects, which is exactly why you record both the scanner and the format in your manifest.

The versioning rule for all of these is the same: record the format name, the format version or specification, the producing tool and version, and the checksum. Exdir's design point is the most generalizable insight in this group. Formats that split data across many files are friendlier to version control than monolithic binaries [13].

A Practical Versioning Architecture for a Working Lab

The following architecture is what most genomics labs converge on after a few painful incidents. It has five layers.

Layer 1: Immutable Raw Zone

Raw sequencing data lands in a read-only directory or bucket prefix. Permissions prevent deletion or modification by analysts. Each delivery gets a manifest with file name, size, checksum (SHA-256 or MD5 as provided by the sequencing center), and the run identifier. Register the manifest in Git.

The point of this layer is that any downstream failure can be recovered by re-running from raw. Labs that skip it eventually discover they cannot reproduce a result because the only copy of a FASTQ was trimmed in place eighteen months earlier.

Layer 2: Versioned Reference Bundle

Reference genomes, annotation GFF3 or GTF, variant databases, and indexes are dependencies, not inputs. Pin them.

Create a reference directory per assembly with a manifest.tsv listing file, source URL, accession, release date, and checksum. Record the exact release, not just the assembly name. Then record the bundle version in every pipeline config.

Layer 3: Code in Git With Reviewed Changes

All scripts and workflow definitions live in Git. Protect the main branch. Require review for changes that alter filtering thresholds, reference versions, or tool versions, because those changes alter results.

Tag releases. A tag such as v1.4.0-analysis should correspond to a specific figure or submission. That tag is the thing you cite in a methods section.

Layer 4: Data Tracked by Hash, Not by Name

Intermediate and processed data use content hashes. DVC is the lowest-friction implementation for most labs. Each processing step produces a new file with a new hash, and the .dvc file records which upstream files it came from.

The rule is one output, one name, one hash, never modified. If a step needs to change, the new run produces a new file. Storage is cheap relative to the cost of an irreproducible result.

Layer 5: Environment Pinning

Record the container digest for every tool, and prefer locked environments. Conda environments exported without explicit build strings drift. Docker tags move. The CREDO project's design goal, storing package information and dependencies in a GitHub-compatible format so Docker images can be rebuilt from scratch, is the right model: the environment definition must be versioned alongside the code so it can be reconstructed, not just referenced [1].

Versioning Naming Conventions and Checksums That Actually Work

A naming scheme fails when it requires judgment. Any field that an analyst must decide ("is this final?") will diverge across people.

Use only machine-derivable fields:

• Project code: fixed list, assigned once.
• Assay type: fixed vocabulary (WGS, RNAseq, ATAC, ChIP).
• Sample ID: from the sample sheet, never edited afterward.
• Read or file role: R1, R2, I1, or a fixed role name.
• Date: ISO 8601 (2026-03-14) so lexical sort equals chronological sort.
• Reference build: assembly plus release.
• Processing state: a controlled vocabulary such as raw, trimmed, aligned, called.

Avoid final, new, old, backup, copy, and personal initials. Each of those is a judgment call.

Checksums should be SHA-256 for archival integrity and are commonly provided as MD5 by sequencing centers. Record whichever the source provides, and record which algorithm was used. A checksum without an algorithm label is ambiguous and occasionally dangerous.

Verify on receipt and before analysis:

sha256sum -c manifest.sha256

For a large archive, generate the manifest once and store it in Git. The manifest is small, text, and diffable, which is exactly what Git is good at.

Worked Scenarios

Scenario A: A Two-Person Lab With 4 TB of RNA-seq

The realistic stack is Git for scripts, DVC with an S3 remote for FASTQ and BAM, and a pinned reference bundle. Raw FASTQ arrives read-only, is registered in DVC, and is never touched again. Each analysis run creates tagged Git commits. The cost is one afternoon of setup and one command (dvc pull) for a new collaborator.

The failure mode to avoid is treating DVC as optional. If one analyst pushes data with DVC and another copies files manually, the hashes diverge and the lineage breaks.

Scenario B: A Core Facility Serving Twenty Labs

The core cannot rely on per-lab tooling. The right answer is content-addressed storage with a delivery manifest per project, plus a fixed directory contract that downstream labs can assume. Each delivery includes checksums, and the core's internal pipeline provenance is exported alongside the data.

The core should also pin container digests for every pipeline version and publish the digest list. Labs that re-run analyses years later need the digest, not the tag.

Scenario C: A Consortium With Controlled-Access Human Data

Encrypted storage plus Crypt4GH containers addresses the policy problem, since the format enables native access to encrypted data stored in SAM/BAM/CRAM/VCF/BCF without creating decrypted copies [4]. Version control applies to the access policies, the key management records, and the analysis code. The genomic data itself stays under controlled-access governance with checksums and immutable versioning at the storage layer.

Scenario D: A Methods Paper Where the Supplement Is the Deliverable

Bundle everything into one container with a manifest. The COMBINE Archive/OMEX model, a ZIP with a manifest, optional metadata, and the constituent files, is directly applicable [3]. Add a README, a manifest.sha256, and the exact tool versions. A reviewer should be able to unpack and verify without contacting the authors.

Common Mistakes and Limitations

Versioning the filename instead of the content. Two files named sample_v2.bam produced by two analysts are not the same file. Without a checksum, you cannot tell. This is the single most common failure.

Committing large binaries to Git. It works until the repository is 40 GB and every clone times out. Recovery requires history rewriting, which changes every commit hash and breaks references in papers and issue trackers.

Trimming in place. Modifying raw FASTQ destroys the ability to re-run with different parameters. Always write a new file.

Recording tool names without versions or digests. "Used BWA" is not reproducible. Record the version and the container digest.

Recording the assembly but not the release. GRCh38 spans multiple releases with different sequences in some regions. Patch releases matter, and annotation releases matter more.

Index and data drift. A BAM and its .bai, or a VCF and its .tbi, must be versioned as a unit. An index built from a different data version can produce wrong results without an error message.

Assuming a format is stable because it is widely used. Niu and colleagues showed that even BED, a very common format, lacks robust interoperability testing across tools and that edge-case handling varies [2]. Record the specification version.

Assuming a format has one variant. FASTQ had at least three incompatible quality encodings before a common definition was published [5]. If your pipeline reads FASTQ, either validate the encoding or use a tool that handles all variants.

Trusting latest tags. Container tags are mutable. Pin digests.

No manifest for reference bundles. Without a manifest, nobody can prove which reference was used two years later.

Versioning outputs but not the parameters. A hash of a BAM tells you which BAM. It does not tell you the mapping parameters. Record parameters in the config file under Git.

Over-versioning exploratory work. Not every scratch file needs a hash. Version what feeds a result. Exploratory notebooks can live in a separate, disposable workspace.

Assuming the workflow manager covers everything. Provenance is recorded for files produced inside the workflow. Files copied in by hand, downloaded references, and manually edited sample sheets fall outside it unless you add them to the tracked inputs.

Treating a compressed archive as immutable proof. A ZIP or OMEX container is only as good as its manifest. If the manifest lists files without checksums, the container does not establish identity [3].

Ignoring human factors. A versioning system that requires more than one extra command per day will be bypassed. Choose tooling that fits the existing workflow, then enforce the parts that matter, specifically raw immutability and reference pinning.

Frequently Asked Questions

What is the difference between file versioning and backup?

Backup makes copies so you can recover from loss. Versioning records identity and lineage so you can prove which input produced which output. A lab needs both, and backup alone will not let you reproduce a result from two years ago.

Should I use Git or DVC for sequencing data?

Use Git for code, configs, sample sheets, and small text files, and use DVC or content-addressed storage for FASTQ, BAM, CRAM, and other large binaries. Commit the DVC pointer files to Git so that the code history and data history link at every commit.

How do I version a reference genome?

Treat it as a pinned dependency with an assembly name, a release identifier, a source URL, a retrieval date, and a checksum, all recorded in a manifest that lives under version control. Record the release, not only the assembly.

Is FASTQ or FASTA better for versioning?

Neither is inherently better, but FASTQ is raw instrument output with quality scores and must be kept immutable, while FASTA references are configuration-like dependencies that need accession and checksum pinning. The format determines the versioning treatment, not the quality of the format.

Do workflow managers replace a versioning system?

No. Workflow managers record provenance for files they produce from tracked inputs. They do not version raw deliveries, manually downloaded references, or files moved outside the workflow, so you still need immutability at the raw layer and pinned references.

How many versions of an intermediate file should I keep?

Keep every version that contributed to a reported result, and delete scratch intermediates once the result is superseded. Storage is inexpensive compared with re-running a multi-day pipeline because an intermediate was overwritten.

Sources

1. CREDO: a friendly Customizable, REproducible, DOcker file generator for bioinformatics applications
2. Assessing and assuring interoperability of a genomics file format
3. COMBINE archive and OMEX format: one file to share all information to reproduce a modeling project
4. Crypt4GH: a file format standard enabling native access to encrypted data
5. The Sanger FASTQ file format for sequences with quality scores, and the Solexa/Illumina FASTQ variants
6. SamQL: a structured query language and filtering tool for the SAM/BAM file format
7. DeNoFo: a file format and toolkit for standardised, comparable de novo gene annotation
8. DeNoFo: a file format and toolkit for standardized, comparable de novo gene annotation
9. The K-mer File Format: a standardized and compact disk representation of sets of k-mers
10. Toffee - a highly efficient, lossless file format for DIA-MS
11. mzDB: A File Format Using Multiple Indexing Strategies for the Efficient Analysis of Large LC-MS/MS and SWATH-MS Data Sets*
12. MMTF-An efficient file format for the transmission, visualization, and analysis of macromolecular structures
13. Experimental Directory Structure (Exdir): An Alternative to HDF5 Without Introducing a New File Format
14. A user-friendly tool to convert photon counting data to the open-source Photon-HDF5 file format
15. Comparison and image analysis of different file formats on whole-slide images for colorectal and non-small cell lung carcinomas