Tracking Container Versions in Bioinformatics Pipelines

By Dr. Zubair Khalid, DVM, MS, PhD · 2026-11-11

Two analysts run the same pipeline on the same sequencing data and get different variant calls. The code is identical. The difference sits inside the containers: one analyst pulled the image tagged 1.17 in March, the other pulled a rebuilt image carrying the same tag in June. Tracking container versions in git makes that failure impossible to miss, because every result points back to the exact image bytes that produced it.

By the end of this guide you will be able to pin container images by digest, keep a container lockfile under version control, record which image every task actually used, upgrade containers deliberately with tests, and reproduce an analysis from any past release. You will need git 2.30 or newer, Docker Engine or Docker Desktop on your workstation, Apptainer 1.x if you work on a shared cluster, about 2 GB of free disk space for the example images, and optionally Nextflow 23.10 or later or Snakemake 8 or later. An account on a container registry such as Docker Hub or GitHub Container Registry helps with the archiving step.

Why Container Versions Decide Whether Your Results Reproduce

Software is part of the scientific method in bioinformatics, and it is the part that changes most quietly. When five commonly used 16S rRNA pipelines analyzed the same amplicon data, the number of ASVs (amplicon sequence variants, the units used to count distinct sequences) ranged from 1,958 to 20,140, and the case-control association results differed across pipelines [1]. A separate reanalysis of a case-control microbiota study compared four pipelines that differed in tool versions and databases and showed that those choices changed the biological conclusions that could be drawn [2]. "We used QIIME" is therefore not a methods statement. "We used QIIME 1.9.1 inside image X with digest Y" is.

Containers exist to make that second sentence writable. The nf-core framework builds community pipelines in Nextflow with native support for Docker and Singularity precisely so that analyses run the same way across institutions [3]. Workflow managers in general are the accepted route to reproducible, scalable, and shareable analysis pipelines [4]. The pattern extends to individual tools: the dshiver project wrapped the shiver HIV-1 assembly pipeline in Docker so that other groups could run it without fighting dependency installation [5].

Packaging only helps if the reference to the package cannot silently change. A tag is a label that a maintainer can move to a different image at any time. A digest is a content hash of the image itself, and it changes only if the bytes change. Tracking container versions in bioinformatics pipelines means recording that digest in git, next to the code it belongs to, so that any result you report can be traced to exact software.

Tags, Digests, and the Git Vocabulary You Need

A container image is a packaged filesystem holding software, libraries, and metadata, built in layers so updates reuse unchanged layers. A registry is a server that stores and serves images. A tag is a human-readable label such as 1.17 or latest attached to an image, and tags are mutable pointers. A digest is a SHA-256 hash of the image manifest, written as sha256: followed by 64 hexadecimal characters, and digests are immutable content addresses. A lockfile is a committed file that records the exact resolved versions your project depends on.

For scientists meeting git for the first time, four terms cover most of what follows. A repository is a folder whose full history git tracks. A commit is a named snapshot of the tracked files. A tag in git is a permanent name pinned to one commit. A clone is your local copy of a repository. Git tags and container tags share a name and nothing else: git tags do not move once pushed, while container tags can.

How to Track Container Versions in Git, Step by Step

The walkthrough uses a small RNA-seq quality-control pipeline with two tools, FastQC and samtools, served from the quay.io/biocontainers registry. Every digest shown is an example. Your digests will differ, which is exactly why you record your own.

Step 1. Create a repository that holds code and container records together

mkdir rna-qc-pipeline && cd rna-qc-pipeline
git init

Expected output:

Initialized empty Git repository in /home/alice/rna-qc-pipeline/.git/

Create a plain-text inventory of every external image the pipeline can run:

# containers.yml
quay.io/biocontainers/fastqc:0.12.1--hdfd78af_0
quay.io/biocontainers/samtools:1.17--h0041a0a_1

The BioContainers tag format packs the tool version and the build string together, so the FastQC tag means version 0.12.1, build hdfd78af_0. Two builds of the same version can differ, which is one more reason the tag alone is not enough.

Step 2. Pull each image and record its digest

docker pull quay.io/biocontainers/fastqc:0.12.1--hdfd78af_0
docker images --digests

Expected output, with columns truncated:

REPOSITORY                     TAG                  DIGEST
quay.io/biocontainers/fastqc   0.12.1--hdfd78af_0   sha256:8f4b6c1e2a9d7f3c5b0e1a8d4c6f2b9e7a3d5c1f8e0b4a6c2d8e4f0a6b8c2d4e

The -digests flag adds the digest column to the listing. Write the digest into a lockfile:

# containers.lock.yml
fastqc:
  image: quay.io/biocontainers/fastqc
  tag: "0.12.1--hdfd78af_0"
  digest: "sha256:8f4b6c1e2a9d7f3c5b0e1a8d4c6f2b9e7a3d5c1f8e0b4a6c2d8e4f0a6b8c2d4e"
  pinned: "2026-11-11"
samtools:
  image: quay.io/biocontainers/samtools
  tag: "1.17--h0041a0a_1"
  digest: "sha256:3a7c9e1b5d2f8a4c6e0b9d3f1a5c7e9b2d4f6a8c0e1b3d5f7a9c1e3b5d7f9a2c"
  pinned: "2026-11-11"

For more than a handful of images, generate the lockfile instead of typing it:

#!/usr/bin/env bash
# scripts/pin_containers.sh
set -euo pipefail
: > containers.lock
while read -r image; do
  docker pull "$image"
  digest=$(docker inspect --format '{{index .RepoDigests 0}}' "$image")
  echo "$image $digest" >> containers.lock
done < containers.yml

set -euo pipefail stops the script at the first failed pull, so you never record a half-pinned inventory. The -format template prints the first entry of RepoDigests, which is the digest the registry actually served.

Step 3. Reference the pinned images in your workflow

Nextflow process:

process FASTQC {
    container 'quay.io/biocontainers/fastqc:0.12.1--hdfd78af_0'

    input:
    path reads

    output:
    path "*_fastqc.html"

    script:
    """
    fastqc ${reads}
    """
}

Snakemake rule:

rule fastqc:
    input:
        "data/sample_R1.fastq.gz"
    output:
        "qc/sample_fastqc.html"
    container:
        "docker://quay.io/biocontainers/fastqc:0.12.1--hdfd78af_0"
    shell:
        "fastqc {input} -o qc/"

Run Nextflow with nextflow run main.nf -with-docker and Snakemake 8 with snakemake, software-deployment-method apptainer (older releases use -use-singularity). If you build on nf-core, the modules already ship with pinned container references, and your job is to record the pipeline release rather than hand-pin every tool [3].

Step 4. Commit code and container records in the same commit

git add containers.yml containers.lock.yml main.nf
git commit -m "Pin fastqc 0.12.1 and samtools 1.17 by digest"
git tag -a v1.0.0 -m "First validated release, test dataset outputs in tests/expected"
git log --oneline --decorate

Expected output:

a1b2c3d (HEAD -> main, tag: v1.0.0) Pin fastqc 0.12.1 and samtools 1.17 by digest
e4f5a6b Add FASTQC and samtools index processes

The -a flag creates an annotated tag, which stores a message, a date, and your name, and is the right kind of tag for releases. The git tag and the lockfile now describe the same moment in history, so a result labeled v1.0.0 in your notebook maps to a commit holding both the code and the exact image digests.

Step 5. Verify what actually ran

Confirm the local image matches the lockfile:

docker inspect --format '{{index .RepoDigests 0}}' quay.io/biocontainers/fastqc:0.12.1--hdfd78af_0

quay.io/biocontainers/fastqc@sha256:8f4b6c1e2a9d7f3c5b0e1a8d4c6f2b9e7a3d5c1f8e0b4a6c2d8e4f0a6b8c2d4e

Check that the binary inside matches the version you expect:

docker run --rm quay.io/biocontainers/samtools@sha256:3a7c9e1b5d2f8a4c6e0b9d3f1a5c7e9b2d4f6a8c0e1b3d5f7a9c1e3b5d7f9a2c samtools --version

samtools 1.17

For Nextflow, every task writes a .command.run script into its work directory, and that file contains the image reference the task used:

grep "biocontainers" work/12/ab3cd4/.command.run

Recent Nextflow versions can also record a container column in the run trace when you start the pipeline with -with-trace trace.tsv. On a cluster, the equivalent check runs against the .sif file you pulled, which is why the Apptainer section below recommends pulling by digest. Stamp reports with the exact release using git describe, tags, which prints v1.0.0 on the release commit and v1.0.0-2-ga1b2c3d when you are two commits past it.

Step 6. Upgrade on purpose, not by accident

When a new samtools release matters to you, pull it, record it, test it, and only then commit:

docker pull quay.io/biocontainers/samtools:1.19.2--h50ea8bc_1
docker inspect --format '{{index .RepoDigests 0}}' quay.io/biocontainers/samtools:1.19.2--h50ea8bc_1

Update containers.lock.yml and the process directive, then review the change:

git diff containers.lock.yml

 samtools:
-  tag: "1.17--h0041a0a_1"
-  digest: "sha256:3a7c9e1b5d2f8a4c6e0b9d3f1a5c7e9b2d4f6a8c0e1b3d5f7a9c1e3b5d7f9a2c"
+  tag: "1.19.2--h50ea8bc_1"
+  digest: "sha256:d9e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3d4e5f6a7b8c9d0e1"

Run your test dataset and compare outputs against the previous release before you tag. Domain best-practice guidance, such as the published workflow recommendations for single-cell RNA-seq analysis, is worth consulting when an update changes an analysis step rather than just a patch level [6]. Benchmark-style reference material is the standard way to check that a pipeline change behaves as intended [7]. Then commit and tag:

git add containers.lock.yml main.nf
git commit -m "Upgrade samtools 1.17 to 1.19.2, test dataset outputs unchanged"
git tag -a v1.1.0 -m "samtools 1.19.2"

Step 7. Reproduce a result from an old release

git checkout v1.0.0
docker pull quay.io/biocontainers/samtools@sha256:3a7c9e1b5d2f8a4c6e0b9d3f1a5c7e9b2d4f6a8c0e1b3d5f7a9c1e3b5d7f9a2c
nextflow run main.nf

The checkout restores the old code and the old lockfile together, and the pull fetches the exact bytes by digest regardless of what has happened to the tag since. Return to the present with git checkout main. If the registry no longer serves the image, restore it from your archive, which is the next step.

Step 8. Archive the images you depend on

Registries remove images, and pull access can disappear with an account or a rate limit. Archive in two ways. First, a tarball with a recorded checksum:

docker save -o images/samtools_1.19.2.tar quay.io/biocontainers/samtools@sha256:d9e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3d4e5f6a7b8c9d0e1
sha256sum images/samtools_1.19.2.tar

c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0c1d2e3f4a5b6c7d8e9f0a1b2c3d4  images/samtools_1.19.2.tar

Commit the checksum in a file such as archive-checksums.txt and push the tarball to institutional storage. Second, a copy in a registry you control:

docker tag quay.io/biocontainers/samtools@sha256:d9e1f2a3... myregistry.lab.org/pinned/samtools:1.19.2
docker push myregistry.lab.org/pinned/samtools:1.19.2

For public releases, attach the archive to a GitHub release with generated release notes, and deposit a copy with a DOI. Archiving a tagged release on Zenodo, which issues a citable DOI per version, is established practice in research software, and the FastaGuard project does exactly this with its v1.1.0 release [8].

Declare Container Versions as Dependencies, Not Comments

A version record only protects you if the workflow engine respects it when reusing results. Workflow engines cache aggressively, and caching is safe only when every declared dependency covers everything that can change a result. A replay of 38,083 commits from 714 public Data Version Control pipelines measured how often that condition fails in practice. Reusing stage results according to declared dependencies avoided 83.4 percent of stage executions, which is the reward. The risk appeared in 3.3 percent of commits, where a stage was reused even though imported code or referenced configuration had changed, affecting 31 percent of pipelines, and a manual audit confirmed a genuinely undeclared dependency in 36 of 40 sampled events. Evaluation stages, whose outputs become the evidence, were the most exposed [9].

Translate that into three habits for container tracking:

1. Write the container reference inside each process definition, so it travels with the code and appears in every diff.
2. When a digest changes, treat every downstream result as stale. Rerun with a fresh work directory or clear the cache, because an output produced by the old image must not survive into the new release.
3. Write the container version into the outputs themselves, for example into a summary file the pipeline emits, so each result file carries its own provenance.

The same study tested release gates that decide whether reused evidence describes the candidate being released. A reference gate that checked declared dependencies and verified Ed25519 signatures blocked eight of nine manipulation scenarios, a gate that only verified signatures blocked three, and a gate that trusted the planner blocked one. A dependency the policy never declared passed every gate [9]. The lesson scales down to a two-person lab: an undeclared dependency is invisible to every check you did not think to write.

Containers Do Not Pin Your Reference Databases

A digest pins the software. It says nothing about the reference genome, the annotation release, the k-mer database, or the versioned classifier your pipeline reads at runtime, and those inputs change results as surely as tool versions do. Record them in the same repository with the same discipline: the download source, the version or build date, and a checksum of every file the pipeline reads. Put the checksum in a file the workflow treats as an input, so a changed database invalidates the affected steps instead of passing through silently.

Version control is extending in exactly this direction. Database version control systems now track schema and data together through promotion workflows that keep development, testing, and production consistent [10], and data-oriented systems bring git-like operations (clone, tag, diff, merge, revert) to terabyte-scale tables so teams can branch, review, and test data changes the way they review code. You do not need terabyte-scale machinery to start. A databases.lock.yml with three checksummed entries, committed next to containers.lock.yml, captures most of the value.

Running Pinned Images on an HPC Cluster

Most shared clusters do not run the Docker daemon, and users do not have root. Apptainer (formerly Singularity) is the standard answer, and nf-core pipelines support it natively alongside Docker [3]. Apptainer pulls Docker images directly and converts them into a single .sif file:

apptainer pull samtools_1.17.sif docker://quay.io/biocontainers/samtools:1.17--h0041a0a_1
apptainer exec samtools_1.17.sif samtools --version

samtools 1.17

Pull by digest so the file name you record and the bytes inside it stay locked together:

apptainer pull samtools.sif docker://quay.io/biocontainers/samtools@sha256:3a7c9e1b5d2f8a4c6e0b9d3f1a5c7e9b2d4f6a8c0e1b3d5f7a9c1e3b5d7f9a2c

Two things to watch. Architecture first: pulling an image built for x86 onto an ARM laptop produces a warning like WARNING: The requested image's platform (linux/amd64) does not match the detected host platform, and production runs should use a matching architecture rather than emulation. Credentials second: a pull that fails with a registry error naming authentication usually means the cluster cannot read a private image, so pull on a machine that can, copy the .sif file over, and record its checksum in the lockfile.

GitHub for Research Teams: Review Every Container Bump

GitHub for researchers works best when container changes get the same review as code changes. Three settings do most of the work:

1. Branch protection on main. Require a pull request for every change to containers.lock.yml, so a container bump cannot land without a second reviewer and a passing test run.
2. Conventional commit messages. Write Upgrade samtools 1.17 to 1.19.2 (digest in lockfile) rather than update stuff. Commits, release notes, README files, and merged pull requests are structured artifacts that can be searched and even summarized automatically, which makes disciplined messages a research asset [11].
3. A release ritual. Tag the release, attach the archived image tarball, and let GitHub generate release notes from the commits since the last tag.

The Git ecosystem scales from two-person labs to distributed teams, and even to dedicated self-hosted platforms that teach bioinformatics software development with repository management and access control built in [12]. The nf-core community shows the mature end of the same path, with peer-reviewed pipelines, automated tools for testing, deployment, and synchronization, and coordinated releases across many pipelines [3]. If hosted GitHub does not fit because data cannot leave your institution, the same workflow runs on a self-hosted Git server [12].

Next Steps

Adopt one habit at a time. Convert your current pipeline to digest-pinned references and commit the first lockfile. Add a test dataset with stored expected outputs so upgrades become measurable. Turn on branch protection for the lockfile. Deposit your first public release on Zenodo. Once the loop from pull to pin to test to tag feels routine, extend the same treatment to reference databases and then to the data files themselves.

Common Mistakes and Limitations

1. Running images by mutable tag. latest is a moving target, and even version tags get rebuilt. Pin by digest for anything you will report.
2. Recording digests you never use. A lockfile entry does nothing if CI and the cluster still pull by tag. Make the digest the reference your pipeline actually resolves.
3. Unpinned base images. FROM python:3.11 rebuilds differently every week. Pin the base image by digest the same way you pin your tools.
4. Splitting container changes from code changes. One commit should carry both, because the pairing is what makes a release tag meaningful.
5. Stale caches after a bump. Cached task outputs from the old image must be invalidated, or a new release quietly reports old numbers.
6. Ignoring reference databases and random seeds. The digest pins the software layer, not your data, references, or seed. Record all of them in the repository.
7. No archive. Registry deletion and pull rate limits are routine events, not edge cases. Keep tarballs with checksums and a private registry copy.
8. Treating tracking as validation. Identical versions make reruns comparable. They do not tell you the pipeline answers the biological question. Validation and benchmark design for NGS pipelines remain their own discipline, with dedicated resources and open challenges [13].
9. Trusting printed version strings. Check the version inside the container with samtools, version rather than assuming the tag describes the binary.

Digest pinning records the computational method. It does not certify that the method is correct, that the reference data fit the question, or that the biological interpretation holds, and those judgments need domain expertise on each individual analysis. A version bump can also change conclusions, which is precisely why upgrades get tested against a reference dataset before they become a release.

Frequently Asked Questions

Should I pin container versions by tag or by digest?

Pin by digest whenever a result matters, because a tag can be moved or deleted while a digest is a permanent fingerprint of the image bytes. Record both in a lockfile so the digest guarantees the bytes and the tag keeps the entry readable for humans.

How do I find the digest of an image I am already using?

Run docker images, digests to list digests for local images, or docker inspect, format '{{index .RepoDigests 0}}' IMAGE:TAG for one image. The registry's web page for the image also shows the digest after a pull.

Do I still need to track container versions if I use nf-core?

Yes. nf-core modules ship with pinned container references, but you should record the pipeline release, plugin versions, and any custom containers in your own repository so the exact combination that produced your results is captured.

How often should I update container versions in a pipeline?

On a schedule you choose, not automatically. Update when a release fixes a bug you care about or adds a feature you need, rerun your test dataset, and commit the lockfile change together with any code change.

What is the difference between a tag and a digest?

A tag is a human-readable label such as 1.17 that a maintainer can point at a different image at any time. A digest is a content hash of the image layers, so it changes only when the image bytes change.

How do I reproduce an analysis I ran two years ago?

Check out the git tag from that period, read the digest from the lockfile committed at that tag, and pull the image by digest. If the registry no longer serves it, restore the image from your archived tarball or private registry copy.

Can I use Docker images on an HPC cluster without root access?

Yes. Clusters run Apptainer (formerly Singularity), which pulls Docker images directly with apptainer pull docker://IMAGE and runs them without a daemon or root privileges.

Does a workflow manager make git unnecessary?

No. The workflow manager executes the pipeline and reports what ran, while git stores the pipeline code, the container lockfile, and the history of every change, which is what you need to reconstruct any past run.

Sources

1. A comparison of bioinformatics pipelines for compositional analysis of the human gut microbiome
2. The Impact of Bioinformatics Pipelines on Microbiota Studies: Does the Analytical “Microscope” Affect the Biological Interpretation?
3. The nf-core framework for community-curated bioinformatics pipelines
4. Reproducible, scalable, and shareable analysis pipelines with bioinformatics workflow managers
5. Comparative Evaluation of Open-Source Bioinformatics Pipelines for Full-Length Viral Genome Assembly
6. Practical bioinformatics pipelines for single-cell RNA-seq data analysis
7. [The challenges of designing a benchmark strategy for bioinformatics pipelines in the identification of antimicrobial resistance determinants using next generation sequencing technologies [version 2; peer review: 2 approved]](https://www.semanticscholar.org/paper/b5d203c1add8acd03b300a6ab716cd491c674d33)
8. FastaGuard: FASTA preflight QC for modern bioinformatics pipelines
9. Reusing Assurance Evidence in AI Pipelines: Validity Conditions, a Fail-Closed Release Gate, and a Replay of 714 Public Data Version Control Pipelines
10. A Unified Distributed Version Control System for SQL and Graph Databases
11. Automated Feature Extraction from Version Control Artifacts in Github Repositories
12. Implementing a Version Control Platform as an Interactive Educational Framework for Bioinformatics and Artificial Intelligence in Medicine
13. The challenges of designing a benchmark strategy for bioinformatics pipelines in the identification of antimicrobial resistance determinants using next generation sequencing technologies