# Automated Deployment of Bioinformatics Workflows By the end of this guide you will be able to build a Docker image for a bioinformatics tool, wire that image into a Nextflow or Snakemake workflow, run the workflow end to end on a laptop or a high performance computing (HPC) cluster, parameterize it with a single configuration file, and verify that the outputs are reproducible. You will also know how to install and run published, containerized workflows such as PyMethylProcess and the COMBAT-TB Workbench without building anything from scratch. To follow the walkthrough you need a computer with Docker installed, at least 8 GB of RAM, roughly 20 GB of free disk space, Java 17 or newer if you want to use Nextflow, and Python 3.9 or newer if you want to use Snakemake. All commands below were written for a Linux or macOS shell. On Windows, use WSL2 so that Docker and the workflow engines behave the same way they do on Linux. ## Why Deploy Bioinformatics Workflows with Docker A typical genomics pipeline chains together many independent programs, each with its own library versions, reference data, and parameter surface. IsoProt, a proteomics workflow for isobarically labeled mass spectrometry data, makes the scale of the problem explicit: its authors note that a bioinformatics workflow can consist of multiple tools requiring hundreds of parameters, so a methods section alone is often insufficient to reproduce an analysis [1]. Containerization attacks that problem at its root. A Docker image packages a program together with its operating system libraries and pinned versions, so the same image yields the same result on any machine that runs it. Docker uses a client-server model. The Docker daemon builds and runs images. An image is a read-only template with layered filesystem snapshots. A container is a running instance of an image. When you deploy a bioinformatics workflow with Docker, each processing step executes inside a container built from a pinned image, and the workflow engine coordinates inputs, outputs, parallelism, and retries between those steps. The published literature shows this pattern across very different domains: - The COMBAT-TB Workbench bundles the IRIDA [data management platform](/blog/guides/data-management-platform-what-it-is-and-how-to-choose-one) and the Galaxy workflow engine into a single, easy-to-install Docker application for Mycobacterium tuberculosis whole-genome sequencing analysis in public health laboratories [2]. - Kuura, an automated workflow for whole-exome and whole-genome sequencing analysis, is written in the Nextflow pipeline language and uses Docker to manage and deploy the workflow, with four stages from quality control through variant annotation [3]. - PyMethylProcess, a parallelized [preprocessing](/blog/data-analysis/what-is-data-preprocessing) pipeline for [DNA methylation](/blog/guides/dna-methylation) array data, is set up and deployed through Docker and PIP, with the package distributed on PyPI and its image on Docker Hub [4]. - The BioDepot-workflow-builder (Bwb) lets users assemble workflows from graphical widgets, where each widget represents a Docker container executing one modular task, and share the result in a native format or export it as a shell script [5]. Two properties recur across these systems. First, the deployment barrier drops: a laboratory that could never compile a variant caller from source can run `docker pull` and a launch command. Second, reproducibility becomes a property of the artifact you ship, not of the machine you ship it to. ## Choosing a Bioinformatics Workflow Manager: Nextflow vs Snakemake A workflow manager (also called a workflow engine) reads a declarative description of your analysis, resolves dependencies between steps, executes the steps in the right order with the right parallelism, and tracks what has already completed. Docker handles software packaging. The manager handles orchestration. You need both for automated deployment. Nextflow and Snakemake are the two dominant open-source engines in bioinformatics. The honest comparison: | Capability | Nextflow | Snakemake | |--|--|--| | Language | Groovy-based DSL | Python-based | | Container directive | `container 'repo:tag'` per process | `container: "docker://repo:tag"` per rule | | Runtime flag | `docker.enabled = true` in nextflow.config | `-use-singularity` or `-sdm apptainer` | | Parameter file | YAML passed with `-params-file` | YAML loaded with `configfile:` | | Resume after failure | `-resume` reuses cached outputs | automatic mtime-based reruns, `-rerun-incomplete` | | Dry run | `nextflow run main.nf -preview` | `snakemake -n` | | Typical strength | Large distributed and cloud deployments, nf-core ecosystem | Python integration, rule-based prototyping, local clusters | Neither engine is universally better. Nextflow has the larger catalog of ready-made community pipelines (the nf-core collection) and first-class cloud execution. Snakemake integrates naturally with Python code, which matters when your pipeline includes custom statistics or plotting. Both pin Docker images per step, both cache completed work, and both run unchanged on a laptop and an HPC scheduler. For a first deployment, pick the engine whose existing pipelines are closest to your analysis, because reading and adapting a mature pipeline teaches the format faster than writing one from zero. ## Worked Example: Deploy a Bioinformatics Workflow with Docker and Nextflow ![Seven steps: create project skeleton, write Dockerfile, build and test image, write Nextflow workflow, run it, add params file, resume after failure.](/article-images/illustrations/deploy-bioinformatics-workflow-docker-deploy-steps.webp) This is a complete deploy bioinformatics workflow docker example that you can run in under an hour. The workflow runs FastQC on a pair of FASTQ files. The same pattern scales to real pipelines: every additional tool is one more pinned image and one more process block. ### Step 1. Create the project skeleton ```bash mkdir wf-demo && cd wf-demo mkdir -p data results cp /path/to/sampleA_1.fastq.gz /path/to/sampleA_2.fastq.gz data/ ``` If you have no sequencing data on hand, download two small test FASTQ files from a public repository of your choice and rename them to match the pattern above. ### Step 2. Write the Dockerfile template Create a file named `Dockerfile` in the project root. This Dockerfile template installs two Bioconda tools into a minimal micromamba base image: ```dockerfile FROM mambaorg/micromamba:1.5.10 RUN micromamba install -y -c conda-forge -c bioconda \ fastqc=0.12.1 seqkit=2.7.0 \ && micromamba clean --all --yes ``` Every element here matters for reproducibility: - `FROM` pins the base image to an exact version tag. Never use `latest` in a workflow image, because the same tag resolves to different content over time. - `-c bioconda -c conda-forge` pulls from the Bioconda channel, which distributes most bioinformatics software as prebuilt conda packages, and conda-forge, which supplies the shared dependencies. - Explicit version pins (`fastqc=0.12.1`) freeze the tool versions that your results depend on. - `micromamba clean` removes package caches so the final image stays small. ### Step 3. Build and test the image ```bash docker build -t yourdockerhubid/wf-demo:0.1.0 . ``` Expected output, abbreviated: ``` [+] Building 42.1s (8/8) FINISHED => [2/2] RUN micromamba install -y -c conda-forge -c bioconda ... => naming to docker.io/yourdockerhubid/wf-demo:0.1.0 ``` Test that the tool actually runs inside the container: ```bash docker run --rm yourdockerhubid/wf-demo:0.1.0 fastqc --version ``` Expected output: ``` FastQC v0.12.1 ``` If you will run the workflow on other machines or in the cloud, push the image to a registry so any host can pull it by name: ```bash docker login docker push yourdockerhubid/wf-demo:0.1.0 ``` ### Step 4. Write the Nextflow workflow Create `main.nf`: ```groovy nextflow.enable.dsl = 2 workflow { reads_ch = channel.fromFilePairs('data/*_{1,2}.fastq.gz', checkIfExists: true) FASTQC(reads_ch) } process FASTQC { tag "${sampleId}" container 'yourdockerhubid/wf-demo:0.1.0' input: tuple val(sampleId), path(reads) output: path "${sampleId}_fastqc" script: """ mkdir ${sampleId}_fastqc fastqc -o ${sampleId}_fastqc -t 2 ${reads} """ } ``` The `container` directive is the line that connects the process to your Docker image. When the process runs, Nextflow starts a container from that image, mounts the work directory and inputs into it, executes the script block inside the container, and copies declared outputs back out. Create `nextflow.config` in the same directory: ```groovy docker.enabled = true process.cpus = 2 process.memory = '4.GB' ``` ### Step 5. Run the workflow ```bash ./nextflow run main.nf ``` Expected output, abbreviated: ``` N E X T F L O W ~ version 24.04.4 Launching `main.nf` [sharp_lovelace] DSL2 - revision: (none) executor > local (2) [8a/1b2c3d] process > FASTQC (sampleA) [100%] 2 of 2, succeeded: 2 ✔ ``` Check the result: ```bash ls results find work -name '*_fastqc.html' ``` You should find one HTML report per input file inside the Nextflow `work` directory, plus a symlinked copy of the process output directory in your project root. Open the HTML report in a browser and confirm per-base sequence quality looks sane for your data type. ### Step 6. Parameterize with a template file Hard-coding paths makes a workflow personal to one directory. A parameter file makes it portable. Create `params.yaml`: ```yaml reads: "data/*_{1,2}.fastq.gz" outdir: "results" ``` Reference it in `main.nf` with `params.outdir` wherever you want the value, then run: ```bash nextflow run main.nf -params-file params.yaml -resume ``` This single-parameters-file approach is exactly the design Kuura adopted for its whole-exome and whole-genome workflow, where a single parameters file drives the four-stage analysis and makes the pipeline easy to extend by amending the parameter list rather than editing code [3]. ### Step 7. Resume after a failure Interrupt a run with Ctrl-C partway through, fix the problem, then rerun: ```bash nextflow run main.nf -params-file params.yaml -resume ``` Nextflow hashes each task (its script, inputs, container, and parameters). Unchanged tasks are skipped and print `Cached` in the log, and only the failed or changed tasks rerun. This is the single most valuable feature for interactive pipeline development, and it works identically on a laptop and a cluster. ## The Same Deployment in Snakemake If you prefer Python, the identical analysis in Snakemake demonstrates the `docker://` format, which Snakemake accepts in its `container` directive and resolves through Apptainer or Singularity at runtime. Create `Snakefile`: ```python configfile: "config.yaml" rule all: input: expand("results/{sample}_1_fastqc.html", sample=config["samples"]) rule fastqc: input: r1="data/{sample}_1.fastq.gz" output: html="results/{sample}_1_fastqc.html" container: "docker://yourdockerhubid/wf-demo:0.1.0" shell: "fastqc -o results {input.r1}" ``` Create `config.yaml`: ```yaml samples: - sampleA ``` Run it: ```bash snakemake --cores 2 --use-singularity ``` Expected output, abbreviated: ``` Building DAG of jobs... Using shell: /bin/bash Job stats: 2 jobs [1/2] fastqc sampleA [2/2] all Downloading and converting Docker image... Finished job 0. 2 of 2 steps (100%) done ``` The `-use-singularity` flag makes Snakemake pull the image referenced in `docker://yourdockerhubid/wf-demo:0.1.0` format and run the rule inside it. The first run is slow because of the conversion to the Singularity image format, and later runs reuse the cached converted image. This is the standard path for running Docker-defined workflows on HPC systems where users cannot run the Docker daemon directly. ## Installing and Running Published Containerized Workflows Building your own pipeline is often unnecessary. Four deployed, published workflows show the range of what you can simply install. ### PyMethylProcess: pip plus Docker PyMethylProcess preprocesses DNA methylation array data in parallel and was designed to be set up and deployed through Docker and PIP [4]. Because the image is on Docker Hub and the package is on PyPI, deployment is two commands: ```bash docker pull joshualevy44/pymethylprocess pip install pymethylprocess ``` Pull the image, check the project home page for the current entrypoint commands, and run the preprocessing steps against your own IDAT files. This pattern, image on Docker Hub plus a thin command-line layer, is the lowest-friction deployment format for a published pipeline. ### Bwb: drag-and-drop containers The BioDepot-workflow-builder represents each Docker container as a widget in a graphical editor [5]. Each widget carries a form for parameter entry and a console for intermediate results, and you connect widgets to build the workflow. Saved workflows can be shared in Bwb's native format or exported as shell scripts, which makes it a practical on-ramp for collaborators who do not work in a terminal. If your deployment audience is a wet lab without command-line experience, deploying through a graphical builder is often the difference between a pipeline that gets used and one that does not. ### COMBAT-TB Workbench: a full platform in one container For clinical and public health settings, the COMBAT-TB Workbench combines the IRIDA platform for web-based [data management](/blog/guides/data-management-basics-principles-processes-and-best-practices) with the Galaxy workflow engine, packaged as a single Docker application [2]. Its M. tuberculosis sample analysis workflow reports drug resistance variants and handles metadata updates from analysis results. Deploying a platform like this is a server operation rather than a laptop operation: you provision a host, install Docker, follow the project's installation instructions to start the bundled stack, and users then interact with it through a browser. ### Kuura: a Nextflow template for germline sequencing Kuura runs quality control, mapping with quality score recalibration, variant calling, and variant annotation as four stages, and generates a consensus variant list from multiple callers including GATK HaplotypeCaller, DeepVariant, VarScan2, FreeBayes, and Strelka2 [3]. As a deploy bioinformatics workflow docker template, a pipeline of this shape is more useful than a toy example, because you can fork it, swap the variant callers or parameters through its parameters file, and adapt it to your data type without redesigning the orchestration layer. ### IsoProt: reproducible proteomics with a browser interface IsoProt deploys a complete isobaric labeling (iTRAQ/TMT) proteomics workflow in a portable container with a browser-based interface for configuring its many parameters [1]. After execution it produces an HTML document that includes the R code used for [statistical analysis](/blog/guides/statistical-analysis), giving reviewers a complete record of the analysis. The lesson for your own deployments is to emit a machine-readable run report capturing tool versions, parameters, and software versions alongside your results. ## Deploying on an HPC Scheduler or in the Cloud Both engines switch execution targets through configuration, not code changes. For Nextflow on a Slurm cluster, add to `nextflow.config`: ```groovy process.executor = 'slurm' singularity.enabled = true singularity.autoMounts = true ``` Then run `nextflow run main.nf -params-file params.yaml`. Nextflow submits each task as a Slurm job, and with `singularity.enabled` it runs each containerized task through Apptainer, which is the standard container runtime on shared clusters because it does not require root privileges. For cloud deployment, Nextflow supports grid, Kubernetes, and AWS Batch executors through the same configuration mechanism, so the workflow file you tested locally is the file you deploy in production. For Snakemake on a cluster, run with a profile: ```bash snakemake --cores 100 --use-singularity --profile slurm ``` A profile is a directory of YAML defaults. Community-maintained profiles for Slurm, LSF, and cloud Kubernetes clusters remove most cluster-specific boilerplate. ## Validating a Deployment An automated deployment is not finished when the pipeline exits cleanly. Check four things: 1. **Outputs exist and are internally consistent.** Count files against expected samples, and compare checksums across a rerun: `md5sum results/*.html > run1.md5` then rerun with `-resume` and recompute. A fully cached rerun should produce identical checksums. 2. **The container versions are what you think they are.** Record the image digest, not just the tag: `docker inspect, format='{{index .RepoDigests 0}}' yourdockerhubid/wf-demo:0.1.0` prints a `sha256:` digest that uniquely identifies the image bytes. 3. **Provenance is captured.** Nextflow writes a full trace with `nextflow run main.nf -with-trace trace.tsv`, including each task's container image, exit status, and runtime. Snakemake writes provenance with `-wms-metadata` options or, more simply, `snakemake, report report.html` produces a self-contained report of the DAG, code, and runtime. 4. **Biology checks out.** FastQC reports should show acceptable per-base quality for your platform, and a variant-calling workflow should reproduce known calls on a reference control sample if one exists for your organism. ## Common Errors and Exact Fixes ![Decision path from the error message: exit 127, exit 137, pull rate limit, missing input, or Apptainer conversion failure, each with its fix.](/article-images/illustrations/deploy-bioinformatics-workflow-docker-error-triage.webp) These are the failures you will actually hit when you deploy a bioinformatics workflow with Docker for the first time. **`docker: permission denied while trying to connect to the Docker daemon socket`** Your user is not in the `docker` group. Fix: ```bash sudo usermod -aG docker $USER newgrp docker docker run --rm hello-world ``` **`Cannot connect to the Docker daemon at unix:///var/run/docker.sock`** The daemon is not running. Start Docker Desktop on macOS or Windows, or `sudo systemctl start docker` on Linux. **Nextflow: `Process FASTQC terminated with an error exit status (127)`** Exit 127 means the command was not found inside the container. Either the tool is not in the image or the executable name differs from the one in your script block. Debug interactively: ```bash docker run --rm -it yourdockerhubid/wf-demo:0.1.0 bash which fastqc ``` Then inspect the task's captured logs with `nextflow log -f exit_status 127` and read the `.command.err` and `.command.sh` files in the failing work directory. **Docker Hub: `toomanyrequests: You have reached your pull rate limit`** Anonymous pulls are rate limited per IP. Authenticate with `docker login`, or mirror the images you need into your own registry if many jobs pull concurrently on a cluster. **Snakemake: `MissingInputException` for a rule input** The pattern in the rule does not match any file on disk. Run `snakemake -n` (the dry run) and compare the resolved input paths against `ls data/`. A trailing underscore or a missing `_1`/`_2` suffix is the usual culprit. **Apptainer conversion failure on a cluster** Some Docker images use features Apptainer cannot convert, typically images that require a writable root filesystem. Rebuild the image with tools installed into a standard prefix, or pull a biocontainers variant of the same tool built for Singularity compatibility. | Symptom | Likely cause | Fix | |--|--|--| | Exit status 127 | Tool missing in image | Add package to Dockerfile, rebuild, bump tag | | Exit status 137 | Task killed, usually out of memory | Raise `process.memory` or scheduler memory request | | Empty outputs, clean exit | Wrong output paths declared | Check `.command.out` and `ls work//` | | Works locally, fails on cluster | Runtime switched from Docker to Apptainer | Test the workflow with `singularity.enabled = true` locally first | | Results differ between machines | Unpinned base image or floating tool versions | Pin every tag, record digests, rebuild from Dockerfile | ## Common Mistakes and Limitations - **Deploying mutable tags.** An image tagged `:latest` changes under you. Pin exact versions in the Dockerfile and in every `container` directive, and reference digests for anything you consider frozen. - **Assuming the container holds your data.** Containers are ephemeral. Data must be mounted in (Nextflow and Snakemake handle this for you) and outputs copied to declared paths. Anything written elsewhere inside a container is lost when it exits. - **Ignoring resource requests.** A process that requests 1 GB will be OOM-killed on a 30 GB job, usually with exit status 137 and no clear error text. Size `cpus` and `memory` directives per process, not globally, and verify peak usage with the trace file before scaling up. - **Treating a clean exit as correctness.** Pipelines can finish successfully with garbage results when a reference genome is mismatched, a sample sheet is wrong, or a parameter is silently defaulted. Spot-check real outputs and run a control sample through any clinical or publication-facing workflow. Individual dataset decisions, particularly around variant interpretation and clinical reporting, need review by someone with domain expertise in the relevant analysis. - **Conflating deployment with reproducibility.** Docker removes software drift, but your workflow code, parameters, reference data versions, and seed values must also be version controlled. Push the repository and the images to a registry together, and record the workflow revision (`nextflow log` prints it) alongside results. - **Over-automating before validating.** Automated deployment multiplies whatever the pipeline does, including its bugs. Run the full workflow manually on one sample before you point it at a hundred. ## Frequently Asked Questions ### Do I need to write my own Dockerfile to deploy a workflow? No. Most published pipelines ship their images on Docker Hub or BioContainers, so you reference them by name in the workflow file and pull them at runtime. You only write Dockerfiles when you combine custom scripts or a tool version that no existing image provides. ### What is the difference between a Docker image and a container? An image is a read-only, versioned package of software and libraries. A container is a running instance of that image. You build and push images once, and the workflow engine creates and destroys containers for every task. ### Nextflow vs Snakemake, which should I choose? Choose Nextflow if your target is large distributed runs or you want to reuse the large catalog of community-maintained genomics pipelines. Choose Snakemake if your analysis mixes in substantial Python code or you want fine-grained rule-based control on a local or cluster system. Both deploy Docker containers per step and both support resumable runs. ### Can I deploy a workflow on an HPC cluster without root access? Yes. Use Apptainer (Singularity) as the container runtime instead of Docker. Nextflow handles this with `singularity.enabled = true` in its configuration, and Snakemake with the `-use-singularity` flag, and neither requires administrator privileges. ### How do I update a deployed workflow without breaking reproducibility? Bump the image tag and the workflow version together, rerun with the resume flag so only changed tasks re-execute, and archive the previous image digest with your old results. Never overwrite a tag you have already used to produce published results. ### How much computing power do I need? A two-sample QC workflow like the example in this article runs on a laptop with 8 GB of RAM. Genome-scale variant calling, such as the Kuura workflow, is designed for high-throughput sequencing data and generally needs a server or cluster, and resource needs scale with the number of samples rather than the pipeline design. ### Can I share my deployed workflow with collaborators? Yes, and this is a main reason to deploy with Docker. Share the repository, the Docker image by its digest, and the parameter file. Tools like the BioDepot-workflow-builder add a graphical sharing path, exporting workflows in a native widget format or as plain shell scripts [5]. ### How do I verify the deployment produced the right results? Run a control sample with known expected outputs, confirm output checksums are identical across a rerun, check that the recorded image digests and tool versions match your documentation, and review biological quality metrics such as per-base quality scores before trusting downstream analysis. ## Sources 1. [IsoProt: A Complete and Reproducible Workflow To Analyze iTRAQ/TMT Experiments.](https://pubmed.ncbi.nlm.nih.gov/30855969/) 2. [The COMBAT-TB Workbench: Making Powerful Mycobacterium tuberculosis Bioinformatics Accessible.](https://pubmed.ncbi.nlm.nih.gov/35138128/) 3. [Kuura-An automated workflow for analyzing WES and WGS data.](https://pubmed.ncbi.nlm.nih.gov/38236904/) 4. [PyMethylProcess-convenient high-throughput preprocessing workflow for DNA methylation data.](https://pubmed.ncbi.nlm.nih.gov/31368477/) 5. [Building Containerized Workflows Using the BioDepot-Workflow-Builder.](https://pubmed.ncbi.nlm.nih.gov/31521606/)