Scaling Variant Calling Pipelines in the Cloud

By Dr. Zubair Khalid, DVM, MS, PhD · 2026-11-11

After reading this guide you will be able to plan a cohort variant calling campaign on cloud infrastructure, choose between scatter-gather, serverless, Spark, and MPI architectures, launch a reference pipeline on AWS with Nextflow, estimate cost from vCPU-hours before you commit, and validate the resulting callset. You will need an AWS account with Batch permissions, Java 17 or later, Nextflow, Docker or Apptainer, sequencing data as FASTQ, BAM, or CRAM, and an S3 bucket you can write to.

Variant calling is the step where aligned sequencing reads become a list of genetic differences per sample. At one genome it is a laptop job. At two thousand genomes it is an infrastructure problem, and the infrastructure, not the caller, is usually what fails. This article covers how to scale variant calling pipelines in the cloud using patterns that have been demonstrated at population scale, with real commands and honest cost math.

Why Variant Calling Outgrows a Single Machine

Three kinds of growth force teams into the cloud. First, sample count grows. National biobank projects such as UK Biobank, the All of Us Research Program, Singapore's PRECISE, Biobank Japan, and Korea's national bio-big-data project now generate whole-genome sequencing data at a scale that demands cloud-based computing solutions alongside improved callers and optimized storage formats [1]. Second, per-sample compute grows. Deep tumor sequencing for somatic variant detection produces volumes of raw data that take days of compute for a single sample under standard tool chains [2]. Third, joint analysis grows. Merging thousands of per-sample calls into one cohort callset is a step that naive approaches handle badly.

The published record shows the cloud handles this if the pipeline is built for it. A 2015 study processed 2,535 genomes from 1000 Genomes Phase 3 in five days on Amazon Web Services through the DNAnexus platform, identifying 68.3 million variants at a compute cost of $7.33 per sample [3]. A decade later, the All of Us program's Celeste infrastructure processes fluctuating workloads of up to 9,000 whole-genome samples per month inside a single AWS Region, coordinating roughly 1.8 million jobs over twelve months of production [4]. The patterns in between those two endpoints are what this article teaches.

Architectural Patterns That Scale Variant Calling in the Cloud

There is no single right architecture. There are four proven patterns, and the choice depends on sample count, latency needs, and who operates the system.

Pattern 1, per-sample scatter-gather

Most of a germline pipeline embarrassingly parallel. Alignment, sorting, duplicate marking, and per-sample calling are independent per sample. You fan out one job per sample across many virtual machines, then gather results for the joint step. This is the model behind the 2,535-genome run, which achieved near-linear scaling by executing the population caller in parallel batches [3]. It is also the default behavior of most workflow managers on cloud executors.

The gather phase is where scaling gets hard, and a dedicated section below covers it.

Pattern 2, serverless orchestration on AWS

Celeste, the All of Us genomics infrastructure, is built on serverless technology and container orchestration. In twelve months of production it executed around 200 million serverless functions and over 20 million messages to coordinate 1.8 million bioinformatics, quality control, and clinical reporting jobs [4]. Serverless means each pipeline step is a container invoked by a managed function service, with a message queue tracking state between steps. The advantage is resilience under fluctuating load, which matters when sequencing instruments deliver batches of uneven size. The cost is engineering overhead. You would not build Celeste for a 50-genome project, but if you are planning recurring batches in the thousands of samples, its design is the closest open reference point for large-scale AWS genomics operations [4].

Pattern 3, in-memory cluster engines, Spark and MPI

Some pipelines instead make the compute itself cluster-native. Halvade somatic runs the GATK somatic best-practice steps, alignment with BWA, sorting, duplicate marking, base quality score recalibration, and Mutect2 calling, on Apache Spark. On a single 36-core node it reduces runtime from 84.5 hours to 19.5 hours, a 4.3-fold speedup, and scales further across nodes [2]. GenMPI takes a different route, implementing the workflow with the Message Passing Interface so that compressed reads stream in parallel to alignment processes and alignment output feeds a distributed queue for sorting and calling. At 300x coverage, its alignment stage scales almost linearly up to 64 nodes and 8,192 CPU cores, and it runs on any private or public cluster or cloud infrastructure [5].

These engines trade portability work for throughput. They suit tumor-normal somatic work and very high depth data, where a single sample can saturate a node.

Pattern 4, hybrid cloud plus on-premise HPC

Large institutions sometimes split work. A hybrid framework used cloud AWS resources, supercomputers, and local high-performance computing together to jointly call, phase, and impute variants for over 5,300 whole-genome samples from the CHARGE consortium in under six weeks, using four independent callers and a binning approach designed to reach 10,000 samples [6]. Hybrid designs are worth considering when you already own idle HPC allocation, or when data governance requires that some stages stay on premises.

Older platforms took a fourth path, running an OpenStack private cloud with the GlusterFS clustered file system, as Bionimbus did for projects including an acute myeloid leukemia resequencing effort, where each sample's alignment needed eight CPUs for about twelve hours and produced 5 to 10 GB BAM files [7]. That generation of infrastructure has largely been replaced by managed services, but it established the cost accounting most teams still use.

Choosing a pattern

Reference Pipeline Templates You Can Reuse

Do not write a variant calling pipeline from scratch to run in the cloud. Four open templates cover the common cases, and each has published scaling behavior.

nf-core/sarek is a Nextflow pipeline for germline and somatic variant calling and annotation that works with any genome with a known reference. Version 3 was a full rewrite that cut storage requirements by writing CRAM instead of BAM and cut runtime by increasing intra-sample parallelization, together yielding a 70 percent cost reduction on commercial clouds [8]. It is the most practical starting template for most teams, because the nf-core test profiles and AWS integration are maintained for you.

Parliament2 is a consensus structural variant calling framework that bundles multiple best-in-class short-read SV callers, pre-installed and optimized to run in parallel inside one container. It processed all 2,691 1000 Genomes samples on GRCh38 in under a day and achieved the highest reported F1 score, 74.27 percent, against the Genome in a Bottle structural variant gold standard [9]. It is distributed as a Docker image from its GitHub repository, which makes it straightforward to drop into a scatter-gather design.

DeepVariant plus GLnexus is the reference pattern for cohort-level germline calling. DeepVariant produces highly accurate per-sample calls, and GLnexus merges them into a joint callset that scales across cohort sizes and sequencing depths, with published quality improvements over GATK best practices at reduced cost [10]. The authors released individual-level and cohort calls for the 1000 Genomes Project through a public cloud storage bucket, which doubles as a validation reference for your own pipeline.

Halvade somatic, described above, is the template for GATK-style somatic workflows that need to run faster than one node allows [2]. Swift/T represents an earlier generation of workflow engines designed for variant calling at production scale across many cluster schedulers, and it remains a useful case study in what a workflow language must provide for robust production pipelines, including transparent operation across PBS, SLURM, and cloud environments [11].

Worked Example, How to Scale Variant Calling in the Cloud on AWS

This example runs 100 germline whole-genome samples through nf-core/sarek on AWS Batch, then sketches the cohort merge. The same skeleton works for 2,000 samples, which is the point of doing it this way.

Step 1, install the local tooling

You need Java 17 or later, Nextflow, and a container runtime. On a Mac or Linux workstation:

curl -sSL https://get.nextflow.io | bash
nextflow -v

Expected output looks like:

nextflow version 24.10.0.4934

You do not need heavy local compute. All real work runs remotely.

Step 2, prepare the sample sheet

The input format for sarek is a CSV. The essential columns are a sample identifier, the FASTQ pair, and the library layout:

sample,fastq_1,fastq_2,library
NA12878,s3://my-bucket/fastq/NA12878_R1.fastq.gz,s3://my-bucket/fastq/NA12878_R2.fastq.gz,1
NA12891,s3://my-bucket/fastq/NA12891_R1.fastq.gz,s3://my-bucket/fastq/NA12891_R2.fastq.gz,1

For samples already aligned, the format accepts BAM or CRAM paths in place of FASTQs, which saves you the alignment cost on reprocessing runs. Generate the sheet programmatically rather than by hand, because a typo in one row of 2,000 is expensive to find after launch.

Step 3, configure the AWS executor

Create a nextflow.config beside your sample sheet:

plugins {
  id 'nf-amazon'
}

workDir = 's3://my-bucket/work'

process {
  executor = 'awsbatch'
}

aws {
  region = 'us-east-1'
  batch {
    cliPath = '/usr/local/bin/aws'
  }
}

Before this works you need, once, in your AWS account, an S3 bucket, an ECR container repository or public images, a Batch compute environment built on Spot-capable EC2 instances, a job queue, and an IAM role that grants Batch jobs read on the FASTQ prefix and read-write on the work and results prefixes. AWS documents this setup under its genomics guidance, and the nf-core documentation carries a tested AWS Batch recipe. Spot capacity matters, because variant calling steps tolerate retries and Spot pricing runs at deep discounts to On-Demand rates.

Step 4, launch the run

nextflow run nf-core/sarek \
  -r 3.4.0 \
  -profile test \
  --input samplesheet.csv \
  --outdir s3://my-bucket/results \
  --genome GRCh38 \
  --tools deepvariant

Run -profile test first. It substitutes tiny synthetic data and validates your entire cloud configuration in minutes for pennies. When it succeeds, drop the test profile and run your real sheet. Expected console output during a live run looks like:

[-] process > NFCORE_SAREK:SAREK:INPUT_CHECK:SAMPLESHEET_CHECK (100 samples)
[-] process > NFCORE_SAREK:SAREK:...:BWA_MEM (NA12878)   12% | 12/100

Each line is one Batch job. At this scale the fan-out is automatic. One hundred alignment jobs start within minutes of each other, limited only by your compute environment's maximum vCPU setting.

Step 5, monitor and handle failure

Watch progress with:

nextflow log -f status FAILED

and resume with:

nextflow run nf-core/sarek -r 3.4.0 -resume

Three errors account for most first-run failures. If you see 403 Forbidden against an S3 path, the job role lacks permissions on that prefix, so fix the IAM policy rather than the code. If you see a Nextflow message of the form Process 'X' terminated for an unknown reason, that is usually a Spot instance reclaimed mid-task, and the fix is a retry block in the config with errorStrategy 'retry' and a maxRetries of 2 or 3, which Nextflow re-queues automatically. If jobs fail pulling the container with an image-not-found error from the container runtime, the ECR repository path in your config is wrong or the image is in a different Region.

Step 6, merge into a cohort callset

Per-sample calls in gVCF or BCF format feed the joint step. The DeepVariant plus GLnexus combination is the pattern with the strongest published cohort-quality evidence [10]. In outline, you export each per-sample callset to BCF, then run the GLnexus CLI over the batch:

# schematic, check the GLnexus repository README for current flags
glnexus_cli --config DeepVariantWGS sample_*.bcf > cohort.bcf

GLnexus is open source and hosted on GitHub [10]. For a simpler merge of already-normalized VCFs, bcftools merge works, but it does not address the joint genotyping statistics that cohort callers handle, and it becomes slow past a few hundred samples.

A different merge bottleneck is pileup generation, the tally of bases at each genomic position. A cloud-native distributed implementation of pileup using custom read partitioning and MD-tag processing ran up to 6.5 times faster with up to 2 times less memory than other distributed tools, and deploys on managed Kubernetes or Hadoop services in public clouds [12]. If pileup or joint genotyping becomes your wall-clock bottleneck at thousands of samples, that class of tool is the fix.

Step 7, validate before you scale

Run every new configuration against a reference sample with a truth set before trusting it. The Genome in a Bottle call sets are the standard benchmark, and Parliament2's published evaluation against the GIAB structural variant call set across all size classes shows exactly how to report the comparison, with F1 score broken out by variant type and size class [9]. Minimum checks on your output:

bcftools stats cohort.bcf > cohort.stats.txt

Expect a Ti/Tv ratio near 2.0 to 2.1 for whole-genome human calls, a het/hom ratio in a sane range for your cohort, and no chromosome with a variant count wildly out of line with the others. Then confirm a handful of known clinical variants by hand in a genome browser.

Workflow and File Formats for Cloud Bioinformatics

Format choices are scaling decisions, not style decisions.

Workflow languages. Nextflow, WDL, and CWL are the three formats you will encounter. Nextflow dominates the community pipeline space in cloud bioinformatics and integrates natively with AWS Batch and cloud object storage. WDL is the native format on platforms built around the Cromwell engine, and CWL is the most portable across academic schedulers. The same analysis workflow can be published in all three formats on a shared registry, which is exactly what recent cloud workflow releases have done, making the pipeline runnable on multiple cloud platforms from one source of truth [13]. Pick the format your team can hire for, wrap every tool in a container, and resist bespoke glue scripts, because they are the part that breaks at scale.

Read and call formats. CRAM versus BAM is the highest-leverage single change for cost. CRAM stores the same alignment information with reference-based compression, and its adoption was a major contributor to the 70 percent cost reduction reported for the rewritten sarek pipeline on commercial clouds [8]. National-scale projects likewise credit optimized data storage formats among the advances that make population-scale WGS processing tractable [1]. For calls, gVCF captures per-sample genotype evidence needed for joint calling, and BCF is the compressed binary counterpart of VCF used as interchange between cohort tools. When converting, keep the MD tag intact in CRAM, because distributed pileup implementations rely on it to reconstruct alignment events without decoding the full read [12].

Reference data. Host the reference genome, indices, and known-sites resources once in your own S3 bucket or a shared public one, version them, and never point a production run at a mutable path. Silent reference drift is a reproducibility failure that no amount of orchestration catches.

Cost Modeling, With Real Numbers

Compute cost scales close to linearly with sample count for per-sample work, which is why cloud economics work at all. The published anchor is the 1000 Genomes Phase 3 run, 2,535 samples, five days, $18,590 in completed-job compute costs, or $7.33 per sample [3]. A fast sanity model for your own campaign:

estimated compute cost
  = samples x (vCPUs per sample x hours per sample) x price per vCPU-hour

Worked calculation at the anchor point. If a sample needs 32 vCPUs for 12 hours of alignment and calling, that is 384 vCPU-hours per sample. For 100 samples, 38,400 vCPU-hours. Multiply by your negotiated or listed price per vCPU-hour, halve it if Spot capacity absorbs most of the workload, and add storage. Then verify the model empirically on a ten-sample pilot, because the real cost drivers are usually the steps people forget, reheadering, sorting shuffles, and the joint step.

Three levers reduce cost without changing results. First, use Spot or similar interruptible capacity with automatic retry, since variant calling steps are stateless and restartable. Second, adopt CRAM, which directly cut storage and cost in the sarek rewrite [8]. Third, increase intra-sample parallelization so each sample finishes faster on a bigger machine, paying for peak cores only for the hours they run [8]. Per-sample calling cost is not the whole bill, though. Joint calling and cohort merging scale super-linearly in naive implementations, which is why purpose-built cohort tools matter, and why the CHARGE hybrid framework needed a binning strategy to push joint calling past 5,000 samples at all [6].

Joint Calling and Cohort Merging at Population Scale

Two strategies coexist. Joint calling, where all samples are genotyped together, produces the most consistent cohort callsets and is the basis of the CHARGE binning approach that scaled joint calling, imputation, and phasing to over 5,300 whole-genome samples in under six weeks [6]. Merge-then-clean strategies, where per-sample calls are combined and refined, are what GLnexus optimizes, and the DeepVariant plus GLnexus combination delivered callsets with better recall and precision and better Mendelian consistency in trios than the existing best-practice pipeline across a range of cohort sizes and depths, at lower cost [10].

Long-read data changes the calculus again. The All of Us long-read effort processed over a thousand individuals at roughly 8x PacBio HiFi coverage through cloud-native pipelines, built cohort-level structural variant calling methods, and produced a scalable workflow to impute more than 750,000 structural variants into existing short-read datasets, which then supported association testing across hundreds of conditions [14]. If structural variation is your scientific target, plan for long-read aware cohort methods rather than forcing short-read callers past their resolution limits.

The general lesson from all of these projects is that the merge stage deserves its own engineering budget. Teams that budget only for alignment and calling hit a wall at gather time.

Security, Privacy, and Somatic Calling Pitfalls

Cloud genomics carries obligations beyond compute. For somatic cancer pipelines, an underappreciated risk is germline leakage, where a somatic pipeline mistakenly reports germline variants as somatic. Across 259 whole-genome somatic SNV prediction sets produced by 21 teams in the ICGC-TCGA DREAM Somatic Mutation Calling Challenge, the median prediction set contained 4,325 somatic SNVs and leaked one germline polymorphism, with leakage inversely correlated with somatic prediction accuracy and higher in tumors with more normal-cell contamination [15]. Somatic variant data may therefore be more re-identifiable than commonly assumed. If you release somatic callsets from a cloud pipeline, run a germline filtering step against a population reference before publication [15].

Operationally, apply the standard cloud hygiene set. Encrypt data at rest and in transit, use per-project bucket prefixes with least-privilege IAM, log all access, and keep PHI out of job names and log strings, because orchestrator logs are copied more widely than people expect. Any clinical interpretation that comes off the pipeline needs review by a qualified professional before it affects care.

Common Mistakes and Limitations

The mistakes below appear in nearly every cloud genomics project at least once.

Running the pilot on the same instance family as production. A tiny test instance hides memory and I/O bottlenecks that appear only at real read depth. Pilot on the actual instance type with one real sample.

Ignoring storage throughput. One thousand jobs all reading the same reference from one bucket prefix can hit request-rate limits and fail with throttling errors. Stage reference data per Region, use appropriately structured prefixes, and let the workflow's caching layer handle repeat reads.

Leaving compute environments running. Autoscaling Batch fleets scale to zero only if configured to. Idle fleets are the most common source of surprise bills in AWS genomics projects.

Assuming joint calling scales like per-sample calling. It does not. The gather stage needs dedicated tooling and its own cost line.

Skipping validation because the pipeline is published. A published pipeline validated on one reference build and read length can still misbehave on yours. Benchmark against a truth set on your own data.

Confusing reproducibility with containerization alone. Containers fix the software layer. Fix versions of references, parameters, and workflow code together, and record them in a single manifest per run.

Underestimating somatic privacy risk. Treat somatic callsets as potentially re-identifiable until germline leakage has been quantified and filtered [15].

One further limitation applies to this guide as a whole. Published per-sample costs come from specific years, price books, and instance generations, and cloud pricing moves. Treat the published figures as order-of-magnitude anchors, rebuild the estimate from current listed prices, and validate on your own ten-sample pilot. For clinical use, variant results require confirmation and interpretation by appropriately certified professionals.

Frequently Asked Questions

How much does it cost to scale variant calling in the cloud?

Compute dominates, and it scales roughly linearly with sample count once your workflow is fixed. Published population-scale runs have landed in the single-digit dollars per sample for compute on AWS, with storage and transfer as secondary lines. Model your cost from vCPU-hours per sample, apply Spot discounts where your workflow tolerates retries, and confirm the model with a small pilot before the full batch.

Which workflow manager should I use for cloud variant calling?

Nextflow is the pragmatic default for most teams because of its native AWS Batch integration and the nf-core ecosystem. WDL suits teams already on Cromwell-based platforms, and CWL maximizes portability across academic infrastructure. Any of the three works if every tool is containerized, which matters more than the language choice.

Do I need Kubernetes to run variant calling in the cloud?

No. AWS Batch, or a serverless orchestration design, runs large variant calling workloads without operating a Kubernetes cluster. Kubernetes becomes worth operating when you need distributed engines such as Spark-based pipelines or fine-grained control over many concurrent services.

Per-sample calling or joint calling, which should I run?

Run both stages. Per-sample calling parallelizes perfectly and handles the bulk of the compute. Joint calling or cohort merging then produces the consistent cohort-wide genotypes that association studies need, and that stage requires purpose-built scalable tooling rather than naive VCF concatenation.

Can I reuse an existing pipeline template instead of writing one?

Yes, and you should. nf-core/sarek covers germline and somatic short-read calling with published cost improvements, Parliament2 covers consensus structural variant calling in a single container, and DeepVariant with GLnexus covers cohort-level germline calling. Each is open source with published scaling behavior.

How do structural variant callers handle cloud scale?

Consensus frameworks bundle multiple SV callers to run in parallel inside one job container, which is how Parliament2 processed all 2,691 1000 Genomes samples in under a day. For the highest sensitivity to complex structural variation, long-read sequencing processed through cloud-native pipelines is the emerging population-scale approach.

How do I verify that a scaled cloud pipeline is correct?

Benchmark against a Genome in a Bottle truth set on your own data, report F1 score or recall and precision by variant type and size class, check cohort-level metrics such as Ti/Tv ratio and Mendelian consistency in trios, and confirm known variants manually. A published pipeline still needs validation on your reference build, read length, and sample ancestry mix.

Is patient genomic data safe in the cloud?

It can be, if you control it properly. Use encryption at rest and in transit, least-privilege access roles, audit logging, and a business associate agreement where applicable. For somatic data specifically, remember that pipelines can leak germline variants into somatic callsets, so filter against a germline reference before any public release.

Sources

1. Lessons from national biobank projects utilizing whole-genome sequencing for population-scale genomics.
2. Halvade somatic: Somatic variant calling with Apache Spark.
3. Inexpensive and Highly Reproducible Cloud-Based Variant Calling of 2,535 Human Genomes.
4. Celeste: A cloud-based genomics infrastructure with variant-calling pipeline suited for population-scale sequencing projects.
5. GenMPI: Cluster Scalable Variant Calling for Short/Long Reads Sequencing Data.
6. A hybrid computational strategy to address WGS variant analysis in >5000 samples.
7. Bionimbus: a cloud for managing, analyzing and sharing large genomics datasets.
8. Scalable and efficient DNA sequencing analysis on different compute infrastructures aiding variant discovery.
9. Parliament2: Accurate structural variant calling at scale.
10. Accurate, scalable cohort variant calls using DeepVariant and GLnexus.
11. Managing genomic variant calling workflows with Swift/T.
12. Cloud-native distributed genomic pileup operations.
13. rMATS-cloud: Large-scale Alternative Splicing Analysis in the Cloud.
14. Population-scale Long-read Sequencing in the All of Us Research Program.
15. Germline contamination and leakage in whole genome somatic single nucleotide variant detection.