# Variant Calling Pipeline Errors in the Cloud: Causes and Fixes

Variant calling pipeline errors in the cloud almost always trace back to one of a handful of causes: a task ran out of memory or disk, a container or dependency changed under your feet, the reference genome did not match the BAM or VCF you fed into it, a permissions or data transfer step failed silently, or the workflow engine lost track of task state. The first thing to check is the exit code and the last few lines of the log for the failed task. Most cloud failures announce themselves there. An exit code of 137 usually means the Linux out-of-memory (OOM) killer terminated the process, a 403 from object storage means a permissions problem, and a truncated FASTQ or BAM shows up as a file size mismatch in a checksum comparison.

This article walks through variant calling pipeline errors in the cloud from the most common cause to the rarest, explains the mechanism behind each failure, gives you a diagnostic sequence you can follow task by task, and closes with the prevention practices that large production systems use. Whether you run GATK Best Practices on AWS Batch, Snakemake on Google Cloud, or Cromwell on Azure, the failure modes are the same even when the error text differs.

## Why Cloud Variant Calling Fails Differently Than Local Runs

A variant calling pipeline converts raw sequence reads into a Variant Call Format (VCF) file, a text file that lists the variants at each genomic position. The steps are familiar: align reads to a reference genome, mark duplicates, recalibrate base qualities, call variants per sample or jointly across a cohort, and filter the calls. On a laptop or a single server, you control every variable. In cloud bioinformatics, that control disappears. Your compute nodes are ephemeral, your disks are provisioned and destroyed with each task, your data lives in object storage behind a permissions layer, and your software runs inside containers pulled from a registry.

That loss of control is also the point. Population-scale projects would be impossible otherwise. A cloud-based pipeline for joint variant calling of 2,535 human genomes from the 1000 Genomes Project processed the full cohort in five days at a compute cost of $7.33 per sample, and the authors concluded that near-linear scalability makes the cloud a practical platform for even larger studies [1]. At the other end of the scale spectrum, the All of Us Research Program built Celeste, a cloud architecture on Amazon Web Services (AWS) that uses serverless technology and container orchestration to analyze fluctuating workloads of up to 9,000 whole-genome sequencing (WGS) samples per month, coordinating roughly 1.8 million jobs in twelve months of production through around 200 million serverless functions and over 20 million messages [2]. Infrastructure that robust does not happen by accident. It survives at that scale because every failure mode described below was encountered, named, and handled deliberately.

The tradeoff is that a pipeline that runs correctly once can fail the next day for reasons that have nothing to do with your variant caller. Benchmarking work for clinical assays has documented this directly: analytical performance depends on the hardware, operating system, and software libraries underneath the pipeline, which makes it impossible to reliably repeat an assessment when any underlying dependency changes [3]. Cloud systems change those dependencies constantly, which is exactly why cloud-specific failure modes deserve their own troubleshooting discipline.

## A Diagnostic Sequence You Can Follow Every Time

![Decision path that sorts a failed cloud variant calling task by exit code into out of memory, corrupt input, software error or container problem.](/article-images/illustrations/variant-calling-pipeline-errors-cloud-exit-code-triage.webp)

When a [variant calling workflow](/blog/guides/variant-calling-workflow-design-decisions-before-you-start) fails in the cloud, resist the urge to rerun it and hope. Work through this sequence in order. It resolves or narrows down the majority of failures in under thirty minutes.

1. **Pull the exit code and stderr.** Open the log for the failed task in your workflow engine's UI (Cromwell, Nextflow Tower, Terra, AWS Batch console) or pull it from CloudWatch or Stackdriver. Note the exit code and the last error line before termination.
2. **Classify the exit code.** 137 means SIGKILL, almost always OOM. 139 means SIGSEGV, often a corrupt input file or a library conflict. 1 or 2 with a stack trace is a software or configuration error. A nonzero exit with no log at all points to the container, the image pull, or the instance itself.
3. **Verify the inputs reached the task.** Run `samtools quickcheck` on the BAM, run `gzip -t` on the FASTQ, and compare file sizes against the manifest. Cloud transfers fail silently more often than people expect.
4. **Confirm the reference.** Check that the FASTA, the `.fai` index, the sequence dictionary (`.dict`), and any known-sites resource all come from the same build and that contig names match your BAM (chr1 versus 1 is the classic mismatch).
5. **Check resource requests against actual usage.** Compare the memory and disk you requested with what the task consumed. Inspect the log for lines about swap, cgroup limits, or "no space left on device."
6. **Look at the node, not just the task.** If tasks fail randomly across a cohort, suspect spot instance interruptions, image registry throttling, or a region-level service problem rather than bad data.
7. **Reproduce the failing task locally.** Pull the exact container tag and run the failing command on a small slice of the input. If it succeeds locally, the problem is cloud-specific. If it fails locally, the problem is your command or data.

This sequence works because [variant calling pipelines](/knowledge/bioinformatics/variant-calling-pipelines-gatk-deepvariant) are long chains of deterministic file transformations. Almost every error is either a bad input entering a task, a resource shortfall inside a task, or an environment difference between tasks.

## Troubleshooting Table: Symptom, Cause, Confirmation, Fix

| Symptom | Likely cause | How to confirm | Fix |
|--|--|--|--|
| Task killed partway through large BAM processing, exit code 137 | Out of memory, OOM killer | Log shows "Killed" or cgroup OOM event near the end | Increase memory in the task definition, sort by query name instead of coordinate where possible, or shard by chromosome intervals |
| "No space left on device" during sorting or merging | Provisioned disk too small for intermediates | Compare BAM size to disk volume size in the task config | Request larger scratch disk, use instance-local NVMe storage, stream instead of writing intermediates |
| Random subset of tasks fails with no error pattern | Spot instance interruption | Cloud console shows instance reclaimed mid-task | Checkpoint at scatter boundaries, retry on non-spot capacity, or move long tasks to on-demand |
| Task fails before it starts, "image pull" or 403 from registry | Container image missing, private, or throttled | Run the pull command manually from a similar node | Pin the image tag, fix registry credentials, replicate the image into the same region |
| GATK errors that contig names or dictionary do not match | Reference build or contig-naming mismatch | `samtools view -H aln.bam` versus reference `.dict` headers | Reheader the BAM or regenerate all indices from one canonical reference bundle |
| Validation fails at download, exit before any compute | Object storage permissions or region mismatch | `aws s3 ls s3://bucket/key` from the same account and region | Correct IAM policy, copy data into the compute region, use VPC endpoints |
| Same workflow produces different VCFs on reruns | Floating inputs (latest tags), nondeterministic parallelization, or uninitialized parameters | Diff container digests and workflow revision between runs | Pin digests, commit workflow code to version control, fix seeds and thread counts |
| Variant counts differ wildly between two pipelines on the same data | Different caller versions or settings, not cloud infrastructure | Run `bcftools stats` and compare Ti/Tv and counts per caller version | Record exact tool versions, revalidate against a truth set after every version bump |
| Joint genotyping fails at cohort merge with malformed VCF errors | Dropped or truncated intermediate files from earlier failed tasks | Validate each per-sample VCF with `gatk ValidateVariants` | Make downstream tasks depend on validated outputs, delete stale intermediate cache |
| Calls look fine but performance metrics are poor in specific regions | Real sensitivity limits of the pipeline, not infrastructure errors | Benchmark against GIAB truth sets stratified by variant type and region [4] | Treat as an analytical tuning problem, add region-aware filters or a second caller |

## Cause 1: Resource Exhaustion (The Most Common Failure)

### The mechanism

Whole-genome alignment and marking of duplicates routinely require tens of gigabytes of RAM per task. A single 30x WGS BAM is around 100 GB, and sorting it doubles the scratch space needed. Cloud schedulers enforce the resources you request through cgroups, so when the task exceeds its memory limit, the kernel kills the process. The task then reports exit code 137, often after processing most of the file, which is why the error feels random.

Disk exhaustion follows the same logic. Sort and merge operations write temporary files many times the size of the final output. A task that provisioned 200 GB of scratch for a 100 GB BAM will fail during `MarkDuplicates` or during a joint genotyping merge across hundreds of samples.

### How to fix it

![Worked calculation that takes a measured peak memory and adds a 20 to 30 percent margin to set the task memory request.](/article-images/illustrations/variant-calling-pipeline-errors-cloud-memory-margin.webp)

- Size memory requests from measured usage, not guesses. Run one sample end to end, record peak RSS with tools such as `/usr/bin/time -v`, and set requests 20 to 30 percent above that.
- Use streaming where the tool supports it, so intermediates never touch disk.
- Scatter variant calling by genomic intervals. Calling 10,000 intervals of a few megabases each keeps every task small and retryable, and it is how production systems at the 9,000 WGS samples per month scale stay reliable [2].
- Keep retries cheap. Because each small task retries independently, a spot interruption or OOM event costs minutes, not days.

### A note on safety with human data

Genomic data is identifiable health information. When you grant tasks broad permissions to debug a resource failure, or when you copy a failing BAM to a personal bucket for local reproduction, you may create a compliance incident larger than the pipeline error. Scope credentials to specific buckets, and delete debug copies when the investigation ends.

## Cause 2: Spot Interruptions and Ephemeral Compute

### The mechanism

Cloud providers reclaim preemptible or spot capacity with short notice. If your workflow places a 12-hour joint calling task on spot capacity, an interruption destroys both the compute and any local scratch files. The task appears to fail "randomly" because the failure correlates with capacity markets, not with your data.

### How to fix it

- Reserve spot capacity for embarrassingly parallel, short, restartable tasks such as per-sample alignment and per-interval calling. This is exactly the structure that let a 2,535-genome joint calling job run in parallel and complete in five days at low cost [1].
- Pin long-running, hard-to-restart tasks such as a final cohort merge to on-demand or fixed capacity.
- Enable automatic retry at the workflow level. Most engines (Cromwell, Nextflow, Snakemake) support retry counts per task, and retry is often the only fix a spot interruption needs.
- Write checkpoints to durable storage at scatter boundaries so a retried task resumes from validated intermediates rather than starting over.

## Cause 3: Container and Dependency Drift

### The mechanism

A workflow that references `biocontainers/gatk:latest` or an unpinned conda environment is not a workflow, it is a lottery ticket. When the registry publishes a new build, tasks launched the next morning run different software than tasks launched the night before. This produces the maddening symptom of a pipeline that works in staging and produces different variant counts in production, or that fails only for samples processed after a certain date.

Reproducibility work in clinical settings makes the stakes explicit. A cloud-based benchmarking workflow for germline variant calling was built specifically so that analytical assessment does not depend on the technician executing it or the underlying hardware, because those dependencies otherwise make results impossible to repeat [3]. Modular pipelines using Snakemake and Docker were adopted in a multi-site cancer immunotherapy research network for the same reason, with benchmarking demonstrating improved reproducibility, precision, and recall across validated truth sets [5].

### How to fix it

- Pin container images by digest, not by mutable tag. In practice this means referencing `image@sha256:...` in your task definitions.
- Commit workflow code and parameter files to version control, and record the exact commit hash with every run.
- Freeze dependency versions in conda environment files rather than resolving them at runtime.
- Never allow two production runs to use the same mutable tag. If a tag changes, treat it as a new pipeline that requires revalidation.

## Cause 4: Reference Genome and Metadata Mismatches

### The mechanism

Variant callers compare evidence against an expected reference sequence. If the reference used for alignment and the reference used for calling differ, or if contig names disagree (GRCh37's `1` versus GRCh38's `chr1`), callers abort or, worse, emit calls against the wrong coordinates. Silent coordinate corruption is the most dangerous failure in this category because the pipeline finishes and hands you a plausible-looking VCF.

Related failures include a missing `.dict` sequence dictionary for GATK, a stale `.fai` index after a reference edit, and a known-sites resource for base quality recalibration built against a different release.

### How to fix it

- Build a single canonical reference bundle per genome build and store its checksums alongside the files. Every task in the workflow reads from that bundle.
- Validate before compute. `samtools quickcheck` catches truncated BAMs in seconds, and `gatk ValidateVariants -V calls.vcf -R reference.fa` catches malformed or coordinate-inconsistent VCFs before they contaminate downstream steps.
- Record reference build and transcript set in output metadata so that any comparison against other callsets, including public controls, starts from known assumptions.

## Cause 5: Storage Permissions, Transfers, and Region Errors

### The mechanism

In aws genomics workflows, data typically lives in Amazon S3 while compute runs in a specific AWS Region. Three things go wrong. First, the task's execution role lacks a `GetObject` permission, so the download fails with a 403 before any analysis starts. Second, data sits in a different Region from the compute, so transfers are slow or fail outright. Third, partial or corrupted uploads from a failed transfer pass through undetected and crash a task hours later.

The same pattern applies to Azure Blob Storage and Google Cloud Storage. Because the failure happens before the tool runs, logs often show nothing about variant calling at all, which misleads people into debugging the wrong layer.

### How to fix it

- Copy or replicate data into the same Region as the compute before launching workloads. Cross-Region transfer is both slow and a common failure point.
- Test storage access with the same credentials and network path the workflow uses, for example `aws s3 ls s3://your-bucket/prefix/` executed inside the compute environment, not from your workstation.
- Verify checksums (MD5 or SHA-256) on arrival, and make the workflow fail fast on mismatch.
- Use VPC endpoints for storage traffic so large BAM transfers do not depend on public internet paths.

## Cause 6: Workflow Engine and Orchestration Errors

### The mechanism

Workflow engines (Cromwell for WDL, Nextflow, Snakemake, Apache Airflow) manage task state, retries, and dependencies. Errors here include a malformed inputs file, a task that references an output the upstream task never published, call-caching reuse of a stale result after you changed an input, and merge tasks that proceed despite failed scatter branches. At the scale of the All of Us production system, where millions of jobs were coordinated in a year, orchestration is precisely the layer designed to absorb and route these failures [2].

Pipelines built on different engines show the same classes of error. A Microsoft team running variant calling on PacBio HiFi long reads used Cromwell on Azure and merged genomic results with FHIR-format health records inside one scalable pipeline [6]. A fungal genomics workflow, TheiaEuk, was written in the Workflow Description Language (WDL) with containerized components for the Terra platform, a design chosen so that laboratory scientists could run analysis without coding, which in turn required the orchestration layer to catch configuration errors before they reached compute [7].

### How to fix it

- Treat the workflow definition itself as versioned code. Validate WDL or Snakemake syntax with a dry run before submitting thousands of tasks.
- Disable or carefully audit call caching after any change to inputs, references, or container digests, because a cached result can silently reuse pre-change output.
- Make every downstream task depend on validated upstream outputs, so a failed scatter branch stops the merge instead of producing a corrupt cohort VCF.
- Use the engine's metadata (task statuses, timings, retry counts) as your primary debugging surface. It is usually more informative than the raw logs.

## Cause 7: Caller-Level and Analytical Errors That Look Like Infrastructure Problems

Some errors survive a perfect infrastructure run. The callset is complete, the VCF validates, and the results are still wrong. These are the hardest variant calling pipeline errors in cloud settings because every layer of the stack reports success.

### Caller disagreement and version effects

Concordance among variant callers and alignment algorithms is generally limited, which is why ExScalibur was built as an automated, scalable exome pipeline that integrates multiple alignment and variant calling algorithms and reports close to 99 percent sensitivity and specificity across its analytical modules [8]. When variant counts change after a caller version bump, that is expected behavior, not a cloud fault. Confirm it with `bcftools stats`, compare Ti/Tv ratios and per-type counts, and revalidate against a truth set.

### Low allele fraction artifacts

Somatic and low-frequency calling is especially error prone. The ArCH pipeline for clonal hematopoiesis exists because distinguishing low-frequency mutations from sequencing artifacts is difficult, and it combines the output of four variant calling tools with filtering based on variant characteristics and sequencing error rate estimation, running as an end-to-end cloud-based pipeline with customizable parameters [9]. If your cloud pipeline reports a suspicious spike of low-allele-fraction calls, suspect artifacts before you suspect batch effects in the cloud environment.

### Cohort-level statistical inflation

When you merge your calls with public datasets as controls, pipeline differences can generate false positives. A systematic analysis of rare-variant association tests found that inflated distributions of test statistics decreased substantially when the same variant caller and filtering pipelines were used across cases and controls, while differences in library prep kits and sequencers did not drive the effect and joint versus separate calling did not either [10]. The practical implication for cloud work is direct: when combining your callset with external data, rerun external samples through your own pinned pipeline or confirm the external pipeline's configuration in detail, because pipeline provenance matters more than wet-lab platform differences.

### Human review of hard calls

Automated filtering cannot settle every ambiguous call. SV-plaudit provides a cloud-based framework in which each [structural variant](/blog/guides/structural-variant) prediction is rendered as an image of coverage and alignment signals, reviewed through a web application, and exported as an annotated VCF. In a proof of principle, nine researchers collectively evaluated thousands of structural variant calls in about an hour of curation each [11]. Building a curation step into your cloud workflow converts unresolved ambiguous calls into documented human decisions.

## Validation and Benchmarking: Turning Errors Into Measurements

A cloud pipeline that never sees a truth set is a pipeline whose errors you have not met yet. The Genome in a Bottle (GIAB) consortium publishes benchmark small variant calls produced by a reproducible cloud-based pipeline integrating multiple sequencing datasets, and those benchmarks reliably identify errors in existing callsets [4]. The GIAB releases expanded earlier resources substantially, adding 17 percent more benchmark single nucleotide variations, 176 percent more indels, and 12 percent larger benchmark regions [4].

Benchmarking tools compare your callset to truth in a stratified way. The standard options referenced in the benchmarking literature include hap.py, vcfeval, and vcflib [3]. A practical validation loop looks like this:

1. Run your pipeline on one or more GIAB samples with the same workflow and container digests you use in production.
2. Compare with hap.py or vcfeval against the appropriate GIAB benchmark regions.
3. Stratify results by variant type and genomic context, because performance is not uniform across regions and variant types [3].
4. Record the metrics with the run. Any future change to caller version, container digest, or reference bundle should produce a new benchmark row, and a drop in precision or recall should block production use until explained.

Domain-specific pipelines follow the same discipline. A pharmacogenetics pipeline, Pgxtools, validated its variant calls against the 1000 Genomes WGS dataset and against CDC Pharmacogenetics Reference Materials before release [12]. Long-read pipelines for the All of Us program compared short-read and PacBio HiFi performance on the same samples, found HiFi reads produced the most accurate results for both small and large variants, and built a cloud pipeline to optimize SNV, indel, and structural variant calling at scale [13]. Whatever your data type, the pattern is the same: measure against reference material, stratify the metrics, and version the measurements.

## Common Mistakes and Limitations

- **Rerunning before reading the log.** A retry without diagnosis burns compute budget and hides the cause. Read the exit code first, every time.
- **Assuming the cloud is the problem.** Many "cloud" failures reproduce perfectly on a local machine with the same container and command. Reproduce locally before blaming the platform.
- **Pinning nothing.** Unpinned images, unpinned references, and unpinned parameters are the leading cause of irreproducible callsets. Pin by digest and record hashes.
- **Treating benchmark metrics as transferable.** A pipeline's measured precision and recall apply to the truth set, genome, and variant types tested. Performance varies by region and variant type, so a high genome-wide F1 score says little about a difficult gene or a homopolymer [3], [4].
- **Ignoring intermediate validation.** Teams validate the final VCF and nothing else. Validate every major intermediate, especially BAM integrity and per-sample VCF well-formedness, so failures surface minutes after they occur instead of hours later at merge.
- **Conflating caller differences with infrastructure errors.** Two pipelines producing different variant counts is not a cloud failure. It is expected caller behavior, documented across many studies [10], [8], and it calls for truth-set benchmarking, not for hunting ghosts in the scheduler.
- **Debugging with over-broad credentials.** Widening IAM permissions to "just get it running" leaves standing access to human genomic data. Fix the specific permission, then revert.
- **Trusting a green checkmark.** A workflow that completes is not a workflow that is correct. Completion plus validation plus benchmarking is the minimum bar for a production callset.

## Prevention Practices That Hold Up at Scale

The systems that process petabytes successfully share a set of habits, and you can adopt them at any scale.

**Build a canonical reference service.** Store one validated reference bundle per genome build with checksums and all indices. Every workflow references it. Nobody regenerates indices per run. This single practice eliminates the reference mismatch class of errors.

**Pin everything, then verify the pins.** Container digests, workflow commit hashes, reference checksums, and parameter files all get recorded with each run. The multi-site cancer research network that rebuilt its WES and RNA-Seq pipelines on Snakemake and Docker did so explicitly to improve reproducibility and benchmark performance against validated truth sets [5].

**Design for cheap retries.** Shard work into small, independent, restartable tasks. Scatter by genomic interval for variant calling. Put retries at the workflow layer. Production systems handling fluctuating workloads of thousands of WGS samples per month depend on this structure to absorb routine failures without human intervention [2].

**Validate inputs and outputs at every boundary.** `samtools quickcheck` on BAMs, `gatk ValidateVariants` on VCFs, checksums on transfers. Fail fast, fail loudly, and fail close to the cause.

**Benchmark continuously, not once.** Cloud-based benchmarking workflows exist precisely so that analytical performance can be reassessed continually as dependencies change, independent of the technician or the hardware [3]. Schedule benchmark runs on the same schedule as your dependency updates.

**Keep a run journal.** Every production run gets a record with the workflow version, container digests, reference checksums, engine version, and benchmark metrics. When a callset looks wrong six months later, this record is the difference between a diagnosis and a guess.

**Monitor cost per sample alongside success rate.** Cloud economics reward parallelism and near-linear scalability, as demonstrated in the 2,535-genome joint calling study [1], but a pipeline whose failure rate forces repeated compute silently multiplies cost. Track cost per completed sample, not cost per launched task.

**Match the pipeline to the data.** Long-read data needs different calling tools and different benchmarks than short-read data, and mixing them under one workflow configuration is a common source of poor performance. Dedicated cloud pipelines for long-read SNV, indel, and SV calling improved results at cohort scale [13]. Review cloud architecture choices, such as serverless versus batch, when workloads shift in size or type rather than forcing an old design onto new data.

## Frequently Asked Questions

### Why does my variant calling pipeline run fine locally but fail on the cloud?

The most common reasons are resource limits smaller than your local machine, different container or library versions resolved in the cloud, and storage permissions or Region placement that block data access. Reproduce the failing command locally with the exact container digest. If it succeeds locally, compare the cloud task's memory, disk, and image against your local setup and check that the input files arrived intact.

### How do I know if a failed task ran out of memory?

Look for exit code 137 in the task log, often with a "Killed" message or an out-of-memory event in the system log near the end of the task. Confirm by measuring peak memory on a successful rerun with increased limits, then size the task definition about 20 to 30 percent above the measured peak.

### Should I joint-call my cohort or call samples separately and merge?

Both work, and the statistical literature shows joint versus separate calling does not by itself inflate association test statistics [10]. Joint calling concentrates memory and disk demand in one place, while per-sample calling with later genotyping scatters cleanly across cloud tasks and retries cheaply. Choose based on your cohort size and your callers' requirements, and validate either way against a truth set.

### Why do reruns of the same workflow give different variant counts?

Different reruns using mutable image tags, unpinned dependencies, or stale cached results will produce different software environments and therefore different calls. Pin containers by digest, version-control the workflow, and audit call caching. Genuine caller version changes also change counts, so record tool versions with every run.

### How do I benchmark my cloud variant calling pipeline?

Run your exact production workflow on Genome in a Bottle reference samples and compare your calls to the published benchmark VCFs using tools such as hap.py or vcfeval, then stratify metrics by variant type and genomic region [3], [4]. Repeat the benchmark after every change to callers, containers, or references and keep the metrics with your run records.

### Which AWS services matter most for genomics pipelines?

Amazon S3 for data, a compute orchestration layer such as AWS Batch or serverless functions for tasks, and container images for reproducible software form the core pattern. The All of Us Celeste architecture runs on this combination, using serverless technology and container orchestration to coordinate millions of analysis jobs [2]. Place your data and compute in the same Region and scope task permissions tightly.

### How can I reduce cloud costs without hurting reproducibility?

Use spot or preemptible capacity for short, restartable, embarrassingly parallel tasks, and keep long or merge tasks on fixed capacity with retries enabled. Shard by genomic interval so interrupted work is cheap to redo. Cost per sample improves with parallelism, as shown in large cohort joint calling studies [1], while reproducibility is protected by pinned containers and versioned workflows, not by expensive hardware.

### Are public datasets safe to use as control cohorts in my analysis?

They are useful but risky for rare-variant association testing, because differences between your pipeline and the public dataset's pipeline can inflate false-positive findings [10]. The safest approach is to process public samples through your own pinned pipeline or to verify the external pipeline's caller, version, and filtering in detail before merging.

<script type="application/ld+json">
{
  "@context": "https://schema.org",
  "@type": "FAQPage",
  "mainEntity": [
    {
      "@type": "Question",
      "name": "Why does my variant calling pipeline run fine locally but fail on the cloud?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "The most common reasons are resource limits smaller than your local machine, different container or library versions resolved in the cloud, and storage permissions or Region placement that block data access. Reproduce the failing command locally with the exact container digest. If it succeeds locally, compare the cloud task's memory, disk, and image against your local setup and check that the input files arrived intact."
      }
    },
    {
      "@type": "Question",
      "name": "How do I know if a failed task ran out of memory?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Look for exit code 137 in the task log, often with a Killed message or an out-of-memory event near the end of the task. Confirm by measuring peak memory on a successful rerun with increased limits, then size the task definition about 20 to 30 percent above the measured peak."
      }
    },
    {
      "@type": "Question",
      "name": "Should I joint-call my cohort or call samples separately and merge?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Both approaches work, and research shows joint versus separate calling does not by itself inflate association test statistics. Joint calling concentrates memory and disk demand in one place, while per-sample calling scatters cleanly across cloud tasks and retries cheaply. Choose based on cohort size and caller requirements, and validate against a truth set either way."
      }
    },
    {
      "@type": "Question",
      "name": "Why do reruns of the same workflow give different variant counts?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Reruns using mutable image tags, unpinned dependencies, or stale cached results produce different software environments and therefore different calls. Pin containers by digest, version-control the workflow, and audit call caching. Caller version changes also change counts, so record tool versions with every run."
      }
    },
    {
      "@type": "Question",
      "name": "How do I benchmark my cloud variant calling pipeline?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Run your exact production workflow on Genome in a Bottle reference samples and compare your calls to the published benchmark VCFs using tools such as hap.py or vcfeval, then stratify metrics by variant type and genomic region. Repeat the benchmark after every change to callers, containers, or references and keep the metrics with your run records."
      }
    },
    {
      "@type": "Question",
      "name": "Which AWS services matter most for genomics pipelines?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Amazon S3 for data, a compute orchestration layer such as AWS Batch or serverless functions for tasks, and container images for reproducible software form the core pattern. The All of Us Celeste architecture runs on this combination, coordinating millions of analysis jobs. Place data and compute in the same Region and scope task permissions tightly."
      }
    },
    {
      "@type": "Question",
      "name": "How can I reduce cloud costs without hurting reproducibility?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "Use spot or preemptible capacity for short, restartable, parallel tasks, and keep long or merge tasks on fixed capacity with retries enabled. Shard by genomic interval so interrupted work is cheap to redo. Cost per sample improves with parallelism, while reproducibility is protected by pinned containers and versioned workflows, not by expensive hardware."
      }
    },
    {
      "@type": "Question",
      "name": "Are public datasets safe to use as control cohorts in my analysis?",
      "acceptedAnswer": {
        "@type": "Answer",
        "text": "They are useful but risky for rare-variant association testing, because differences between your pipeline and the public dataset's pipeline can inflate false-positive findings. Process public samples through your own pinned pipeline, or verify the external pipeline's caller, version, and filtering in detail before merging."
      }
    }
  ]
}
</script>

## Sources

1. [Inexpensive and Highly Reproducible Cloud-Based Variant Calling of 2,535 Human Genomes.](https://pubmed.ncbi.nlm.nih.gov/26110529/)
2. [Celeste: A cloud-based genomics infrastructure with variant-calling pipeline suited for population-scale sequencing projects.](https://pubmed.ncbi.nlm.nih.gov/40343041/)
3. [Benchmarking workflows to assess performance and suitability of germline variant calling pipelines in clinical diagnostic assays.](https://pubmed.ncbi.nlm.nih.gov/33627090/)
4. [An open resource for accurately benchmarking small variant and reference calls.](https://pubmed.ncbi.nlm.nih.gov/30936564/)
5. [Modular and cloud-based bioinformatics pipelines for high-confidence biomarker detection in cancer immunotherapy clinical trials.](https://pubmed.ncbi.nlm.nih.gov/40857351/)
6. [A cloud-based pipeline for analysis of FHIR and long-read data.](https://pubmed.ncbi.nlm.nih.gov/36726729/)
7. [TheiaEuk: a species-agnostic bioinformatics workflow for fungal genomic characterization.](https://pubmed.ncbi.nlm.nih.gov/37593727/)
8. [ExScalibur: A High-Performance Cloud-Enabled Suite for Whole Exome Germline and Somatic Mutation Identification.](https://pubmed.ncbi.nlm.nih.gov/26271043/)
9. [ArCH: improving the performance of clonal hematopoiesis variant calling and interpretation.](https://pubmed.ncbi.nlm.nih.gov/38485690/)
10. [Inflated expectations: Rare-variant association analysis using public controls.](https://pubmed.ncbi.nlm.nih.gov/36696392/)
11. [SV-plaudit: A cloud-based framework for manually curating thousands of structural variants.](https://pubmed.ncbi.nlm.nih.gov/29860504/)
12. [A New Cloud-Native Tool for Pharmacogenetic Analysis.](https://pubmed.ncbi.nlm.nih.gov/38540411/)
13. [Utility of long-read sequencing for All of Us.](https://pubmed.ncbi.nlm.nih.gov/38281971/)