Batch vs. Real-Time Processing for Genomic Workflows

By Dr. Zubair Khalid, DVM, MS, PhD · 2026-11-11

Batch processing collects samples and runs them together on a schedule, while real-time processing starts work the moment data arrives and produces results within seconds to minutes. The rule of thumb for choosing between them is simple: if a person, an instrument, or a clinical decision depends on a result within minutes of the data existing, process in real time. If the result can wait hours or days without harming anyone or blocking any decision, batch it.

This article walks through the difference between batch and real-time genomics processing on its own terms, compares them on the criteria that actually decide infrastructure choices, works through concrete scenarios, and covers the hybrid architectures and cloud patterns that most production genomics groups end up running.

The difference between batch and real-time genomics processing

The difference between batch and real-time genomics processing comes down to when work begins and what the system optimizes for.

Batch processing waits for data to accumulate, groups it into a job, and runs that job from start to finish on a fixed pool of compute. The MapReduce model and its open source implementation Hadoop, long the default for large-scale data analysis, were designed from the ground up for exactly this kind of batch and high-throughput work. In a genomic context, a batch job might take 500 FASTQ files delivered over the past week, align each one to a reference genome, call variants, and write consolidated results at the end.

Real-time processing, sometimes called stream processing, treats each incoming record or event as something to act on immediately. When demand for low-latency, interactive, and streaming workloads outgrew what batch-only systems could offer, a family of new architectures and engines emerged specifically to handle real-time queries and data streams. In a genomic context, a real-time system might basecall each nanopore read as it exits the pore, map it to a reference within seconds, and flag a run-quality problem while the flow cell is still sequencing.

Two clarifications prevent most confusion later:

1. Real-time does not mean faster compute. A real-time pipeline and a batch pipeline can run the same aligner at the same speed on the same hardware. Real-time refers to the latency between data arriving and work starting, not to how fast the work itself finishes.
2. Batch does not mean slow overall. A batch system running thousands of samples in parallel often produces more finished genomes per day than any real-time setup. It simply does not promise anything about when any single sample starts.

Genomic data also has a natural shape that matters here. A sequencing run generates data in bursts, files are large and self-contained, and most downstream analysis (secondary analysis such as alignment and variant calling, and tertiary analysis such as annotation and interpretation) operates on complete files rather than on a dribble of individual records. That structure favors batch far more than the firehose-style sensor data that motivated much of the original stream processing literature. At the same time, instruments themselves have become streaming systems, which keeps real-time design relevant to genomics in specific, well-defined places.

Batch processing in genomic workflows

Batch is the default pattern for secondary analysis in genomics, and for good reasons rooted in how the data and the science behave.

What a batch pipeline looks like

A typical batch workflow is a directed acyclic graph (DAG), meaning a pipeline of steps where data flows in one direction and no step feeds back into an earlier one. Tools such as Nextflow, Cromwell (which runs workflows written in WDL), and Snakemake express these DAGs, track which steps finished, and allow a failed run to resume from the last completed step.

A standard germline variant calling pipeline illustrates the pattern:

1. Start with FASTQ files (raw compressed sequence reads with quality scores).
2. Align reads to a reference genome with an aligner such as BWA-MEM, producing a SAM file that is compressed into BAM or CRAM format.
3. Mark duplicates and recalibrate base quality scores.
4. Call variants per chromosome interval, a decomposition strategy known as scatter-gather.
5. Merge per-chromosome calls into a single VCF (variant call format) file and annotate it.

Each sample is an independent unit of work. That property, sometimes called embarrassingly parallel, is why batch scheduling works so well here. A scheduler can spread 500 samples across 500 containers without the samples ever interacting.

Why batch fits genomics

Batch processing offers four properties that genomics heavily rewards:

• Reproducibility. A batch job pins its software (usually in a container image), its reference genome build, and its parameter set at launch. Rerunning the same job on the same inputs produces the same outputs. This matters for publications, regulatory submissions, and any analysis you may need to defend later.
• Checkpointing and restart. If a run fails at step 4 of 5, batch frameworks resume from step 4 rather than repeating the whole pipeline. On long jobs this saves enormous amounts of compute.
• Throughput optimization. Batch schedulers pack work tightly onto machines, fill idle capacity with queued jobs, and bid for discounted spare capacity (spot or preemptible instances in cloud terms) that real-time systems generally cannot tolerate because a spot reclaim kills the running job.
• Simplicity of reasoning. You know exactly what is running, on what data, with what software version. Debugging, auditing, and cost accounting are all straightforward.

The cohort dimension

Some genomic questions only make sense at batch scale. Joint genotyping across a cohort, population-level allele frequency estimation, and genome-wide association studies all require having many samples finished and available before the analysis can run. No amount of per-sample real-time processing substitutes for a scheduled, reproducible batch step that aggregates the cohort. This is the strongest form of "when to use batch": whenever the scientific question itself is defined over a collection rather than over a single sample.

Real-time processing in genomic workflows

Real-time processing in genomics is narrower than the general hype suggests, but where it applies it changes what is scientifically and operationally possible.

Streaming at the instrument

Modern sequencers are themselves real-time systems. Nanopore instruments, for example, emit sequence signal continuously, and basecalling (the conversion of raw electrical signal into bases) can run as each read finishes rather than after the run ends. That opens the door to live decisions: watching mapping rates and read quality accumulate during the run, and stopping or adjusting the experiment early when the data already answers the question. This kind of online feedback loop, where processing results feed back into the running process, is the defining feature of real-time pipelines.

Interactive querying of finished results

A second real-time pattern has nothing to do with instruments and everything to distance from it: answering analyst questions in seconds rather than hours. Distributed query engines such as Presto (now widely deployed through the Trino community) were built precisely to move organizations from batch-only processing toward interactive analytics, real-time reporting, and dashboards, running unified SQL over heterogeneous storage systems without copying the data first [1]. For a genomics group, that means an analyst can issue a query against variant tables or sample metadata in a data lake and get results back in seconds, without submitting a batch job and waiting in a queue.

Real-time feedback in analysis platforms

Web-based bioinformatics platforms have pushed the same idea further. OmicStudio, a composable bioinformatics cloud platform, was designed around real-time feedback so that users running analyses and generating publication-quality visualizations see results immediately as they adjust parameters, rather than submitting a job and waiting [2]. This is real-time processing applied to the human analysis loop: the latency that matters is the wait between changing an input and seeing the figure.

The healthcare analogy

The genomics patterns above have close parallels in healthcare data processing more broadly. One IoT-cloud framework for healthcare separates two requirements that frequently appear together: continuous real-time analysis of streaming sensor data so that care decisions happen promptly, and long-term batch processing of the accumulated data to train machine learning models for prediction [3]. Genomics groups face the same pairing. Streaming covers instrument monitoring and fast triage. Batch covers cohort model training and periodic reannotation of the whole database against updated reference resources.

Side-by-side comparison

The table below is the core of choosing between batch and real-time processing for genomic workflows.

When to use batch

Use batch processing when at least three of the following are true:

1. A human decision window of hours or more is acceptable. Research analysis, population screening reports, and periodic clinical reporting all tolerate batch turnaround.
2. Data arrives in groups. Plate-based sequencing, weekly instrument schedules, and retrospective cohort studies naturally produce batches.
3. The analysis is defined over the collection. Joint genotyping and cohort statistics cannot run on one sample at a time anyway.
4. Cost per sample is a constraint. Batch jobs fill machines, tolerate spot interruptions, and can run overnight at discounted rates.
5. Reproducibility carries legal or scientific weight. A pinned batch job is an auditable artifact.

Worked scenario: a 1,000-sample cohort reanalysis

Suppose your lab needs to reprocess 1,000 whole genomes against a new reference build and updated variant catalog. Each 30x human genome produces FASTQ input on the order of a hundred gigabytes. This is unambiguously a batch problem.

A concrete cloud implementation:

1. Lay out inputs in Amazon S3 (simple storage service) under a per-sample prefix, with a manifest table listing sample identifiers and file paths.
2. Write the pipeline once in Nextflow or WDL. The alignment and variant calling steps scatter across genomic intervals and across samples.
3. Submit work to AWS Batch, which runs each task in a container on EC2 instances drawn from a queue you configure. Spot instances cut compute cost substantially in exchange for occasional task interruptions, which the workflow engine retries automatically.
4. Resume on failure with nextflow run germline.nf -resume, which skips completed tasks.
5. Write final CRAM and VCF outputs back to S3 and register them in a sample metadata catalog so downstream query tools can find them.

Total wall time depends on fleet size. The defining feature is that no sample needs an answer in the next five minutes, so the scheduler is free to optimize for throughput and cost, which is exactly what batch does best.

The energy and scheduling dimension

Batch workloads also have levers that real-time workloads lack. Work on scheduling Spark applications has shown that a scheduler can exploit time-series profiles of application energy use and execution time, then apply dynamic voltage and frequency scaling (DVFS, the practice of lowering processor voltage and clock speed to save power) to reduce datacenter energy consumption while still meeting performance requirements [4]. For a genomics group running large batch fleets, this matters in two ways. Overnight batch jobs can run on slower, cheaper, more energy-efficient capacity without anyone noticing. And organizations with sustainability commitments can report real energy reductions from shifting flexible work into batch windows rather than keeping hardware provisioned around the clock.

When real-time earns its keep

Use real-time processing when at least two of the following are true:

1. The value of the result decays within minutes. Run-quality decisions, contamination alerts, and time-critical clinical triage all lose value while a batch queue waits.
2. The data arrives as a continuous stream. Signal-level instrument output and telemetry are inherently streaming.
3. The consumer is an interactive human. Analysts exploring a cohort, clinicians watching a dashboard, and platform users tuning visualizations all need sub-minute feedback.
4. Downstream automation acts on events. Automated laboratory systems that trigger a follow-up assay when a result lands need event-driven processing, not a nightly job.

Worked scenario: live QC during a long-read sequencing run

During a nanopore run, basecalling and alignment run continuously as reads are produced. Within the first fraction of the run, the streaming statistics already reveal whether the library is working: read length distribution, mapping rate to the expected organism, and adapter contamination. If those numbers look wrong, the operator stops the run, saves the remaining reagent and instrument time, and fixes the library. A batch-only workflow would deliver the same bad news the morning after the run ended, after the flow cell and reagents were spent. This is the clearest genomics case where the streaming approach pays for itself, because the early result changes a physical action in the lab.

Worked scenario: interactive cohort exploration

After a large batch pipeline finishes, an analyst wants to know how many samples carry rare variants in a gene panel, stratified by ancestry and sequencing center. Submitting a batch job for each exploratory question would take hours per question and would kill exploratory analysis entirely. A query engine that provides unified SQL over the data lake without moving data first solves this, which is precisely the role Presto-style engines were designed to fill at internet scale, supporting millions of queries per day across heterogeneous storage [1]. The batch pipeline remains responsible for producing the analytical tables. The real-time engine is responsible for making them answer questions in seconds.

Worked scenario: continuous clinical monitoring

The healthcare streaming pattern translates directly to genomics-adjacent clinical settings. In the IoT-cloud healthcare model, streaming analysis of incoming patient data supports immediate care decisions while the same accumulated data feeds long-term batch model training [3]. A molecular diagnostics lab can mirror this: streaming QC and result flagging during acquisition, and batch reprocessing of the historical record whenever references, panels, or calling algorithms improve.

Hybrid architectures: Lambda, Kappa, and ZeroETL

Most real genomics groups end up running both modes, and the big data community has converged on named architectures for doing so.

Lambda architecture

The Lambda architecture runs two parallel paths over the same data: a batch layer that periodically recomputes complete, accurate views over the full history, and a speed layer that processes the live stream for up-to-the-minute results, with a serving layer merging the two.

Applied to genomics, the batch layer is your production secondary analysis pipeline, and the speed layer is instrument streaming, live QC, and event-driven alerting. A concrete implementation from the energy-efficient homes domain shows the shape of the components: TimescaleDB for short-term storage, Apache Hudi for long-term storage, Kafka for data streaming, GraphQL-powered visualizations for real-time insight, and batch processing for advanced analytics and machine learning model training [5]. A genomics group would substitute its own tools but recognize the structure immediately, because the dual demands are identical: end users wanting live answers, and data scientists wanting complete historical datasets for analysis.

A smart farming platform built on the same Lambda pattern makes the motivation explicit: combining real-time monitoring with batch analytics lets stakeholders act on current conditions while still benefiting from retrospective analysis over the full record [6]. Genomics operations centers, which monitor instrument fleets, queue depths, and sample flow in real time while batch pipelines crunch the science, are structurally the same system.

Kappa architecture and ZeroETL

The Kappa architecture drops the batch layer entirely and treats everything as a stream, replaying the stream from storage when batch-style recomputation is needed. It reduces the two-engine complexity of Lambda but demands that every transformation be expressible as stream processing.

More recently, the ZeroETL idea has emerged to remove the extract-transform-load boundary between streaming and batch paths altogether. The Streaming View engine in AnalyticDB at Alibaba Cloud was built because both Lambda and Kappa typically require bolting on external stream processing engines, which creates challenges around complexity, efficiency, and consistency, and because existing incremental view maintenance (IVM, the technique of updating precomputed results by processing only the changed data) implementations focused on query acceleration rather than high-throughput transformation workloads [7]. The lesson for genomics infrastructure planners is direct: every additional engine you bolt on to serve both modes costs you consistency guarantees and operational headroom, so prefer platforms that close the gap natively when they exist.

The genomics synthesis

For genomics, the practical synthesis most groups converge on looks like this:

1. Instruments and acquisition systems stream raw data and QC metrics into object storage and a message bus.
2. Event triggers start per-sample processing the moment files land, but the per-sample processing itself runs as an idempotent batch job (idempotent meaning running it twice produces the same result as running it once).
3. Scheduled batch passes handle cohort aggregation, reannotation, and model retraining.
4. A query layer serves interactive questions over the finished results.

Note the subtle resolution in step 2. The trigger is real-time. The triggered work is batch. This hybrid, event-driven batch, captures most of the latency benefit of streaming without inheriting the full complexity of a stream processing engine, and it keeps the reproducibility guarantees that genomics requires.

Choosing between batch and real-time processing for genomic workflows

When you have to commit, work through these six questions in order:

1. Who consumes the result, and how fast do they need it? A clinician waiting to act, an operator deciding whether to stop a run, and an analyst exploring data have latency needs measured in seconds to minutes. A publication, a cohort database, and a regulatory submission have latency needs measured in days. Consumer latency is the first and hardest constraint, and it decides the mode more often than any technical consideration.
2. How does the data arrive? Bursty, file-shaped data (the native shape of sequencing output) suits batch. Continuous signal streams suit real-time. Fighting the natural arrival pattern of your data always costs more than accommodating it.
3. What does failure cost, and how must recovery behave? Batch gives you checkpointing and resume for free. Streaming gives you event replay and dead-letter handling, which is a different and more demanding discipline. Match the failure model to your team's actual skills.
4. What is the true cost profile? Batch clusters can scale to zero between jobs. Streaming systems hold capacity open continuously, and cost analyses of cloud-native serverless streaming show that total cost depends heavily on the split between active streaming time and idle time, with service selection producing substantial cost differences across providers [8]. If your streaming path idles most of the day, you are paying for nothing during those hours unless you chose a scale-to-zero design.
5. Can you meet the latency need with event-driven batch instead? As described above, triggering a batch job the instant data lands often satisfies the real requirement at batch economics. Many groups that believed they needed streaming infrastructure discovered that event-triggered batch was sufficient.
6. Do you have the hybrid architecture budget? Running both modes means two sets of tooling, two sets of failure modes, and consistency questions between the layers. The ZeroETL literature documents these costs concretely. Budget for them or simplify.

If the answers point in different directions, weight the consumer latency question highest. Compute is replaceable. A missed decision window usually is not.

Cloud bioinformatics and AWS genomics

Cloud bioinformatics means running genomic data storage, pipelines, and analysis on rented, elastic infrastructure rather than on local clusters, scaling capacity up and down with workload. The batch-versus-real-time decision interacts with the cloud in specific ways.

Storage-first design

Both modes start from the same foundation: raw reads, alignments, and variant calls live in durable object storage (Amazon S3 in the AWS ecosystem), organized by sample and versioned. This storage-first, or data lake, pattern lets batch pipelines, streaming triggers, and query engines all read the same bytes without duplicating them. Query engines that run unified SQL directly over heterogeneous storage without copying data make this pattern practical at scale.

Batch on AWS

AWS Batch is the managed service most genomics groups use for the batch path. You define job queues and compute environments, register a job definition that points at a container image, and submit jobs from the command line or from a workflow engine. A typical submission looks like this:

aws batch submit-job \
  --job-name germline-NA12878 \
  --job-queue wgs-standard \
  --job-definition gatk-germline-4-6 \
  --parameters sampleId=NA12878,fastqR1=s3://my-lake/fastq/NA12878_R1.fastq.gz

In practice, most groups let Nextflow or Cromwell drive the submission so the DAG, retries, and resumption logic live in the workflow layer. Spot capacity for batch workers, with automatic retry of interrupted tasks, is the single largest cost lever in cloud batch genomics.

The streaming and event path on AWS

For the real-time path, S3 event notifications can fire the moment an object lands, kicking off a lightweight function or a Step Functions state machine that starts processing. Note a naming trap here: AWS Lambda is a serverless function service, completely unrelated to the Lambda architecture pattern described earlier. Both terms circulate in the same design documents, and new engineers confuse them constantly.

For sustained stream processing, managed streaming services and container-based consumers handle instrument telemetry and message-bus traffic. AWS also offers HealthOmics, a managed service purpose-built for storing, querying, and processing omics data, which covers several of these patterns with genomics-specific defaults.

Cost discipline

Cost behavior differs sharply between the modes, and the FinOps discipline (the practice of managing cloud spending collaboratively between engineering and finance) exists largely because these differences surprise organizations. Analyses of cloud-native near-real-time serverless streaming show that costs depend heavily on processing management and service selection, and that the same architectural pattern can carry very different price tags across cloud providers [8]. Two practical disciplines follow. First, attribute every batch cost to a job or project, which is easy because batch jobs have clean boundaries. Second, measure idle time on streaming infrastructure and force a decision about it, because idle streaming capacity is the most common source of avoidable spend in event-driven designs.

Multi-cloud and portability

Some genomics organizations span providers, whether for resilience, data residency, or negotiation leverage. Multi-cloud real-time and analytics architectures face real compatibility friction: different APIs, data formats, and configuration models across providers hinder integration, while containerization with Docker and orchestration with Kubernetes let workloads interconnect regardless of the underlying cloud environment [9]. For genomics, the practical translation is to keep pipelines in containers, keep workflow definitions provider-neutral (WDL and Nextflow both run on multiple backends), and accept that the managed-service layer (the part that differs most between clouds) is where portability is lost.

Common Mistakes and Limitations

These are the errors that show up repeatedly when groups choose between batch and real-time genomics processing.

• Confusing latency with compute speed. Teams adopt a streaming engine believing it will make variant calling faster. It will not. The aligner runs at the same speed either way. Streaming changes when work starts, not how fast it runs.
• Streaming everything to look modern. Stream processing adds failure modes, consistency challenges, and operational complexity that batch does not have. The ZeroETL and streaming-warehouse literature documents these integration costs explicitly [7]. A weekly cohort job run through a streaming engine is strictly worse than the same job run through a batch scheduler.
• Ignoring the idle cost of always-on capacity. Streaming clusters and provisioned consumers bill around the clock. Cost analyses of serverless streaming show idle time is a first-class component of total spend, not an afterthought [8]. Measure it or it will surprise you at invoice time.
• Forgetting energy and sustainability trade-offs. Provisioning decisions have energy consequences. Scheduler-level techniques such as DVFS demonstrate that the trade-off between performance requirements and energy consumption is real and manageable, and batch windows are where the flexibility to exploit it lives [4].
• Breaking reproducibility in the real-time path. Batch jobs pin their software and references. Streaming logic evolves continuously, so results from March may not be reproducible from June unless you version the processing logic and replay capability deliberately.
• Confusing AWS Lambda with the Lambda architecture. One is a serverless function service. The other is a hybrid batch-plus-speed-layer architecture pattern. Design documents that use both terms without distinction cause genuine confusion in reviews.
• Assuming multi-cloud is free. Cross-provider integration runs into API, format, and configuration incompatibilities that require deliberate standardization work [9]. Budget the engineering or stay on one provider.
• Storing results where query engines cannot reach them. If cohort results land in formats or locations that the interactive query layer cannot read without a copy step, the real-time analyst experience collapses back into batch waiting times. Keep the lake and the query layer aligned from day one.
• Applying the healthcare or farming architecture wholesale. The Lambda architecture papers in adjacent domains are useful templates, but genomics file sizes, tooling, and regulatory context differ. Adapt the pattern, do not import the stack,.

A note on limitations. This article addresses infrastructure and processing architecture, not the interpretation of any individual genomic result. Clinical interpretation belongs in an accredited laboratory setting under qualified professionals, and research pipelines do not substitute for that.

Frequently Asked Questions

What is the difference between batch and real-time genomics processing?

Batch processing collects data and runs jobs on a schedule, optimizing for throughput and cost, while real-time processing starts work the moment data arrives, optimizing for latency. The same aligner and caller run at the same speed in both modes. The difference lies in when work begins and what the system guarantees about waiting time.

When should I use batch processing for my genomic pipeline?

Use batch when results can tolerate hours or days of turnaround, when data arrives in groups, when the analysis is defined over a cohort, or when reproducibility and cost per sample dominate. Cohort reanalysis, joint genotyping, and bulk transcriptomics are all naturally batch workloads. Choose real-time only when a decision depends on results within minutes.

Can I run both batch and real-time pipelines on the same infrastructure?

Yes, and most production groups do, using a hybrid pattern. Instruments and triggers feed the real-time path, the triggered per-sample work runs as idempotent batch jobs, and scheduled batch passes handle cohort aggregation. The Lambda architecture formalizes this dual-path design, with a batch layer for accuracy and a speed layer for freshness.

Does real-time processing cost more than batch processing?

Generally yes per unit of work, because real-time systems hold capacity open continuously and batch systems can scale to zero and use discounted spare capacity. Cost analyses of cloud-native streaming show total spend depends strongly on the ratio of streaming time to idle time and on service selection across providers. Event-driven batch often delivers the latency you actually need at batch economics.

How do the Lambda and Kappa architectures apply to genomics?

Lambda architecture runs a batch layer for complete, reproducible views and a speed layer for live results, which maps cleanly onto genomics batch pipelines plus instrument streaming. Kappa collapses everything into stream processing, which fits genomics poorly because most genomic analysis is file-shaped and batch-native. Newer ZeroETL approaches try to remove the boundary between the layers to reduce complexity and consistency problems.

Which AWS services are commonly used for genomics workflows?

S3 for storage, AWS Batch with EC2 (often spot capacity) for batch compute, Nextflow or Cromwell as the workflow layer, S3 event notifications with Step Functions for event-driven triggers, and HealthOmics for managed omics storage and analysis. Query engines running SQL directly over the data lake serve the interactive path. Containerized pipelines keep the workflow portable if you later add another cloud.

Sources

1. From Batch Processing to Real Time Analytics: Running Presto® at Scale
2. OmicStudio: A composable bioinformatics cloud platform with real-time feedback that can generate high-quality graphs for publication
3. An IoT-Cloud Based Solution for Real-Time and Batch Processing of Big Data: Application in Healthcare
4. A Holistic Energy-Efficient Real-Time Scheduler for Mixed Stream and Batch Processing Workloads
5. A Comprehensive Approach to Real-Time and Batch Processing for Energy-Efficient IoT Homes: Leveraging Lambda Architecture and Data Lakes
6. A Scalable Smart Farming Big Data Platform for Real-Time and Batch Processing Based on Lambda Architecture
7. Streaming View: An Efficient Data Processing Engine for Modern Real-time Data Warehouse of Alibaba Cloud
8. FinOps in Cloud-Native Near Real-Time Serverless Streaming Solutions
9. Optimizing hybrid and multi-cloud architectures for real-time data streaming and analytics: Strategies for scalability and integration