# Nextflow RNA-seq on AWS Batch: Step-by-Step Tutorial By the end of this tutorial you will be able to run a production-grade bulk RNA-seq analysis, based on the nf-core/rnaseq pipeline, on AWS Batch. You will [install Nextflow](/knowledge/bioinformatics/install-nextflow), stage FASTQ files in S3, build a Batch compute environment and job queue, launch the pipeline in the cloud, monitor it, collect MultiQC reports and count matrices from S3, and re-run failed steps without repeating completed work. You will also learn the configuration options that matter, the errors you are most likely to see, and how Nextflow compares with Snakemake and other workflow managers when you are choosing a system for your lab. Here is what you need on hand before you start: - An AWS account with permission to create S3 buckets, IAM roles, and Batch resources. - A local machine (Linux, macOS, or Windows with WSL2) with Java 11 or later installed. Check the Nextflow download page for the exact Java version required by the release you plan to use. - The AWS CLI, version 2, installed and configured with credentials. - Roughly 10 to 20 GB of free disk space for the local dry run and caching. - A set of paired-end RNA-seq FASTQ files, either your own data or public data you download from a repository such as SRA or ENA. If you have never used a bioinformatics workflow manager before, the idea is simple. Instead of running each command by hand, you write (or adopt) a workflow that declares every step, its inputs, and its container. The manager runs the steps, parallelizes them across samples, tracks what succeeded, and lets you rerun only what failed. Nextflow, Snakemake, and the R-based systemPipeR environment all follow this general pattern. systemPipeR, for example, provides a uniform workflow interface across NGS applications with automated report generation and support for running both R and command-line software on local machines and clusters [1]. Nextflow's distinguishing feature for this tutorial is that it speaks natively to AWS Batch, so the same pipeline you test on a laptop runs unchanged on hundreds of cloud CPUs. ## Why Nextflow for RNA-seq RNA-seq analysis is a chain of many independent per-sample jobs followed by a few aggregation steps. Quality control with FastQC, adapter trimming, genome alignment, quantification, gene-level counting, and QC report generation are all embarrassingly parallel across samples. A laptop handles three samples. A 96-sample experiment needs something that can fan out. Nextflow solves this by wrapping each step in a process with a declared container, then delegating execution to an executor. The `awsbatch` executor submits each process as an AWS Batch job, Batch schedules it onto EC2 instances, and Nextflow tracks state from your local machine (or a small always-on head node). The work directory lives on S3 so every job, wherever it lands, can read its inputs and write its outputs. The Nextflow ecosystem for [transcriptomics](/knowledge/bioinformatics/modern-transcriptomics-bulk-single-cell-spatial) is mature. CircRNAFlow, for instance, is a free, open-source Nextflow pipeline that takes paired-end FASTQ files as input and produces tables and graphical reports of identified circular RNAs, integrating tools such as FastQC, Flexbar, Bowtie2, STAR, and DCC [2]. nf-core/rnaseq, the pipeline used in this tutorial, follows the same architectural pattern for bulk [gene expression](/blog/guides/gene-expression) analysis. The reason this matters is reproducibility and comparability. Reanalysis studies repeatedly show that inconsistent use of reference genomes, annotation versions, and tool versions complicates cross-study comparison, and that standardized, reproducible workflows are the fix [3]. A Nextflow pipeline pinned to specific containers and reference files gives you that standardization for free, which is one reason RNA-seq has become robust enough for routine use in settings with real turnaround requirements, including clinical profiling [4]. ## Step 1: Install and Verify the Local Toolchain Nextflow itself is a single self-updating executable. Install it with: ```bash curl -s https://get.nextflow.io | bash chmod +x nextflow sudo mv nextflow /usr/local/bin/ ``` Then verify the installation and your Java runtime: ```bash nextflow -version ``` Expected output looks like this (version numbers will differ): ``` N E X T F L O W version 24.10.2 build 5948 created 20-12-2024 14:41 UTC cite: Di Tommaso et al, 2017, Nature Biotechnology https://doi.org/10.1038/nbt.3820 ``` Install and authenticate the AWS CLI: ```bash aws --version aws configure ``` `aws configure` prompts for an access key, secret key, default region, and output format. Set the region to the one where you will build your Batch resources, for example `us-east-1`. Mixing regions is one of the most common silent failures in cloud workflows, so pick one region and stay with it for this tutorial. If your organization uses AWS SSO, run `aws sso login` instead and confirm that `aws sts get-caller-identity` returns your expected account. You do not need Docker installed locally for the cloud run, but you do need it for the local dry run in Step 4. Install Docker Desktop or the Docker Engine, then verify: ```bash docker run hello-world ``` ## Step 2: Prepare S3 and Your Sample Sheet Create a bucket and a simple folder layout. Pick a globally unique name and use the same region as everything else: ```bash aws s3 mb s3://sc-rnaseq-demo-2026 --region us-east-1 aws s3 mkdir 2>/dev/null aws s3 cp ./fastq/ s3://sc-rnaseq-demo-2026/data/ --recursive --exclude "*" --include "*.fastq.gz" ``` The second command in that block is unnecessary noise, so the clean version is just: ```bash aws s3 mb s3://sc-rnaseq-demo-2026 --region us-east-1 aws s3 cp ./fastq/ s3://sc-rnaseq-demo-2026/data/ --recursive aws s3 ls s3://sc-rnaseq-demo-2026/data/ ``` Expected output of the listing is one line per file: ``` 2026-11-10 14:02:11 124839102 CTRL1_R1.fastq.gz 2026-11-10 14:02:29 126443871 CTRL1_R2.fastq.gz 2026-11-10 14:02:47 121990254 TREAT1_R1.fastq.gz 2026-11-10 14:02:58 123771600 TREAT1_R2.fastq.gz ``` nf-core/rnaseq takes a CSV sample sheet. The current schema has four columns: a sample identifier, the read 1 FASTQ, the read 2 FASTQ, and library strandedness. S3 URIs are accepted directly, which is exactly what you want for a Batch run. Create `samplesheet.csv`: ```csv sample,fastq_1,fastq_2,strandedness CTRL_REP1,s3://sc-rnaseq-demo-2026/data/CTRL1_R1.fastq.gz,s3://sc-rnaseq-demo-2026/data/CTRL1_R2.fastq.gz,auto CTRL_REP2,s3://sc-rnaseq-demo-2026/data/CTRL2_R1.fastq.gz,s3://sc-rnaseq-demo-2026/data/CTRL2_R2.fastq.gz,auto TREAT_REP1,s3://sc-rnaseq-demo-2026/data/TREAT1_R1.fastq.gz,s3://sc-rnaseq-demo-2026/data/TREAT1_R2.fastq.gz,auto TREAT_REP2,s3://sc-rnaseq-demo-2026/data/TREAT2_R1.fastq.gz,s3://sc-rnaseq-demo-2026/data/TREAT2_R2.fastq.gz,auto ``` Set `strandedness` to `auto` if you are unsure of your library type. The pipeline includes a strandedness check step that infers it from a small subset of reads and reports the result in MultiQC, which is far better than guessing. Upload the sample sheet itself to S3 as well, because the head process needs to fetch it: ```bash aws s3 cp samplesheet.csv s3://sc-rnaseq-demo-2026/samplesheet.csv ``` One design note before moving on. The pipeline you are using does QC, trimming, alignment with STAR, quantification with Salmon and RSEM, gene-level counting with featureCounts, and differential expression. If your data are small RNAs rather than messenger RNA, this is the wrong pipeline. Small RNA analysis has its own requirements, including isomiR handling and careful annotation of incompletely characterized species, which is why dedicated workflows exist for that problem [5]. Match the pipeline to the library type before you spend money on compute. ## Step 3: Create the AWS Batch Compute Environment and Job Queue ![Four step flow: IAM role, compute environment, job queue and Nextflow config built in order before launch.](/article-images/illustrations/nextflow-rna-seq-aws-batch-tutorial-aws-setup-order.webp) AWS Batch has three pieces you need to understand before clicking through the console: 1. A **compute environment** defines the EC2 instances Batch may launch, including instance families, vCPU limits, whether to use Spot instances, and the IAM role those instances assume. 2. A **job queue** sits in front of the compute environment and holds jobs in priority order. 3. A **job definition** is the template for a single job, including its container image. Nextflow registers job definitions automatically for each pipeline process, so you do not create these by hand. You can build all of this through the console, with an infrastructure-as-code tool, or with the guided setup that nf-core and Seqera publish for AWS. The console route, condensed to the decisions that matter, is: 1. In IAM, create an instance role for the Batch EC2 instances. Attach a policy granting S3 access to your bucket and the managed `AmazonSSMManagedInstanceCore` policy. This is the role your jobs use to read FASTQs and write results. 2. In IAM, confirm your user or execution context has Batch permissions to submit, describe, cancel, and tag jobs, plus permission to register ECS task definitions and to pass the roles above. The Nextflow documentation lists the exact policy actions required. 3. In the Batch console, create a compute environment. Choose EC2, managed, and either on-demand or Spot for the allocation strategy. For a first run, pick on-demand so Spot interruptions do not confuse your debugging. Set the maximum vCPUs to your budget. A ceiling of 256 vCPUs handles a 12-sample mammalian RNA-seq run comfortably. 4. Select instance types. A mix of `m5` and `r5` families is a sensible default for RNA-seq because alignment and quantification steps benefit from both compute and memory. 5. Create a job queue with priority 1 and attach the compute environment. 6. Note the exact queue name. You will put it in your Nextflow configuration. The container images for pipeline processes are pulled from public registries such as Docker Hub and quay.io by default. If your data are sensitive or your network policy restricts egress, copy the required images into Amazon ECR and point the pipeline at your copies. ## Step 4: Dry Run the Pipeline Locally Never launch a cloud run on a pipeline you have not executed. nf-core pipelines ship a `test` profile with tiny reference data that runs in minutes on a laptop. This costs nothing on AWS and validates your Nextflow installation, your Docker setup, and the pipeline version: ```bash nextflow run nf-core/rnaseq -profile test,docker --outdir results_test ``` Expected console output, condensed: ``` N E X T F L O W ~ version 24.10.2 Launching `nf-core/rnaseq` [distracted_curie] DSL2 - revision: 3.14.0 ------------------------------------------------------ Looking forward! Updating to v3.14.0 Pulling docker images ... executor > local (186) [3a/8b21cf] process > NFCORE_RNASEQ:RNASEQ:... [100%] Completed at: 10-Nov-2026 14:22:04 Pipeline Status: SUCCESS ``` The numbers differ between versions and machines. What matters is that the final status is `SUCCESS` and the local `results_test` directory contains a `multiqc` folder with an HTML report you can open in a browser. If this step fails, fix your local environment before touching AWS, because every cloud error will otherwise be tangled with a local one. This dry-run habit is also where benchmark thinking starts. Community resources that provide curated datasets and code templates for benchmarking gene expression analysis steps make it easier to test pipeline variants against data with expected structure before committing to a production configuration [6]. ## Step 5: Configure Nextflow for AWS Batch Nextflow reads configuration from `nextflow.config` files and from files passed with `-c`. Create a file named `aws.config` in your project directory: ```groovy workDir = 's3://sc-rnaseq-demo-2026/work' aws.region = 'us-east-1' process { executor = 'awsbatch' queue = 'my-rnaseq-job-queue' errorStrategy = { task.attempt <= 2 ? 'retry' : 'terminate' } maxRetries = 2 withLabel:process_high_memory { cpus = 16 memory = 120.GB } } ``` Each parameter here does a specific job: - `workDir` points at S3 rather than a local path. With the Batch executor this is mandatory, because the jobs run on EC2 instances that have no access to your laptop's disk. Nextflow stages inputs from S3, runs the process, and copies outputs back to S3 between every step. - `aws.region` must match the region of your bucket, queue, and compute environment. - `process.executor = 'awsbatch'` switches execution from local to Batch. - `process.queue` names the job queue from Step 3. You can override the queue for specific processes with a `withLabel` block, which is how you send memory-hungry alignment jobs to high-memory instances and lightweight QC jobs to cheap ones. - `errorStrategy` with `task.attempt` gives you automatic retries. Transient failures, especially Spot reclaims and registry rate limits, are common enough on cloud runs that unconditional termination on first failure wastes money and time. - Memory values use Nextflow unit syntax, so `120.GB` means 120 gigabytes. Batch maps the requested vCPU and memory to an instance type that satisfies both. Two optional blocks are worth knowing. If you install Fusion, a file system layer that streams data between S3 and jobs without staging copies, you add: ```groovy fusion.enabled = true fusion.s3Endpoint = 'https://s3.us-east-1.amazonaws.com' ``` Fusion removes most of the per-job S3 copy overhead and typically shortens wall time on many-sample runs. It is a Seqera product, available in the community edition, and the nf-core AWS guidance recommends it for production runs. Second, older Nextflow versions required `aws.batch.cliPath` and related `aws.batch` settings, and many tutorials on the internet still show them. Current Nextflow releases have deprecated most of those keys. If you copy a config from an old blog post and see warnings about unknown or deprecated attributes, delete those lines and rely on the modern keys shown above. ## Step 6: Launch the RNA-seq Run With the dry run passing and the config in place, launch the real analysis: ```bash nextflow run nf-core/rnaseq \ -profile docker \ -c aws.config \ --input s3://sc-rnaseq-demo-2026/samplesheet.csv \ --outdir s3://sc-rnaseq-demo-2026/results \ --genome GRCh38 \ -with-report -with-trace -with-timeline -with-dag ``` Parameter by parameter: - `-input` points at the sample sheet in S3. Note that pipeline-level parameters use a double hyphen, which distinguishes them from Nextflow's own single-hyphen flags like `-profile` and `-c`. - `-outdir` puts all published results in S3. Publishing directly to S3 avoids a second sync step later. - `-genome GRCh38` selects a bundled iGenomes reference, so you do not have to supply FASTA and GTF files yourself. For a custom organism or a specific annotation release, use `-fasta` and `-gtf` instead. Pinning the exact reference build and annotation version is not cosmetic. A recent reanalysis of fungal pathogen RNA-seq studies showed near-perfect correlation with published results even across annotation version differences, but only because the team standardized the workflow and tracked provenance explicitly [3]. Record your reference choice in your project notes the moment you launch. - `-with-report -with-trace -with-timeline -with-dag` produce four files documenting resource usage, task status, timing, and the DAG. The report and timeline files are what you consult when you later tune CPU and memory requests. On startup, Nextflow prints its banner, pulls the pipeline code from GitHub, and begins submitting Batch jobs. Each line like `[3a/8b21cf] process > NFCORE_RNASEQ:RNASEQ:FASTQC ...` represents tasks running remotely. The two-character prefix in brackets is the hash of the task work directory on S3, which is how you locate a job's files when debugging. For runs longer than a few hours or runs you want to watch from anywhere, connect the pipeline to Seqera Platform with `-with-tower`. It gives you a web view of every task, live logs, and cost-oriented summaries, and it keeps the run alive even if your laptop disconnects. ## Step 7: Monitor the Run and Read the Logs ![Three question decision path checking queue and region, then memory and exit status 137, then retries for Spot reclaims.](/article-images/illustrations/nextflow-rna-seq-aws-batch-tutorial-failed-task-triage.webp) Three layers of monitoring exist, and you should know all of them: 1. **Nextflow's own view.** Run `nextflow log` in the launch directory to see every task with its status, container, and duration. `nextflow log -f status FAILED` filters to failures. The `.nextflow.log` file contains the full debug output, including the exact Batch job IDs. 2. **AWS Batch and CloudWatch.** In the Batch console, open the job queue to see pending and running jobs. Click a failed job and follow the link to CloudWatch Logs to read the container's stdout and stderr. This is where you see the actual tool error, for example a STAR failure from a malformed GTF. 3. **The reports from Step 6.** The trace file, typically named `trace-.txt`, lists every task with peak CPU and memory. Use it to shrink over-provisioned processes and to find the processes that drive cost. A healthy run shows the per-sample processes (FastQC, trimming, alignment, quantification) fanning out as dozens of parallel Batch jobs, then collapsing into sequential aggregation steps (MultiQC, the differential expression step). If the run sits with zero jobs for more than a few minutes at the start, the queue name or region in your config is wrong, and the head log will say so. ## Step 8: Collect and Check the Results When the run finishes, Nextflow prints a completion summary. Pull the published results to your machine for inspection: ```bash aws s3 sync s3://sc-rnaseq-demo-2026/results ./results ls ./results ``` Expected top-level directories include `multiqc`, `pipeline_info`, `star_salmon`, and `fastqc`. Your sanity checklist should be: - **Sample completeness.** The count matrix under `star_salmon` should have one column per sample in the sample sheet, no more and no fewer. - **QC report.** Open `multiqc/multiqc_report.html`. Check per-sample duplication rates, adapter content, and alignment rates. An alignment rate far below your tissue's expectation usually indicates wrong reference, wrong strandedness, or contaminated libraries. - **Strandedness inference.** The pipeline's strandedness check output appears in the MultiQC report. If it contradicts what you declared, correct the sample sheet and rerun. - **Provenance files.** The `pipeline_info` directory contains the software versions report and the execution reports. Archive these with your results. Any future reanalysis, whether by a collaborator or by an automated agent, depends on knowing exactly which reference, annotation, and tool versions produced the counts [3]. The counts matrix is the handoff point to downstream statistics. How you model the data matters. A systematic TCGA-based analysis of platinum-sensitivity biomarkers in ovarian cancer compared DESeq2, edgeR, limma-voom, and the Wilcoxon rank-sum test on the same cohorts and found only partial overlap in the resulting differentially expressed gene sets, with patient selection and pipeline choice both influencing outcomes [7]. The practical lesson for your pipeline run is to fix the counting method, the annotation, and the statistical model before you start comparing groups, and to treat any single pipeline's gene list as one estimate rather than ground truth. ## Step 9: Resume, Retry, and Clean Up The single most valuable Nextflow feature on a paid cloud platform is caching. If a 12-hour run fails at step 90 of 100, you do not repeat the first 89 steps. Fix the cause, then rerun with `-resume` and the identical command: ```bash nextflow run nf-core/rnaseq \ -profile docker \ -c aws.config \ --input s3://sc-rnaseq-demo-2026/samplesheet.csv \ --outdir s3://sc-rnaseq-demo-2026/results \ --genome GRCh38 \ -resume ``` Nextflow hashes every task's inputs, container, script, and resources. Tasks whose hashes match cached outputs on S3 are skipped instantly, and the console marks them as cached rather than re-executed. Two rules keep this working. First, change nothing except what you intend to change. Editing a script line or a container tag invalidates every downstream task. Second, do not delete the S3 work directory until the project is truly finished, because the cache lives there. For cleanup, the work directory is the expensive part. A full run stages intermediates for every task, and a mammalian RNA-seq run can leave hundreds of gigabytes of BAM files and per-sample intermediate directories in S3. Once you have the published results and the provenance files, delete `s3://sc-rnaseq-demo-2026/work/` deliberately: ```bash aws s3 rm s3://sc-rnaseq-demo-2026/work/ --recursive ``` Add a lifecycle rule to the bucket so unfinished work directories age out automatically. Storage is cheap per gigabyte but not free at the scale transcriptomics projects accumulate. ## Key Options Reference | Option or setting | Where it goes | What it controls | Practical default | |--|--|--|--| | `workDir = 's3://bucket/work'` | `nextflow.config` | Location of task scratch directories | S3 prefix dedicated to one project | | `aws.region` | `nextflow.config` | Region for S3 and Batch API calls | Same region as bucket and queue | | `process.executor = 'awsbatch'` | `nextflow.config` | Uses AWS Batch instead of local execution | Required for Batch runs | | `process.queue` | `nextflow.config` | Batch job queue that receives jobs | One queue per project or priority tier | | `errorStrategy = 'retry'` | `nextflow.config` | Automatic resubmission of failed tasks | Retry up to twice | | `cpus`, `memory` per label | `nextflow.config` | Resource requests mapped onto Batch jobs | Start from the pipeline's defaults, tune with the trace file | | `-resume` | command line | Reuse cached tasks from a previous run | Always, on every rerun | | `-profile test,docker` | command line | Tiny built-in test dataset with Docker | Run before any cloud launch | | `-input`, `-outdir`, `-genome` | command line | Pipeline inputs, output location, reference | S3 URIs for cloud runs | | `-with-report`, `-with-trace` | command line | Resource usage records for tuning | Enable on every real run | ## Practical Tips for AWS Batch Runs **Tip 1: Test locally, always.** The `test` profile run costs minutes of laptop time and catches 80 percent of configuration errors. Every serious nf-core and Nextflow guide repeats this advice because it works. **Tip 2: One region, everywhere.** Bucket, queue, compute environment, and `aws.region` must agree. Cross-region reads fail or crawl, and the error messages rarely name the mismatch directly. **Tip 3: Use labels to tier your resources.** RNA-seq has a natural resource hierarchy. QC and trimming are light, alignment and quantification are heavy, MultiQC is trivial. Use `withLabel` blocks so the heavy steps get 16 or more vCPUs and the light steps do not hold large instances hostage. The `-with-trace` output from a real run tells you which labels deserve adjustment. The nf-core defaults are a reasonable starting point, but your organism and read lengths shift the balance. **Tip 4: Budget for the work directory, then delete it.** The published results are usually a small fraction of the intermediate data. Set a calendar reminder to clean S3 work directories after project sign-off. **Tip 5: Keep provenance with the data.** Archive the trace, report, timeline, and software versions files alongside results in S3. This is the cheapest possible reproducibility insurance, and it is the same principle that platforms designed for standardized pathogen genomics encode into their architecture, forcing analysis through tracked, versioned tools rather than ad hoc steps [3]. **Tip 6: Use retries deliberately.** Transient cloud failures are normal. Configure `errorStrategy` and `maxRetries` as shown in Step 5 so a single registry hiccup does not kill an overnight run. ## Common Mistakes and Limitations **Mistake 1: Leaving `workDir` on a local path.** With the Batch executor, jobs cannot see your laptop. The error surfaces as tasks failing immediately with file-not-found or permission errors inside the container. Fix: set `workDir` to an S3 URI before launching. **Mistake 2: Launching from an untested pipeline version.** Running `nextflow run nf-core/rnaseq` without a revision tag pulls the latest pipeline release, which may have changed parameters since you read the documentation. Pin a version with `-r 3.14.0` (or whatever release you validated in Step 4) for every production run. **Mistake 3: Resource requests that exceed instance limits.** If a process requests more memory than your largest configured instance type offers, Batch jobs fail or pend forever. Exit status 137 on a task usually means the container was killed for exceeding memory. Fix: check the trace file, compare requested versus peak usage, and either lower the request or add larger instances to the compute environment. **Mistake 4: Treating Spot interruptions as bugs.** On Spot capacity, jobs can be reclaimed mid-run. Without retries, those tasks fail permanently. With `errorStrategy = 'retry'`, Nextflow resubmits them. Spot saves substantially on long runs, but only with retries configured. **Mistake 5: Forgetting that pipeline choice changes results.** The same reads processed by different alignment, counting, and statistical combinations produce different gene lists, sometimes with only partial overlap between methods [7]. This is not an AWS problem, but running your analysis in the cloud makes it easy to fire off many pipeline variants without thinking. Decide your analysis plan first. **Mistake 6: Ignoring reference and annotation consistency.** Cross-study comparisons break when studies use different genome builds or annotation releases, and this problem recurs in RNA-seq reanalyses across organisms and contexts [3]. Pin your reference in version control with the pipeline revision, and record both in the archived provenance files. **Mistake 7: Deleting the S3 work directory before the project closes.** The cache that powers `-resume` lives there. Deleting it means the next rerun repeats everything at full price. **Limitation: this tutorial covers bulk RNA-seq.** The workflow patterns transfer to other assays, but the pipelines differ. Single-cell RNA-seq, for example, has its own family of protocols and dedicated analysis pipelines, from droplet-based methods that use dissolvable hydrogel beads for cell barcoding [8] to benchtop long-read platforms that recover full-length transcripts per cell with dedicated barcode-recovery and quantification software [9]. Full-length single-cell protocols such as FLASH-seq also carry their own [preprocessing](/blog/data-analysis/what-is-data-preprocessing) requirements inherited from the Smart-seq lineage [10]. If your experiment is single-cell, start from a single-cell pipeline rather than forcing bulk tooling onto the data. Similarly, specialized analyses such as circular RNA detection have purpose-built Nextflow pipelines that take the same paired-end FASTQ input but apply chimeric-read detection rather than standard gene counting [2]. **Limitation: cloud costs need governance.** Nothing in the Batch console stops a misconfigured loop from submitting thousands of jobs. Set the compute environment's maximum vCPUs to a number you are willing to pay for, and review the AWS cost dashboard during your first runs. ## Nextflow vs Snakemake: Choosing a Bioinformatics Workflow Manager Both Nextflow and Snakemake are excellent, widely adopted systems, and both can run the RNA-seq analysis described here. The honest answer to "which should I choose" depends on where your compute lives and who maintains the pipeline. Choose Nextflow when cloud execution is central. Nextflow has first-class support for AWS Batch, Google Batch, and Kubernetes, along with container-native design and built-in caching and resumption. The nf-core collection, which includes the RNA-seq pipeline used in this tutorial, is the largest curated ecosystem of community-maintained Nextflow pipelines, and new domain pipelines continue to appear in it, from small RNA analysis [5] to circRNA detection [2] to single-nuclei workflows built as reusable templates [11]. If your goal is to run established, peer-reviewed pipelines with minimal custom code, Nextflow plus nf-core is currently the lowest-friction path, and this tutorial's entire AWS Batch workflow depends on that integration. Choose Snakemake when your analysis is Python-heavy, your rules map naturally onto file targets, and your group already has Snakemake expertise. Snakemake also supports containers and cluster execution, including cloud backends, and its rule-based syntax is elegant for reproducible research compendia. Consider systemPipeR when your team lives in R. It provides workflow construction and automated report generation inside the R and Bioconductor environment, supports both R and command-line tools, and runs on local machines or clusters [1]. It is the natural choice when the analysis products are R Markdown reports and the team maintains Bioconductor expertise rather than wanting a separate Groovy or Python layer. The comparison that matters in practice is not syntax but ecosystem gravity. A bioinformatics workflow manager earns its keep through the pipelines, configs, and institutional knowledge that surround it. If the pipeline you need already exists as a well-tested nf-core workflow with AWS profiles, adopting Nextflow costs you less than porting that pipeline into another system. If you are writing a bespoke workflow from scratch for a small analysis, the differences shrink, and your team's existing expertise should decide. ## Frequently Asked Questions ### Do I need to modify the nf-core/rnaseq pipeline code to run it on AWS Batch? No. You run the standard pipeline with a configuration file that sets the Batch executor, queue, region, and S3 work directory. Pipeline logic and execution infrastructure stay separate by design, which is the core value of the workflow manager layer. ### How much does a Nextflow RNA-seq run on AWS Batch cost? Cost scales with the number of samples, read depth, and reference genome size, plus S3 storage for intermediates and results. No single figure fits all projects, so run the test profile first, then use the `-with-trace` report from a small pilot to extrapolate before committing to a full run. ### Can I use Singularity or Apptainer instead of Docker on AWS Batch? AWS Batch jobs run Docker containers by default, and nf-core pipelines provide profiles for multiple container engines. On Batch, Docker is the straightforward path. Apptainer matters more on HPC systems where Docker daemons are not permitted. ### What does exit status 137 mean in my failed Batch jobs? It usually means the container was killed for exceeding its memory allocation. Open the CloudWatch logs for the job, confirm the out-of-memory pattern, and either raise the process memory request or add higher-memory instance types to the compute environment. ### How do I rerun only the failed steps without paying for everything again? Use `-resume` with the identical command. Nextflow caches every completed task's outputs in the S3 work directory and skips any task whose inputs and code hash match, so only failed and downstream tasks execute. ### Do I need a special genome or can I use my own reference? The `-genome` flag selects a bundled iGenomes reference for common organisms. For other organisms or specific annotation releases, supply `-fasta` and `-gtf` directly, and record the exact file versions in your provenance archive. ### Nextflow vs Snakemake, which one should my lab adopt? Choose Nextflow if cloud execution and community pipelines like nf-core are your priority. Choose Snakemake if your analyses are Python-centric and your team already knows it. Both are sound choices, and ecosystem support usually outweighs syntax differences. ### Can Nextflow pipelines handle data types beyond bulk RNA-seq? Yes. The Nextflow ecosystem includes pipelines for small RNA analysis, circRNA detection, single-cell assays, and many other NGS applications, and the same executor and configuration patterns shown in this tutorial apply to them. ## Sources 1. [systemPipeR: NGS workflow and report generation environment.](https://pubmed.ncbi.nlm.nih.gov/27650223/) 2. [Circular RNA Identification and Characterization with CircRNAFlow: A Bioinformatics Approach.](https://pubmed.ncbi.nlm.nih.gov/40886271/) 3. [Standardizing RNA-seq Analysis of Fungal Pathogens Using BRC-Analytics and Agentic AI: A Candidozyma auris Case Study.](https://pubmed.ncbi.nlm.nih.gov/41509213/) 4. [The actionable transcriptome: a framework for incorporating RNA sequencing into precision oncology.](https://pubmed.ncbi.nlm.nih.gov/41492079/) 5. [Small RNA-seq analysis of single porcine blastocysts revealed that maternal estradiol-17beta exposure does not affect miRNA isoform (isomiR) expression.](https://pubmed.ncbi.nlm.nih.gov/30081835/) 6. [Dex-Benchmark: datasets and code to evaluate algorithms for transcriptomics data analysis.](https://pubmed.ncbi.nlm.nih.gov/37953774/) 7. [Assessing the Impact of Patient Selection and Analytical Methods on mRNA Biomarker Identification in Ovarian Cancer Using TCGA Data.](https://pubmed.ncbi.nlm.nih.gov/42555814/) 8. [Hydrop enables droplet-based single-cell ATAC-seq and single-cell RNA-seq using dissolvable hydrogel beads.](https://pubmed.ncbi.nlm.nih.gov/35195064/) 9. [BenchDrop-seq: a microfluidics-free platform for benchtop single-cell long-read RNA sequencing.](https://pubmed.ncbi.nlm.nih.gov/41959170/) 10. [Full-Length Single-Cell RNA-Sequencing with FLASH-seq.](https://pubmed.ncbi.nlm.nih.gov/36495447/) 11. [Multidimensional Single-Nuclei RNA-Seq Reconstruction of Adipose Tissue Reveals Adipocyte Plasticity Underlying Thermogenic Response.](https://pubmed.ncbi.nlm.nih.gov/34831295/)