# Handling Job Dependencies and Retries in Snakemake on HPC By the end of this guide you will be able to run a Snakemake pipeline on a SLURM-based HPC cluster with automatic job retries, recover cleanly from node failures and preemptions, rerun only the jobs that actually failed, and verify that the finished workflow is complete and correct. You will also understand exactly how Snakemake resolves dependencies between jobs, why it does not use `sbatch, dependency` chains, and what to do instead. Before you start, have the following on hand: - A Snakemake installation of version 7 or newer. Check with `snakemake, version`. On most clusters you load it with `module load snakemake` or run it inside a Conda environment (`conda create -n snakemake -c conda-forge snakemake bioconda::snakemake`). - A cluster account with a scheduler. This guide uses SLURM commands (`sbatch`, `squeue`, `sacct`, `scancel`) because it is the most common scheduler at universities and research institutes in the United States. The concepts transfer to other schedulers. - A working pipeline directory with a `Snakefile`, or the example pipeline included in this article. - Familiarity with basic shell use. You do not need prior experience with cluster job arrays. Large scientific workflows routinely run for hours or days on parallel machines, and a [workflow management](/knowledge/bioinformatics/snakemake-vs-nextflow-workflow-architectures) system (WMS) is the software layer that controls this parallel execution, schedules tasks, and manages the data they consume [1]. Snakemake is one of the most widely used bioinformatics workflow managers in that category. Getting dependencies and retries right is the difference between a pipeline that finishes overnight unattended and one that wakes you up at 2 a.m. because one alignment job was killed by the out-of-memory killer. ## How Snakemake Handles Dependencies on an HPC Cluster ### The DAG in plain terms When you run `snakemake target`, Snakemake builds a directed acyclic graph, or DAG. A DAG is a map of every job needed to produce your target files, where arrows point from inputs to outputs and no path loops back on itself. Each node is one instance of one rule, for example `align_reads` applied to `sampleA`. Snakemake derives the graph entirely from the `input:` and `output:` file declarations in your rules. You never write the order of jobs yourself. This is the single most important mental model for dependency handling on a cluster: **Snakemake, not SLURM, owns the dependency graph.** Snakemake submits a job to the scheduler only when all of its input files exist. When a job finishes successfully, Snakemake wakes up, checks which new outputs appeared, and submits the next batch of ready jobs. Scheduling at this scale, where many long-running tasks compete for cores and memory on large machines, is exactly the execution-control problem that workflow management systems are designed to solve, and efficient scheduling of these hybrid workloads is a recognized challenge in WMS design [1]. ### Why Snakemake does not use sbatch dependency chains A common first instinct is to chain jobs with SLURM's own dependencies, for example `sbatch, dependency=afterok:12345`. Snakemake deliberately does not do this. There are two reasons. First, the DAG is dynamic. A rule like `bwa index` produces one index per reference genome, and Snakemake only knows how many instances exist after the wildcards are resolved. SLURM dependency chains must be declared at submission time. Second, retries and reruns become trivial for Snakemake and painful for a static chain. If job 40 of a 200-job chain dies, a SLURM chain keeps the downstream jobs queued against a dead job ID. Snakemake simply resubmits job 40 when a retry slot opens, then evaluates downstream readiness again. Snakemake does offer `-immediate-submit`, which submits every job at once and pushes the dependency problem onto the scheduler through the submit command. This only makes sense when your submit script itself translates the `dependencies` file Snakemake provides into scheduler dependencies. For almost everyone, the default behavior, where Snakemake submits ready jobs as inputs appear, is the correct choice. If you did not specifically configure immediate submission, do not assume SLURM knows your dependencies. ### What the scheduler actually sees From SLURM's point of view, your pipeline looks like a stream of independent `sbatch` submissions arriving over time, each asking for the cores, memory, wall time, and partition you configured. The scheduler balances these against other users. Two consequences follow: - Each job must declare enough resources (memory, time) to survive, because a job killed by SLURM is a job Snakemake has to retry. - Snakemake itself runs as a long-lived process, usually on a head or compute node, and that process is a single point of coordination for the whole run. ## Step-by-Step: Enabling Retries and Dependency-Safe Execution on SLURM This walkthrough uses a small read-mapping pipeline as the worked example: quality control with FastQC, alignment with `bwa mem`, and coordinate sorting with `samtools sort`. Every step and command works the same way for your own pipeline. ### Step 1: Confirm the environment and Snakemake version ```bash module load snakemake snakemake --version which sbatch sinfo -s ``` Expected output looks like this (versions will differ): ``` 7.32.4 /usr/bin/sbatch CLUSTER: general* up 1-00:00:00 1 8 ``` If `snakemake, version` reports 6.x, the profile syntax in this guide will partly differ. Upgrade to 7.x or 8.x before continuing. If `which sbatch` returns nothing, you are not on a cluster with SLURM, so adapt the scheduler-specific parts. ### Step 2: Write the Snakefile with explicit inputs, outputs, threads, and memory ```python configfile: "config.yaml" SAMPLES = config["samples"] REF = config["reference"] rule all: input: expand("results/{sample}/sorted.bam", sample=SAMPLES) rule fastqc: input: "data/{sample}_1.fastq.gz" output: "qc/{sample}_fastqc.html" threads: 1 resources: mem_mb = 1000, runtime = "00:10:00" shell: "fastqc -o qc {input}" rule bwa_map: input: reads = "data/{sample}_1.fastq.gz", ref = REF output: "results/{sample}/mapped.bam" threads: 8 resources: mem_mb = 16000, runtime = "04:00:00" shell: "bwa mem -t {threads} -M {input.ref} {input.reads} > {output}" rule samtools_sort: input: "results/{sample}/mapped.bam" output: "results/{sample}/sorted.bam" threads: 2 resources: mem_mb = 4000, runtime = "01:00:00" shell: "samtools sort -@ {threads} -m 2G -T results/{wildcards.sample}/tmp -o {output} {input}" ``` Note two details that matter for retries. The sort rule passes `-T results/{wildcards.sample}/tmp` so temporary sort files live in a per-sample directory instead of the default shared location, which avoids collisions when a restarted job overlaps with a dead predecessor. Declaring `mem_mb` and `runtime` as resources lets the cluster profile translate them into SLURM directives, which is what stops the out-of-memory killer from turning your retries into a permanent failure loop. ### Step 3: Inspect the DAG before you submit anything ```bash mkdir -p qc results snakemake --dry-run snakemake --dag | dot -Tsvg > dag.svg ``` Expected dry-run output: ``` Building DAG of jobs... Job stats: job count --------------- ----- all 1 bwa_map 3 fastqc 3 samtools_sort 3 total 10 ``` Read this before every run. The counts tell you how many scheduler jobs the pipeline will create. If you expected 3 samples and you see 30 `bwa_map` jobs, stop here and fix the wildcard expansion, not after 30 jobs hit the queue. ### Step 4: Write the cluster profile Snakemake reads all cluster settings from a profile directory. Create `profiles/slurm/config.yaml` in your project root: ```yaml cluster: "sbatch --partition=general --cpus-per-task={threads} --mem={resources.mem_mb} --time={resources.runtime} --job-name={rule} --output=logs/slurm/{rule}-%j.out --error=logs/slurm/{rule}-%j.err" cluster-cancel: scancel jobs: 100 restart-times: 3 latency-wait: 120 max-status-checks-per-second: 1 max-jobs-per-second: 10 rerun-incomplete: true keep-incomplete: true use-conda: true ``` Each line does a specific job: - `cluster`: the submit command run for every job. `{threads}`, `{resources.mem_mb}`, and `{resources.runtime}` are filled in per rule, so your alignment job requests 8 CPUs and 16 GB while FastQC requests 1 CPU and 1 GB. - `cluster-cancel: scancel`: when you press Ctrl-C or a job fails permanently, Snakemake cancels its own running jobs instead of leaving orphans burning allocation. - `jobs: 100`: the maximum number of jobs in flight at once. Keep this within what your allocation and cluster etiquette allow. - `restart-times: 3`: the retry setting. A failed job is resubmitted up to 3 times before the pipeline stops with an error. - `latency-wait: 120`: seconds Snakemake waits for a job's output file to appear after the job reports completion. This is the single most valuable parameter on shared filesystems, for reasons covered in the failure section below. - `rerun-incomplete: true`: if a previous run was killed mid-job, Snakemake reschedules the incomplete jobs instead of refusing to start. - `keep-incomplete: true`: keeps partially produced outputs so you can inspect what a failed job actually wrote. On Snakemake 8 the plugin replaces the raw submit string. After `pip install snakemake-executor-plugin-slurm`, the equivalent profile uses: ```yaml executor: slurm jobs: 100 default-resources: slurm_partition: "general" mem_mb: 8000 ``` and rule-level `resources: mem_mb=...` still override the default per job. The plugin handles SLURM status polling internally, so `latency-wait` and status-check settings from the generic cluster workflow are largely replaced. ### Step 5: Run a pilot with a small job limit ```bash mkdir -p logs/slurm snakemake --profile profiles/slurm --jobs 2 ``` Expected live output: ``` Building DAG of jobs... Using shell: /bin/bash Job stats: job count --------------- ----- all 1 bwa_map 3 fastqc 3 samtools_sort 3 total 10 Select jobs to execute... [Mon Nov 10 14:02:11 2026] rule fastqc: input: data/sampleA_1.fastq.gz output: qc/sampleA_fastqc.html jobid: 1 resources: mem_mb=1000, runtime=00:10:00 Submitted cluster job 1 as slurm job 501101 ``` The pilot with `-jobs 2` confirms that the submit string renders correctly, that SLURM accepts the resource requests, and that outputs land where the DAG expects them. Common pilot failures and their fixes appear in the failure section below. Fix everything at this scale before you scale out. ### Step 6: Scale up and monitor ```bash snakemake --profile profiles/slurm --jobs 100 ``` While it runs, watch both sides of the system: ```bash squeue -u $USER -o "%.8i %.10P %.20j %.8T %.10M %.6D %R" tail -f logs/slurm/bwa_map-501104.err ``` To see the SLURM accounting record for any job, including its final state and exit code: ```bash sacct -j 501104 --format=JobID,State,ExitCode,Elapsed,MaxRSS ``` Expected output: ``` JobID State ExitCode Elapsed MaxRSS -------------- ---------- -------- ---------- ---------- 501104 COMPLETED 0:0 00:37:12 11.5G 501104.batch COMPLETED 0:0 00:37:12 11.5G ``` The `MaxRSS` column is your memory truth. If an alignment job peaks at 11.5 GB and you allocated 16 GB, you are fine. If it peaks at 15.9 GB, raise `mem_mb` for that rule, because the next larger sample will get killed. ### Step 7: Recover from failures and interrupted runs When a job fails and retries are exhausted, Snakemake stops and prints the rule name, job ID, log paths, and the SLURM exit code. To resubmit just the failed work after you fix the cause: ```bash snakemake --profile profiles/slurm --jobs 100 --rerun-incomplete ``` If the Snakemake master process itself died, for example the head node was rebooted or you lost your terminal session, the workflow directory holds a stale lock. First confirm no Snakemake process is alive, then remove the lock: ```bash ps aux | grep snakemake snakemake --unlock ``` Expected output: ``` Unlocking working directory ``` Then relaunch the pipeline. Snakemake compares file timestamps and metadata against the DAG and resumes from wherever it stopped, which is the payoff of the file-based dependency model. To see exactly what Snakemake believes changed and would rerun: ```bash snakemake --summary ``` Expected output: ``` output_file date rule version input-file update-failure ------------------------------------------------------------------------------------------------------------------ results/sampleA/sorted.bam Mon Nov 10 14:44 samtools_sort 7.32.4 results/sampleA/mapped.bam qc/sampleB_fastqc.html Mon Nov 10 12:02 fastqc 7.32.4 data/sampleB_1.fastq.gz ``` An empty `update-failure` column means every file is consistent. Entries there point at files whose metadata could not be verified, usually after a crash, and Snakemake will rebuild them. ### Step 8: Verify the finished run A clean finish is not the same as a correct finish. Three checks: ```bash snakemake --dry-run ``` Expected output when everything is complete: ``` Nothing to be done (all requested files are present and up to date). ``` Then spot-check integrity of the biological outputs themselves: ```bash samtools quickcheck results/*/sorted.bam && echo "all BAMs valid" fastqc -q qc/*.zip ``` Finally, compare per-job `MaxRSS` and `Elapsed` from `sacct` against the resources declared in your rules. Jobs that used a fraction of their allocation are candidates for downsizing, which shortens the queue wait for everyone and, in turn, reduces the number of retries your pipeline needs. ## Key Snakemake Options for Dependencies and Retries | Option | What it controls | Typical value on HPC | |--|--|--| | `-restart-times N` | Number of resubmissions after a job fails before the pipeline aborts | 3 | | `-latency-wait SEC` | Seconds to wait for output files to become visible on the shared filesystem after a job reports done | 120 | | `-jobs N` | Maximum jobs in flight at once | Match your allocation, often 50 to 200 | | `-rerun-incomplete` | Reschedule jobs marked incomplete from a previous killed run | Always set in the profile | | `-keep-incomplete` | Preserve partial outputs from failed jobs for debugging | On while developing | | `-max-status-checks-per-second N` | Rate limit on scheduler status polling | 1 | | `-max-jobs-per-second N` | Rate limit on job submission | 10, lower it if the scheduler complains | | `-cluster-cancel scancel` | Cancel Snakemake-owned SLURM jobs when the run stops | Recommended | | `-cores N` / `-resources name=N` | Cap local cores or named resources such as database connections | Match node limits | | `-rerun-triggers` | Which metadata changes force a rerun (Snakemake 7+) | `mtime,params,input` for stability | ## Reading Failure Messages and Exit Codes Correctly Most retry problems on HPC reduce to three failure classes. Diagnose the class before touching the profile. ### Exit code 137 and 143 ![Decision path that sorts a failed Snakemake job by exit code 137, exit code 143, or missing output files into the right fix.](/article-images/illustrations/snakemake-job-retries-hpc-exit-code-triage.webp) SLURM reports the job exit code in `sacct` and in the Snakemake error banner. Exit code 137 means the process received SIGKILL. On clusters this is almost always the Linux out-of-memory killer or an explicit `scancel`. Exit code 143 means SIGTERM, which is what SLURM sends when the job hits its wall-time limit. Both are retryable but both will fail every time until you change something. A job killed with 137 needs more `mem_mb`. A job killed with 143 needs a longer `runtime` resource or a faster rule. Retries do not fix either, and `restart-times: 3` just guarantees you fail three times slower. ### "Missing output files" after a successful job The error looks like this: ``` Error in rule bwa_map: output: results/sampleA/mapped.bam jobid: 7 Missing output files for rule bwa_map: results/sampleA/mapped.bam ``` You check `ls results/sampleA/mapped.bam` and the file is right there. This is the classic shared-filesystem latency problem. The compute node finished writing the BAM, but the metadata has not propagated to the node where the Snakemake master process polls, so Snakemake sees the job as complete but the file as absent. On busy lustre and GPFS filesystems this delay routinely exceeds the default wait. The fix is `latency-wait`, raised from the default of 60 seconds to 120 or 180 seconds on busy filesystems. This is not a code bug and no amount of resubmitting fixes it permanently, which is why it belongs in the profile. ### Incomplete files and stale locks If a run was killed hard, the next launch may refuse to start: ``` IncompleteFilesException: The file results/sampleA/mapped.bam is incomplete. Use --rerun-incomplete to reschedule the incomplete jobs. ``` Run `snakemake, rerun-incomplete`, or, if the master process is truly gone, `snakemake, unlock` followed by a normal relaunch. Never hand-edit metadata files in the `.snakemake` directory, because that is where Snakemake tracks job provenance and corruption there forces full reruns. ## Tips for Long-Running Pipelines on HPC **Make every rule restart-safe.** A restarted job should either start clean or resume correctly, a property programmers call idempotency. Two habits get you there. Write output through a temporary file and rename at the end, for example `samtools sort ... -o sorted.bam.tmp && mv sorted.bam.tmp sorted.bam`, so a killed job never leaves a truncated file at the final path. For rules that download data or create many side files, add `shadow: "minimal"` to the rule, which gives each job attempt a private working directory symlinked to the inputs. **Use checkpoints for data-dependent dependencies.** When the number of downstream jobs is unknown until a job runs, for example de novo assembly producing a variable number of contigs, a plain rule would snapshot the empty output at DAG-build time. The `checkpoint` decorator tells Snakemake to re-evaluate the DAG after the checkpoint finishes: ```python checkpoint assemble: input: "results/{sample}/mapped.bam" output: directory("results/{sample}/assembly") threads: 16 resources: mem_mb = 32000, runtime = "12:00:00" shell: "metaspades.py -t {threads} -1 {input} -o results/{wildcards.sample}/assembly" def aggregate_contigs(wildcards): checkpoint_output = checkpoints.assemble.get(**wildcards).output return multiwildcard.expand("results/{sample}/assembly/{contig}.fa", sample=wildcards.sample, contig=glob_wildcards(os.path.join( checkpoint_output, "{contig}.fa")).contig) ``` Without the checkpoint, downstream rules would see an empty or stale assembly directory and the DAG would silently drop jobs, a bug that produces missing data rather than an error. **Group small jobs.** A pipeline that generates 5,000 one-minute jobs wastes more time in scheduler queue overhead than in computation. Grouping combines them into fewer SLURM allocations: ```bash snakemake --profile profiles/slurm --group-components fastqc=20 --jobs 50 ``` This submits FastQC in batches of 20 per job, cutting your scheduler footprint by 20 times for that rule. **Lower submission and polling rates if administrators ask.** `-max-jobs-per-second 5` and `-max-status-checks-per-second 1` keep Snakemake polite to the scheduler without meaningfully slowing most pipelines. Workflow management systems running on large clusters must balance execution control against scheduler load, and research systems built for this setting rely on carefully designed distributed data layers to keep that coordination overhead low across hundreds of cores [1]. **Plan for preemptible partitions.** Many clusters offer cheap or free preemptible nodes that can revoke your job at any moment. Snakemake handles this naturally as long as the rule is restart-safe, so the combination of a preemptible partition plus `restart-times: 3` plus restart-safe rules is often the cheapest way to run a large pipeline. Never point a checkpoint or a rule with side effects outside its declared outputs at a preemptible partition without those safeguards. ## Common Mistakes and Limitations **Treating retries as a memory or time fix.** The most common mistake in the "snakemake job retries hpc" space is setting `restart-times: 10` on a job that dies of exit code 137 every time. Retries are for transient failures, such as filesystem hiccups, preemption, and flaky database servers. Deterministic failures need resource changes, and no retry count will save you. **Relying on default `latency-wait` on shared filesystems.** The default of 60 seconds is calibrated for local disks. On parallel filesystems under load, output visibility can lag for minutes. If your log alternates between "Missing output files" for a job and finding that exact file present, this is your problem. **Running Snakemake on a login node with default limits.** The master process coordinates thousands of file existence checks. Login nodes often have process and memory limits that kill it mid-run, and clusters prohibit compute on login nodes by policy. Submit the master process as its own SLURM job, or run it inside a `screen` or `tmux` session on a dedicated head node if your center provides one. **Running two Snakemake instances in the same working directory.** Snakemake locks the working directory precisely to prevent this. The second instance fails with a directory-lock error, and if you bypass the lock with `-nolock` you get silent DAG corruption. Give concurrent pipelines separate directories. **Forgetting that parameter changes trigger reruns by design.** Snakemake 7 introduced `-rerun-triggers`. With the default `all` trigger, editing a line of shell code in a rule reruns every job that used it, which is correct behavior but alarming the first time it deletes a day of compute. During development, restrict triggers with `-rerun-triggers mtime` while you are only changing unrelated files, and let the full trigger run before final analysis. **Leaving orphaned SLURM jobs after aborting.** Without `cluster-cancel: scancel`, Ctrl-C on the master process leaves running jobs alive, burning allocation and potentially writing outputs that confuse the next run. The cancel setting is cheap insurance. **Assuming the retry count applies per failure type.** `restart-times` counts total resubmissions of a job, not per cause. A job preempted twice on a volatile partition plus one genuine failure exhausts a `restart-times: 3` budget. **A limitation to keep in mind.** Snakemake's master-process design puts all coordination in one process on one node, which is simple and reliable for the scale most labs operate at. Systems built for very large parallel executions, such as the d-Chiron workflow management system, have explored distributed in-memory architectures for the execution-control data layer specifically to keep coordination scalable and highly available on clusters of hundreds of cores [1]. If your pipelines grow past the point where a single master process is comfortable, that class of architecture, and alternative engines discussed in the Nextflow comparison below, is where to look. ## Nextflow vs Snakemake for Dependency Handling and Retries Both are mature bioinformatics workflow managers, and the nextflow vs snakemake question has no universal answer. The practical differences for dependency and retry handling: - **Dependency model.** Both build a DAG from declared inputs and outputs and submit ready jobs. Snakemake expresses everything in Python and files on disk. Nextflow uses a Groovy-based DSL with dataflow channels, which makes streaming and data-dependent logic first-class but adds a language layer. - **Cluster integration.** Snakemake 7 uses profile-based `sbatch` strings, Snakemake 8 uses the SLURM executor plugin. Nextflow ships SLURM, PBS, LSF, and other executors built in with `process` directives. In practice both handle SLURM well. - **Retry mechanics.** Snakemake retries with `restart-times`, keyed to restart-safe rules. Nextflow has `errorStrategy 'retry'` with `maxRetries` and backoff, plus task-level `errorStrategy` per process. The concepts map almost one-to-one. - **Ecosystem.** Snakemake integrates tightly with Conda and containers and dominates in fields built on QIIME 2-style tooling. Nextflow dominates in genomics pipelines distributed through nf-core. Many centers support both. Choose the one your institution already supports and your collaborators can read. Migrating a working pipeline between them is rarely worth the downtime unless you hit a hard architectural limit. ## Frequently Asked Questions ### What does, restart-times do when a SLURM job fails? ![Worked example showing that restart-times 3 gives a job four total chances, and that preemptions use up part of that budget.](/article-images/illustrations/snakemake-job-retries-hpc-retry-budget.webp) It tells Snakemake to resubmit a failed job up to that many times before the whole pipeline stops with an error. With `restart-times: 3`, a job gets four total chances: the original attempt plus three retries. The retry reuses the same command and resources, so it only helps for transient problems. ### Why does Snakemake report "Missing output files" when the file exists? The job finished and wrote the file, but the shared filesystem has not propagated the new file's metadata to the node running the Snakemake master process yet. Raise `-latency-wait` to 120 or more seconds on busy parallel filesystems and the error disappears. ### Does Snakemake submit jobs with SLURM dependencies using sbatch, dependency? No. Snakemake manages dependencies itself by submitting each job only when its input files exist, which is more flexible than static dependency chains because the DAG can grow dynamically through wildcards and checkpoints. Only use `-immediate-submit` if your submit script explicitly translates Snakemake's dependency file into scheduler dependencies. ### How do I rerun only the jobs that failed or were interrupted? Run `snakemake, rerun-incomplete` with your usual profile. Snakemake compares file timestamps and metadata against the DAG and reschedules exactly the jobs whose outputs are missing or incomplete, leaving completed work untouched. ### Can I run the Snakemake master process on a login node? You can, but you should not. Login nodes have resource limits and usage policies that make a long-running coordination process unwelcome and unreliable. Submit the master process as its own SLURM job or run it in `tmux` on a designated head node. ### How do I clear a stale lock error after a node failure? Confirm no Snakemake process is still running with `ps aux | grep snakemake`, then run `snakemake, unlock` in the working directory. This removes the lock file only, and the next launch rebuilds state from the output files and the `.snakemake` metadata. ### What do exit codes 137 and 143 mean in my Snakemake error logs? Exit code 137 means the job was killed by SIGKILL, on clusters usually the out-of-memory killer or a manual `scancel`. Exit code 143 means SIGTERM, which SLURM sends at the wall-time limit. Fix the underlying resource declaration (`mem_mb` or `runtime`) rather than adding retries. ### Nextflow vs Snakemake: which handles HPC retries better? Both handle retries well with equivalent mechanisms: Snakemake's `restart-times` and Nextflow's `errorStrategy 'retry'` with `maxRetries`. Choose based on your institution's existing support and your team's language preference, since the retry semantics map closely between the two systems. ## Sources 1. [Distributed in-memory data management for workflow executions.](https://pubmed.ncbi.nlm.nih.gov/34013039/)