# How to Retry Failed Runs in Nextflow and Snakemake

A six-hour RNA-seq run that dies at the alignment step should not cost you another six hours. Nextflow and Snakemake, the two most widely used bioinformatics workflow managers, both record what finished and rerun only what is broken. The catch is that each tool records progress differently, and a handful of settings decide whether a retry takes ten minutes or starts over from scratch.

This guide shows you how to retry failed workflow runs in Nextflow and Snakemake step by step. By the end you will be able to:

- locate the exact task or rule that failed, along with its command, exit code, and logs
- resume a Nextflow run so completed tasks are reused from cache
- configure automatic retries that request more memory on each attempt
- recover a Snakemake run after a crash, including stale locks and incomplete files
- verify that a rerun did what you intended and nothing more
- avoid the common mistakes that silently throw away a cache

A bioinformatics workflow manager runs a multi-step analysis as a directed acyclic graph (DAG), a graph of jobs where each job waits for files produced by earlier jobs. Both tools run the same analysis on a laptop or on a cluster scheduler such as SLURM. Every command below assumes a Unix shell.

Have the following on hand before you start:

- A Unix shell (Linux, macOS, or WSL2 on Windows).
- Java 11 or newer, and Nextflow. [Install Nextflow](/knowledge/bioinformatics/install-nextflow) with `curl -s https://get.nextflow.io | bash`.
- Python 3.7 or newer for Snakemake 7, or Python 3.11 or newer for Snakemake 8. Install with `pip install snakemake`, or create an environment with `mamba create -n smk -c conda-forge -c bioconda snakemake`.
- A small test dataset, so you can practice reruns in

## How the Two Tools Decide What to Rerun

Before touching a single command, it pays to understand the mental model each tool uses, because almost every retry problem traces back to a mismatch between what you think the tool remembers and what it actually records.

**Nextflow** remembers individual tasks. Every task gets a hash that is computed from the process name, the script text, the input files and values, the container image or conda environment, and any variables declared in the process `env` block, as described in the Nextflow documentation. The results of each task live in a unique directory under the `work/` folder. When you resume a run, Nextflow recomputes the hash for each task it needs, and if a directory with that hash already exists and contains valid outputs, the task is marked as cached and skipped. If anything in the hash changed, the task executes again, and everything downstream of it executes again too.

**Snakemake** remembers files, not tasks. It builds a directed acyclic graph from the rules in your Snakefile and the target files you request. For each output file, Snakemake consults metadata stored under `.snakemake/metadata` and decides whether the file is up to date. Since Snakemake 7.8, that decision considers not just timestamps but so-called rerun triggers: changes to the rule code, the params, the inputs, the software environment, and modification times, as described in the Snakemake documentation. Snakemake 8 enables all of these triggers by default, which makes reruns stricter than many long-time users expect.

This difference drives most of the practical distinctions later in this guide:

| Aspect | Nextflow | Snakemake |
|--|--|--|
| Unit of cache | Task (a process applied to a specific input set) | Output file |
| Cache location | `work/` directories plus a cache index | `.snakemake/` directory in the working directory |
| What a resume skips | Tasks whose hash matches a cached result | Files judged up to date under the rerun triggers |
| After a crash mid-job | The interrupted task reruns on resume | Interrupted jobs rerun, often flagged as incomplete |
| Main resume flag | `-resume` | No flag needed for a normal rerun; auxiliary flags handle edge cases |

One consequence worth internalizing: in Nextflow, "retry a failed run" almost always means the same thing, `nextflow run ... -resume` from the same launch directory. In Snakemake, "retry" can mean simply running the same command again, or it can mean adding flags for incomplete files, stale locks, or forced rules depending on how the run died. Knowing which situation you are in is half the work.

## Step by Step: Retrying a Failed Nextflow Run

### Step 1: Find out exactly what failed

When a run stops, the console shows the last few lines, but the full picture lives in hidden files. In your launch directory, `.nextflow.log` contains the complete session history, including scheduler submissions and task failures. A quick way to isolate failures:

```bash
grep -i "error\|failed" .nextflow.log | tail -20
```

For the failing task itself, go to its work directory. Nextflow prints the path when a task fails, and it looks like `work/ab/12abcdef.../`. Inside that directory you will find, per the Nextflow documentation:

- `.command.sh`: the exact script that ran
- `.command.err`: standard error, usually where the real cause is
- `.command.out`: standard output
- `.command.log`: the combined runtime log
- `.exitcode`: the exit status of the command
- `.command.run`: the wrapper script, useful for checking what resources and environment the task actually received

Read `.command.err` first. Nine times out of ten the failure is an out-of-memory kill, a missing input file, a typo in the script, or a container that does not contain the tool version you need. Fixing the root cause is always the next step, because resuming a run without fixing anything just repeats the failure at the same task.

If you did not enable a trace file the first time, you can enable it on the resume run so you get an auditable record of what was reused:

```bash
nextflow run mypipeline.nf -resume -with-trace trace.tsv
```

### Step 2: Resume from the same launch directory

The simplest resume is:

```bash
nextflow run mypipeline.nf -resume
```

The `-resume` flag must appear after the pipeline reference. Run it from the same directory where you launched the original run, because Nextflow looks up prior sessions in its run history for that launch directory. If you use profiles, parameters, or a config file, include them exactly as before. Any parameter that changes can change task inputs, and changed inputs invalidate the cache for every affected task.

If you set pipeline parameters, the safest habit is to keep them in a `nextflow.config` file or a profile rather than on the command line, so the resume command is identical to the failed command. If your parameters are already in the config, the resume command stays short and there is nothing to mistype.

### Step 3: Resume a specific past run

Sometimes you launched several runs and want to resume one in particular. Nextflow assigns each run a name and a session identifier. List them with:

```bash
nextflow log
```

This prints run names, session identifiers, status, and launch times. You can resume by run name or by session identifier:

```bash
nextflow run mypipeline.nf -resume sleepy_curie
```

You can also inspect what a past run did, field by field:

```bash
nextflow log sleepy_curie -f process,task_id,status,exit
```

This is useful when you want to know, before resuming, how far the old run got.

### Step 4: Confirm what was reused

After the resume finishes, check the trace file if you enabled one:

```bash
column -t -s $'\t' trace.tsv | less -S
```

Tasks with `CACHED` in the status column were reused; tasks marked as completed were executed fresh. Another quick check is wall time: a resume that genuinely reuses an alignment-heavy workflow should finish in minutes, not hours. If it runs for hours, something invalidated the cache, and the troubleshooting section below covers the usual suspects.

## Automatic Retries in Nextflow: errorStrategy and Dynamic Resources

Transient failures, especially scheduler timeouts and out-of-memory kills, often succeed on a second attempt with more resources. Nextflow handles this declaratively in `nextflow.config`.

### Basic retry configuration

```groovy
process {
    errorStrategy = 'retry'
    maxRetries = 2
}
```

With this in place, any task that fails is resubmitted automatically, up to two retry attempts. You do not need to rerun the pipeline manually; Nextflow keeps going within the same session. The `maxRetries` value counts the retry attempts, so a task can run up to `maxRetries + 1` times in total.

The `errorStrategy` directive also accepts `terminate` (kill the whole run), `ignore` (mark the task failed but keep going), and `finish` (stop submitting new tasks but let running ones complete). The default is `terminate`.

### Escalating resources per attempt

The powerful pattern is combining `retry` with `task.attempt`, a variable that equals 1 on the first run of a task, 2 on the first retry, and so on. A common arrangement:

```groovy
process {
    errorStrategy = 'retry'
    maxRetries = 2

    withLabel: 'mem_medium' {
        memory = { 16.GB * task.attempt }
        time   = { 8.h  * task.attempt }
    }

    withLabel: 'mem_high' {
        memory = { 64.GB * task.attempt }
    }
}
```

On the first attempt a `mem_medium` task asks for 16 GB; if it is killed for exceeding memory, the retry asks for 32 GB. Because resource requests are evaluated per attempt, this lets you start cheap and escalate only for the tasks that actually need it, which matters a great deal on busy shared clusters.

You can also scope retries to specific processes by name:

```groovy
process {
    withName: 'STAR_ALIGN' {
        errorStrategy = 'retry'
        maxRetries = 3
    }
}
```

### What retry does and does not fix

The retry mechanism only fires when a task exits with a nonzero status that Nextflow attributes to the task itself. It will not rescue a workflow whose failure is deterministic, such as a wrong file path or a syntax error in your code. It also will not fix a command that fails because it reads a corrupted input. When you find yourself adding retries everywhere, treat that as a signal to look at the underlying cause in `.command.err` rather than at the retry settings.

## Step by Step: Retrying a Failed Snakemake Run

### Step 1: Read the failure

Snakemake prints the rule name, wildcards, job identifier, and a snippet of the log when a job fails. If your rules declare a `log:` directive, the error message tells you where that log file was written, which is the single best reason to always declare `log:` in your rules. The full log of the run itself is stored under `.snakemake/log/` with a timestamped filename.

A useful overview command is:

```bash
snakemake -summary
```

This lists every output file with its status, such as whether it is up to date, whether its inputs were updated, or whether parameters or code changed. It is the Snakemake equivalent of reading a trace file, and it works even before you rerun anything.

### Step 2: Handle incomplete files

If the process was killed while a job was writing its output, for example because you pressed Ctrl-C, the cluster preempted the job, or the node went down, Snakemake may report incomplete files on the next run. Snakemake tracks these in `.snakemake/incomplete/`. The fix is to rerun with:

```bash
snakemake -rerun-incomplete -cores 4
```

This tells Snakemake to discard the partial outputs and rerun those jobs from scratch. In practice you will use this flag routinely after any interruption, and it is harmless to include when there are no incomplete files.

### Step 3: Clear a stale lock

Snakemake locks the working directory so two runs do not interfere. If a run was killed hard, the lock can remain and you will see a message about a directory being locked. Before unlocking, verify that no Snakemake process is actually still running:

```bash
ps aux | grep snakemake
```

If the coast is clear, remove the lock:

```bash
snakemake -unlock
```

Then rerun normally. Unlocking while a real run is in progress is one of the classic self-inflicted wounds in this ecosystem, so treat that `ps` check as mandatory.

### Step 4: Rerun, and know what will rerun

A plain rerun is just:

```bash
snakemake -cores 4
```

Snakemake examines the DAG, finds files that are already complete and up to date under the rerun triggers, and skips them. Jobs that never ran, or whose outputs are missing or stale, run now. There is no separate resume flag for the common case; resume behavior is the default.

When you want to force specific rules to rerun regardless of the triggers, use `-R`, the short form of the force-run option:

```bash
snakemake -cores 4 -R trim_reads
```

This is handy after you fix a bug in one rule and want to be certain it executed, or after editing a script that Snakemake's triggers do not detect for some reason in your setup. To force only the first job and let dependencies flow normally, `-f` forces the first target. You can also bound a rerun with `-until` (run until a given rule completes) or `-omit-from` (start the DAG after a given rule).

If you changed a rule's parameters and want to see which files would rerun without actually running anything, combine a dry run with the summary view:

```bash
snakemake -cores 4 -n
```

The dry run prints every job that would execute, which is your preview before committing.

## Automatic Retries in Snakemake

Snakemake supports retries both on the command line and inside rules.

On the command line:

```bash
snakemake -cores 4 -retries 2
```

Or per rule, which is often better because only flaky rules should retry:

```python
rule align_reads:
    input:
        reads="reads/{sample}.fq.gz"
    output:
        "aligned/{sample}.bam"
    retries: 2
    shell:
        "some_aligner {input.reads} > {output}"
```

You can combine retries with escalating resources. Inside a resource expression, the `attempt` variable reflects the current attempt number, which is the documented pattern for retry-aware resource requests:

```python
rule heavy_step:
    output:
        "results/{sample}.out"
    retries: 3
    resources:
        mem_mb=lambda wildcards, attempt: 8000 * attempt
    shell:
        "heavy_tool -mem {resources.mem_mb} {output}"
```

Each retry therefore asks for more memory than the last, mirroring the Nextflow pattern of `task.attempt` based escalation. Snakemake also accepts wait times between retry attempts, so a rule that fails because a filesystem is briefly unavailable can be given a pause before the next try; consult the option list in the Snakemake documentation for the exact flag in your installed version.

A practical note: retries on a cluster executor are especially valuable for node failures and preemptions, which are transient by nature. A rule that fails deterministically, for instance because of a malformed input file, will fail all of its retries and waste your allocation, so treat retries as a complement to reading logs, not a substitute.

## Worked Example: Recovering a Multi-Sample Run in Both Tools

Abstract explanations fade fast. The following miniature sandbox lets you rehearse the whole failure, fix, and resume cycle in ten minutes on a laptop.

### The Nextflow sandbox

Create a file `toy.nf`:

```groovy
nextflow.enable.dsl = 2

process MAKE_FASTA {
    input:
    val sample
    output:
    path "results/${sample}.fa"
    script:
    """
    mkdir -p results
    printf ">${sample}\nATGC\n" > results/${sample}.fa
    """
}

workflow {
    Channel.fromList(['sampleA', 'sampleB', 'sampleC']) | MAKE_FASTA
}
```

Run it:

```bash
nextflow run toy.nf
```

Now introduce a failure. Edit the script line to write to a directory that does not exist:

```groovy
    script:
    """
    mkdir -p results
    printf ">${sample}\nATGC\n" > missing_dir/${sample}.fa
    """
```

Rerun with `nextflow run toy.nf -resume`. Two things happen: tasks that already succeeded come back as cached, and the pipeline fails at `MAKE_FASTA` for the remaining samples. Fix the script back, then run `nextflow run toy.nf -resume` again. You should see some tasks reported as cached and only the previously failed ones executed. Add `nextflow log` to your notes, and you have now rehearsed the complete diagnostic loop: failure, root cause in the work directory, fix, resume, verification.

### The Snakemake sandbox

Create a `Snakefile`:

```python
SAMPLES = ["sampleA", "sampleB", "sampleC"]

rule all:
    input:
        expand("results/{sample}.fa", sample=SAMPLES)

rule make_fasta:
    output:
        "results/{sample}.fa"
    log:
        "logs/{sample}.log"
    shell:
        "mkdir -p results logs && printf '>{wildcards.sample}\\nATGC\\n' > {output} 2> {log}"
```

Run `snakemake -cores 2`. To simulate a crash, run again with a deliberately broken shell command, watch it fail partway, then fix it. Now interrupt a run with Ctrl-C while a job is in flight, and rerun with `snakemake -cores 2 -rerun-incomplete`. Kill a run hard with `kill -9` on the Snakemake process, observe the lock message, verify with `ps aux` that nothing is running, and clear it with `snakemake -unlock`. Finally, run `snakemake -summary` and read the status column.

Doing this once in a sandbox is worth more than reading ten guides, because when a real six-hour run dies at midnight, you want the recovery commands to be muscle memory rather than a memory of something you skimmed.

## Nextflow vs Snakemake: How Retry Semantics Differ

![Two column comparison of Nextflow task cache and Snakemake file cache across unit, location, resume flag and what a rerun skips.](/article-images/illustrations/retry-failed-workflow-nextflow-snakemake-cache-unit-compare.webp)

The two tools overlap heavily but their retry behavior differs in ways that matter day to day.

**When a failure happens mid-run.** Nextflow lets currently running tasks finish but stops submitting new tasks, then you resume with a flag. Snakemake, with default settings, stops scheduling new jobs after a failure but you can tell it to keep going; some users configure it to continue past failures and collect all errors in one pass. Either way, in Snakemake a subsequent plain rerun picks up where the DAG left off without a special flag.

**What counts as "done".** Nextflow's cache is task based and content addressed, so a resumed run skips tasks whose hashes match. Snakemake's cache is file based, so a rerun skips files that exist and pass the rerun triggers. A subtle consequence: in Nextflow, deleting a task's intermediate output does not force a rerun as long as the cached directory still exists, because the outputs are matched by hash rather than looked up by path. In Snakemake, deleting an output file forces that job to rerun, and if the file was declared `temp()`, its absence triggers regeneration whenever it is needed again.

**Granularity of invalidation.** If you change one process's script in Nextflow, only tasks of that process and their downstream dependents get new hashes; everything else stays cached. If you edit one rule's shell block in Snakemake, the code trigger marks that rule's outputs for rerun along with their dependents. The granularity is conceptually similar, which is reassuring: both tools invalidate locally rather than globally.

**Automatic retries.** Nextflow expresses retries through `errorStrategy` and `task.attempt` in the config. Snakemake expresses them through `retries` and `attempt` in rules or through command-line flags. Both support escalating memory per attempt. Nextflow centralizes this in configuration, which suits large pipelines with standard labels; Snakemake places it in the rule, which suits workflows where only one or two steps are flaky.

**Version drift.** Snakemake's rerun trigger defaults changed materially between versions 7 and 8, so behavior you relied on in one version may tighten in the next. Nextflow's cache semantics are comparatively stable, but new versions occasionally change which process properties feed the hash. The operational habit that protects you in both tools is the same: run a dry run or summary check before a resume, and read the release notes when upgrading the workflow manager mid-project.

## Common Mistakes That Break a Cache

These are the errors that turn a ten-minute retry into a full rerun, ordered roughly by how often they bite people.

**1. Passing different parameters on the resume run.** If the original run got one set of parameters and you type the resume command from memory with a slightly different value, every task that consumed that parameter gets a new hash. Keep all parameters in `nextflow.config` or a profile so the resume command is byte-for-byte identical. In Snakemake, the equivalent mistake is editing a value in the Snakefile or a config.yaml between the failure and the retry, which fires the params trigger.

**2. Deleting the work directory or the hidden metadata.** In Nextflow, `work/` is the cache. In Snakemake, `.snakemake/` holds metadata, locks, incomplete markers, and logs. Cleaning either one is a decision to rerun from scratch. If disk pressure forces a cleanup, prune selectively: Nextflow offers cache cleanup subcommands for old sessions, and Snakemake users often archive completed outputs rather than deleting metadata wholesale.

**3. Changing the container tag instead of the digest.** If your process references a floating tag and the registry updated the image between runs, the image contents changed even though your string did not. Whether the cache notices depends on how the reference is hashed, but reproducibility suffers either way. Pinning by digest is the durable fix in both tools.

**4. Touching inputs gratuitously.** A `touch` on an input file, or any operation that rewrites its timestamp without changing content, can trigger reruns in Snakemake under the mtime trigger, especially in version 8 where the trigger set is strict by default. Nextflow hashes file content, so a pure timestamp change usually does not invalidate its cache, but copying files around can change content-irrelevant metadata that some steps consume.

**5. Resuming from a different launch directory.** Nextflow associates run history with the launch directory. If you launch the resume from a different directory, the history lookup may not find your session. Either resume from the original directory or supply the session identifier explicitly, which you can obtain with `nextflow log` in the original directory.

**6. Editing shared code that many rules import.** In Snakemake, a change to a shared function or module fires the code trigger for every rule that uses it. Sometimes that is exactly what you want; often it is an accidental mass rerun. In Nextflow, editing a function used inside several process scripts invalidates each of them. Localize shared helpers when you can, and check the dry run output before resuming.

**7. Mixing execution environments between the failed run and the retry.** If the first run used a container and the retry runs with conda, or you switch profiles between runs, cached tasks may not be reusable because the environment is part of what defines the task. Match the original invocation.

**8. Assuming publish or output-copying steps are cached.** In Nextflow, the `publishDir` step copies results out of the work directory, and republishing behavior has its own settings; in Snakemake, outputs in the final results directory are the tracked entities themselves. Confusion here leads people to "rerun the publishing" with force flags when a simpler targeted command would do.

## Troubleshooting a Retry That Refuses to Resume

![Decision tree asking whether the run is Nextflow or Snakemake, whether a lock or incomplete files remain, and whether the failure is transient.](/article-images/illustrations/retry-failed-workflow-nextflow-snakemake-retry-triage.webp)

**Everything reruns in Nextflow despite `-resume`.** Work through this list: confirm you are in the same launch directory; confirm parameters match; confirm you did not edit any process script; check whether the container reference changed; and check whether environment variables in an `env` block changed. If you truly need to resume a run whose original invocation you cannot reproduce, resuming by explicit session identifier, obtained via `nextflow log` in the original launch directory, is the escape hatch.

**Everything reruns in Snakemake after a version upgrade.** Snakemake 8's stricter default rerun triggers mean upgrades can cause a one-time mass rerun. If you want the older, timestamp-only behavior, restrict the rerun triggers on the command line to mtime, understanding that you then lose automatic detection of code and parameter changes. Read the dry run output first so you know what you are about to execute.

**"Directory cannot be locked" persists.** This means either a Snakemake process is genuinely alive, or a previous run died without releasing the lock. Do the `ps` check, then `snakemake -unlock`, then rerun. On shared filesystems, also confirm no colleague is running the same workflow in the same directory; locking is per directory, not per user.

**Incomplete file warnings every time you rerun.** Persistent incomplete markers usually mean the incomplete flag was never passed, or jobs keep getting killed mid-write because they request too little memory or time. The durable fix is the retries and escalating resources described above; the flags merely clear the symptom.

**Retry loops that never succeed.** If a task fails identically across attempts, more attempts will not help. Read `.command.err` in Nextflow, or the rule's declared log in Snakemake, and distinguish transient causes, such as preemption and transient I/O errors, from deterministic ones, such as corrupt input, bad flags, or version incompatibilities. Escalating resources fix the first category only.

**The resume succeeds but outputs look wrong.** Cached tasks return whatever they produced originally, so if a bug existed upstream and you fixed only the failing step, the cache will happily feed stale upstream results forward. This is why the verification step below matters, and why a targeted force of the upstream rules, rather than a full rerun, is usually the right correction.

## Verifying That a Rerun Did What You Intended

A retry is not complete until you have evidence about what ran and what was reused.

In Nextflow, enable a trace on the resume run and inspect the status column, where cached and executed tasks are distinguished. You can also count tasks per process:

```bash
awk -F'\t' 'NR>1 {print $2, $NF}' trace.tsv | sort | uniq -c
```

The `nextflow log` subcommand gives you a per-run history including exit statuses, which is a lightweight post-mortem without rerunning anything.

In Snakemake, `-summary` is the primary verification tool, listing each output file and why it is in its current state. The companion commands `-list-code-changes`, `-list-params-changes`, and `-list-input-changes` tell you exactly which outputs would rerun due to each trigger, which is the cleanest way to confirm that your fix will do precisely what you expect before you spend compute. After the run, compare the summary against your expectations: the number of updated files should equal the number of jobs you intended to fix, plus their downstream dependents.

Finally, verify biology and bookkeeping, not just workflow mechanics: spot-check log files for warnings, confirm sample counts in the outputs, and make sure published or final-directory results correspond to the new run rather than leftovers from an older one. A surprising number of "the fix did not work" reports are actually cases where the fix worked fine but the analysis was reading a stale copy in the results directory.

## Quick Reference: The Commands That Matter

**Nextflow**

```bash
# Diagnose
grep -i "error" .nextflow.log
cat work/<hash-dir>/.command.err

# Retry with cache reuse
nextflow run mypipeline.nf -resume

# Retry a specific past run
nextflow log
nextflow run mypipeline.nf -resume <run-name>

# Verify
nextflow run mypipeline.nf -resume -with-trace trace.tsv
```

**Snakemake**

```bash
# Diagnose
snakemake -summary
ls .snakemake/log/

# Clear a stale lock after confirming nothing is running
ps aux | grep snakemake
snakemake -unlock

# Retry after an interruption
snakemake -cores 4 -rerun-incomplete

# Force a specific rule
snakemake -cores 4 -R rule_name

# Preview before committing
snakemake -cores 4 -n
```

**Retry configuration one-liners**

- Nextflow: `errorStrategy = 'retry'` and `maxRetries` in the process scope, with resources scaled by `task.attempt`.
- Snakemake: `retries:` in a rule or `-retries` on the command line, with resources scaled by the `attempt` variable.

If you internalize only three habits from this guide, make them these: read the error artifacts before resuming, keep the resume invocation identical to the failed invocation, and always verify with a dry run, summary, or trace what the retry is about to do. Those three habits are what separate a ten-minute recovery from an accidental six-hour rerun, and they transfer to every workflow you will ever maintain with either tool.