# How to Resume a Nextflow Pipeline After a Crash By the end of this article you will be able to restart a crashed Nextflow run so that completed steps are skipped, only the interrupted and downstream tasks execute again, and you can prove from the logs that the reuse actually happened. You will also know exactly which changes to a pipeline, its parameters, or its directories silently destroy cached work, and how to avoid each one. To follow along you need the machine (or a copy of it) where the pipeline was launched, the launch directory containing the hidden `.nextflow` folder and `.nextflow.log` file, the `work` directory that holds task outputs, and a working Nextflow installation with a supported Java runtime. If your pipeline came from a repository such as an nf-core pipeline, you also need the revision number (the version tag you launched with) and your parameters file. ## What a Workflow Manager Does for You When Things Fail A bioinformatics workflow manager is software that runs a multi-step analysis as a directed graph of tasks, resolving dependencies, parallelizing work across cores or a scheduler, and recording what ran. Nextflow, an open-source workflow manager written for the Java Virtual Machine and configured with Groovy-style DSL2 syntax, is one of the two managers most commonly used in genomics alongside Snakemake. Its single most valuable property in daily practice is that a run that ends in a crash, a killed job, or an exhausted cluster walltime is not a lost run. It is a paused run. That recovery ability rests on a small amount of bookkeeping that Nextflow writes continuously while the pipeline runs. Understanding that bookkeeping turns a scary crash into a routine ten-minute recovery. Skipping that understanding is what produces the classic bad outcome, where a user reruns everything from scratch and burns two days of cluster time on tasks that were already finished. ## How Nextflow Resume Actually Works Three pieces of state make resume possible. Each one lives in a specific place and each one is required. **The work directory.** This is the directory, named `work` by default, where Nextflow executes every task. Each task gets its own subdirectory whose name starts with a short hash fragment. Inside a task directory you find the rendered script (`.command.sh`), the wrapper the scheduler ran (`.command.run`), the captured standard output and error (`.command.out` and `.command.err`), and, when the task succeeded, an empty marker file named `.exit`. The published results you actually care about usually live elsewhere, in the directory given to `publishDir`, but the intermediate files that let resume skip work live here. **The cache database.** Nextflow records the state of every task in a small database under `.nextflow/cache`, organized in a subdirectory named after the session ID, a unique identifier assigned to each run. This database, not the log file, is what resume reads. **The log file.** `.nextflow.log`, a hidden file in the launch directory, is a running diagnostic log. It does not drive resume, but it tells you what happened before the crash, which task was running, and what error appeared. When you launch with the resume option, Nextflow computes a task hash for every task it is about to run. The hash is derived from the task's inputs, its script, its container image or conda environment, its declared resources, and other deterministic inputs. If a task directory holding the outputs of that exact hash exists and contains a `.exit` file marking a successful completion, the task is skipped and its outputs are linked forward. If the directory is missing the `.exit` file, the task is treated as incomplete and runs again. Tasks downstream of a rerun task also run again, because their inputs changed. Two consequences follow from this design and are worth internalizing before any crash ever happens: 1. Resume is not a saved checkpoint file. It is a comparison between what Nextflow wants to run now and what already exists on disk. Anything that changes the hash, such as editing a module script, breaks the match. 2. Resume never validates output content. A task directory with a `.exit` file and plausible outputs is reused even if a file inside was truncated by a filesystem problem. You verify biology and file integrity separately from resume. ## Step by Step: How to Resume a Nextflow Pipeline After a Crash This is the core walkthrough. The worked example is a pipeline launched from `/scratch/rnaseq-run`, which crashed at about 60 percent completion when the cluster node running the head process was rebooted. ### Step 1. Do not delete anything, then look at the log Go to the launch directory and look at the end of the log. The log is hidden, so list hidden files first. ```bash cd /scratch/rnaseq-run ls -a tail -100 .nextflow.log ``` Expected last lines before a hard crash look something like this, with details varying by version: ``` Jan-09 14:32:07.960 [Task monitor] DEBUG n.n.s.p.TaskRunner - Submitted process > TRIM (sample_B) Jan-09 14:32:08.104 [Task monitor] DEBUG n.processor.TaskProcessor - [TRIM] Process terminated with an error exit status (137) ``` An exit status of 137 on Linux means the task was killed with SIGKILL, which in practice almost always means the scheduler or the operating system out-of-memory killer terminated it. If instead the log simply stops with no error line, the head process itself died, for example during a node reboot, and the last running tasks were interrupted. Nextflow rotates logs, so if the crash happened during an earlier phase, also check `.nextflow.log.1` and `.nextflow.log.2`, which hold older material. ### Step 2. Identify the run you want to resume From the same launch directory, list recent runs: ```bash nextflow history ``` Example output (the column layout varies slightly between Nextflow versions): ``` 2026-11-09 14:32:11 grim_kirby 3f5c2a1b-9d4e-4c1a-b2f8-7e6d5a0c1e44 ERR 2026-11-08 09:11:02 keen_lovelace 8b1f0d2c-3a5b-4e9f-a1c2-d4e5f6a7b8c9 OK ``` Each row gives the launch timestamp, the auto-generated run name, the session ID, and the final status. A crashed run shows an error status or, if the process itself was killed, may show no clean status at all. Write down the run name and session ID of the crashed run. If `nextflow history` shows nothing, you are in the wrong launch directory. The cache database and history live with the launch directory, not with your home directory. If the launch directory itself was lost but you know the session ID, you can still resume by passing the session ID directly to the resume flag, provided the work directory survived and you know where it is. ### Step 3. Confirm where the work directory is Resume matches against the task directories in the work directory, so you need the one the crashed run used. ```bash nextflow log grim_kirby -f process,tag,status,work_dir | head -20 ``` Expected output: ``` process tag status work_dir INDEX - COMPLETED /scratch/rnaseq-run/work/1a/2b3c4d TRIM sample_A COMPLETED /scratch/rnaseq-run/work/8c/9d10e1 TRIM sample_B FAILED /scratch/rnaseq-run/work/3e/4f5a6b ALIGN sample_A CACHED /scratch/rnaseq-run/work/5g/6h7i8j ``` If the work directory is a shared scratch path or a custom path passed with the `-w` option at launch time, confirm it still exists and has not been wiped by a scratch-cleaning policy. Cluster scratch systems that delete files after a fixed number of days are the most common silent killer of resume. If the work directory contents are gone, resume cannot recover those tasks and a rerun is unavoidable. ### Step 4. Relaunch with resume From the original launch directory, rerun the exact same command you used before, adding the resume flag. For a local pipeline directory: ```bash nextflow run . -resume ``` For a remote pipeline such as an nf-core pipeline, pin the same revision you used originally and pass your parameters through a file so nothing drifts between attempts: ```bash nextflow run nf-core/rnaseq \ -r 3.14.0 \ -profile docker \ -params-file my-params.yml \ -resume ``` Your parameters file might look like this: ```yaml input: "samplesheet.csv" outdir: "results" ``` Note that pipeline-level parameters such as `input` are normally written on the command line with a prefix of two hyphens before the name. A parameters file avoids repeating them, keeps the recovery command identical between attempts, and is the format nf-core documents for its pipelines. If you prefer the command line, the same parameters are passed with that two-hyphen prefix followed by the parameter name. You can also resume a specific past run rather than the most recent one, which matters when several runs share a launch directory: ```bash nextflow run . -resume grim_kirby ``` The argument after the resume flag is the run name or the full session ID from `nextflow history`. ### Step 5. Read the launch output and confirm reuse is happening The beginning of a resumed run tells you immediately whether caching engaged. Expected output: ``` Launching `./main.nf` [soggy_brown] DSL2 - revision: 4a1b2c3d45 executor > local (4), slurm (2) [1a/2b3c4d] process > INDEX (1) [100%] 1 of 1, cached: 1 ✓ [8c/9d10e1] process > TRIM (sample_A) [100%] 1 of 1, cached: 1 ✓ [3e/4f5a6b] process > TRIM (sample_B) [100%] 1 of 1 ✓ [7k/8l9m0n] process > ALIGN (sample_B) [100%] 1 of 1 ✓ ``` The phrase `cached: 1` next to a process line means that task was skipped. The bracketed hash fragment at the start of each line matches the task directory name in the work directory. If every line shows work being done and `cached: 0` everywhere, caching did not engage, which means one of the conditions in the next section changed. Stop the run rather than letting it burn compute, and diagnose first. ### Step 6. Verify the result two ways First, inspect the run record: ```bash nextflow log -f process,tag,status,exit,duration ``` Expected output after a successful resume: ``` process tag status exit duration INDEX - CACHED 0 - TRIM sample_A CACHED 0 - TRIM sample_B COMPLETED 0 3m 12s ALIGN sample_B COMPLETED 0 11m 40s ``` Second, verify the actual biology-level outputs. Confirm that files in your `publishDir` output folder have recent modification times only for products downstream of the rerun tasks, that file sizes are nonzero, and, where it matters, compare checksums of published outputs against a reference from a previous good run. Resume guarantees reuse of whatever is on disk, not correctness of what is on disk, so a checksum pass on key outputs is cheap insurance. ### Step 7. Clean up only after you trust the result Resumed runs leave completed task directories from previous attempts in the work directory. Once the final run is verified, reclaim space: ```bash du -sh work nextflow clean -n grim_kirby nextflow clean -f -k grim_kirby ``` The first command reports work directory size, the second shows a dry run of what would be deleted, and the third performs the deletion while keeping the log files. Never run cleanup while a run is still in progress. ## What Breaks Resume: Every Way the Cache Can Miss ![Decision path asking whether code, parameters, container, revision, work directory path or the cache database changed, ending in a fix for each.](/article-images/illustrations/nextflow-resume-after-crash-cache-miss-triage.webp) Resume is deterministic. When it reruns everything, a specific hash input changed. The list below covers the causes in rough order of how often they appear in practice. **Edited code.** Any change to a process script, a module file, or a function the script calls changes the hash of every task that includes that code. This is the intended behavior, not a bug. If you edited a module between the crash and the resume, tasks built from it rerun. If you did not intend the edit, revert it and resume again. For versioned pipelines such as nf-core pipelines, launching with a different `-r` revision changes all hashes, so always pin the original revision. **Changed parameters.** A task hash includes every value the task actually consumes. Changing a parameter that a process reads reruns that process and everything downstream. Changing a parameter that no process reads leaves all tasks cached, which surprises people in the harmless direction. The practical tip is to change only what you intend to change and check the first minutes of output to see the blast radius. **Changed container or environment.** Moving from a pinned container tag to `latest`, or changing a conda environment specification, invalidates every affected task. Pin versions explicitly in your configuration. **Touched or moved work directory contents.** Never edit, compress, or relocate files inside a task directory. Nextflow treats the work directory as immutable evidence of what already ran. **Different work directory path.** If you resume from a different launch directory or a different mount point without pointing Nextflow at the original work directory, no cached outputs will be found. Restore access to the original path or pass it explicitly with `-w /original/path/to/work`. **Lost cache database.** The resume comparison reads the cache database under `.nextflow/cache`. If only the work directory survived and the launch directory was deleted, resume by session ID may still work if the database is reachable, but if both are gone the tasks rerun. Backing up the small cache database alongside the work directory is a cheap habit. **Changed Nextflow version.** Hash inputs are stable across routine releases, but large version jumps can alter internals enough that some or all tasks rerun. If a resume produces unexpected mass rerunning right after an upgrade, reinstalling the previous Nextflow version and resuming under it is a legitimate recovery move. **File timestamp churn.** On some shared filesystems and object stores, file timestamps change without content changes, which defeats timestamp-sensitive matching. Nextflow provides caching modes for this: setting the process directive `cache = 'lenient'` relaxes the match between cached and new task inputs, and `cache = 'deep'` verifies content more strictly for cases where it should not trust metadata. These directives go in your configuration: ```groovy process { cache = 'lenient' } ``` ## Special Situations After a Crash ### The cluster head node or login node crashed State was being written continuously, so resume works normally. Two follow-ups matter. First, check your scheduler for orphaned jobs submitted by the dead run and cancel them, for example with `scancel` on Slurm, so old jobs do not fight with the resumed run over the same task directories. Second, wait a moment and retry if you see a lock error, covered in the error section below, because a killed process can leave a stale lock that clears once the running process is confirmed gone. ### You are on a different machine or a different launch directory Resume across machines works only if three things travel together or stay reachable: the work directory, the cache database, and the pipeline code at the original revision. Copy the work directory with a tool that preserves structure and metadata, for example `rsync -a`, keep absolute paths identical where possible, and pass the work directory location with `-w` if the path changed. Realistically, cross-machine resume is reliable for small to medium work directories and painful for terabyte-scale ones. For very large work directories, resuming on the original filesystem is the sane path. ### The run failed at a task instead of crashing If Nextflow stopped because one task failed rather than because the machine died, resume behaves exactly as in a crash: upstream tasks are cached, the failed task and its dependents rerun. Before resuming, read the failed task's own logs so you are not just re-running into the same wall: ```bash cd work/3e/4f5a6b cat .command.err ``` If the failure was transient, for example a node with faulty memory, add a retry policy so future failures absorb themselves: ```groovy process { errorStrategy = 'retry' maxRetries = 2 } ``` `errorStrategy` tells Nextflow what to do when a task fails, and `retry` relaunches the identical task up to `maxRetries` times. ### Resuming without network access On an air-gapped cluster or during an outage, add the offline option so Nextflow does not try to re-resolve the pipeline from a remote repository: ```bash nextflow run . -resume -offline ``` This requires that the pipeline project is already present locally, which it is whenever you resumed from the same launch directory. ## Common Error Messages During Recovery Three messages account for most recovery confusion. **A lock message about the session.** A message of the general form "Unable to acquire lock on session" with a session ID means another Nextflow process believes it owns this run. It happens when a run was started twice, when a previous instance is still alive on a compute node, or immediately after an unclean kill. Check with `ps aux | grep nextflow`, kill any genuine leftover process or wait for it to exit, and retry. If you resumed while your own resumed run was already going, stop one of the two. **A message that the session cannot be found.** If you passed a session ID that the local cache database does not know, Nextflow tells you it cannot find the session. The cause is almost always a different launch directory or a different `NXF_HOME` cache root than the original run used. Go back to the original launch directory, or set the cache location explicitly before resuming. **Exit status 137 and task terminations with no error text.** These are not resume problems, they are the reason you are resuming. Exit 137 is the out-of-memory or forced-kill signature. Before the next attempt, either give the offending process more memory in its resource directives, or add the retry policy shown above. Resuming into an unchanged memory limit usually just reproduces the crash one task later. ## Common Mistakes and Limitations - Rerunning from scratch as a reflex. The single most expensive habit in Nextflow practice is deleting the work directory or dropping the resume flag after a crash. Cached reuse is free. Take the ten minutes. - Editing the pipeline between the crash and the resume, then wondering why nothing was cached. Code edits change hashes. Fix the code first, resume second. - Launching from the wrong directory. Resume reads history and the cache from the launch directory. If `cached: 0` appears across the board and you launched from somewhere new, that is why. - Renaming processes. A renamed process has a new hash. Cosmetic refactors before a recovery attempt cost a full rerun of the renamed stages. - Deleting the work directory to free disk space before the resumed run is verified. The work directory is the only thing making resume possible. Confirm outputs, publish what you need, then clean with `nextflow clean`. - Treating resume as validation. Resume reuses any task marked complete. A corrupted output file from a filesystem incident will be faithfully reused and propagated. Run your pipeline's own quality checks and checksum spot checks on key outputs. - Resuming across a large Nextflow upgrade and not noticing a silent mass rerun. Read the first screen of output. If you see new execution where you expected cache hits, stop and diagnose. - Ignoring scratch-cleanup policies on HPC systems. Many clusters delete scratch files on a timer. If your work directory lives in such a location, a resume two weeks later can be an empty shell. Point work directories at durable storage when runs may need recovery. - Assuming resume works without the cache database. The work directory holds the outputs, but the cache database is what identifies them. Keep both. - One general limitation: the exact behavior in any given recovery depends on your pipeline code, configuration, storage system, and scheduler. Rehearse the recovery procedure once on a small test dataset so that when a real crash happens, you know what your setup does. ## Nextflow vs Snakemake for Crash Recovery ![Comparison table of Nextflow and Snakemake across recovery mechanism, what triggers a rerun, language and ecosystem.](/article-images/illustrations/nextflow-resume-after-crash-nextflow-vs-snakemake.webp) Because choosing a bioinformatics workflow manager is partly a decision about failure behavior, a short comparison belongs here. Both Nextflow and Snakemake are mature, open-source, and widely deployed in genomics, and both recover from crashes rather than forcing full reruns. The mechanics differ. Nextflow computes a content-derived hash for each task and, when you opt in with the resume flag at launch, skips tasks whose hashed outputs already exist in the work directory. Snakemake tracks dependencies through metadata it keeps alongside its own directory structure, marks jobs that were incomplete when a run died, and reruns those jobs on the next invocation, with rerun triggers that compare code, parameters, inputs, and software environments between the recorded and current state. In day-to-day terms, both systems give you the same guarantee, that finished work survives a crash, while differing in what invalidates that work. The practical differences that influence the choice are elsewhere. Nextflow runs on the Java Virtual Machine, uses a Groovy-based DSL2 syntax, has first-class container and cloud-native plumbing, and anchors the nf-core collection of community-maintained pipelines, which makes it the default in much of large-scale genomics. Snakemake is Python-based, integrates naturally with conda environments, and tends to suit teams whose pipelines are Python-centric and analysis-script-heavy. If crash recovery is your main criterion, both are adequate, and the decision should rest on your team's language background, your infrastructure, and whether the nf-core ecosystem covers your domain. ## Frequently Asked Questions ### Does `-resume` skip everything or only completed tasks? It skips only tasks whose hash matches a completed record in the work directory. Interrupted, failed, and downstream tasks run again. ### How do I resume a specific past run instead of the most recent one? Pass the run name or session ID to the resume flag, for example `nextflow run . -resume grim_kirby`, and find that name with `nextflow history`. ### Can I resume if the crash happened on a cluster and the head process was killed? Yes. State is written continuously, so a killed head process is recoverable. Cancel any orphaned scheduler jobs from the dead run first, then relaunch with the resume flag from the same launch directory. ### Why does resume want to rerun tasks I already completed? Something changed a task hash, most often edited code, changed parameters, a different container tag, a different pipeline revision, or a work directory path Nextflow is not looking at. Check the work directory location and your recent edits. ### Can I resume after the work directory was deleted or moved? If it was deleted, no, the outputs are gone and those tasks must rerun. If it was moved, point Nextflow at the new location with the `-w` option and resume from a launch directory that still has the cache database. ### Is it safe to upgrade Nextflow before resuming? Routine upgrades are generally safe, but a large version jump can change hashing behavior and trigger reruns. If a resume right after an upgrade reruns unexpectedly, reinstall the previous version and resume under it. ### Can I resume from a different launch directory or a different user account? Only if the cache database travels with you or the session ID can be resolved against it, and the work directory is reachable. Copy the `.nextflow` directory and work directory together and keep paths consistent, or simply resume from the original launch directory. ### Nextflow vs Snakemake, which handles crash recovery better? Both recover well, with finished work preserved after a crash. Nextflow uses opt-in content-hash caching keyed to a work directory, while Snakemake records metadata and reruns jobs it marks as incomplete on the next run. Choose based on your team's language and infrastructure, not on resume mechanics.