# Nextflow Process Failed Exit Code: How to Choose A Nextflow process fails when the command inside it exits with a non-zero exit code, and the `errorStrategy` directive you attach to that process decides what Nextflow does next. The rule of thumb is simple. Retry failures that look transient, such as a task killed because it ran out of memory. Terminate on failures that will never fix themselves, such as a missing input file or a typo in a command. Reserve `ignore` for failures you have deliberately decided to tolerate, and use `finish` when you want a clean partial run that you can extend later with `-resume`. This article explains what an exit code is in Nextflow, walks through each of the four `errorStrategy` options on its own terms, compares the best nextflow process failed exit code handling choices side by side, and works through real bioinformatics troubleshooting scenarios, including the classic RNA-seq error where an aligner dies with exit code 137. The behavior described here follows the official Nextflow documentation, and you should verify the details against the documentation for the specific Nextflow version your pipeline runs on, because directive behavior can change between releases. ## What a failed exit code means in Nextflow An exit code (also called an exit status or return code) is a small integer that every Unix process reports to the shell when it finishes. By long-standing Unix convention, zero means success and any non-zero value means failure. Nextflow adopts this convention directly. When the script block of a process finishes with exit code 0, Nextflow treats the task as complete. When the script finishes with any non-zero value, Nextflow declares the task failed and applies the process error strategy. The number itself carries information. The Unix convention reserves values above 128 for processes terminated by a signal, with the signal number added to 128. Three values come up constantly in bioinformatics troubleshooting: - **137** means the process was killed by SIGKILL, signal 9. On a compute node this is very often the Linux out-of-memory (OOM) killer terminating your tool because it asked for more RAM than the container, cgroup, or scheduler allocation allowed. An out-of-memory (OOM) kill is the single most common cause of an unexpected 137 in alignment and assembly jobs. - **139** means SIGSEGV, signal 11, a segmentation fault. The program accessed memory it should not have. This usually points to a software bug, a corrupted input, or an incompatibility rather than to an undersized allocation. - **143** means SIGTERM, signal 15, a polite termination request. Schedulers and batch systems often send SIGTERM when a job exceeds its requested wall time or when the job is preempted. - **127** means the command was not found. In Nextflow this often reveals a missing tool in the container or environment, a misspelled binary name, or a PATH problem. - **126** means the command was found but not executable, which usually points to file permissions or a corrupted binary. One distinction matters before you choose anything. The exit code belongs to the tool that ran inside the process, not to Nextflow itself. If Nextflow fails for its own reasons, for example it cannot stage an input file or the container image cannot be pulled, you get a Nextflow error rather than a tool exit code, and no `errorStrategy` will paper over it. Choosing an error strategy only makes sense for failures that the task script itself produced. ### Where to find the exit code when a task fails Bioinformatics troubleshooting starts with locating the number. Nextflow gives you four places to look. First, the console and log message. When a task fails, Nextflow prints an error naming the process and the exit status, something to the effect that the process terminated with an error exit status followed by the number in parentheses. The full run log lives in the hidden `.nextflow.log` file in the launch directory, and searching that file for the failed process name almost always leads to the underlying tool output. Second, the task work directory. Every task executes in a unique work directory under the `work/` folder by default. Inside you find `.command.sh` (the exact script that ran), `.command.err` (standard error output), `.command.log` (the combined execution log), and `.exitcode`, the file where Nextflow records the final exit status of the task. Reading `.command.err` and the last lines of `.command.log` tells you not just that the tool failed but why. Third, the trace report. Launching a run with `-with-trace` writes a tab-separated `trace.txt` file with one row per task, including a column for the exit status and the task hash. This is the fastest way to survey many failures at once, for example after a large RNA-seq run where dozens of sample-level tasks might have died for different reasons. Fourth, the `nextflow log` command run against a past session, which lets you inspect the history of executed tasks from a previous run without rerunning anything. ## The four errorStrategy options Nextflow supports four values for the `errorStrategy` directive, which is a process directive (a per-process setting) that you can write either inside the process definition in your `.nf` script or in the `process` scope of `nextflow.config`. Each option answers the same question differently: when this task exits non-zero, what should the rest of the pipeline do? ### terminate, the default `terminate` is the default error strategy, so you get it even if you never write it. When a task fails under `terminate`, Nextflow aborts the run immediately. Tasks that were waiting to be submitted are discarded, and the run stops with an error. The failed task itself is the only one whose failure you see directly. The strength of `terminate` is safety and speed. Most failures in a well-built pipeline indicate a real problem that would contaminate every downstream result: a missing reference file, a corrupted FASTQ, an indexing failure, a malformed sample sheet. Stopping right away prevents the pipeline from burning hours of compute producing results built on a broken step. It also surfaces the error to you while the run is still fresh, which makes bioinformatics troubleshooting much easier than debugging a cascade of downstream failures hours later. The weakness is brittleness. A single flaky task, for example one sample whose aligner job hit the OOM killer at hour six of a twelve-hour run, kills an otherwise healthy pipeline. For that situation you want `retry`, not `terminate`. In DSL2 (the current Nextflow scripting language), a process that relies on the default needs no directive at all, which is exactly why many pipelines look like they never thought about failure handling. They did. They chose the default. ### ignore `ignore` tells Nextflow to treat a failed task as if it had succeeded. The task is marked complete, and downstream processes are submitted using whatever outputs exist, which may be none. The legitimate use cases are narrow but real. Quality screening steps sometimes use `ignore` so that samples failing a QC tool do not stop the run, on the theory that a missing report is acceptable while a halted run is not. Multi-sample reporting pipelines sometimes tolerate a single failed sample so that one corrupt file does not block results for hundreds of others. The pros and cons of `ignore` deserve blunt language, because it is the most dangerous of the four options. When a task is ignored, every downstream process that consumes its output runs anyway. Some will crash quickly, which at least is loud. Others will happily produce results of apparently normal shape and quality from missing or partial input, which is silent data corruption. In a clinical, publication, or production context this is the worst possible outcome, worse than a crash, because nothing alerts you. If you use `ignore`, pair it with explicit checks in downstream processes that verify the expected outputs exist and fail loudly when they do not. Never adopt `ignore` as a general way to make an annoying error go away. If a step fails deterministically, fix the step. ### finish `finish` waits for tasks that are already running to complete, submits nothing new, and then ends the run in a failed state. It occupies a deliberate middle ground between `terminate` and `ignore`. The practical difference from `terminate` shows up with caching. Tasks that completed before the failure are recorded in the cache, so when you resume the run with `-resume`, Nextflow skips every task that finished successfully and picks up from the point of failure. Under `finish`, the long-running tasks that were in flight when the error surfaced get to finish and join that cache, which means hours of compute are preserved instead of abandoned. Under `terminate`, in-flight tasks are killed and their work is lost. `finish` therefore suits long pipelines with expensive middle stages, where an error late in the run should not invalidate half a day of completed alignment or quantification work. It also suits interactive development, where you want the run to drain gracefully, report all the errors that occurred, and leave you with the largest possible resumable state. The cost is time. The pipeline keeps running and paying for compute after the error, which is exactly what you want with expensive tasks and exactly what you do not want when the failure makes every remaining task pointless. ### retry ![Worked example where an aligner asks for 8 GB on the first run, 16 GB on the first retry and 24 GB on the second.](/article-images/illustrations/nextflow-process-failed-exit-code-attempt-memory-scaling.webp) `retry` tells Nextflow to execute the failed task again, up to `maxRetries` times. By default you get one retry unless you raise `maxRetries`. A companion directive, `maxErrorRetries`, sets a cap on the total number of retries across the whole pipeline, which is useful when many different processes have retry enabled and you want to bound the total cost of flakiness. On its own, retrying the identical command usually reproduces the identical failure, so `retry` earns its keep through one of the most useful features in Nextflow: `task.attempt`. The `task.attempt` variable is 1 on the first execution of a task, 2 on the first retry, and so on. You can make resources scale with the attempt, so each retry gets a bigger allocation than the last. A standard configuration pattern looks like this: ```groovy process { errorStrategy = 'retry' maxRetries = 2 withLabel: 'aligner' { memory = { 8.GB * task.attempt } cpus = { 8 * task.attempt } time = { 4.h * task.attempt } } } ``` With this configuration, a process carrying the `aligner` label that fails with exit code 137 from an OOM kill is retried with 16 GB instead of 8 GB, then 24 GB if needed. The label is attached in the process itself with `label 'aligner'`. This pattern converts the most common RNA-seq error in modern pipelines, a memory-starved aligner, into a self-healing behavior rather than a page at 3 a.m. Two cautions apply. First, `retry` is for transient failures. A retry cannot fix a missing input, a syntax error in a tool call, or a corrupted reference genome. Retrying deterministic failures wastes the full retry budget on every affected task and delays the moment you learn about the real problem. Second, retried tasks that eventually succeed are cached like any other completed task, so a later `-resume` run does not repeat the successful retry. ## Side-by-side comparison of the options The following table is the core of the choice. It compares the best nextflow process failed exit code handling options on the criteria that actually decide real runs. | Criterion | terminate | ignore | finish | retry | |--|--|--|--|--| | What happens on failure | Run aborts immediately, pending tasks discarded, running tasks killed | Task marked complete, run continues | Running tasks complete, no new tasks submitted, run then ends in failure | Task runs again up to maxRetries | | Downstream tasks | Never run | Run regardless of missing outputs | Never run | Run once a retry succeeds | | Default? | Yes | No | No | No | | Safe for silent bad results | Yes | No, this is the main risk | Yes | Yes, if retries eventually succeed | | Preserves cached work | Yes, for tasks that already finished | Yes | Yes, including tasks in flight at failure time | Yes | | Cost after the error | None | Continues spending on downstream work | Keeps paying for in-flight tasks | Pays for each retry | | Best for | Deterministic failures you want surfaced fast | Deliberately optional steps with downstream validation | Long expensive pipelines where cached work matters | Transient failures, OOM kills, preemptions | | Main risk | One flaky task stops everything | Silent corruption of downstream results | Wasted compute if remaining tasks are pointless | Delays discovery of deterministic failures | Reading the table by row tells you the trade structure. `terminate` and `retry` protect correctness, `finish` protects cached effort, and `ignore` protects run continuity at the price of correctness. No single value is the best nextflow process failed exit code strategy for every situation, which is why Nextflow lets you set the directive per process and even compute it per failure. ## Choosing the strategy from the exit code itself ![Decision path that sends signal-range exit codes such as 137 and 143 to retry, and ordinary codes such as 1, 126 and 127 to terminate.](/article-images/illustrations/nextflow-process-failed-exit-code-exit-code-triage.webp) The exit code is not just a fact to record in a trace file. It is the strongest signal available for deciding what should happen next, and Nextflow lets you key the error strategy directly to it. The `errorStrategy` directive accepts a closure, a small block of Groovy code evaluated at failure time, and the `task.exitStatus` variable inside that closure holds the exit code that triggered the failure. The canonical example from the official Nextflow documentation retries the task when the exit code falls in a range typical of resource kills and terminates otherwise: ```groovy process ALIGN { errorStrategy { task.exitStatus in 137..140 ? 'retry' : 'terminate' } maxRetries 2 script: """ hisat2 -x ${index} -1 ${reads1} -2 ${reads2} -S ${sample_id}.sam """ } ``` The expression uses the ternary operator, `condition ? value_if_true : value_if_false`. If the aligner died with 137, the run retries. If it died with 1 because a read file does not exist, the run terminates immediately, because no allocation size will conjure a missing file into existence. This pattern scales cleanly across your whole pipeline. A common production configuration retries signal-range deaths on every process while letting ordinary exit code 1 failures stop the run: ```groovy process { errorStrategy { task.exitStatus in 137..144 ? 'retry' : 'terminate' } maxRetries = 3 maxErrorRetries = 10 } ``` The reasoning per exit code class looks like this. - **137 and neighboring signal codes.** These are usually resource or scheduler events, not tool bugs. OOM kills, preemptions, and wall-time enforcement all land here. Retry with growing resources is almost always right, and it is the single highest-value default you can add to a genomics pipeline. - **139.** A segmentation fault occasionally reflects a transient environment problem, but it more often reflects a real bug or bad input. Retrying once is cheap and sometimes works, particularly where a tool is known to segfault under rare memory layouts. Retrying three times on every task is usually wasted compute. - **143.** SIGTERM frequently means the scheduler stopped the job for time or priority reasons. The task itself may be fine. Retry is appropriate, ideally with the `time` directive scaled up by attempt so the retry asks for a longer allocation. - **1 through 4 and other ordinary tool errors.** These usually mean the tool inspected its inputs and rejected them, or a file was missing. Terminate. The failure will reproduce identically until a human intervenes. - **126 and 127.** Command not found or not executable. Terminate immediately. This is an environment and packaging problem, and retrying it in a loop just delays the fix. For pipelines handling RNA-seq error scenarios at scale, the exit-code-keyed dynamic strategy combined with attempt-scaled resources is the configuration most practitioners converge on, and it mirrors what mature community pipelines ship by default. ## Worked scenarios ### An RNA-seq aligner dies with exit code 137 You run 60 samples through an RNA-seq workflow. Alignment runs for several hours, then the pipeline stops with an error reporting exit status 137 on one sample. Reading `.command.err` in the work directory shows the aligner ran out of memory. The genome index for a large reference exceeds the memory you allocated. This is the textbook case for `retry` with attempt-scaled memory. Set `errorStrategy = 'retry'`, raise `maxRetries`, and multiply memory by `task.attempt` for the aligner label. The failing sample gets a second run with a larger allocation, succeeds, and the pipeline continues as if nothing happened. The other 59 samples are never delayed because their tasks never failed. If instead you had left the default `terminate` strategy, the entire run would have stopped at hour six, and your options would be manual resubmission or `-resume`, both slower than an automatic retry that finishes the job in flight. If the same run shows 137 across many samples rather than one, treat that as information about the pipeline rather than about the samples. The allocation is probably too small for every task with that profile, and the right fix is to raise the base memory, not to lean on retries for all 60 jobs. ### A corrupt FASTQ keeps killing a QC step One sample arrived from the sequencer with a truncated FASTQ. A quality control task on that sample fails with a small non-zero code on every attempt. Here `retry` is actively wrong. The failure is deterministic, the file will not repair itself, and each retry burns the tool's full runtime before failing identically. The right call is `terminate`, which stops the run at the failure and points you straight at the corrupt file. The alternative worth considering is `ignore` plus a downstream guard, but only if the pipeline's purpose survives a missing sample, for example a cohort report where you would rather deliver 59 of 60 samples today. In that case write the downstream consumer to check for the expected output and fail loudly when it is absent. If the run is meant to produce a complete, consistent dataset, terminate and replace the file. ### Reference data download fails intermittently A process fetches a reference genome index from a remote FTP server. The server drops connections a few times a day. Failures here are transient, cheap to retry, and unrelated to your inputs. `retry` with a couple of attempts and a short delay between attempts is exactly right. Add `maxErrorRetries` at the pipeline level so that a genuinely dead mirror cannot trigger unbounded retries across many download tasks. The general lesson generalizes to any task whose outcome depends on an external service, including database downloads, API calls, and cloud transfers. Transient external failures are the friendliest possible retry target because retries are cheap, independent, and often successful. ### A long pipeline fails at hour 20 of 24 A metagenomics or single-cell pipeline runs for a day, and a late stage fails. You want to fix the problem and resume without repeating the first 20 hours. `finish` is the strategy built for this moment. It lets in-flight tasks complete so their results enter the cache, then ends the run. After you fix the root cause, `nextflow run` with `-resume` replays everything from cache and executes only the failed and downstream tasks. With the default `terminate`, in-flight tasks would have been killed mid-execution and their hours lost. The difference between the two strategies on a day-long run can be many hours of recomputation, which is why `finish` deserves a place in any pipeline designed for expensive, long-lived runs. ## Common mix-ups Several mistakes appear so often in real pipelines that they deserve their own list. **Confusing terminate with finish.** People frequently assume `finish` is a gentler name for the default. It is not. `terminate` stops the run and kills running tasks. `finish` lets running tasks complete before stopping. If your pipeline has tasks that run for hours, the choice between them is measured in compute hours. **Retrying deterministic failures.** Setting `retry` globally feels safe, and it is safe for correctness, but it converts every genuine bug into N failed attempts separated by full task runtimes. On a 24-hour assembly task with three retries, a broken input costs four days before the pipeline gives up. Scope retry to exit codes that plausibly indicate transient trouble. **Raising maxRetries without raising resources.** A retry that repeats the exact same command with the exact same memory allocation on an OOM kill is a coin flip weighted against you. The retry pattern works because `task.attempt` lets you change something. Tie resource growth to the attempt. **Mixing up maxRetries and maxErrorRetries.** `maxRetries` bounds retries for a single task. `maxErrorRetries` bounds the total across the pipeline. A pipeline with a dozen retrying processes and no `maxErrorRetries` cap can, in a pathological run, spend far more compute on retries than on the original work. **Masking the real exit code inside the script.** The exit code Nextflow sees is the exit code of the last command in your script block. If a wrapper script catches an error, prints a message, and then ends with a command that succeeds, Nextflow sees 0 and marks the task complete even though the real work failed. Similarly, in a shell pipeline like `cat file | grep pattern`, the exit code comes from `grep` unless you set `pipefail`. Nextflow launches the script with bash in a strict mode that aborts on simple command errors and unset variables, but pipeline exit semantics still follow Unix rules, so a tool that exits 0 after failing will defeat any error strategy you choose. When you wrap tools, make sure the wrapper propagates the real failure with an explicit non-zero `exit` statement. **Treating ignore as a fix.** When someone cannot work out why a step fails, the temptation is to set `ignore` on it and move on. The step still fails. It just fails silently now, and everything downstream of it runs on nothing. Every downstream surprise in such a run traces back to that choice. **Forgetting that errorStrategy only covers task failures.** Nextflow-level failures, such as unresolvable input declarations, container pull errors, and syntax errors in the workflow itself, are not process failures and are not affected by any of the four options. ## Common Mistakes and Limitations A few specific, practical points close out the guidance. - Do not use `ignore` on a process whose outputs feed results you will publish, deposit, or act on, unless every downstream consumer explicitly validates its inputs. Ignored failures have produced silently incomplete results in real analyses, and the trace file will even show the ignored task as completed, so nothing in the run summary warns you. - Do not key your dynamic strategy to the wrong exit codes. Ranges like 137 to 140 target signal deaths. Extending the retry range to cover ordinary errors such as exit code 1 defeats the point of distinguishing transient from deterministic failures. - Do not assume the exit code you see is the tool's own. Wrapper scripts, workflow-level validation, and shell pipes all sit between the tool and Nextflow. When a failure seems inexplicable, open `.command.sh` and check whether the reported exit code could have come from a wrapper rather than from the tool you think you ran. - Exit code conventions above 128 come from Unix signal handling, not from Nextflow, so a specific code can vary with the shell, the container, and the scheduler. Treat 137 as strong evidence of an OOM kill rather than as a certified diagnosis, and confirm with the tool's own log before changing allocations. - Directive behavior, defaults, and available options can change between Nextflow releases. Check the official Nextflow documentation for the version pinned in your environment before relying on a specific behavior in production. - An error strategy manages execution, not analysis validity. A pipeline that never fails can still be wrong if its inputs, references, or parameters are wrong. These settings reduce operational failure, and no setting substitutes for reviewing outputs. One limitation applies to all of the above. The right choice depends on your specific pipeline, scheduler, cost model, and tolerance for missing results, and there is no configuration that is correct for every laboratory. The patterns here cover the decisions that recur across most genomics workflows. ## Frequently Asked Questions ### What exit code makes a Nextflow process fail? Any non-zero exit code marks the task as failed. Zero means success. Codes above 128, such as 137, 139, and 143, indicate the process was killed by a Unix signal, with 137 most commonly meaning an out-of-memory kill. ### What is the default error strategy in Nextflow? The default is `terminate`. When a task exits non-zero, the run aborts immediately, pending tasks are discarded, and running tasks are stopped. You get this behavior whether or not you write it in your configuration. ### Should I ever use the ignore strategy? Only when you have deliberately decided that a step is optional and that a missing output is acceptable, and only when downstream processes validate their inputs and fail loudly when they are absent. Otherwise `ignore` risks silent data corruption, because downstream tasks run on whatever exists, including nothing. ### How do I retry only out-of-memory failures? Use a dynamic error strategy keyed to the exit status, for example `errorStrategy { task.exitStatus in 137..140 ? 'retry' : 'terminate' }`, combined with `maxRetries` and resources scaled by `task.attempt` so each retry requests more memory and CPUs than the last. ### What is the difference between terminate and finish? `terminate` aborts the run immediately and kills in-flight tasks. `finish` lets currently running tasks complete, submits nothing new, and then ends the run in a failed state, which preserves more cached work for a later `-resume` run. ### How do I find the exit code of a failed task? Check the error message printed by Nextflow, the `.exitcode` file in the task work directory, the `.command.err` file for the tool's error output, or the exit column of a trace report generated with `-with-trace`.