# Nextflow Pipeline Debugging Tools The single biggest difference between Nextflow debugging tools is scope. The trace file records one line per task with exit codes and resource use, the HTML report turns those numbers into plots, the timeline shows when every task ran, and the log plus work directory tell you exactly which command failed and why. The rule of thumb is simple. Use the trace and report to answer what and how much, use the work directory and `.nextflow.log` to answer why, and use dry-run options to answer whether the pipeline would run at all. This guide covers the standard toolset that ships with Nextflow, the options that control it, the pros and cons of each artifact, and worked troubleshooting scenarios that mirror the failures seen most often in production bioinformatics, including the classic RNA-seq error classes. Everything here reflects the behavior described in the official Nextflow documentation and long-standing Unix conventions, which means the commands and file names are stable and reproducible across recent Nextflow releases. ## The Nextflow debugging toolset at a glance Every option in the table below is activated on the command line or in the configuration file, and most of them write to your launch directory. The table is the reference point for the rest of the article. Each row gets its own section with commands, examples, and tradeoffs. | Tool | What it captures | Best for | Main limitation | |, - |, - |, - |, - | | Trace file (`-with-trace`) | One row per task: status, exit code, process name, hash, timestamps, duration, CPU, memory | Post-run audits, finding failed or slow tasks, cost accounting | Must be enabled at launch, plain TSV, no per-command detail | | HTML report (`-with-report`) | Same metrics rendered as resource plots | Spotting memory or CPU outliers across the run | Only reflects the current launch, larger file, slower to generate on huge runs | | Timeline (`-with-timeline`) | Per-task chronology from submit to completion | Finding serial bottlenecks and idle time | Same launch-time restriction | | DAG (`-with-dag`) | Graph of process dependencies | Explaining or auditing pipeline structure | Large pipelines produce unreadable graphs | | `.nextflow.log` | Engine-level events, scheduler messages, stack traces | Diagnosing crashes before any task runs | Very verbose, needs patience | | `nextflow log` command | History of past runs and their tasks | Comparing runs, recovering a lost run ID | Only sees runs launched from that directory | | Work directory files | `.command.sh`, `.command.err`, `.command.out`, `.exitcode` per task | Reproducing and understanding one specific failure | You must locate the task hash first | | `-preview`, `nextflow inspect` | Resolved configuration and process definitions without execution | Catching config and parameterization errors before compute | Does not test data-dependent logic | | `-dump-channels`, `view()` | Channel contents as JSON or console output | Debugging DSL2 data flow | Verbose on wide channels | | `-stub-run` and test profiles | Fast dry runs with placeholder outputs | Validating logic, CI testing | Stubs hide real tool behavior | | `errorStrategy` and retry | Automatic re-execution of failed tasks | Surviving transient resource or queue failures | Can mask a real bug if overused | | Seqera Platform (`-with-tower`) | Live monitoring, logs, metrics in a web interface | Team runs, long-running production jobs | External service, requires setup | No single tool answers every question. A failed RNA-seq alignment step might look clean in the trace (exit code 137, killed) and still require the work directory to explain why the container was killed. The rest of this article walks through each tool in that order, from what to data to why. ## Where pipeline failures actually live ![Decision path that routes a Nextflow failure to the log, channel dumps, work directory or timeline based on when it failed.](/article-images/illustrations/nextflow-trace-report-debugging-failure-layer-triage.webp) Bioinformatics troubleshooting with Nextflow gets easier once you sort failures into four layers, because each layer has a matching tool. 1. **Configuration and parameter errors.** The run dies before any task executes. Wrong paths, an invalid profile name, a parameter with a typo, or an executor that does not exist. The `.nextflow.log` and `-preview` output are the tools here. 2. **Script and channel logic errors.** The pipeline starts, channels do not connect as expected, a process receives zero items or receives items in the wrong format, or a DSL2 module is not included. The DAG, `view()`, and `-dump-channels` are the tools here. 3. **Task execution errors.** A single process exits nonzero. This covers the vast majority of mid-run failures, including most RNA-seq error classes such as a fastq pair with swapped read orientation, a genome index built for the wrong reference, or an out-of-memory kill during STAR or bwa alignment. The trace file, report, and work directory are the tools here. 4. **Resource and environment errors.** Tasks are killed by the scheduler, containers fail to pull, or the shared filesystem is slow. The timeline, trace metrics, and log are the tools here. A quick diagnostic habit worth building: before opening any file, read the last thirty lines of `.nextflow.log` after a failure. Nextflow prints the failed task's hash, the process name, and the work directory path right at the end of a failed run. That single habit routes you to layer three or four most of the time, and from there you go straight to the offending work directory. ## Trace, report, and timeline files for Nextflow debugging This is the core of nextflow trace report debugging. Three artifacts share the same data source (the task execution records that Nextflow collects during a run) and differ only in presentation. All three must be enabled before the run starts. If a pipeline fails and you did not enable them, you cannot recover the metrics for that run, though rerunning with `-resume` plus the flags will re-execute only the failed tasks and produce records for them. ### Enabling the trace file The fastest activation is on the command line: ```bash nextflow run main.nf -with-trace ``` This writes a tab-separated file named `trace.txt` in the launch directory. Every completed task gets one row. The columns you will use most are the task identifier, the hash, the process name, the tag, the status, the exit code, the submit, start, and complete timestamps, the duration, the CPU percentage, the peak resident set size, the memory percentage, and the attempt number. On Linux executors where cgroup accounting is available you also get bytes read and written per task, which is useful for estimating I/O cost. A more sustainable pattern is to put the settings in the configuration file so every run produces them by default: ```groovy trace { enabled = true overwrite = true file = 'trace.txt' } report { enabled = true file = 'report.html' } timeline { enabled = true file = 'timeline.html' } dag { enabled = true file = 'dag.html' } ``` The `overwrite` setting matters more than most people expect. Without it, repeated runs in the same launch directory append rows from different runs into one trace file, which mixes metrics from a fixed run with metrics from a broken one. With `overwrite = true`, each launch replaces the previous file. Alternatively, give each run its own directory or pass a filename directly, for example `-with-trace run_2026-05-12.trace`. You can also trim the output with the `trace.fields` setting if you want a narrower file for downstream parsing. ### Reading the trace file like a troubleshooter ![Worked example showing that exit code 137 equals 128 plus signal 9, the SIGKILL used by the out-of-memory killer.](/article-images/illustrations/nextflow-trace-report-debugging-exit-code-signal.webp) Open the trace after a failure and sort by the status column. Three patterns account for most findings. - **A row with status `FAILED` and a nonzero exit code.** Note the hash and the process name, then go to the work directory (covered below) to see the actual command output. The exit code narrows the cause before you read a single line of log: 137 means the task received a SIGKILL, which on a scheduler is nearly always the out-of-memory killer, 139 means a segmentation fault inside the tool or a corrupted input file, and 143 means the task received a SIGTERM, often from a scheduler timeout or a manual cancellation. These are standard Unix conventions where the shell reports 128 plus the signal number. - **A row with status `CACHED`.** The task was resolved from the cache during a resumed run. If you expected fresh execution, the hash did not change, which usually means an input file or parameter change did not actually reach the task. - **A row with an attempt number above 1.** The task failed at least once and was retried, which indicates either an `errorStrategy 'retry'` directive or a resource escalation rule. If retries keep failing with the same exit code, the problem is deterministic, not transient, and no amount of memory will fix a wrong reference file. The duration and peak memory columns also support capacity planning. If every task in a process consistently uses half of its requested memory, you can reduce the request and shorten the queue wait. If a task consistently hits its limit, raise it before the next run rather than discovering the failure three hours in. ### The HTML report The report is the same task data rendered with resource plots. Activate it with `-with-report` and open the resulting `report.html` in a browser. It shows CPU, memory, and time distributions per process, which makes outliers obvious in a way a sorted TSV does not. When one sample out of forty blows the memory budget during gene quantification, the report plot shows it as a single tall bar. The pros are readability and speed of diagnosis. The cons are that it captures only the current launch, it grows noticeably on runs with tens of thousands of tasks, and it cannot answer questions the trace did not record, such as the content of a failed command's stderr. ### The timeline `-with-timeline` produces `timeline.html`, which lays out each task as a horizontal bar on a wall-clock axis from submission to completion. Its unique value is concurrency diagnosis. Long idle stretches where only one process runs (a serial bottleneck), or tasks piled up waiting on a slow upstream step, show up immediately. If your run takes ten hours but the timeline shows total useful compute of two hours, the pipeline has a structural bottleneck, not a resource problem. The limitation is the same as the report: launch-time activation only, and on very large runs the rendering can be heavy. ### Pros and cons of the trace, report, and timeline together The honest summary of nextflow trace report debugging options is this. The trace is cheap, always worth enabling, and parseable by scripts, but it is a post-hoc audit rather than a live view. The report and timeline cost a little more and add visualization that saves real time on runs over a few hundred tasks. None of them record anything for a run that already finished without them, which is the single most common frustration new users hit, and the mitigation is simply to enable all three in the config file once and never think about it again. ## The DAG for structure questions `-with-dag` renders the directed acyclic graph (DAG), the dependency graph of processes in the pipeline. The file extension determines the format, so `-with-dag dag.html` produces an interactive HTML page. The DAG answers questions about structure, not execution: does the trimming process feed both alignment and quality control, does the multiqc process depend on every upstream channel, and is there an accidental cycle or orphan process. Use it when you inherit an unfamiliar pipeline or when a process receives no input and never runs, because the graph will show a disconnected node. Skip it for execution failures. A pipeline can have a perfect DAG and still fail every task on a wrong genome path. ## Log files and the nextflow log command Two distinct things share the word log, and confusing them is a common mix-up. The `.nextflow.log` file is the engine's own log for the current (most recent) run in the launch directory. The `nextflow log` subcommand is a query interface for the history of runs stored in the `.nextflow` directory. ### Reading .nextflow.log When a run crashes, `.nextflow.log` contains the engine's account of events: configuration parsing, executor submission, scheduler responses, and Java stack traces if the crash was in Nextflow itself rather than a task. Because the file is dense, search rather than read. On a failed run, try: ```bash grep -i error .nextflow.log | tail -30 ``` If log color codes make the output hard to read, disable them with `NXF_ANSI_LOG=false` or the `-ansi-log false` option and the log becomes plain text. Rotated logs from earlier runs in the same directory appear as `.nextflow.log.1`, `.nextflow.log.2`, and so on, which matters when several people share a launch directory and the latest log belongs to someone else's run. ### Querying run history with nextflow log `nextflow log` with no arguments lists every run launched from that directory with its run ID, timestamp, and status. This is how you recover a run ID after a terminal session is gone. With a run ID, you can query individual tasks: ```bash nextflow log nextflow log f7x2k9 -f process,hash,status,exit,duration nextflow log f7x2k9 -F 'status == FAILED' ``` The `-f` option selects fields and the `-F` option applies a filter, so the last command lists only failed tasks for that run, with their hashes ready to paste into a work directory path. The pros are that this works entirely offline against files that already exist, with no flags needed at launch time. The cons are that it only knows about runs launched from that specific directory, and it records task metadata rather than tool output, so it tells you which task failed but not why. ## Inside the work directory: the ground truth of a failed task Every task executes in its own directory under `work/`, organized by the first two characters of its hash, for example `work/3f/8a2c.../`. This is where nextflow pipeline debugging becomes concrete. The files in that directory are stable and documented: - `.command.sh` contains the exact script the process executed, with interpolated variables resolved to their final values. If you suspect a parameter arrived empty or a file path was wrong, this file settles it. - `.command.run` contains the full execution wrapper Nextflow generated, including environment variables, container invocation, and scheduler directives. It is long but authoritative when a task behaves differently interactively than in the pipeline. - `.command.err` holds the task's standard error stream. A truncated BAM, a missing index file, or an illegal memory access from a GPU tool will show its message here. - `.command.out` holds standard output. - `.command.log` combines both streams in some execution modes. - `.exitcode` is a plain file holding the numeric exit code, convenient for scripting over many task directories. - `.command.begin` marks task startup, which helps distinguish a task that never started from one that started and failed. To jump straight to a failed task from the log output, copy the hash printed at the end of a failed run: ```bash cd work/3f/8a2c9d1e.../ cat .command.sh cat .command.err cat .exitcode ``` The pros of work directory debugging are unmatched precision. You see the literal command, the literal error, and you can often rerun the command by hand in that directory to iterate on the fix. The cons are that you need the hash (recovered from the log or trace), that files inside the work directory should not be modified because Nextflow uses them for caching, and that heavily parallel runs produce thousands of directories, so navigation without the hash is impractical. One caveat that saves an hour of confusion: some bioinformatics tools write progress and error messages only to standard output, leaving `.command.err` empty even when the task failed. If `.command.err` is empty, read `.command.out` and `.command.log` before concluding the tool said nothing. ## Resume, caching, and rerunning after failure The `-resume` option re-executes only tasks whose inputs, code, container, and resources changed since a previous run. Successful tasks are reused from the cache, and failed tasks always re-execute, which makes the combination of `-resume` with the observability flags a debugging strategy in itself: ```bash nextflow run main.nf -resume -with-trace -with-report ``` This reruns the failed tasks and any downstream tasks, produces a trace with fresh metrics for exactly the failures you care about, and leaves everything else cached. It is the standard move when you forgot to enable the trace on the original run. Three cache rules drive most resume confusion. First, the cache key is a hash of the task's code, input values, container, and relevant settings, so editing even a comment inside a process script invalidates every task from that process. Second, files produced by the tool inside the work directory are what get cached, so `-resume` cannot recover a task that failed, only skip ones that succeeded. Third, the run history that enables resume lives in the `.nextflow` directory of the launch directory, so moving your pipeline elsewhere and running from a new directory starts a fresh history with no cache to reuse. A frequent workflow failure pattern is fixing a bug in a script, rerunning with `-resume`, and discovering the pipeline "still fails" identically. Check whether the launch directory actually picked up your edit, whether you are resuming the run you think you are (the run ID appears in the banner at startup), and whether the file you edited is inside a container image that was itself pinned and therefore unchanged. ## Debugging pipeline structure before compute When the failure happens before any real work, or a process mysteriously receives no data, the execution-focused tools above are the wrong instruments. Three options target setup and data flow. ### Preview and inspect `-preview` prints the resolved process definitions and configuration without executing anything: ```bash nextflow run main.nf -profile test -preview ``` The `nextflow inspect` subcommand takes this further and prints the resolved configuration along with the process graph and their resource settings. Use both when a run fails in the first seconds, when you suspect a profile in `nextflow.config` is overriding your command-line parameters, or when you want to confirm which executor and queue a process will target before burning scheduler allocation. The con is that neither option executes the pipeline, so data-dependent mistakes (a wrong glob pattern that matches zero files) remain invisible. ### Channel-level debugging DSL2 pipelines move data through channels (named queues of values between processes). When a downstream process runs zero times, the cause is almost always an upstream channel that is empty or mis-shaped. Two tools expose channel contents directly. The `view()` operator prints a channel's contents as the pipeline runs: ```groovy ch_reads = Channel.fromPath("${params.fastq_dir}/*_R{1,2}.fastq.gz") ch_reads.view() ``` The `-dump-channels` option writes channel contents to JSON files you can inspect after or during a run: ```bash nextflow run main.nf -dump-channels ``` The JSON output shows every value in the channel, which turns a silent data flow problem into a visible one. A classic RNA-seq error of this class is a glob such as `*_R{1,2}.fastq.gz` that misses files named with `_R1_001.fastq.gz` (the Illumina bcl2fastq convention), leaving the alignment process starved of input. `view()` or `-dump-channels` reveals the empty or partial channel instantly. ### Stub runs and test profiles `-stub-run` executes processes using their `stub:` blocks, which generate placeholder output files without running the actual tool. Combined with a test profile, this validates that the pipeline graph, channel wiring, and file publishing all work end to end in minutes rather than hours: ```bash nextflow run main.nf -profile test -stub-run ``` nf-core pipelines ship a `-profile test` configuration with small bundled datasets precisely for this purpose, and running it is the first recommended check for any nf-core pipeline error. The pro is speed and isolation of logic errors from tool errors. The cons are that stubs hide tool-specific failures entirely, and test profiles use bundled data, so a passing test run does not prove your production data will work. ## Error strategies, retries, and resource labels Not every debugging tool observes failures, some prevent them from stopping the run. The `errorStrategy` directive controls what happens when a task exits nonzero, and the two most useful configurations are automatic retry and resource escalation: ```groovy process STAR_ALIGN { errorStrategy 'retry' maxRetries 2 withLabel: process_high { cpus = 16 memory = 32.GB } } ``` The `attempt` variable increments across retries, so you can write a dynamic strategy that requests more memory on each attempt, a pattern the nf-core template uses in its resource configuration with the `process_high`, `process_medium`, and `process_low` labels. This turns transient out-of-memory failures into silent self-heals. The pros are obvious, fewer aborted runs and less babysitting. The cons deserve equal weight. A retry masks the failure from anyone reading only the final status, so the trace's attempt column and the report become the only evidence that something was wrong. If the underlying cause is deterministic (a corrupted input file, a wrong reference), retries just burn compute and delay the failure. Use retries for transient causes, and treat a process that exhausts its retries as a debugging priority rather than a solved problem. ## Seqera Platform and remote monitoring For team environments and production services, `-with-tower` connects a run to Seqera Platform (formerly Tower), which provides live status, logs, resource metrics, and run history in a browser, with the ability to inspect a failed task's logs without access to the compute node. This matters on shared clusters where the work directory lives on a filesystem you can reach but the scheduler's live state does not, and for long runs where you want failure alerts instead of periodic polling. The pros are centralized history across many users and pipelines, rich live dashboards, and retained logs after the work directory has been cleaned. The cons are that it is an external service requiring account setup and network access, and for a single-user laptop debugging session it adds complexity with no benefit over the local trace and log. A lighter-weight middle ground is the `-with-weblog` option, which streams run events as JSON to any HTTP endpoint, useful for custom monitoring. ## Choosing the right tool: criteria and worked scenarios Four criteria decide which of the nextflow pipeline debugging tools to reach for. 1. **Timing.** Can the tool be added to a run that is already failing, or must it be enabled at launch? Trace, report, and timeline require launch-time activation. The work directory, `.nextflow.log`, and `nextflow log` always exist. 2. **Question type.** Metrics (trace, report), chronology (timeline), structure (DAG), engine state (log), command behavior (work directory), or data flow (channels). 3. **Cost.** Trace files are near free, reports add rendering overhead, dashboards require external infrastructure. 4. **Audience.** A solo troubleshooter works in the terminal. A team needs the report or a platform view that non-developers can read. ### Scenario one: an RNA-seq error at the alignment step A bulk RNA-seq run fails after an hour with an error in the alignment process. The tail of `.nextflow.log` shows the failed task hash. In the work directory, `.exitcode` contains 137 and `.command.err` ends abruptly with no tool message, the signature of the out-of-memory killer rather than a software bug. The trace (enabled in the config since the first run) shows peak memory for that task at the ceiling of what was requested, while sibling tasks used half. The fix is a resource label raising memory for that process, or a dynamic retry that escalates memory on attempt two. A resume run then re-executes only the alignment tasks and their downstream steps. ### Scenario two: a silent RNA-seq error where a process never runs A run completes successfully, but the quantification output folder is empty and no differential expression results appear. There is no failure to debug, which is the trap. The timeline shows that the quantification process executed zero tasks. `view()` on the input channel reveals it received nothing. The cause is a sample sheet column mismatch, so the paired-end read paths never formed. The lesson is that some of the worst pipeline failures produce exit code zero, and the timeline plus channel dumps are the tools that catch them. ### Scenario three: an intermittent failure that succeeds on retry A workflow fails about once in every ten runs, always in the same trimming process, always with exit code 143, and rerunning the task by hand succeeds. The trace shows the failed attempts exceeded their `time` directive on a busy partition. This is scheduler contention, not a code bug. The fix is a longer `time` directive or a dynamic error strategy that retries on that signal, and the trace confirms over subsequent runs that attempts stopped exceeding one. ### Scenario four: debugging an inherited pipeline You receive a pipeline written by a departed colleague. Before running it on real data, `nextflow inspect` shows the resolved configuration and process list, and `-with-dag dag.html` renders the structure. You spot a process not connected to the published output chain. You then run `-profile test` with `-stub-run` to verify the graph executes end to end in minutes. Only after that do you launch on real data with the trace, report, and timeline enabled. This order, structure before logic before compute, is the cheapest debugging path for unfamiliar code. ## Common Mistakes and Limitations - **Enabling observability too late.** The trace, report, and timeline only record the current launch. If you add `-with-trace` after a failure, the metrics for the failed run are gone. Mitigate by putting all three in the config file unconditionally. - **Reading a trace mid-run as if it were final.** Rows for running tasks are incomplete or absent. Treat a trace as valid only after the run ends, or re-read it after failure. - **Confusing CPU percentage with core count.** In the trace, `%cpu` reflects usage over the task's wall time, so a single-threaded task pegging one core can show roughly 100 percent, and a four-core task can show up to roughly 400. Comparing this number directly to the `cpus` directive without this mental conversion leads to wrong conclusions about efficiency. - **Trusting peak memory after a kill.** When the kernel kills a task with SIGKILL, sampled memory metrics can miss the final spike, so the recorded peak may be lower than what caused the death. An exit code of 137 with modest recorded memory still points to memory as the cause. - **Using -resume as a fix.** Resume reuses cached successes, but it does not repair bad data, and editing a process script invalidates its cache by design. If a resumed run reruns everything, the likely cause is that inputs or parameters changed, not a Nextfault. Check the run banner and hash behavior before assuming a caching bug. - **Debugging in the work directory by editing files there.** Nextflow relies on work directory contents for cache integrity. Reproduce the failing command in a scratch directory instead, leaving the task directory untouched. - **Ignoring the attempt column.** Silent retries change the interpretation of every other metric. Duration and memory belong to the attempt in that row, which may not be the attempt that finally succeeded. - **Attributing every failure to the executor.** Before blaming the scheduler or the cluster, verify the tool itself did not fail. Empty `.command.err` files are common, and the real error often sits in `.command.out`. - **Unpinned containers.** When a pipeline suddenly fails after months of working, a floating container tag (for example a default tag that follows the latest release) is a frequent cause. Pin versions to make bioinformatics troubleshooting reproducible. - **Assuming the report explains the failure.** The report visualizes metrics, not messages. It narrows where to look, never what happened. The work directory remains the ground truth. One structural limitation runs through all of these tools. Individual pipeline failures vary with data, environment, and tool versions, so a fix that works for one run needs verification on the next. Treat each debugging session as producing a hypothesis and a config change, then confirm the change with a fresh trace on the following run. ## Frequently Asked Questions ### What is the difference between the Nextflow trace file and the HTML report? The trace file is a plain TSV with one row per task and raw values such as exit code, duration, and peak memory, suited to scripting and auditing. The HTML report visualizes the same underlying metrics as plots per process, suited to human review. Enable both at launch, since neither can recover metrics for a run that already finished without them. ### How do I find which command failed in my Nextflow pipeline? Read the end of `.nextflow.log` after the run, where Nextflow prints the failed task's process name, hash, and work directory path. Then open that task's directory and read `.command.err`, `.command.out`, and `.exitcode`. You can also list failed tasks for any past run with `nextflow log -F 'status == FAILED'`. ### Can I get trace and resource metrics for a run that already failed? Not for the original run if the trace was not enabled at launch, because the metrics are collected only while the run executes. The standard workaround is to rerun with `-resume` together with `-with-trace` and `-with-report`, which reuses all successful tasks from cache and re-executes only the failed ones, producing fresh records for exactly the tasks you need to inspect. ### What does exit code 137 mean in a Nextflow task? Exit code 137 means the task was terminated by signal 9, SIGKILL, following the Unix convention of 128 plus the signal number. On a scheduler this almost always indicates the out-of-memory killer, so the fix is usually more memory for that process or a retry strategy that escalates memory. For contrast, 139 indicates a segmentation fault and 143 a SIGTERM, often a scheduler timeout. ### How do I debug an RNA-seq error in Nextflow? Match the failure layer to the tool. A task that exits nonzero needs the work directory files to see the tool's message, which for RNA-seq pipelines commonly reveals reference index mismatches, swapped read mates, or memory kills during alignment. A process that never runs needs `view()` or `-dump-channels` to inspect the input channel, often exposing a sample sheet or glob pattern problem. A run that fails before any task needs `-preview` and `.nextflow.log`. ### Does the -resume option rerun everything? No. `-resume` reuses every task whose code, inputs, container, and settings are unchanged, and only re-executes failed tasks and their downstream dependents. If a resumed run reruns more than expected, the usual causes are an edit inside a process script, a changed input file, or resuming a different run ID than you intended.