# How to Read a Nextflow Log File Every Nextflow run writes a complete record of what it did, and every failed run writes an explanation. The catch is that the record lives in several files, and the explanation is easy to miss if you only read the last lines of the console. This guide teaches you how to read a Nextflow log file the way an experienced pipeline developer does. By the end of this article you will be able to: - Locate the runtime log, the run history, the trace file, and the per-task files in the work directory - Trace any failure from the high level error line down to the exact tool message that caused it - Interpret exit codes, memory fields, and timestamps - Fix the underlying problem, resume the run, and verify that the fix worked - Aggregate resource usage across runs so you can right-size future ones What you need on hand: Nextflow with a supported Java runtime, a terminal, and a launch directory containing at least one run (a failed run is the best practice material). A spreadsheet program, or R or Python, helps once you start reading trace tables. If you manage many runs and want dashboards instead of ad hoc scripts, a reporting database such as Nextpie can take over the aggregation [1]. The commands assume a bash shell on Linux or macOS, and they behave the same way under WSL on Windows. ## What a Nextflow Log File Is Three terms first, because the logs use them constantly. A pipeline (or workflow) is the whole program you wrote or downloaded, built out of processes. A process is a reusable step definition, such as aligning reads or counting genes. A task is one concrete execution of a process on one item of data, so an alignment process run over ten samples produces ten tasks. Nextflow executes tasks through an executor, which is either your local machine or a scheduler such as SLURM (the job scheduler common on HPC clusters) on a high performance computing system. Each task gets a private folder under the work directory, named `work` by default, where Nextflow stages its inputs and captures its outputs. The recording is thorough. Nextflow records the submission time, start time, completion time, CPU usage, memory usage, and disk usage of every task it runs, and it stores these records in log files that are often scattered across the file system rather than collected in one tidy place [1]. That scattering is why reading the logs well is a skill. No single file answers every question, but four of them together answer almost all of them. ## The Files That Hold the Story One naming trap before the list. The command `nextflow log` and the file `.nextflow.log` are two different things. The command prints the history of your runs. The file is the detailed narrative of one run. Confusing them wastes a lot of time. ### The Runtime Log: .nextflow.log Every `nextflow run` writes a narrative to `.nextflow.log` in the launch directory, which is the directory where you typed the command. This file contains every submission, every completion, every retry, and the full error block for every failed task. It is the file you will read most. The file rotates. When you start a new run in the same launch directory, the previous log becomes `.nextflow.log.1`, and older copies shift to `.nextflow.log.2` and beyond. If you are investigating yesterday's run, you may need a numbered backup rather than the current file. The console output you watched scroll past is a condensed, animated view. The file keeps the full sequence, including lines the console trims and the complete history of retries. ### The Run History Run level metadata (when each run started, how long it lasted, whether it succeeded) lives in the Nextflow home directory, which defaults to `~/.nextflow` and can be relocated with the `NXF_HOME` environment variable. You never open these records by hand. You query them with the `nextflow log` command, which prints the history table. ### The Work Directory Each task owns a folder addressed by a hash, a short string computed from the process definition, its inputs, and its configuration. The first two characters of the hash name the parent folder, so a task with hash `2b7c9d1e4f5a6` lives in `work/2b/7c9d1e4f5a6`. Nextflow stages inputs into that folder (as copies or symbolic links) before the script runs, then captures the results there. The folder holds: - `.command.sh`, the exact script the task executed - `.command.run`, the generated wrapper that set up the environment and ran the script - `.command.out` and `.command.err`, the task's standard output and standard error, the two text streams every Unix program writes - `.exitcode`, the numeric exit code the script returned - `.command.begin`, a timestamp for when the task started, and on many systems also `.command.env` and `.command.trace` When a task fails, these files are the ground truth. Everything else, including the error block in the log, is a copy of what is in here. ### Trace, Report, and Timeline Files Launch options add machine readable records. `-with-trace` writes `trace.txt`, a tab separated table with one row per task attempt. `-with-report` and `-with-timeline` write HTML summaries of resource use and timing. These files are optional, but running with `-with-trace` by default is one of the highest value habits in bioinformatics troubleshooting, because it turns log reading into table reading. ## Reading a Log Line by Line Open the runtime log and you will see lines like these: ``` Nov-11 09:14:05.772 [main] INFO nextflow.Session - Launching `main.nf` [mad_curie] DSL2 - revision: master Nov-11 09:14:12.004 [Submit-3] INFO nextflow.processor.TaskProcessor - Submitted process > `ALIGN` (sample_1) Nov-11 09:41:57.880 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > `SORT` (sample_1) ``` Each line has five parts: 1. A timestamp (`Nov-11 09:41:57.880`), in the local time of the machine running Nextflow 2. A thread name in square brackets (`[main]`, `[Submit-3]`, `[Task monitor]`) 3. A level: `DEBUG`, `INFO`, `WARN`, or `ERROR` 4. The logger class (`nextflow.processor.TaskProcessor`), which tells you which subsystem wrote the line 5. The message Levels in practice: `INFO` is normal progress. `WARN` flags something suspicious that did not stop the run. `ERROR` marks a failed task or an aborted session. Almost everything you need lives in `INFO` and `ERROR` lines. ### The Error Block The most valuable structure in the file is the error block. When a task fails, Nextflow appends a block that quotes the cause, the exact command, the exit status, the tail of the tool's stderr, and the work directory: ``` Nov-11 09:41:57.880 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > `SORT` (sample_1) Caused by: Process `SORT` (sample_1) terminated with an error exit status (137) Command executed: samtools sort -o sample_1.sorted.bam sample_1.bam Command exit status: 137 Command output: (empty) Command error: .command.sh: line 4: 4711 Killed samtools sort -o sample_1.sorted.bam sample_1.bam Work dir: work/2b/7c9d1e4f5a6 ``` Read the block top to bottom. The `Caused by` line is the wrapper level verdict. `Command executed` shows the script. `Command exit status` gives the tool's exit code, the numeric status a command returns when it ends, where 0 means success. `Command error` carries the tail of stderr, which here holds the operating system's `Killed` message. `Work dir` tells you where the full files live. The block also names the container and prints a tip about enabling per task debug output when one applies. ### Console Output Versus the Log File The console shows progress bars, percentages, and the final error block. The file keeps all of it, in order, including earlier attempts that the console overwrote with animation. When someone reports that a pipeline "crashed" and all they have is a screenshot of the console, the first thing to ask for is the `.nextflow.log` file. ## Nextflow Log File Analysis, Step by Step This walkthrough follows a debugging session from the outside in: history, log, task table, work directory, trace, fix, resume. Run every command from the launch directory. ### Step 1. Confirm Which Run You Are Looking At ``` nextflow log ``` You get the history table, one row per run: ``` Timestamp Duration Session ID Status Ramp-up % CPUs User % CPU Peak RSS Command 2025-11-11 09:14:02 28m 31s 1f2e3d4c5b6a7f8e9d0c1b2a3f4e5d6c ERR 0% 8 zk 14% 1.4 GB nextflow run main.nf -profile docker 2025-11-10 16:02:11 1h 12m 9a8b7c6d5e4f3a2b1c0d9e8f7a6b5c4d OK 0% 8 zk 11% 2.1 GB nextflow run main.nf -profile docker ``` `Status` is what matters here: `OK` for a clean run, `ERR` for a failed one. Column sets can vary slightly between Nextflow versions, so treat the printed header as the authority. ### Step 2. Read the End of the Runtime Log ``` tail -n 200 .nextflow.log ``` Failures sit near the end, but cleanup and shutdown messages follow them, so the last line is rarely the interesting one. Read upward until you reach the error block. ### Step 3. Pull Every Error and Warning With Line Numbers ``` grep -n -E "ERROR|WARN" .nextflow.log | tail -n 40 ``` ``` 412:Nov-11 09:41:57.880 [Task monitor] ERROR nextflow.processor.TaskProcessor - Error executing process > `SORT` (sample_1) ``` The line number lets you open the surrounding block directly: ``` sed -n '405,470p' .nextflow.log ``` ### Step 4. Get the Run Name ``` grep -m 1 "Launching" .nextflow.log ``` ``` Nov-11 09:14:05.772 [main] INFO nextflow.Session - Launching `main.nf` [mad_curie] DSL2 - revision: master ``` The run name is `mad_curie`. You need it for the task level commands in the next step. ### Step 5. List the Failed Tasks ``` nextflow log mad_curie -f 'status == FAIL' ``` This prints the task table for that run, filtered to failures. To control the columns yourself, pass a field list: ``` nextflow log mad_curie -f hash,name,status,exit,duration ``` ``` hash name status exit duration 2b7c9d1e4f5a6 SORT FAIL 137 21m 4s 8a3b1c9d2e4f5 ALIGN CACHED 0 0 ``` One row per task. The `hash` column is your address into the work directory. ### Step 6. Open the Work Directory ``` cd work/2b/7c9d1e4f5a6 ls -la ``` ``` .command.begin .command.err .command.out .command.run .command.sh .exitcode sample_1.bam -> /data/aligned/sample_1.bam ``` The staged input appears as a symbolic link back to its original location. The dot files are the task's record. ### Step 7. Read the Tool's Own Error ``` cat .exitcode less .command.err ``` ``` 137 ``` ``` .command.sh: line 4: 4711 Killed samtools sort -o sample_1.sorted.bam sample_1.bam ``` Exit code 137 means 128 plus 9, the signal number for SIGKILL, the operating system's force kill signal. In pipelines this is the out of memory signature: the kernel's OOM (out of memory) killer or the scheduler decided the task used too much RAM and terminated it. You can confirm the kernel side with the kernel's own log: ``` dmesg -T | grep -i "killed process" ``` ### Step 8. Quantify Resource Use With a Trace File If you launched with `-with-trace`, open `trace.txt`. The header row names every column, and the standard layout includes `task_id`, `hash`, `native_id`, `name`, `status`, `exit`, `submit`, `duration`, `%cpu`, `%rss`, `peak_rss`, `rchar`, and `wchar`, among others. Always confirm against your own header row before writing column numbers into scripts. Three definitions you will use constantly: - `peak_rss` is the peak resident set size, the maximum physical memory the task held - `%cpu` and `%rss` are averages over the task's lifetime, not peaks - `rchar` and `wchar` approximate the bytes the task read and wrote, which is how disk usage gets recorded per task [1] One more useful field: `native_id` is the task's process ID on the node, or its job ID on a scheduler, which is what you use to find the task in the scheduler's own accounting. To rank tasks by memory (column 13 in the standard layout): ``` awk -F'\t' 'NR>1 { mem = $13 + 0 if ($13 ~ / TB/) mem = mem * 1024 * 1024 if ($13 ~ / GB/) mem = mem * 1024 print mem " MB", $4, $5 }' trace.txt | sort -n | tail -n 10 ``` ``` 12288 MB SORT FAIL 9216 MB ALIGN COMPLETED 6144 MB ALIGN COMPLETED ``` Failed or killed tasks sometimes report a dash instead of a number, so treat missing values as part of the story rather than as a bug in your script. ### Step 9. Fix the Configuration and Resume For the out of memory failure above, raise the memory directive for that process in `nextflow.config`: ``` process { withName: SORT { cpus = 4 memory = 16.GB time = 2.h } } ``` Or let Nextflow escalate automatically across retries, where `task.attempt` counts attempts starting at 1: ``` process { withName: SORT { errorStrategy = { task.attempt <= 2 ? 'retry' : 'terminate' } memory = { 8.GB * task.attempt } } } ``` Then resume: ``` nextflow run main.nf -resume ``` Unchanged tasks are found in the cache and print as `Cached process >`, the fixed task reruns, and the closing summary reports how many tasks were cached. Resume works by recomputing each task's hash: only tasks whose inputs, script, or configuration changed, and everything downstream of them, actually run again. ### Step 10. Aggregate Across Runs One trace answers "what happened in this run". Many traces answer "what should I request by default". Hand parsing scattered log files becomes cumbersome quickly, and aggregating resource usage is what makes pipelines cheaper to run and easier to reproduce [1]. Nextpie was built for exactly this: it loads Nextflow resource records into a relational database behind a web interface, supports interactive visualizations of a pipeline's resource usage, and ships with an SQLite database of test data [1]. The code and documentation are at https://github.com/bishwaG/Nextpie [1]. ## Commands and Fields at a Glance | Command or file | What it gives you | Example | |, - |, - |, - | | `nextflow log` | Run history table with status and duration | `nextflow log` | | `nextflow log RUN -f 'status == FAIL'` | Only the failed tasks of one run | `nextflow log mad_curie -f 'status == FAIL'` | | `nextflow log RUN -f field,list` | Chosen columns for every task | `nextflow log mad_curie -f hash,name,status,exit,duration` | | `.nextflow.log` | Full narrative, including complete error blocks | `tail -n 200 . ## Comparing the Four Files: Which One Answers Which Question ![Comparison of the runtime log, run history, trace file and work directory by scope, best question answered and whether the record is machine readable.](/article-images/illustrations/nextflow-log-file-analysis-four-files-compared.webp) By now you have met several files that all claim to describe the same run. The fastest way to lose an hour of troubleshooting is to search the wrong one, so it pays to have a mental map of who answers what. | File or record | Scope | Best question it answers | Machine readable? | |, - |, - |, - |, - | | `.nextflow.log` | One run, chronological | Why did this task fail, and what did Nextflow itself do about it? | No, but greppable | | Run history via `nextflow log` | All runs from this launch directory | Which run am I looking at, and did it even finish? | Semi (table output) | | `trace.txt` | One run, one row per task attempt | Which tasks used the most memory, and how long did each take? | Yes, tab separated | | Work directory files | One task | What exactly did the tool print, and what code did it return? | The ground truth | | Report and timeline HTML | One run | Where did the wall clock time go across the whole pipeline? | No (visual) | A useful habit is to move outward in rings. The work directory tells you what a single task did. The runtime log tells you how Nextflow managed that task and everything around it. The trace file tells you how all tasks behaved statistically. The history tells you how runs behave over weeks. A question like "why did sample_7 fail" lives in the innermost ring, while "how much should I request for this process next month" lives in the outermost one, which is exactly the distinction that motivated tools that collect per task records into one queryable place rather than leaving them scattered across the file system [1]. A related point that surprises people: the HTML report and timeline are snapshots written at the end of the run. If your run died halfway through, these files may be missing or incomplete, but `trace.txt` is written incrementally as tasks complete, so it often survives crashes with partial content. When a run aborts, check for a trace file first; it may hold completed task statistics that the HTML files never got around to summarizing. ## Reading Exit Codes Like a System Administrator ![Decision path splitting a failed task by exit code: low codes send you to the tool's stderr, 128 and above to the kernel or scheduler record.](/article-images/illustrations/nextflow-log-file-analysis-exit-code-triage.webp) Exit codes are the most compressed diagnostic signal in the entire log, and they follow a convention inherited from Unix that is worth memorizing once. Exit status 0 means success. Small positive numbers, typically 1 through 127, are values the program itself chose to report failure. Everything from 128 upward means the process was terminated by a signal, and the signal number is the exit status minus 128. This is why the article's worked example read 137 as 128 plus 9: signal 9 is SIGKILL, the force kill that the kernel's OOM killer or a scheduler delivers when it decides a process must die immediately [1] is not involved here, it is purely the operating system acting. The signal family you will meet most often in pipelines: - 128 plus 9 equals 137: SIGKILL, the classic out of memory signature, or an explicit `kill -9` from an administrator or scheduler - 128 plus 15 equals 143: SIGTERM, a polite termination, often a scheduler canceling a job that overran its wall time request - 128 plus 11 equals 139: SIGSEGV, a segmentation fault, which usually means a genuine bug in the tool, corrupted input, or a binary compiled for the wrong architecture - 130: SIGINT, the interrupt you generate by pressing Ctrl C, worth recognizing so you do not debug your own finger - 126: the command was found but could not be executed, frequently a permissions problem or a broken container entrypoint - 127: the command was not found at all, which in Nextflow usually means a typo in the script block, a missing conda package, or a tool that never made it into the container The distinction between 1 to 127 and 128 upward matters because it splits your investigation in two. If the code is in the low range, the tool ran and complained, so `.command.err` will contain the tool's own error message and that message is authoritative. If the code is 128 or higher, the tool was killed externally, stderr may be nearly empty, and you must look outward to the kernel log or the scheduler record. The single most common mistake is treating 137 as if it were a samtools or STAR failure and Googling the tool name, when the tool never got to finish anything at all. One caveat: a few programs violate the convention and exit with values like 137 of their own accord. It is rare, but when the stderr shows a clean tool level message alongside a signal range exit code, trust the message over the arithmetic. ## Common Mistakes in Nextflow Log File Analysis Watching people learn this material reveals the same failures over and over. Each of the following has cost someone real time. **Grep from the bottom and stopping at the first hit.** Because `tail` shows you the end first, it is natural to search for `ERROR` and take the last match. But a run can contain many task failures across many retries, and the final error block may describe a downstream consequence, not the root cause. If the `ALIGN` process failed silently at hour one and everything downstream then failed loudly, the bottom of the log shows the loud failures. Always scan upward for the first ERROR and ask whether it could be the ancestor of the others. **Reading the console instead of the file.** The console is an animated, condensed view that overwrites progress lines. Screenshots of it lose retry history entirely. When a colleague says "the pipeline crashed" and sends a screenshot, the useful reply is: send me the `.nextflow.log` from the launch directory. **Forgetting the rotation.** If you reran the pipeline this morning, yesterday's failure now lives in `.nextflow.log.1`, and the current `.nextflow.log` describes a different, possibly successful run. Before investigating, confirm which run the file describes by checking the `Launching` line near the top, and compare the session identifier against the history table. **Trusting the peak versus average confusion.** The `%cpu` and `%rss` columns in a trace file are lifetime averages, not maxima. A task that idles for 40 minutes and then spikes will look modest in the average column and brutal in `peak_rss`. Sizing resources from the average column is a classic way to underprovision, because it hides exactly the spike that caused the kill. **Cleaning the work directory before diagnosing.** The work folder holds `.command.err`, `.exitcode`, and the staged inputs. Deleting it to "tidy up" before you understand the failure destroys the evidence, and depending on what changed, it can also break the cache that makes resume cheap. **Assuming a dash in the trace means the script is broken.** Killed tasks sometimes have no recorded peak memory, so trace cells contain a dash instead of a number. Parsers that crash on a dash, or analysts who delete those rows, throw away the very tasks that were the problem. Treat missing values as data about the kill, not noise. **Diagnosing in the wrong launch directory.** Nextflow writes `.nextflow.log` to the directory where you typed the command. If your team launches the same pipeline from different folders, there are multiple log files named identically on different paths, and the wrong one looks convincingly authoritative. Confirm the session identifier before believing any file. ## RNA-seq Error Walkthrough: From a Failed Alignment Task to a Working Resume RNA-seq pipelines concentrate failures in a few predictable places, so a second worked example, following the same outer to inner method, is worth the space. The scenario: a bulk RNA-seq pipeline with [FASTQ quality control](/blog/guides/fastq-quality-control-a-researcher-s-guide-to-read-level-diagnostics), alignment, sorting, quantification, and counting. Run name `jolly_lovelace`. **Step 1. Identify the run and the failing process.** ``` nextflow log ``` The history shows the most recent run with status `ERR`. Then: ``` nextflow log jolly_lovelace -f hash,name,status,exit,peak_rss ``` One row stands out: ``` hash name status exit peak_rss c4d5e6f7a8b9c0 ALIGN FAIL 137 - ``` The dash in `peak_rss` is itself informative: the task was killed so abruptly that the monitoring never recorded a final value. **Step 2. Read the error block in the runtime log.** ``` grep -n "Error executing" .nextflow.log | head ``` Find the block for `ALIGN` (sample_7) and read the `Command error` section. Here it is empty, which is typical of a SIGKILL: the tool did not get to print a farewell. This is the moment to resist the urge to search for STAR bugs. Go to the work directory instead. **Step 3. Confirm the kill in the work directory.** ``` cd work/c4/d5e6f7a8b9c0 cat .exitcode # prints 137 cat .command.err # empty or shows only the shell's Killed line cat .command.sh # shows the exact STAR invocation ``` **Step 4. Get external corroboration.** ``` dmesg -T | grep -i "killed process" ``` On a scheduler, the equivalent evidence comes from the job record; the `native_id` column of the trace file gives you the job identifier to look up. This cross checking, between what Nextflow recorded and what the underlying system recorded, is the pattern that separates guessing from diagnosis. It is also the pattern that aggregation tools formalize: rather than manually reconciling Nextflow's records with scheduler records run after run, a reporting layer pulls the per task resource records into one database where comparisons become queries instead of archaeology [1]. **Step 5. Distinguish the two usual RNA-seq memory stories.** Alignment against a large genome is generally the most memory demanding step of an RNA-seq workflow, because the aligner holds its compressed genome index in memory. If your `ALIGN` tasks are the ones dying with 137, and the trace shows they are also the tasks with the largest peaks among those that succeeded, the diagnosis is nearly certain. The fix is the same pattern shown earlier: raise the memory directive for the aligning process, or configure an escalating retry so the first attempt runs modest and the retries climb. **Step 6. Handle the second common RNA-seq failure mode.** Suppose instead the trace points at `COUNT`, and `.command.err` ends with the counting tool reporting that it cannot open the genome annotation file. Exit code 1, low memory, a real tool message. Now the problem is upstream data flow: the GTF file either was not staged into the work directory, was staged from the wrong path, or the parameter pointing at it was wrong for this sample set. Check the staged files with `ls -la` in the work directory, confirm the symlink target exists, and check the `params` block in the log for the value actually used. A very common RNA-seq variant of this: the annotation and the genome index disagree about chromosome naming, and the failure appears not as an error at all but as a count table of zeros, which is why reading per task output files, not just errors, belongs in your routine. **Step 7. Fix, resume, and verify.** Apply the configuration change, run with resume, and then verify rather than assume. The closing summary reports how many tasks were cached; the log should show `Cached process >` for everything unchanged and a fresh submission for `ALIGN`. Then re-run the trace and confirm the previously failing task completed, and that its peak sits comfortably below what you now request. ## Debugging on an HPC Cluster: Reading Scheduler Logs Alongside Nextflow Logs On a cluster, every task passes through two systems with two independent record keeping habits, and reading only one of them gives you half the truth. Nextflow's records tell you what Nextflow asked for and what it observed. The scheduler's records tell you what the scheduler actually granted, where the job ran, and why it ended. Discrepancies between the two are where cluster specific failures hide. Start from the trace file's `native_id` column, which holds the scheduler job identifier for tasks run through a scheduler executor. With that identifier in hand, SLURM systems offer two lookups. `scontrol show job JOBID` shows a live job's full definition, including the node it ran on and the resources requested. For finished jobs, the accounting command is the one to learn properly: ``` sacct -j JOBID -o JobID,State,Elapsed,MaxRSS,AllocCPUS,NodeList ``` The `State` column is gold. States beginning with `OUT_OF_MEMORY` or `TIMEOUT` confirm a resource story before you even open `.command.err`. A `CANCELLED` state with an administrator's name attached tells you someone's maintenance script killed your job. A `NODE_FAIL` means the hardware died, the task is not your fault, and the fix is simply to rerun, not to reconfigure anything. There is one cluster specific trap worth naming. On systems where the scheduler enforces memory limits per step, a task that exceeds its allocation is killed by the scheduler rather than the kernel, and the kill evidence appears in the scheduler's own messages rather than in `dmesg` on the node. The practical symptom is a 137 with an empty `.command.err` and nothing in the kernel log you can see. When the kernel log shows nothing, escalate to the scheduler accounting before concluding the evidence does not exist. Queue time is the other thing cluster logs reveal that local logs cannot. In the trace file, compare the `submit` and `start` columns per task. A long gap means the task sat in the scheduler queue, which is not a performance problem in your code but can be a scheduling problem in your request: asking for more CPUs than the partition typically has free, for example, lengthens every queue wait. The timeline HTML file visualizes exactly these gaps across the whole run, which makes it the fastest way to see whether a slow pipeline is slow because of compute or because of waiting. Finally, resist reading scheduler emails as if they were Nextflow errors. They describe the job wrapper's fate, and the wrapper's fate usually mirrors the task's fate, but the authoritative per task record remains the work directory files. Use the scheduler to explain the kill, and use the work directory to explain the computation. ## Using DEBUG Level Output and Verbose Logging The console shows INFO and above, but the `.nextflow.log` file is richer than the console by default: it includes DEBUG level lines, the diagnostic detail that the animated console hides. When a failure is not in a task but in Nextflow's machinery itself, such as staging oddities, executor communication problems, or cache and resume questions, the DEBUG lines are often the only place the mechanism is described. Search them deliberately: ``` grep -n "DEBUG" .nextflow.log | tail -n 50 ``` Around a failed submission you will typically find DEBUG lines describing the exact staging decisions, the constructed command wrapper, and the executor interaction. Reading these is how you answer questions the error block never poses, such as why an input file was staged as a copy rather than a symlink, or which environment variables were exported into the task. Three habits make verbose logs more usable. First, give each run its own log file with the `-log` option, for example `nextflow run main.nf -log run-sample7.log`, so investigations never depend on rotation position. Second, when running in scripts or continuous integration, disable the animated console with the `NXF_ANSI_LOG` environment variable set to `false`, which produces a plain, greppable stream instead of a redrawn display. Third, when you file an issue with a pipeline or with Nextflow itself, the maintainers will ask for the full log file, not the console excerpt, for exactly this reason: the DEBUG layer is where their interest lives. One more record deserves a mention here. Inside the work directory, some executions also leave a per task trace file recording resource usage observed from inside the task wrapper. When the outer trace file reports a dash for a killed task, this inner file is a second chance at the numbers, and it is one more reason the work directory is described as ground truth throughout this guide. ## Scripting the Trace File: Robust Parsing Recipes Once you always run with a trace file, the file becomes a dataset, and ad hoc awk one liners eventually give way to a small parser you reuse. Two properties of `trace.txt` should shape that parser: fields are tab separated, and both missing values (dashes) and unit suffixed numbers (MB, GB, TB) appear in the data. A compact, robust reader in Python: ``` import csv, sys def to_mb(value): if value is None or value in ("-", ""): return None value = value.strip() mult = 1.0 for suffix, factor in ((" TB", 1024*1024), (" GB", 1024)): if value.endswith(suffix): value, mult = value[: -len(suffix)], factor break try: return float(value) * mult except ValueError: return None with open("trace.txt") as handle: rows = list(csv.DictReader(handle, delimiter="\t")) ranked = sorted( (r for r in rows if r.get("peak_rss")), key=lambda r: to_mb(r["peak_rss"]) or 0, reverse=True, ) for r in ranked[:10]: print(r["name"], r["status"], r["peak_rss"]) ``` Note what this script gets right that naive column indexing does not: it reads the header row to map names to columns, so it survives the column set varying between Nextflow versions; it converts suffixed units to a common scale before comparing; and it silently drops or tolerates dashes rather than crashing on them. A second recipe worth keeping is the queue time profile. Using the `submit` and `start` columns, compute the gap per task and rank it. On a scheduler, this tells you in one command whether your wall clock pain is compute or waiting: ``` awk -F'\t' 'NR>1 && $8 != "-" && $9 != "-" { print $9 - $8, $4 }' trace.txt | sort -rn | head ``` Confirm the column positions against your own header row before running this; the draft's earlier warning about treating your header as the authority applies to every script you ever write against trace files. For teams running many pipelines and many runs, hand rolled parsers eventually become a maintenance burden, which is the point where loading the records into a relational database behind a reporting interface starts to pay for itself [1]. The boundary is roughly this: parse by hand while you are diagnosing, aggregate in a database once you are budgeting. ## Common Misconceptions About Nextflow Logging Several beliefs circulate that sound plausible and cost people real diagnostic time. Correcting them explicitly is faster than discovering each in production. **"The last line of the log is the cause."** Usually false. Shutdown and cleanup messages follow the error, and multi failure runs bury the root cause higher up. Read upward and look for the earliest error that could explain the later ones. **"Cached means the outputs are still on disk."** Cached means the task's hash matched something in the cache and Nextflow did not rerun it. If you have cleaned the work directory or moved it, cached status in a future run is not guaranteed, and a run that reports fewer cached tasks than you expected is telling you that some inputs changed. Treat the cached count as a fingerprint of what the run considers unchanged. **"Editing the memory directive reruns only that task."** It changes the task's hash, which reruns that task and everything downstream of it, but exactly which configuration values participate in the hash has varied across Nextflow versions. The honest procedure is empirical: make the change, resume, and read the log's cached versus submitted counts. If more reran than you expected, the log has told you something true about your version's hashing that no amount of assuming would have. **"A successful run needs no log reading."** The most valuable reading often happens on green runs. Comparing peak memory against requested memory across successful tasks is how you right-size future runs and stop paying for headroom you never use, and it is the entire premise of aggregating resource records across runs into one place [1]. **"The trace file will be there when I need it."** Only if you requested it. `trace.txt`, the report, and the timeline exist only for the run that launched with those flags. After a crash you cannot retroactively generate them. This is why running with a trace by default is described elsewhere in this guide as a high value habit. **"The work directory is disposable."** It holds both the evidence and the cache. Cleanup commands exist, and a dry run flag makes them show what they would remove before removing anything, but treat the work directory as live data until the run is fully diagnosed, published, and archived. ## Resume and Cache: What the Log Tells You About What Will Rerun Resume is where log reading earns its keep, because the log is the only place that shows the cache verdict per task. After a resumed run, three log lines matter. `Cached process >` means the hash matched and nothing executed. A fresh `Submitted process >` line for a task you expected to be cached is a signal that something changed: an input file's content, a parameter value, the script block, the container reference, or a configuration value that feeds the hash. A task you expected to rerun but that shows as cached means the opposite: your fix did not actually reach the hash, perhaps because you edited a config file the run is not reading, or edited a directive in a scope that does not apply to that process. Reading these three lines converts resume from a superstition into a verification step. To see the verdicts in table form rather than prose, query the finished run's task table filtered by status, and compare the `CACHED`, `COMPLETED`, and `FAIL` rows against your expectation. The duration column for cached rows is effectively zero, which makes them easy to spot even in unfiltered output. Two resume behaviors deserve a warning. First, changing an input file's contents changes the hash of every task that consumes it and everything downstream, so a small correction to one sample's FASTQ quietly reruns a large fraction of the pipeline. That is correct behavior, but people are routinely surprised by the scale. Second, the closing summary's cached count refers to this session's tasks; a resumed run that failed early can report mostly cached tasks and still be a failure. Read the summary's status, not just its counts. Finally, resume only works against the same session. Starting a run in a fresh directory, or after moving the work directory, produces a run with no cache to consult, and the log will show every task submitted fresh. If you see that after intending a resume, check the launch directory and work directory paths first. ## A Troubleshooting Checklist Condense everything above into an order of operations you can follow under pressure. 1. Confirm the run. `nextflow log`, check status and duration, and match the session identifier against the `Launching` line in `.nextflow.log`. Check for rotation (`​.nextflow.log.1` and beyond) if the failing run is not the latest. 2. Find the first meaningful error. Grep for `ERROR` with line numbers, take the earliest plausible one, and open the surrounding block with `sed`. 3. Extract the exit code. Low codes mean read `.command.err` in the work directory; signal range codes mean look at the kernel log or scheduler accounting using `native_id`. 4. Open the work directory. Read `.command.sh`, `.command.err`, and `.exitcode`. Verify staged inputs actually exist and point where they claim. 5. Corroborate externally. `dmesg`, scheduler records, or the per task trace inside the work directory. Never conclude from a single source that a task was killed for memory. 6. Check the pattern, not just the incident. Open `trace.txt`, rank by peak memory, and ask whether the failing process is also the heaviest among survivors. One killed task is an incident; a process that consistently peaks near its request is a configuration problem. 7. Fix at the correct layer. Tool level messages lead to data or software fixes. Signal level kills lead to resource directives. Scheduler states lead to submission policy. 8. Resume and verify. Read the cached versus submitted counts, confirm the previously failing task completed, and rerun the trace to confirm the new peaks sit below the new requests. 9. Record what you learned. Note the process, the old and new resource requests, and the evidence. Over multiple runs, these notes become the default resource profile, and if you accumulate enough of them, the point arrives where per task records belong in a queryable store rather than in scattered files and memory [1]. 10. Only then clean up. With the diagnosis written down and the outputs published, work directory cleanup is safe, and a dry run flag lets you preview it. The skill this checklist encodes is not really about Nextflow at all. It is the habit of moving from the loudest evidence to the quietest, of trusting raw task records over summaries, and of writing down resource behavior so future runs inherit your conclusions instead of repeating your debugging. Every file Nextflow writes exists to support that habit; the log is only unfriendly until you know which ring of the record to open first. ## Sources 1. [Nextpie: a web-based reporting tool and database for reproducible nextflow pipelines.](https://pubmed.ncbi.nlm.nih.gov/41180966/)