# Snakemake Rerun Triggers: Step-by-Step Guide Snakemake is a widely used bioinformatics workflow manager. It reads a Snakefile, builds a directed acyclic graph (DAG, a network of jobs where arrows point from inputs to outputs and cycles are impossible), and runs the jobs in dependency order. When you invoke it a second time, it neither blindly reruns everything nor blindly trusts existing outputs. It compares the current state of the workflow against metadata it stored during the last run. The properties it compares are called rerun triggers. This guide walks through every trigger with a small working project. By the end you will be able to predict which jobs will rerun before you execute anything, read the reason lines in a dry run, narrow or widen the trigger set with the rerun-triggers option, and diagnose a workflow that reruns everything on every invocation. The same walkthrough doubles as a set of practical tips for keeping large workflows honest, and the closing section compares the design with Nextflow resume, a common point of confusion when teams pick between the two tools. You need a terminal and Snakemake 7.8 or newer. The commands below were checked against the 9.x series and work on any release from 7.8 onward. Install Snakemake with conda or pip. A conda installation makes the software environment demo in Step 9 easier, but every other step needs nothing beyond Snakemake and standard shell tools. Budget about 20 minutes. ## What a Rerun Trigger Is ![Decision path from missing output through incomplete record and enabled triggers to a scheduled rerun or an up-to-date job.](/article-images/illustrations/snakemake-rerun-triggers-rerun-decision.webp) A rerun trigger is a property of a job that Snakemake recorded in its metadata and compares again on every invocation. If any enabled trigger differs from the stored value, Snakemake schedules the job again, even though the output file exists. Jobs whose outputs are missing always run, regardless of triggers. The metadata lives in the project directory under `.snakemake/metadata`. It is a set of JSON files, one per output file, each recording: - The source code of the rule that produced the file. - The evaluated values of the rule's params. - The paths of the input files and their modification times. - The software environment definition (a conda environment file or a container). - Whether the job finished, stored as an incomplete flag. Deleting `.snakemake/metadata` deletes this memory. Nothing else in the project holds it. On each invocation, Snakemake walks the DAG and marks a job for execution when the output is missing, when the stored record marks it incomplete, or when any enabled trigger fires. Forced runs are the exception: the forceall and forcerun options execute jobs regardless of triggers. Current releases enable five triggers by default: mtime, params, code, input, and software-env. A sixth trigger, metadata, is opt-in. The default set changed in Snakemake 7.8 (2022), which is why advice written before that date describes different behavior. The command line help prints the exact defaults for your installed version: ```bash snakemake --help | grep -A3 rerun-triggers ``` Check this once on every cluster you use, because the version installed there decides what your reruns mean. ## The Six Rerun Triggers The official Snakemake documentation defines the trigger names accepted by the rerun-triggers option. Each trigger records one property of a job and fires when that property changes. Here is what each one does, with a bioinformatics example. ### mtime The mtime (modification time) is the timestamp the filesystem stores for each file and updates on every write. Snakemake records the mtime of every input file when a job finishes and compares the stored values with the current ones on the next run. A job also reruns when its output file is older than any of its input files on disk, the classic make-style check. The trigger fires when an input file changed since the job last ran. Example: you replace `reference/genome.fa` with a new release, so every job that used the old FASTA (the standard text format for a reference genome) reruns. The mtime trigger is the one most often fired by accident. Copying files without preserving timestamps, checking out a different git branch, and cloud sync clients all rewrite mtimes without changing content. The fixes appear in the mistakes section below. ### params Params are named values attached to a rule, usually computed from the config. Snakemake evaluates params every time it builds the DAG and stores the resulting values in the metadata. The trigger fires when a stored value differs from the fresh evaluation. Example: `min_quality: 20` in `config.yaml` becomes `min_quality: 25`. Every job whose params include that value reruns, and no other job does. This is the main route through which config edits propagate into reruns, which is why the params block deserves care (see the mistakes section). ### code The code trigger records the source of the rule body: the shell command string, the Python code in a run block, and, in recent releases, the contents of external scripts referenced with the script directive. It fires when the current definition differs from the stored one. Example: you change a filtering threshold inside a shell command. Every job of that rule reruns, because the recorded command no longer matches the file on disk. Only the rule body is recorded. Code in helper functions called from the body, or in modules imported by the Snakefile, is not tracked. A useful consequence: reverting an edit restores the match, and the rerun request disappears. ### input The input trigger records which files a job consumed, independent of their timestamps. It fires when the set of input files changes. Example: you add sample SRR3 to the config. The aggregate rule (usually called all) now lists one more file, so its input set changed and it reruns. A new job runs for SRR3. Jobs for SRR1 and SRR2 keep their existing outputs, because their own input sets did not change. This per-job scope is what keeps large workflows cheap to extend. ### software-env The software-env trigger records the software environment definition of the job: the contents of the conda environment file referenced with the conda directive, or the container image referenced with the container directive. It fires when the definition differs. Example: you add samtools to `envs/align.yaml`. Every job that uses that file reruns. The comparison is against the definition, not the installed result, so editing the file reruns jobs even if the resolved packages end up identical. Pin container images by digest rather than by a moving tag such as latest, because a tag that silently points to new content leaves the recorded definition unchanged and hides the change. ### metadata The metadata trigger is opt-in. It fires when the stored record for an output is missing or marked incomplete. Use it when the `.snakemake` folder was lost, when outputs were produced on another machine or by an older version of the workflow, or when you want a clean rebuild. Interrupted jobs are marked incomplete and rerun on the next invocation regardless of the trigger set. The metadata trigger covers the opposite case, records that are simply gone. ## Snakemake Rerun Triggers, Step by Step The walkthrough uses a tiny sequencing-style project that counts the reads in two FASTQ files (FASTQ is the standard text format for raw sequencing reads). Every command runs locally in seconds. Printed outputs are representative and trimmed, and the exact wording of reason lines varies slightly between releases. ### Step 1: Install and check the trigger defaults ```bash conda create -n snakemake -c conda-forge -c bioconda snakemake conda activate snakemake snakemake --version ``` Expected output: ``` 9.6.0 ``` Any release from 7.8 onward supports the full trigger set. Confirm the defaults on your version: ```bash snakemake --help | grep -A3 rerun-triggers ``` Expected output (abridged): ``` --rerun-triggers {mtime,params,code,input,software-env,metadata} ... Define which parameters of a job shall be used to decide whether a job has to be rerun. Default: mtime, params, code, input, software-env ``` ### Step 2: Create the demo project ```bash mkdir -p demo/data demo/envs cd demo printf '@r1\nACGT\n+\nIIII\n@r2\nACGT\n+\nIIII\n' > data/SRR1.fastq printf '@r1\nACGT\n+\nIIII\n' > data/SRR2.fastq ``` Create `config.yaml`: ```yaml samples: - SRR1 - SRR2 min_quality: 20 ``` Create `Snakefile`: ```python configfile: "config.yaml" SAMPLES = config["samples"] rule all: input: expand("results/{sample}.counts.txt", sample=SAMPLES) rule count_reads: input: "data/{sample}.fastq" output: "results/{sample}.counts.txt" params: min_quality=config["min_quality"] shell: "mkdir -p results && " "grep -c '^@' {input} > {output} && " "echo {params.min_quality} >> {output}" ``` The rule counts lines that start with the @ sign (one per read) and appends the quality threshold from the config. The project holds a Snakefile, a config.yaml, two FASTQ files in data/, and an empty envs/ folder used in Step 9. ### Step 3: Run the workflow ```bash snakemake --cores 1 ``` Expected output (trimmed): ``` Building DAG of jobs... Job stats: job count all 1 count_reads 2 total 3 [Sat Nov 11 10:00:00 2026] rule count_reads: input: data/SRR1.fastq output: results/SRR1.counts.txt jobid: 1 reason: Missing output files: results/SRR1.counts.txt [Sat Nov 11 10:00:00 2026] rule count_reads: input: data/SRR2.fastq output: results/SRR2.counts.txt jobid: 2 reason: Missing output files: results/SRR2.counts.txt 3 of 3 steps (100%) done ``` Snakemake 8 and newer require the cores flag. On 7.x, a plain invocation works. Run it again: ```bash snakemake --cores 1 ``` Expected output: ``` Nothing to be done (all requested files are up to date). ``` That sentence is the trigger system working. Every job matched its stored record. ### Step 4: Inspect the stored metadata ```bash ls .snakemake/metadata ``` Two JSON files appear, one per output file, with hashed names. Open one and you will find the rule code, the evaluated params (min_quality set to 20), the input path with its mtime, and the incomplete flag set to false. This record is the entire basis of every rerun decision for that file. The summary command reports the same state per file: ```bash snakemake --summary ``` Expected output (columns trimmed): ``` output_file date rule log status plan results/SRR1.counts.txt Sat Nov 11 10:00 2026 count_reads - ok no update results/SRR2.counts.txt Sat Nov 11 10:00 2026 count_reads - ok no update ``` ### Step 5: Change a param (params trigger) Edit `config.yaml` and set `min_quality: 25`. Inspect before running: ```bash snakemake --cores 1 --dry-run --reason ``` Expected output (trimmed): ``` rule count_reads: input: data/SRR1.fastq output: results/SRR1.counts.txt jobid: 1 reason: params have changed since last execution rule count_reads: input: data/SRR2.fastq output: results/SRR2.counts.txt jobid: 2 reason: params have changed since last execution ``` Both count_reads jobs are selected. The all rule is not, because it has no params of its own. The list commands isolate one trigger at a time: ```bash snakemake --list-params-changes ``` Expected output: ``` results/SRR1.counts.txt results/SRR2.counts.txt ``` Run for real, then verify the result is right: ```bash snakemake --cores 1 cat results/SRR1.counts.txt ``` Expected output: ``` 2 25 ``` Run the list command again. It now prints nothing, because every job matches its stored params. ### Step 6: Edit a rule body (code trigger) Change the last line of the shell block in the Snakefile to: ```python "echo min_quality={params.min_quality} >> {output}" ``` A dry run now reports the code trigger: ```bash snakemake --cores 1 --dry-run --reason ``` Expected reason line for both count_reads jobs: ``` reason: code has changed since last execution ``` The matching list command narrows to the same files: ```bash snakemake --list-code-changes ``` Now undo the edit, restoring the original echo line, and dry run again. The rerun request disappears, because the stored code matches the current code once more. Reverting an edit un-does the trigger, which is a quick way to test whether a change you made is the one causing a rerun. ### Step 7: Add a sample (input trigger) ```bash printf '@r1\nACGT\n+\nIIII\n' > data/SRR3.fastq ``` Add `SRR3` to the samples list in `config.yaml`, then: ```bash snakemake --cores 1 --dry-run --reason ``` Expected output (trimmed): ``` rule count_reads: input: data/SRR3.fastq output: results/SRR3.counts.txt jobid: 3 reason: Missing output files: results/SRR3.counts.txt localrule all: input: results/SRR1.counts.txt, results/SRR2.counts.txt, results/SRR3.counts.txt jobid: 0 reason: input files have changed since last execution ``` Two jobs are selected and two are not. The new sample has no output yet, and the all rule consumes a different file set. The jobs for SRR1 and SRR2 match their records exactly, so their outputs stand. Run for real to produce the third count file. ### Step 8: Touch an input (mtime trigger) ```bash touch data/SRR1.fastq snakemake --cores 1 --dry-run --reason ``` Expected output (trimmed): ``` rule count_reads: input: data/SRR1.fastq output: results/SRR1.counts.txt jobid: 1 reason: updated input files: data/SRR1.fastq ``` The touch command rewrote the timestamp without changing a byte, and the mtime trigger noticed. Now narrow the trigger set and dry run again: ```bash snakemake --cores 1 --dry-run --rerun-triggers params code input software-env ``` Expected output: ``` Nothing to be done (all requested files are up to date). ``` With mtime excluded, the timestamp change is invisible. Treat this as a scalpel, not a default. If an input's content changes while mtime is excluded, Snakemake will not notice. The gentler fixes for scrambled timestamps are to copy projects with rsync -a (which preserves times) or to wrap stable inputs in the ancient() function, which excludes them from mtime checks permanently. ### Step 9: Change the software environment (software-env trigger) Create `envs/count.yaml`: ```yaml channels: - conda-forge - bioconda dependencies: - coreutils ``` Add a conda directive to the count_reads rule, directly under the output block: ```python conda: "envs/count.yaml" ``` Dry run: ```bash snakemake --cores 1 --dry-run --reason ``` Expected reason line for both count_reads jobs: ``` reason: software environment definition has changed since last execution ``` The dry run detects the change even without conda installed, because the trigger compares the recorded definition with the file contents. When you run for real with the use-conda flag, Snakemake resolves the environment before executing. ### Step 10: Lose the metadata, then use the metadata trigger ```bash rm -r .snakemake/metadata snakemake --cores 1 --dry-run ``` Expected output: ``` Nothing to be done (all requested files are up to date). ``` With the records gone, Snakemake falls back to comparing file timestamps on disk. The outputs are newer than the inputs, so it reports up to date. That verdict says nothing about code or params changes made since the outputs were built. Force an honest rebuild with the opt-in trigger: ```bash snakemake --cores 1 --dry-run --rerun-triggers metadata ``` All three jobs are selected now, because none of them has a record. Run for real, and the metadata folder is repopulated. Note that `-rerun-triggers metadata` alone disables the other triggers for that run, so use it for a one-off rebuild rather than as a standing setting. Two blunt instruments sit next to the trigger system. The forceall flag reruns every job in the DAG regardless of triggers. The forcerun flag takes rule names and reruns every job of those rules: ```bash snakemake --cores 1 --forceall --dry-run snakemake --cores 1 --forcerun count_reads --dry-run ``` ## Rerun Trigger Reference Table 1 summarizes the six triggers and the command that inspects each one. | Trigger | What Snakemake records | Fires when | Inspect with | |--|--|--|--| | `mtime` | Modification times of the job's input files | An input file is newer than its stored value, or newer than the output on disk | `snakemake, dry-run, reason` | | `params` | Evaluated param values | Any stored value differs from the fresh evaluation | `snakemake, list-params-changes` | | `code` | Rule body source, including external script contents in recent releases | The definition differs from the stored one | `snakemake, list-code-changes` | | `input` | The set of input files | Files are added to or removed from the job | `snakemake, list-input-changes` | | `software-env` | Conda environment file contents or container definition | The definition differs from the stored one | `snakemake, list-software-stack-changes` | | `metadata` | Presence and completeness of the record itself (opt-in) | The record is missing or marked incomplete | `snakemake, dry-run, rerun-triggers metadata` | Table 2 collects the control commands used in the walkthrough. | Command | Effect | |--|--| | `snakemake, cores 1, dry-run, reason` | List jobs that would run, with reasons | | `snakemake, cores 1, summary` | Per-file status table (ok, params changed, code changed, and so on) | | `snakemake, cores 1, rerun-triggers mtime` | Restrict rerun decisions to timestamps only (pre-7.8 behavior) | | `snakemake, cores 1, forceall` | Rerun every job regardless of triggers | | `snakemake, cores 1, forcerun count_reads` | Rerun every job of one rule | | `snakemake, cores 1, touch` | Mark outdated outputs as current without running commands | | `snakemake, cores 1, cleanup-metadata results/SRR1.counts.txt` | Delete the stored record for one output | To make a trigger set the standing default for a project, put it in a profile. Create `profiles/default/config.yaml`: ```yaml rerun-triggers: - mtime - params - code - input - software-env ``` Then run with `snakemake, profile profiles/default`. Any flag shown in this article can live in a profile, which keeps cluster invocations short and makes every team member rerun under the same rules. From here, add a dry run as a preflight step before cluster submissions, and read the rerun trigger section of the official Snakemake documentation for your installed version. ## Common Mistakes and Limitations 1. **Nondeterministic params rerun everything forever.** A params function that reads the clock, draws a random value, or reads a frequently updated file produces a new value on every DAG build, so the params trigger fires on every invocation. Fix: compute such values once, store them in the config, or seed randomness. `snakemake, list-params-changes` names the affected files within seconds. 2. **Config values used directly in shell strings escape the trigger system.** Writing `{config[min_quality]}` inside a shell command records nothing. Editing the config then changes nothing from Snakemake's point of view, because neither the params nor the code record changes. Fix: route every config value through the params block, as the demo workflow does. 3. **Params functions that read files without declaring them as inputs.** The evaluated value is recorded, but the file itself is not tracked, so a silent update to that file can slip through if the evaluated value happens to stay the same. Run `snakemake, list-untracked-params-files` to find these cases, then declare the files as inputs. 4. **Helper functions and imported modules are invisible to the code trigger.** Only the rule body is recorded. If a rule body calls `my_filter()` and you edit the function, no rerun fires. Fix: keep per-rule logic in the rule body or in scripts referenced with the script directive, which recent releases track. 5. **Timestamps scrambled by copies, checkouts, and syncs.** rsync without -a, git checkout, and most cloud sync clients rewrite mtimes, so the mtime trigger selects jobs whose inputs did not really change. Fix: preserve times when moving projects, or narrow the triggers for one run as in Step 8. 6. **Moving container tags.** A container directive pinned to a tag such as latest produces an identical recorded definition while the image behind the tag changes. Fix: pin by digest. 7. **Deleting `.snakemake` to reset a project.** The folder holds every rerun record. Deleting it throws away the workflow's memory, and the next run either reruns everything (with the metadata trigger) or trusts on-disk timestamps (with the defaults). Fix: back the folder up with the project, and use `-cleanup-metadata` when you want one file forgotten deliberately. 8. **Using `-touch` as a habit.** Touching marks outputs as current without running anything. Used once, after verifying the outputs are actually correct, it is a legitimate repair. Used to silence reruns you have not diagnosed, it converts a real problem into a silent one. 9. **Commands that write to the wrong path.** If a job exits cleanly but the output file is absent, the next invocation fails with a message in this form: ``` MissingOutputException in rule count_reads in file /path/Snakefile, line 14 Job completed successfully, but some output files are missing: results/SRR1.counts.txt ``` Fix: check that the command writes to `{output}` and that nothing in the command renames or relocates the file. Do not repair this with `-touch`, which would create an empty stand-in. 10. **Locks after crashed runs.** A killed Snakemake leaves a lock, and the next invocation fails: ``` Directory cannot be locked. Please make sure that Snakemake is not running, or wait for it to finish. If you are sure that no other instances of snakemake are running on this directory, the remaining lock was likely caused by a kill event or a system crash. It can be removed with the --unlock argument. ``` Fix: confirm nothing is running, then `snakemake, unlock`. Interrupted jobs are marked incomplete and rerun automatically, which is the behavior you want after a crash. 11. **Old advice describes old defaults.** Before Snakemake 7.8, only mtime was enabled, so forum answers from that era recommend flags and describe behavior that no longer matches. Check `snakemake, help` on the version your cluster actually runs. 12. **Triggers compare definitions, not outcomes.** Editing a comment inside a rule body reruns the job. A conda env edit reruns jobs even if the resolved packages end up identical. On shared filesystems, mtime granularity and clock skew between nodes can both fire false reruns and hide real changes. The system trades a small number of false reruns and rare misses for the ability to track definitions at all. Budget for both when scheduling large reruns on a cluster, and prefer definition-level triggers (params, code, software-env) for anything that affects results. ## Nextflow vs Snakemake: Two Ways to Decide What to Rerun ![Comparison table of Snakemake rerun triggers and Nextflow resume across decision basis, storage location and per-run control.](/article-images/illustrations/snakemake-rerun-triggers-nextflow-vs-snakemake.webp) Teams choosing a bioinformatics workflow manager often ask how rerun behavior differs, because it shapes daily work more than syntax does. Snakemake stores per-output metadata inside the project and compares selectable triggers. You decide what counts as a change, per invocation, with the rerun-triggers option. The records travel with the project folder, and they vanish if you delete it. Nextflow takes a different approach. With the resume flag, each task is identified by a hash, a fixed fingerprint computed from the task's inputs, script, container, and parameters. If a task with the same hash has a cached result in the work directory, Nextflow reuses it and skips execution. Reuse is all or nothing per task: change any hashed component and the hash changes. Practical consequences follow from those designs: - Snakemake lets you say "ignore timestamps this once". Nextflow's cache is tied to task identity, with no per-run dial for which components count. - Snakemake's records live in `.snakemake`, Nextflow's cache in the work directory. Both are heavy folders to back up or deliberately drop. - Both integrate conda environments and containers, and both are mature, widely deployed tools. The choice usually follows team language (Python versus Groovy) and whether you write pipelines or run existing ones from collections such as nf-core. ## Frequently Asked Questions ### What are the default rerun triggers in Snakemake? Current releases enable mtime, params, code, input, and software-env by default, and the metadata trigger is opt-in. Because the defaults changed in Snakemake 7.8, run `snakemake, help` and read the rerun-triggers entry on the version you actually use. ### How do I find out why Snakemake wants to rerun a job? Run a dry run with reasons enabled: `snakemake, cores 1, dry-run, reason`. Each selected job prints a reason line such as "params have changed since last execution". The list commands (`-list-params-changes`, `-list-code-changes`, `-list-input-changes`, `-list-software-stack-changes`) isolate one trigger at a time. ### Where does Snakemake store rerun metadata? In `.snakemake/metadata` inside the project directory, as one JSON record per output file. Deleting that folder deletes the workflow's memory of every job. ### How do I rerun just one rule or one file? Use `-forcerun` with the rule name to rerun every job of that rule, or `-cleanup-metadata` with an output path to delete its record so the next run rebuilds it. `-forceall` reruns everything. ### Why does my rule rerun on every invocation? The usual cause is a param that evaluates to a new value each time, such as a timestamp or a random draw. Confirm with `-list-params-changes`, then make the value deterministic. ### Do rerun triggers cover conda environments and containers? Yes. The software-env trigger records the conda environment file contents or the container definition and fires when it changes. Pin container images by digest so a moving tag cannot hide a change. ### How do I stop Snakemake from rerunning jobs after I edit the Snakefile? Narrow the trigger set for that run with `-rerun-triggers mtime`, which ignores code, params, input, and software-env changes. Better, scope edits to rules whose reruns you accept, and dry run first. ### How does Nextflow resume compare with Snakemake rerun triggers? Nextflow keys each cached task on a hash of its inputs, script, container, and parameters, and reuses results when the hash matches. Snakemake compares stored per-output metadata against triggers you select per run, which gives you a dial Nextflow does not have.