# Snakemake Checkpoint Design


## Key Takeaways

-   Snakemake checkpoints enable dynamic Directed Acyclic Graph (DAG) expansion at runtime, crucial for workflows where the number or identity of output files is unknown until an earlier step completes, such as sample demultiplexing or read clustering.
-   A checkpoint rule, denoted by `checkpoint:`, produces files whose existence or names are determined during execution; downstream rules must use an aggregator function that explicitly calls `checkpoints.<name>.get()` to establish a dependency and discover these files.
-   Avoid checkpoints for scenarios where output filenames can be defined by static patterns with known wildcard values at workflow invocation; regular wildcards or `glob_wildcards` are more appropriate and efficient in such cases.
-   Reproducibility with checkpoints requires careful consideration; rerunning a checkpoint invalidates downstream jobs, and changes in directory modification times without content alteration can lead to missed reruns, necessitating content-based identifiers (e.g., hashes) for critical files.
-   Checkpoints introduce DAG evaluation latency as Snakemake must pause to execute the checkpoint and then expand the graph, which can impact performance in complex workflows with numerous checkpoints or large numbers of dynamically generated files.
-   Common implementation errors include neglecting the `checkpoints.<name>.get()` call in aggregator input functions, leading to `MissingInputException`, and overusing checkpoints for simple file existence checks, which adds unnecessary computational overhead.

---

A Snakemake checkpoint is a special rule that lets your workflow discover files at runtime and then expand the directed acyclic graph (DAG) based on what it finds. Use this guide if you are a bioinformatics pipeline developer who needs to build workflows where the number or identity of output files depends on results from an earlier step (for example, sample demultiplexing, read clustering, or assembly binning). The core framework here covers when to choose checkpoints over alternatives, how to implement them correctly, what quality checks to run, common mistakes that break reproducibility, and the limits of checkpoint reproducibility.

## At a Glance

| Concept | Description |
|---|---|
| Checkpoint rule | A rule decorated with `checkpoint:`. It can produce files whose existence or names are unknown until the rule finishes. |
| Dynamic aggregation | A subsequent rule that uses `checkpoint.<name>.get()` to collect files produced by the checkpoint and to create new input targets. |
| Flag file trick | A lightweight alternative where a regular rule writes a flag file indicating completion, but this does not truly expand the DAG. |
| Runtime DAG expansion | The DAG is frozen until the checkpoint executes, afterward a new sub-DAG is generated for downstream rules. |

## When to Use a Checkpoint: Decision Criteria

Snakemake provides several mechanisms for handling variable file sets: `glob_wildcards`, `dynamic` (deprecated in Snakemake 7+), and checkpoints. The table below shows when a checkpoint is the correct choice.

**Use a checkpoint if** your pipeline must: (1) run a rule that produces an unknown number of output files (e.g., a binning tool that splits contigs into bins), (2) process files whose names depend on content discovered during execution (e.g., sample names only known after read trimming), or (3) allow downstream rules to trigger only after the checkpoint finishes and the new files are enumerated.

**Do not use a checkpoint if** the set of files is known at workflow invocation (use regular wildcards or `glob_wildcards` instead). Also avoid checkpoints for simple file existence checks that can be accomplished with Snakemake’s `input` functions reading from a config file.

A more rigorous way to decide is to ask: can I define the output filenames using a static pattern with known wildcard values? If yes, a regular rule is sufficient. If the pattern itself depends on output from a previous rule, a checkpoint is needed.

## Practical Implementation Workflow

Below is a step-by-step approach for designing a checkpoint. The example assumes a metagenomic binning workflow where the binner outputs an unpredictable number of bins per sample, and each bin must be taxonomically classified.

### Step 1: Identify the Checkpoint Candidate

Find the rule that creates files whose existence cannot be predicted before the rule runs. In this example it is the binning step. Write it as a checkpoint instead of a rule.

```python
checkpoint bin_samples:
    input: "assemblies/{sample}.fa"
    output: directory("bins/{sample}/")
    shell: "run_binner.sh {input} {output}"
```

The `output` directory will contain bin FASTA files with unknown names. The checkpoint keyword tells Snakemake that this rule may produce files that downstream rules need to discover.

### Step 2: Write an Aggregator Rule

An aggregator rule collects the checkpoint outputs and returns a list of filenames. Use `checkpoints.bin_samples.get()` inside an `input` function.

```python
def get_bin_files(wildcards):
    # Wait for the checkpoint to complete
    checkpoint_output = checkpoints.bin_samples.get(
        sample=wildcards.sample
    ).output[0]
    # Return a list of all .fa files in the bin directory
    return expand(
        "bins/{sample}/{bin}.fa",
        sample=wildcards.sample,
        bin=glob_wildcards(os.path.join(checkpoint_output, "{bin}.fa")).bin
    )

rule classify_bins:
    input: get_bin_files
    output: "taxonomy/{sample}/{bin}.tax"
    shell: "classify.py {input} {output}"
```

The `get_bin_files` function runs after the checkpoint rule finishes. It reads the contents of the output directory using `glob_wildcards` and returns a list of file paths. Snakemake then expands the DAG to include one instance of `classify_bins` for each bin file.

### Step 3: Ensure Proper Ordering

The aggregator rule must have an input function, not a simple string or list. The function must call `checkpoints.<name>.get()` to create a dependency on the checkpoint. Without this call, Snakemake will treat the rule as if it can run before the checkpoint, leading to a “MissingInputException” because the files do not exist yet.

### Step 4: Handle Aggregation of Multiple Checkpoints

If your workflow has multiple checkpoints (e.g., per sample and per experiment), chain them by having the downstream checkpoint depend on the upstream one. Snakemake automatically determines the correct execution order. Use the same pattern of `checkpoints.<name>.get()` in each aggregator function.

### Step 5: Test with a Minimal Example

Before embedding checkpoints in a large pipeline, build a tiny test workflow. For instance, create a checkpoint that generates random numbers of empty `.done` files and an aggregator that counts them. Run with `snakemake -n` to verify the DAG, then execute with `-j1` to see that the DAG expands after the checkpoint runs.

## Common Mistakes

**1. Using `dynamic` instead of `checkpoint`.** In Snakemake 7 and later the `dynamic` keyword is deprecated and may cause unintended reruns. Use `checkpoint` plus an input function to achieve the same effect with explicit dependency tracking.

**2. Forgetting to call `checkpoints.<name>.get()` in the aggregator.** Without this call, Snakemake will not know that the aggregator rule depends on the checkpoint. The aggregator will try to evaluate before the checkpoint finishes, causing a file-not-found error.

**3. Using `glob_wildcards` on a path that includes wildcards from the checkpoint output.** Always construct the search path using the actual output directory returned by `get()`, not by using wildcards from the workflow graph.

**4. Overusing checkpoints for simple file existence.** If you only need to confirm that a directory is non-empty, consider writing a regular rule with an `input` function that uses `os.listdir()` inside a `try/except`. Checkpoints add computational overhead in DAG evaluation and I/O for tracking intermediate files.

**5. Misplacing the aggregator in the same rule as the checkpoint.** The aggregator must be a separate rule because it uses the checkpoint’s output after the checkpoint has completed.

## Limits and Uncertainty

Checkpoints are powerful but come with tradeoffs in reproducibility and performance.

**Reproducibility and caching.** Snakemake tracks checkpoint outputs using them as additional input to downstream rules. If the checkpoint is rerun, all downstream rules will be re executed because their inputs change. However, if the checkpoint creates the same file set across runs, but the directory modification time changes, Snakemake may not detect this and could skip the downstream steps. To avoid this, ensure that each bin file has a unique content based identifier (like a hash) and that the checkpoint output is not a single directory that gets reused.

**DAG evaluation latency.** Each checkpoint introduces a pause in DAG evaluation. Snakemake must wait for the checkpoint to finish, then expand the graph. For workflows with many checkpoints or with very large numbers of child files, this can slow down performance. Profile your workflow with a realistic dataset before deploying at scale.

**Debugging difficulty.** When a checkpoint fails, error messages may point to the aggregator function rather than the checkpoint rule itself. Add logging and intermediate file checks inside the aggregator function to narrow down issues.

**Uncertainty with parallelization.** If you run with multiple jobs, the checkpoint must complete before any downstream aggregator rule can start. This creates a synchronization point. Use `snakemake --jobs` carefully and reserve enough resources for the checkpoint rule.

**Cross platform file system behavior.** `glob_wildcards` may behave differently on filesystems with case sensitivity, filesystem caching, or network mounts. Always test the aggregator on the target filesystem. The [Galaxy Training Network](https://training.galaxyproject.org) provides best practices for writing portable workflows that work across HPC and cloud environments.

## Frequently Asked Questions

**1. Can a checkpoint rule have multiple output files with different patterns?** Yes. The checkpoint’s output can be a directory, a single file, or a list of files. The aggregator function should use the appropriate path returned by `get()` for the output it needs. Multiple aggregation rules can target the same checkpoint.

**2. How does Snakemake handle checkpoint reruns when the input to the checkpoint changes?** If the checkpoint rule’s input changes, Snakemake will rerun the checkpoint (as with any rule). After rerunning, the aggregator functions will be re evaluated, and new downstream jobs may be added. Old downstream outputs from previous runs will be invalidated and re executed.

**3. What is the difference between a checkpoint and a regular rule with an input function?** A regular rule with an input function can also compute input names at runtime, but it does so before any rule executes. If the expression evaluates to a list, Snakemake expands the DAG immediately. A checkpoint defers that evaluation until after the checkpoint rule runs, allowing the listing of files that did not exist when the DAG was first constructed.

**4. Can I skip the aggregator rule and use the checkpoint output directly in a downstream rule?** Not directly. Downstream rules must use an input function that calls `checkpoints.<name>.get()` to declare the dependency. Without that call, the rule would appear to have no dependency on the checkpoint and would try to run immediately. The input function itself serves as the aggregator, it can be defined inline or as a separate function.

## Related Clinical & Scientific Guides

* [Observational vs. Experimental Studies: How to Tell Them Apart](/blog/guides/observational-vs-experimental-studies-how-to-tell-them-apart)
* [Astrocyte Single Cell Rna Seq](/blog/guides/astrocyte-single-cell-rna-seq)
* [Structural Genes](/blog/guides/structural-genes)


## References and Further Reading

1. [Snakemake official documentation: Checkpoints](https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#data-dependent-conditional-execution) (the primary developer resource for checkpoint syntax and behavior).
2. [Galaxy Training Network: Advanced workflow topics](https://training.galaxyproject.org) , offers tutorials on handling dynamic data in bioinformatics pipelines, relevant for workflow design concepts.
3. [Bioconductor: Reproducible Research](https://bioconductor.org) , provides guidance on creating reproducible genomic analyses, applicable to checkpoint based pipelines.
4. [EMBL EBI Training: Bioinformatics workflow frameworks](https://www.ebi.ac.uk/training) , includes materials on workflow management best practices that inform checkpoint usage.
5. [NCBI Bookshelf: Bioinformatics Workflows](https://www.ncbi.nlm.nih.gov/books) , authoritative text covering data analysis lifecycles where dynamic file handling is often required.
6. [NCBI Sequence Read Archive: Data organization](https://www.ncbi.nlm.nih.gov/sra) , a practical example of a data source that yields unpredictable sample identifiers, similar to what checkpoints handle.
7. [Snakemake official documentation: Input functions](https://snakemake.readthedocs.io/en/stable/snakefiles/rules.html#using-input-functions) , explains how to write Python functions for dynamic input, foundational for checkpoint design.
8. [Reproducible Research with Snakemake (book)](https://www.reproducible-snakemake.org) , a community guide with chapters on advanced patterns including checkpoints.

## Related Articles

- [Structural Variant Validation](/blog/guides/structural-variant-validation)
- [Repeat Expansion Sequencing](/blog/guides/repeat-expansion-sequencing)
- [Copy Number Variant Analysis](/blog/guides/copy-number-variant-analysis)
- [Somatic Mosaic Variant Calling](/blog/guides/somatic-mosaic-variant-calling)
- [Germline Variant Filtering](/blog/guides/germline-variant-filtering)