Snakemake Checkpoint Design
A Snakemake checkpoint is a special rule that lets your workflow discover files at runtime and then expand the directed acyclic graph (DAG) based on what it finds. Use this guide if you are a bioinformatics pipeline developer who needs to build workflows where the number or identity of output files depends on results from an earlier step (for example, sample demultiplexing, read clustering, or assembly binning). The core framework here covers when to choose checkpoints over alternatives, how to implement them correctly, what quality checks to run, common mistakes that break reproducibility, and the limits of checkpoint reproducibility.
At a Glance
| Concept | Description |
|---|---|
| Checkpoint rule | A rule decorated with checkpoint:. It can produce files whose existence or names are unknown until the rule finishes. |
| Dynamic aggregation | A subsequent rule that uses checkpoint.<name>.get() to collect files produced by the checkpoint and to create new input targets. |
| Flag file trick | A lightweight alternative where a regular rule writes a flag file indicating completion, but this does not truly expand the DAG. |
| Runtime DAG expansion | The DAG is frozen until the checkpoint executes, afterward a new sub-DAG is generated for downstream rules. |
When to Use a Checkpoint: Decision Criteria
Snakemake provides several mechanisms for handling variable file sets: glob_wildcards, dynamic (deprecated in Snakemake 7+), and checkpoints. The table below shows when a checkpoint is the correct choice.
Use a checkpoint if your pipeline must: (1) run a rule that produces an unknown number of output files (e.g., a binning tool that splits contigs into bins), (2) process files whose names depend on content discovered during execution (e.g., sample names only known after read trimming), or (3) allow downstream rules to trigger only after the checkpoint finishes and the new files are enumerated.
Do not use a checkpoint if the set of files is known at workflow invocation (use regular wildcards or glob_wildcards instead). Also avoid checkpoints for simple file existence checks that can be accomplished with Snakemake’s input functions reading from a config file.
A more rigorous way to decide is to ask: can I define the output filenames using a static pattern with known wildcard values? If yes, a regular rule is sufficient. If the pattern itself depends on output from a previous rule, a checkpoint is needed.
Practical Implementation Workflow
Below is a step-by-step approach for designing a checkpoint. The example assumes a metagenomic binning workflow where the binner outputs an unpredictable number of bins per sample, and each bin must be taxonomically classified.
Step 1: Identify the Checkpoint Candidate
Find the rule that creates files whose existence cannot be predicted before the rule runs. In this example it is the binning step. Write it as a checkpoint instead of a rule.
checkpoint bin_samples:
input: "assemblies/{sample}.fa"
output: directory("bins/{sample}/")
shell: "run_binner.sh {input} {output}"
The output directory will contain bin FASTA files with unknown names. The checkpoint keyword tells Snakemake that this rule may produce files that downstream rules need to discover.
Step 2: Write an Aggregator Rule
An aggregator rule collects the checkpoint outputs and returns a list of filenames. Use checkpoints.bin_samples.get() inside an input function.
def get_bin_files(wildcards):
# Wait for the checkpoint to complete
checkpoint_output = checkpoints.bin_samples.get(
sample=wildcards.sample
).output[0]
# Return a list of all .fa files in the bin directory
return expand(
"bins/{sample}/{bin}.fa",
sample=wildcards.sample,
bin=glob_wildcards(os.path.join(checkpoint_output, "{bin}.fa")).bin
)
rule classify_bins:
input: get_bin_files
output: "taxonomy/{sample}/{bin}.tax"
shell: "classify.py {input} {output}"
The get_bin_files function runs after the checkpoint rule finishes. It reads the contents of the output directory using glob_wildcards and returns a list of file paths. Snakemake then expands the DAG to include one instance of classify_bins for each bin file.
Step 3: Ensure Proper Ordering
The aggregator rule must have an input function, not a simple string or list. The function must call checkpoints.<name>.get() to create a dependency on the checkpoint. Without this call, Snakemake will treat the rule as if it can run before the checkpoint, leading to a “MissingInputException” because the files do not exist yet.
Step 4: Handle Aggregation of Multiple Checkpoints
If your workflow has multiple checkpoints (e.g., per sample and per experiment), chain them by having the downstream checkpoint depend on the upstream one. Snakemake automatically determines the correct execution order. Use the same pattern of checkpoints.<name>.get() in each aggregator function.
Step 5: Test with a Minimal Example
Before embedding checkpoints in a large pipeline, build a tiny test workflow. For instance, create a checkpoint that generates random numbers of empty .done files and an aggregator that counts them. Run with snakemake -n to verify the DAG, then execute with -j1 to see that the DAG expands after the checkpoint runs.
Common Mistakes
1. Using dynamic instead of checkpoint. In Snakemake 7 and later the dynamic keyword is deprecated and may cause unintended reruns. Use checkpoint plus an input function to achieve the same effect with explicit dependency tracking.
2. Forgetting to call checkpoints.<name>.get() in the aggregator. Without this call, Snakemake will not know that the aggregator rule depends on the checkpoint. The aggregator will try to evaluate before the checkpoint finishes, causing a file-not-found error.
3. Using glob_wildcards on a path that includes wildcards from the checkpoint output. Always construct the search path using the actual output directory returned by get(), not by using wildcards from the workflow graph.
4. Overusing checkpoints for simple file existence. If you only need to confirm that a directory is non-empty, consider writing a regular rule with an input function that uses os.listdir() inside a try/except. Checkpoints add computational overhead in DAG evaluation and I/O for tracking intermediate files.
5. Misplacing the aggregator in the same rule as the checkpoint. The aggregator must be a separate rule because it uses the checkpoint’s output after the checkpoint has completed.
Limits and Uncertainty
Checkpoints are powerful but come with tradeoffs in reproducibility and performance.
Reproducibility and caching. Snakemake tracks checkpoint outputs using them as additional input to downstream rules. If the checkpoint is rerun, all downstream rules will be re executed because their inputs change. However, if the checkpoint creates the same file set across runs, but the directory modification time changes, Snakemake may not detect this and could skip the downstream steps. To avoid this, ensure that each bin file has a unique content based identifier (like a hash) and that the checkpoint output is not a single directory that gets reused.
DAG evaluation latency. Each checkpoint introduces a pause in DAG evaluation. Snakemake must wait for the checkpoint to finish, then expand the graph. For workflows with many checkpoints or with very large numbers of child files, this can slow down performance. Profile your workflow with a realistic dataset before deploying at scale.
Debugging difficulty. When a checkpoint fails, error messages may point to the aggregator function rather than the checkpoint rule itself. Add logging and intermediate file checks inside the aggregator function to narrow down issues.
Uncertainty with parallelization. If you run with multiple jobs, the checkpoint must complete before any downstream aggregator rule can start. This creates a synchronization point. Use snakemake --jobs carefully and reserve enough resources for the checkpoint rule.
Cross platform file system behavior. glob_wildcards may behave differently on filesystems with case sensitivity, filesystem caching, or network mounts. Always test the aggregator on the target filesystem. The Galaxy Training Network provides best practices for writing portable workflows that work across HPC and cloud environments.
Frequently Asked Questions
1. Can a checkpoint rule have multiple output files with different patterns? Yes. The checkpoint’s output can be a directory, a single file, or a list of files. The aggregator function should use the appropriate path returned by get() for the output it needs. Multiple aggregation rules can target the same checkpoint.
2. How does Snakemake handle checkpoint reruns when the input to the checkpoint changes? If the checkpoint rule’s input changes, Snakemake will rerun the checkpoint (as with any rule). After rerunning, the aggregator functions will be re evaluated, and new downstream jobs may be added. Old downstream outputs from previous runs will be invalidated and re executed.
3. What is the difference between a checkpoint and a regular rule with an input function? A regular rule with an input function can also compute input names at runtime, but it does so before any rule executes. If the expression evaluates to a list, Snakemake expands the DAG immediately. A checkpoint defers that evaluation until after the checkpoint rule runs, allowing the listing of files that did not exist when the DAG was first constructed.
4. Can I skip the aggregator rule and use the checkpoint output directly in a downstream rule? Not directly. Downstream rules must use an input function that calls checkpoints.<name>.get() to declare the dependency. Without that call, the rule would appear to have no dependency on the checkpoint and would try to run immediately. The input function itself serves as the aggregator, it can be defined inline or as a separate function.
References and Further Reading
- Snakemake official documentation: Checkpoints (the primary developer resource for checkpoint syntax and behavior).
- Galaxy Training Network: Advanced workflow topics , offers tutorials on handling dynamic data in bioinformatics pipelines, relevant for workflow design concepts.
- Bioconductor: Reproducible Research , provides guidance on creating reproducible genomic analyses, applicable to checkpoint based pipelines.
- EMBL EBI Training: Bioinformatics workflow frameworks , includes materials on workflow management best practices that inform checkpoint usage.
- NCBI Bookshelf: Bioinformatics Workflows , authoritative text covering data analysis lifecycles where dynamic file handling is often required.
- NCBI Sequence Read Archive: Data organization , a practical example of a data source that yields unpredictable sample identifiers, similar to what checkpoints handle.
- Snakemake official documentation: Input functions , explains how to write Python functions for dynamic input, foundational for checkpoint design.
- Reproducible Research with Snakemake (book) , a community guide with chapters on advanced patterns including checkpoints.