# Snakemake Input Functions: Dynamic File Resolution By the end of this article you will be able to write a Snakemake input function that resolves file names from a sample sheet, wire it into a rule, combine it with `expand()` for aggregate targets, use the checkpoint pattern for outputs that only exist after another rule has run, and diagnose the errors that dynamic file resolution produces. You will need Snakemake 7 or 8 installed (for example with `pip install snakemake` or via Bioconda), Python 3.9 or newer, pandas (`pip install pandas`), and a small directory of test FASTQ files or any placeholder files you can create with `touch`. ## Why dynamic file resolution matters A bioinformatics workflow manager exists to describe multistep computational analyses in a common language, support incremental builds and re-entrancy, and delegate execution details to the tool itself [1]. Snakemake does this by building a directed acyclic graph (DAG) of jobs before running anything. Every job's inputs and outputs must be known as concrete file names at DAG build time. Static input strings cannot handle workflows where the set of files depends on a sample sheet, a configuration choice, or the output of a rule that has not run yet. Input functions are Snakemake's mechanism for deferring that decision to DAG build time. Published Snakemake workflows show why this matters in practice. Colora, a chromosome-scale de novo assembly workflow, accepts Pacific Biosciences HiFi and Hi-C data and optionally Oxford Nanopore reads, so its input layer must add or omit files depending on configuration [2]. MAYA accepts either raw Illumina paired-end FASTQ reads or pre-assembled FASTA sequences, meaning the inputs of downstream rules change with a single config switch [3]. Tourmaline starts from a configuration file that defines parameters and input files, including a reference database, a sample metadata file, and a manifest or archive of FASTQ sequences [4]. All three resolve file names with logic, not literals, and the input function is where that logic lives. ## What a Snakemake input function is An input function is any Python callable that you pass where Snakemake expects an input file. When Snakemake builds the DAG, it needs concrete file paths for every job it schedules. For a job whose wildcards are, say, `sample=S2`, Snakemake calls your function with a `Wildcards` object whose attributes hold the assigned values. Whatever the function returns becomes the rule's input list for that job. The function can return a single string, a list of strings, or, in recent Snakemake versions, a dict of named inputs. The core mental model: 1. Snakemake parses the Snakefile once at startup. Rules are registered, but wildcards are unbound. 2. You name a target, for example `snakemake -n results/summary/S1.txt`. 3. Snakemake matches the target against rule output patterns, binds wildcards, and then calls input functions with those bound wildcards. 4. Returned strings become the concrete inputs of the job. Snakemake repeats the process recursively for those inputs until the full DAG is closed. 5. Input functions are called again on every subsequent DAG build, including every dry run and every partial rerun, so they must be deterministic and free of side effects. An input function only receives the wildcards of the job. It does not receive `params`, `threads`, or `resources`, which are evaluated later, per job. When an input function needs configuration, it reads it from the global `config` object or from values captured in a closure. ## Step-by-step: build a sample-sheet-driven pipeline ### Step 1: lay out the project Create this structure: ``` project/ Snakefile config.yaml config/samples.csv data/raw/ S1_R1.fastq.gz S1_R2.fastq.gz S2_R1.fastq.gz S2_R2.fastq.gz S3_R1.fastq.gz S3_R2.fastq.gz results/ ``` Create the placeholder FASTQ files with `touch data/raw/S1_R1.fastq.gz` and so on for all six files. Content does not matter for this exercise. ### Step 2: write the sample sheet and config `config/samples.csv`: ``` sample,condition,fastq_1,fastq_2 S1,control,S1_R1.fastq.gz,S1_R2.fastq.gz S2,treated,S2_R1.fastq.gz,S2_R2.fastq.gz S3,treated,S3_R1.fastq.gz,S3_R2.fastq.gz ``` `config.yaml`: ```yaml samples: "config/samples.csv" datadir: "data/raw" ``` This mirrors the design used by config-driven pipelines such as Tourmaline, where the user supplies a manifest of sequences and metadata and the workflow derives every per-sample path from it [4]. ### Step 3: load the sheet and write the input functions `Snakefile`: ```python import os import pandas as pd from snakemake.exceptions import WorkflowError configfile: "config.yaml" samples = pd.read_csv(config["samples"], dtype=str).set_index("sample", drop=False) def read1(wildcards): return os.path.join(config["datadir"], samples.loc[wildcards.sample, "fastq_1"]) def read2(wildcards): return os.path.join(config["datadir"], samples.loc[wildcards.sample, "fastq_2"]) def trimmed_pair(wildcards): return [ f"results/trimmed/{wildcards.sample}_R1.trim.fastq.gz", f"results/trimmed/{wildcards.sample}_R2.trim.fastq.gz", ] rule all: input: "results/qc/multiqc_report.html" rule trim: input: r1=read1, r2=read2 output: r1="results/trimmed/{sample}_R1.trim.fastq.gz", r2="results/trimmed/{sample}_R2.trim.fastq.gz" shell: "fastp -i {input.r1} -I {input.r2} " "-o {output.r1} -O {output.r2} " "-h results/trimmed/{wildcards.sample}.html -j results/trimmed/{wildcards.sample}.json" rule multiqc: input: expand("results/trimmed/{sample}.html", sample=samples.index) output: "results/qc/multiqc_report.html" shell: "multiqc results/trimmed -o results/qc" ``` Notice the split of labor. `read1` and `read2` are input functions because the raw file names depend on the sample sheet and cannot be derived from a single glob pattern. The `multiqc` rule uses `expand()` because the set of QC reports is fully known at parse time from the sample list. This division, `expand()` for knowable sets and input functions for per-wildcard lookups, is the single most useful rule of thumb in this topic. ### Step 4: dry run and inspect the DAG Run: ``` snakemake -n -j 4 ``` Expected output: ``` Building DAG of jobs... Job stats: job count all 1 multiqc 1 trim 3 total 5 Reasons: (check individual rules above for details) output of multiqc has to be generated: missing output file results/qc/multiqc_report.html output of trim has to be generated: missing output files results/trimmed/S1_R1.trim.fastq.gz, ... This was a dry-run (flag -n). In order to actually execute it, remove the flag. ``` The three `trim` jobs confirm that the input functions resolved three distinct wildcard bindings, one per sample. If you see only one job, or the wrong file names, the resolution logic is wrong, not the execution. ### Step 5: run for real ``` snakemake -j 4 -p ``` The `-p` flag prints each shell command. Verify results by checking that `results/trimmed/` contains three HTML reports and six trimmed FASTQ files, and that `results/qc/multiqc_report.html` exists. ### Step 6: fail loudly on bad sheets A common production pattern is validation inside the input function, so a typo in the manifest produces a clear message instead of a bare missing-file error: ```python def read1(wildcards): if wildcards.sample not in samples.index: raise WorkflowError( f"Sample '{wildcards.sample}' is not in config/samples.csv. " f"Known samples: {', '.join(samples.index)}" ) path = os.path.join(config["datadir"], samples.loc[wildcards.sample, "fastq_1"]) if not os.path.exists(path): raise WorkflowError( f"{path} listed in config/samples.csv does not exist on disk." ) return path ``` The existence check is safe here because raw reads are true leaves of the DAG. Do not add existence checks for files produced by other rules, for reasons covered in the mistakes section. ## The closure pattern: accessing params and config Because an input function receives only wildcards, it cannot read the `params` block of its own rule. The standard workaround is a factory that captures the values it needs: ```python def fastq_factory(datadir): def fastq(wildcards): return f"{datadir}/{wildcards.sample}_R{wildcards.read}.fastq.gz" return fastq rule trim: input: r1=fastq_factory(config["datadir"]), r2=fastq_factory(config["datadir"]) output: "results/trimmed/{sample}_R{read}.trim.fastq.gz" shell: "seqkit seq {input} > {output}" ``` The factory runs once at parse time, producing one specialized function per call site. The inner function then behaves like any other input function. This pattern also keeps rule definitions reusable when the same path scheme appears in several rules with different prefixes, which is a common need in modular pipelines such as TrAnnoScope, whose design lets users adapt steps to different sequencing platforms and data types [5]. ## Checkpoints: input functions for unknown future outputs Some file sets cannot be known at DAG build time because they are produced by a rule itself. The classic case is splitting a file into an unknown number of chunks. Snakemake solves this with `checkpoint` rules and input functions that consult the checkpoint. ```python import os checkpoint split: input: "results/assembly/{sample}.fa" output: directory("results/split/{sample}") shell: "python scripts/split_fasta.py {input} {output}" def chunk_files(wildcards): checkpoint_output = checkpoints.split.get(**wildcards).output chunk_ids = glob_wildcards( os.path.join(checkpoint_output, "{chunk}.txt") ).chunk return expand( "results/split/{sample}/{chunk}.txt", sample=wildcards.sample, chunk=chunk_ids, ) rule summarize: input: chunk_files output: "results/summary/{sample}.txt" shell: "cat {input} > {output}" ``` How this executes: 1. Snakemake reaches `summarize`, whose input is the function `chunk_files`. 2. The function calls `checkpoints.split.get(**wildcards)`. If the checkpoint job for that sample has not run yet, Snakemake pauses DAG construction, runs the checkpoint job, and then resumes. 3. `glob_wildcards` scans the completed checkpoint directory, for example finding `chunk_ids = ["part0", "part1", "part2"]`. 4. `expand()` turns those into concrete paths, and the DAG closes. This is the only sanctioned way to make a downstream rule depend on an unbounded set of files. The older `dynamic` flag is deprecated in favor of checkpoints. Use the checkpoint pattern whenever chunk counts, cluster counts, or per-entry output files depend on data content. ## Configuration-dependent inputs Input functions shine when a single config value changes the shape of the DAG. Two patterns from published workflows translate directly into code. First, optional data types. Colora runs with HiFi and Hi-C reads and treats Oxford Nanopore reads as optional [2]. The input layer expresses this as a conditional list: ```python def assembly_inputs(wildcards): files = [ f"results/hifi/{wildcards.sample}.bam", f"results/hic/{wildcards.sample}.bam", ] if config.get("ont_reads"): files.append(f"data/ont/{wildcards.sample}.fastq.gz") return files ``` Second, alternative input modes. MAYA accepts either raw paired-end FASTQ or pre-assembled FASTA and routes the workflow accordingly [3]: ```python def assembly_input(wildcards): if config["input_mode"] == "fastq": return expand( "results/trimmed/{sample}_R{read}.trim.fastq.gz", sample=wildcards.sample, read=[1, 2], ) return [f"assemblies/{wildcards.sample}.fasta"] ``` A third pattern is minimal manual input. The ABFE workflow described by Ries and colleagues takes little more than a receptor PDB and a ligand SDF and uses Snakemake for dynamic control of parameters and execution [6]. In Snakemake terms, that means the input functions derive nearly every intermediate path from one or two wildcard values, so the user supplies a minimum and the workflow expands it. ## Quick reference table ![Three-column comparison of string literals, expand and input functions across evaluation time, typical use, wildcard access and main gotcha.](/article-images/illustrations/snakemake-input-function-expand-vs-function.webp) | Mechanism | Evaluated | Typical use | Gotcha | |, - |, - |, - |, - | | String literal | Parse time | Fixed reference files | No wildcard substitution | | `expand()` | Parse time | Known sets over a sample list | Fails if set is unknown at parse time | | Input function | DAG build, per job | Per-sample lookups from sheets, config-dependent lists | Receives only wildcards | | Checkpoint input function | DAG build, mid-execution | Unknown output sets from a split step | Only works with `checkpoint` rules | | `multiext()` | Parse time | Paired extensions like `.bam` and `.bai` | Static pairs only | | `directory()` | Parse time | Directory-valued outputs | Combine with checkpoints for dynamic contents | | `ancient()` | Rerun logic | Inputs whose mtime should not trigger reruns | Exempts file from timestamp logic | | `temp()` | File lifecycle | Intermediates to delete after use | Deleted files are re-created on demand | ## Snakemake input function tips 1. Keep input functions pure. No writing files, no network calls, no global counters. They run on every DAG build, and nondeterministic behavior causes invisible reruns and confusing DAGs. 2. Cache expensive scans. If you must glob a large directory that does not change, do it once at parse time and store the result in a module-level variable, rather than globbing inside a function that is called per job. 3. Validate the sample sheet once, at parse time. Assert unique sample names, required columns, and well-formed file names before any input function is called. Early, loud failure is a feature. RepairNatrix, a read-processing workflow for DNA storage, shows how much downstream value comes from careful handling of the raw input stage before anything else runs [7]. 4. Use `WorkflowError` for user-facing failures. A bare `KeyError` from pandas points at the wrong culprit. Raising `WorkflowError` with the sample name and the offending path turns a stack trace into an actionable message. 5. Prefer `os.path.join` or `pathlib` over string concatenation with slashes, so the same Snakefile works when a config points at nested directories. 6. Keep the wildcard contract tight. If a rule's output pattern is `results/{sample}/{tool}/report.html`, the input function should derive paths from the same wildcards, never invent new ones, or you will trigger wildcard consistency errors. 7. Combine with provenance. uORF-Tools ships a comprehensive annotation file that can serve as an alternative input to the workflow, letting users swap a computed file for a provided one [8]. Designing input functions around swappable sources, computed versus provided, makes workflows easier to test and share. 8. Test resolution in isolation. Put input functions in a plain Python module, import it in a small test script, call it with a fake wildcards object (`SimpleNamespace(sample="S1")` works), and assert the returned paths. This catches most sheet-path bugs without running Snakemake at all. ## Common Mistakes and Limitations **Mistake 1: using `expand()` where a lookup is needed.** If your sample sheet maps `S1` to `lane3_KO_R1.fq.gz`, no amount of `expand("data/{sample}_R1.fq.gz")` will produce the right path. `expand()` substitutes pattern values, it does not consult external mappings. Fix: read the sheet into a pandas frame at parse time and use an input function keyed on `wildcards.sample`. **Mistake 2: side effects or randomness in the function.** A function that shuffles, appends to a list, or reads a file that changes between calls will produce a different DAG on every invocation. Symptoms include jobs that rerun without any file change. Fix: pure functions only, and derive everything from `config`, `samples`, and `wildcards`. **Mistake 3: checking existence of generated files.** An `os.path.exists` check inside an input function that references an intermediate file will fail during the very first dry run, because intermediates do not exist yet, even though the DAG is perfectly valid. Existence checks belong on true leaf inputs (raw reads, databases) or in a dedicated validation step. Fix: reserve checks for files no rule creates. **Mistake 4: missing files a rule creates anyway.** The canonical error looks like this: ``` MissingInputException in rule summarize in file /path/Snakefile, line 42. Missing input files for rule summarize: results/split/S1/part0.txt ``` when those files are produced by a `split` rule. The cause is using a plain rule instead of a `checkpoint` for the producing step. Snakemake must resolve all inputs before execution, so files that appear only after execution require the checkpoint mechanism described above. **Mistake 5: accessing `params` inside the input function.** This raises an error because params are not defined at DAG build time. Fix: the closure pattern, or read the value from the global `config` directly. **Mistake 6: overly greedy wildcard patterns.** An input function that returns paths like `results/anything/S1.txt` can collide with another rule's output pattern and raise: ``` AmbiguousRuleException in rule summarize in file /path/Snakefile, line 42. Rules split (produce results/S1/part0.txt) and summarize (produce results/S1/part0.txt) ambiguous for the requested file. ``` Fix: keep directory prefixes distinct per rule family and add `wildcard_constraints` where file names share structure. **Limitation 1: the DAG is static per run.** Once built, the DAG does not change unless a checkpoint triggers a rebuild. Input functions cannot express "decide the next rule after seeing this result" beyond the checkpoint pattern. For adaptive designs that require repeated data-dependent branching, split the workflow into staged invocations. **Limitation 2: performance on very large sample sets.** Because input functions are called per job per DAG build, a function that re-parses a 100,000-row manifest on each call wastes real time at scale. Precompute derived structures once at parse time. **Limitation 3: parse-time errors are unforgiving.** Any exception raised while Snakemake evaluates input functions aborts the whole build, including dry runs. This is a feature for correctness but means untested functions block quick experiments. The isolation test in tip 8 exists for this reason. ## How published Snakemake pipelines structure their inputs Looking at three real workflows clarifies where input functions sit in a mature project. Colora's input layer handles HiFi and Hi-C reads as required data and ONT reads as optional data, all declared through configuration [2]. MAYA's dual FASTQ and FASTA entry points mean its input functions branch on `input_mode` before any tool runs [3]. TrAnnoScope combines Illumina short-read and PacBio long-read data in one modular pipeline, which in practice means separate input function families per data type, converging on shared downstream rules [5]. uORF-Tools demonstrates the complementary capability, accepting either a workflow-computed annotation file or a provided one as input [8]. The shared lesson is that input functions are the seam between user-supplied data (sheets, configs, raw reads) and the workflow's internal namespace, and mature pipelines invest most of their engineering effort at exactly that seam. ## Nextflow vs Snakemake for dynamic file resolution The comparison comes up constantly, so it is worth stating precisely. In Nextflow, data flows between processes through channels, and resolution is asynchronous and reactive, a process emits values and downstream processes consume them as they arrive. In Snakemake, the DAG is declared and resolved up front, with input functions providing lazy per-job resolution and checkpoints as the escape hatch for genuinely unknown sets. The practical consequences: Nextflow handles streaming, scattered results, and resumable pipelines without an explicit mechanism analogous to checkpoints, while Snakemake's model makes the full job graph visible before execution, which helps with dry runs, resource estimation, and debugging resolution errors like the ones shown above. Neither model is universally better. Jackson and colleagues argue that the right way to choose a bioinformatics workflow manager is prototyping your actual analysis in the candidates rather than reasoning from feature lists, because the fit depends on the workflow's structure and the team's maintenance context [1]. For the specific problem of file-name resolution from sample sheets, both systems handle it well, and the difference is mostly where the logic lives, in channels and closures on the Nextflow side, in input functions and config on the Snakemake side. ## A second worked project: merging multiple sequencing lanes per sample The first walkthrough assumed one FASTQ pair per sample. Real sequencing runs rarely cooperate that way. A lane-split flowcell produces several R1/R2 pairs per sample, and the merge step must know the lane count per sample, which differs from sample to sample. This is a natural input function exercise, and it introduces a pattern that scales to multi-omics projects where each sample has a variable number of component files. ### Sheet design for one-to-many relationships `config/samples.csv`: ``` sample,lane,fastq_1,fastq_2 S1,L001,S1_L001_R1.fastq.gz,S1_L001_R2.fastq.gz S1,L002,S1_L002_R1.fastq.gz,S1_L002_R2.fastq.gz S2,L001,S2_L001_R1.fastq.gz,S2_L001_R2.fastq.gz S3,L001,S3_L001_R1.fastq.gz,S3_L001_R2.fastq.gz S3,L002,S3_L002_R1.fastq.gz,S3_L002_R2.fastq.gz S3,L003,S3_L003_R1.fastq.gz,S3_L003_R2.fastq.gz ``` S1 has two lanes, S2 has one, S3 has three. The key design decision is that the sheet is long format, one row per file pair, not one row per sample. Long format is what allows the workflow to absorb variable lane counts without any workflow changes, which is the same design philosophy behind manifest-driven pipelines such as Tourmaline, where a user-supplied manifest of sequences and metadata drives every downstream path [4]. ### Precompute the mapping at parse time The important performance and correctness habit is to build the lookup structure once, at parse time, not inside the input function: ```python import os import pandas as pd configfile: "config.yaml" sheet = pd.read_csv(config["samples"], dtype=str) lane_map = {} for sample, group in sheet.groupby("sample"): lane_map[sample] = [ ( os.path.join(config["datadir"], row.fastq_1), os.path.join(config["datadir"], row.fastq_2), ) for row in group.itertuples() ] def lane_fastqs(wildcards): if wildcards.sample not in lane_map: raise WorkflowError( f"Sample '{wildcards.sample}' not found in sheet. " f"Known: {', '.join(sorted(lane_map))}" ) pairs = lane_map[wildcards.sample] return [p for pair in pairs for p in pair] ``` The `lane_map` dictionary is built exactly once. The input function is then a dictionary lookup plus a flattening comprehension, which is fast enough to call per job on every DAG build even for large cohorts. ### The merge rule ```python rule all: input: expand("results/merged/{sample}_R{read}.fastq.gz", sample=lane_map, read=[1, 2]) rule merge_lanes: input: lane_fastqs output: r1="results/merged/{sample}_R1.fastq.gz", r2="results/merged/{sample}_R2.fastq.gz" shell: "cat {input} > {output}" ``` Study what happens here. The `all` rule uses `expand()` over the keys of `lane_map`, which is legal because the sample list is known at parse time even though the per-sample file lists are not uniform. The `merge_lanes` rule receives a variable number of inputs: two files for S2, four for S1, six for S3. The shell command `cat {input}` handles all cases identically because Snakemake expands `{input}` to the full resolved list regardless of length. ### Why a list comprehension beats expand() here A common instinct is to write the aggregate input as a nested `expand()`. That fails structurally, because `expand()` with two wildcard keywords produces the Cartesian product of the supplied lists, not the pairwise grouping your sheet describes. When the full file set is known at parse time, a plain list comprehension over the sheet is clearer, faster, and immune to the Cartesian trap: ```python all_raw = [ os.path.join(config["datadir"], f) for f in list(sheet["fastq_1"]) + list(sheet["fastq_2"]) ] ``` Reserve `expand()` for genuinely pattern-shaped sets, and use comprehensions when the sheet itself is the source of truth for pairing. ### Verify with a dry run ``` snakemake -n -j 4 ``` You should see exactly three `merge_lanes` jobs and the `all` job. If you instead see six jobs, the wildcard binding has split by lane rather than by sample, which means the output pattern or the sheet indexing is wrong. The job count in the dry-run summary is the fastest diagnostic for one-to-many input logic: count the jobs you expect by hand from the sheet before running anything. ## Migrating an existing pipeline from static inputs to input functions Many Snakefiles in the wild start life with hardcoded sample names and grow painful as the project scales. A staged refactor gets you to sheet-driven resolution without a rewrite. Here is the progression, using a toy alignment pipeline. ### Stage 0: the hardcoded original ```python SAMPLES = ["S1", "S2", "S3"] rule all: input: expand("results/aligned/{sample}.bam", sample=SAMPLES) rule align: input: "data/raw/{sample}_R1.fastq.gz", "data/raw/{sample}_R2.fastq.gz" output: "results/aligned/{sample}.bam" shell: "bwa mem {input.r1 if False else ''} {input} > {output}" ``` This works only while every raw file follows the `{sample}_R1.fastq.gz` naming convention. The moment a collaborator delivers files named `lane3_KO_1.fq.gz`, the pipeline breaks silently or fails with a missing-file error. ### Stage 1: move the sample list into a sheet Replace the hardcoded list with the index of a parsed CSV: ```python samples = pd.read_csv(config["samples"], dtype=str).set_index("sample", drop=False) rule all: input: expand("results/aligned/{sample}.bam", sample=samples.index) ``` Nothing else changes yet. The DAG is identical, but adding a sample is now a one-line edit to the sheet instead of a Snakefile edit. This is the highest value-per-minute refactor in the entire progression. ### Stage 2: replace naming assumptions with lookups The `align` rule still assumes a naming convention. Replace the literal inputs with lookup functions: ```python def read1(wildcards): return os.path.join(config["datadir"], samples.loc[wildcards.sample, "fastq_1"]) def read2(wildcards): return os.path.join(config["datadir"], samples.loc[wildcards.sample, "fastq_2"]) rule align: input: r1=read1, r2=read2 output: "results/aligned/{sample}.bam" ``` Now arbitrary file names in the sheet resolve correctly. Confirm behavioral equivalence with a dry run before and after: the job stats should be identical, and `snakemake -n -p` should print the same underlying commands with the same concrete file names. ### Stage 3: harden the input layer Add the parse-time validation and the `WorkflowError`-raising lookups described earlier in the article. At this point also move the loading code and functions into a separate `rules/common.smk` or a plain Python module so they can be imported and unit tested. Modular pipelines such as TrAnnoScope, which combine Illumina short-read and PacBio long-read data through a modular rule structure, depend on exactly this separation between input logic and rule bodies [5]. ### A regression net during refactoring Before touching anything, capture the current behavior: ``` snakemake -n -p > before.txt 2>&1 ``` After each stage, regenerate the same file and diff: ``` snakemake -n -p > after.txt 2>&1 diff before.txt after.txt ``` If the only differences are expected ones, such as reordered job listings, the refactor preserved semantics. This dry-run diffing habit turns a scary refactor into a mechanical, verifiable procedure, and it costs nothing because no jobs execute. ## Reading Snakemake resolution errors like a diagnostician Dynamic resolution shifts most pipeline failures from runtime to DAG build time, which is good for correctness but confusing if you have not learned to read the failure modes. Each error below points at a specific layer of the resolution machinery. ### InputFunctionException: the function itself raised ``` InputFunctionException in rule align in file /path/Snakefile, line 30. Error: KeyError: 'S9' Wildcards: sample=S9 ``` This is the most informative error in the set. The `Wildcards:` block shows the exact binding that triggered the failure, which means the wildcard value `S9` reached your function and the function failed on it. In this case a pandas `KeyError` means `S9` is not in the sheet index. Two diagnoses follow: either a rule elsewhere produced the wildcard value `S9` unexpectedly, or the sheet is missing the sample. Trace the wildcard upward: who asked for a job with `sample=S9`? Usually it is an `expand()` over a stale list, or a leftover output file from a removed sample that a rerun trigger picked up. Wrapping lookups in `WorkflowError` with the known sample list, as shown earlier, converts this error from a puzzle into an instruction. ### MissingInputException with a path your function computed ``` MissingInputException in rule align in file /path/Snakefile, line 30. Missing input files for rule align: data/raw/S9_R1.fastq.gz ``` Read this as a report about your function's return value. The path shown is literally what your function returned for that wildcard binding. The function executed successfully; the file does not exist. The debugging move is to ask whether the path is wrong or the file is missing. If the path is wrong, fix the sheet or the join logic. If the path is right and the file should be produced by another rule, you have hit the checkpoint situation described in Mistake 4 earlier: the producing rule must be a `checkpoint`. ### CyclicDependencyException: the function returned its own rule's output An input function that returns a path matching its own rule's output pattern produces a dependency cycle: ``` CyclicDependencyException: Circular dependency between rule align output results/aligned/S1.bam and its input of the same name. ``` This typically happens after a copy-paste edit where the input function was written for a different rule and still points at that rule's namespace. The fix is to enforce the directory-prefix discipline from tip 6: each rule family owns a distinct subtree, and input functions only ever return paths in upstream subtrees. ### Wildcards appearing in the error that you never declared If an error mentions a wildcard such as `read` or `chunk` that your rule never uses, an upstream `expand()` or input function emitted paths containing extra wildcard components, and Snakemake is trying to bind them through your rule. The fix is on the producing side: the returned paths must contain only wildcards that some rule's output pattern actually declares. ### A systematic debugging protocol ![Flowchart branching from a Snakemake resolution failure into function errors, missing files, and cycles, each with its fix.](/article-images/illustrations/snakemake-input-function-resolution-errors.webp) When resolution fails and the cause is not obvious, work through this sequence: 1. Run `snakemake -n` and read the very first error line, not the traceback tail. Snakemake reports the rule, the line number, and the binding that failed. 2. Reproduce the resolution in isolation. In a Python session, load your sheet the same way the Snakefile does and call the function with `SimpleNamespace(sample="S9")`. This separates function bugs from Snakemake mechanics. 3. Add a temporary `print()` inside the input function and run a dry run. Prints during DAG construction appear on stderr before any job runs, which makes them easy to spot. Remove them afterward; debugging prints have no place in committed input functions because they run on every build. 4. Visualize the graph with `snakemake -dag | dot -Tsvg > dag.svg` and inspect the node labels. A wrong edge almost always explains a wrong resolution. 5. Check for stale outputs from previous runs. Files left over from an earlier version of the naming scheme can match wildcard patterns and pull the DAG in unexpected directions. A clean scratch directory isolates resolution logic from history. ## Unit testing input functions with pytest Because input functions are ordinary Python callables with a one-argument contract, they are unusually testable for workflow code. The payoff is large: most sheet-path bugs surface in a test that runs in milliseconds instead of a dry run that takes minutes on a cluster. ### Structure the code for testability Put loading and input functions in a dedicated module, for example `workflow/inputs.py`, and import it from the Snakefile: ```python # workflow/inputs.py import os import pandas as pd from snakemake.exceptions import WorkflowError def load_samples(path): df = pd.read_csv(path, dtype=str).set_index("sample", drop=False) required = {"sample", "fastq_1", "fastq_2"} missing = required - set(df.columns) if missing: raise WorkflowError(f"Sample sheet missing columns: {sorted(missing)}") if df.index.has_duplicates: dupes = df.index[df.index.duplicated()].unique().tolist() raise WorkflowError(f"Duplicate sample names in sheet: {dupes}") return df def make_read_fn(df, datadir, column): def read(wildcards): if wildcards.sample not in df.index: raise WorkflowError(f"Unknown sample '{wildcards.sample}'") return os.path.join(datadir, df.loc[wildcards.sample, column]) return read ``` ```python # Snakefile from workflow.inputs import load_samples, make_read_fn configfile: "config.yaml" samples = load_samples(config["samples"]) read1 = make_read_fn(samples, config["datadir"], "fastq_1") read2 = make_read_fn(samples, config["datadir"], "fastq_2") ``` ### The tests ```python # tests/test_inputs.py from types import SimpleNamespace import pandas as pd import pytest from snakemake.exceptions import WorkflowError import workflow.inputs as inputs @pytest.fixture def samples(): return inputs.load_samples("tests/fixtures/samples.csv") def test_read_resolves_known_sample(samples, tmp_path): read = inputs.make_read_fn(samples, str(tmp_path), "fastq_1") assert read(SimpleNamespace(sample="S1")).endswith("S1_R1.fastq.gz") def test_read_rejects_unknown_sample(samples, tmp_path): read = inputs.make_read_fn(samples, str(tmp_path), "fastq_1") with pytest.raises(WorkflowError, match="Unknown sample 'S9'"): read(SimpleNamespace(sample="S9")) def test_sheet_rejects_missing_columns(tmp_path): bad = tmp_path / "bad.csv" bad.write_text("sample,condition\nS1,control\n") with pytest.raises(WorkflowError, match="missing columns"): inputs.load_samples(str(bad)) def test_every_sample_resolves(samples, tmp_path): for sample in samples.index: read = inputs.make_read_fn(samples, str(tmp_path), "fastq_1") path = read(SimpleNamespace(sample=sample)) assert sample in path ``` The last test is the workhorse: it walks every sample in the sheet and asserts resolution succeeds, so a malformed row fails one test with a named sample rather than failing a cluster run three hours in. `SimpleNamespace` works as a stand-in for the `Wildcards` object because input functions only ever access named attributes. For checkpoint-consuming functions, the same technique applies with a stub object exposing whatever attributes the function reads, though the checkpoint `.get()` call itself can only be exercised inside a real Snakemake run. This testing discipline is worth the effort precisely because input functions run on every DAG build, including every dry run: a broken function blocks not only execution but also the `-n`, `-dag`, and `-summary` diagnostics you would otherwise use to investigate. Tests keep the diagnostic tooling itself usable. ## Named inputs, inline lambdas, and readability at scale The basic walkthrough showed functions returning plain strings and lists. Two refinements matter once rules accumulate multiple inputs. ### Dict-returning input functions Recent Snakemake versions accept a dict from an input function, producing named inputs that are accessible as attributes: ```python def sample_files(wildcards): row = samples.loc[wildcards.sample] return { "r1": os.path.join(config["datadir"], row["fastq_1"]), "r2": os.path.join(config["datadir"], row["fastq_2"]), "ref": config["reference"], } rule align: input: sample_files output: "results/aligned/{sample}.bam" shell: "bwa mem {input.ref} {input.r1} {input.r2} > {output}" ``` The shell command now reads like the tool's own argument list instead of a positional mystery. This also removes the fragile link between input order and shell syntax: reordering the dict does not change any command. For rules with five or more inputs, which are common in variant calling and assembly, named inputs are the difference between reviewable and unreviewable shell blocks. ### Inline lambdas for one-off lookups When a lookup is needed exactly once and is a single expression, a lambda keeps the logic adjacent to the rule: ```python rule call_variants: input: bam="results/aligned/{sample}.bam", ref=lambda wc: config["references"][config["genome_build"]] ``` The lambda receives the wildcards object exactly like a named function. Use them sparingly: anything longer than one expression, or needed by more than one rule, belongs in a named, tested function. The closure pattern from earlier remains the right tool when the function needs configuration captured at parse time. ### Mixing mechanisms in one rule A rule's input block can freely mix strings, `expand()` calls, named functions, and lambdas: ```python rule annotate: input: vcf="results/called/{sample}.vcf.gz", dbsnp=config["dbsnp"], panels=expand("resources/panels/{panel}.vcf.gz", panel=config["panels"]), extra=lambda wc: samples.loc[wc.sample, "extra_bed"] ``` Snakemake evaluates each entry by its own mechanism: strings and `expand()` at parse time, functions and lambdas at DAG build. This mixing is idiomatic and, in mature pipelines, the normal state of a substantial rule. ## Performance engineering for large cohorts Input function overhead is negligible at ten samples and material at ten thousand. The cost model is simple: every function call happens at least once per job per DAG build, and a typical project accrues many builds across dry runs, real runs, cluster submissions, and rerun evaluations. A function that does a disk read or a glob per call multiplies that cost across the entire cohort. ### Vectorize path construction at parse time Instead of building paths inside the function, build them once as columns: ```python sheet["path_1"] = os.path.join(config["datadir"], sheet["fastq_1"]) sheet = sheet.set_index("sample") def read1(wildcards): return sheet.loc[wildcards.sample, "path_1"] ``` The function body is now a single indexed lookup on an in-memory frame. On a 100,000-row manifest this is orders of magnitude cheaper than re-reading and re-joining per call. ### Choose the right structure per access pattern Different lookups favor different precomputed structures: | Access pattern | Precomputed structure | Built | | Lookup one sample by name | dict or indexed DataFrame | Parse time | | Aggregate over all samples | list via comprehension | Parse time | | Group files per sample | dict of lists via groupby | Parse time | | Files known only after a rule runs | checkpoint input function | DAG build, mid-run | The organizing principle: everything derivable from the sheet and config belongs at parse time, and only content-dependent sets belong in functions that run during DAG construction. This mirrors the split that published pipelines make between their configuration layer and their execution layer. ### Watch for accidental per-call work Three common hidden costs inside input functions: re-parsing a CSV, re-globbing a large directory, and network calls to object storage or LIMS systems to check file existence. The first two have the parse-time fixes already described. The third deserves emphasis: a function that contacts a remote service on every call turns every dry run into a network load test and introduces nondeterminism exactly where the article's tips demand purity. If provenance from an external system matters, fetch it once at parse time, cache it to disk, and have the functions read the cache. ## How input functions interact with rerun triggers and timestamps Dynamic resolution has a second life after the first successful run: it participates in deciding what reruns. Snakemake records metadata about jobs, and on subsequent invocations it compares the current state of the world against that metadata to detect changes in code, params, input sets, software environment, or modification times. Because input functions produce the input sets, edits to input function logic are visible to this machinery. The practical consequences are worth internalizing: - Changing an input function so that it returns a different file for the same wildcard is a change in the job's recorded input set, which triggers reruns of the affected jobs on the next invocation. This is usually exactly what you want: the pipeline notices that its notion of "the inputs for S1" changed. - Editing comments or formatting inside the function does not change its return values, so no rerun follows, though depending on how the code change is tracked you may see rerun triggers respond to the file change itself. When you want to suppress timestamp-driven reruns for inputs that change mtime without changing content, `ancient()` remains the tool, as listed in the quick reference table. - Reordering keys in a dict-returning function changes named input metadata but not the set of files, and the sensible outcome is that reruns follow the declared changes rather than the dict order. The habit that keeps this machinery predictable is the one stated in the tips: determinism. A function that returns the same paths for the same wildcards on every call makes rerun behavior legible. A function whose output drifts makes reruns mysterious, because Snakemake will faithfully detect and react to changes you did not intend. RepairNatrix, a read-processing workflow for DNA storage pipelines, illustrates how much downstream reliability depends on disciplined handling of the raw input stage before any processing occurs [7], and rerun determinism is part of that discipline. ## Concept mapping for teams moving between Snakemake and Nextflow The article's comparison section states the models in prose. A term-level mapping makes the comparison concrete for teams that maintain code in both systems, and it clarifies where input functions sit in each worldview. | Task | Snakemake mechanism | Nextflow analog | | Parse a sample sheet at startup | pandas read at parse time | `splitCsv` into a channel | | Per-sample leaf lookup | input function on `wildcards.sample` | channel join or map closure | | Aggregate target over known samples | `expand()` in an `all` rule | collect operator into one process | | Variable file sets per sample | input function over groupby structure | grouped channel (groupKey) | | Unknown sets produced at runtime | checkpoint plus globbing function | channels are inherently lazy, no special mechanism | | Branch on config before execution | conditional logic in input functions | channel branching before process consumption | | See the whole job graph before running | DAG built up front, dry runs, `-dag` | DAG rendered from resolved channel topology | Two asymmetries deserve comment. First, the Snakemake up-front DAG makes resolution errors, like the missing paths and ambiguous patterns shown throughout this article, fail fast and loudly at DAG build, while the equivalent Nextflow mistakes tend to surface as empty or mismatched channels downstream, which can be quieter but harder to trace. Second, the checkpoint mechanism is Snakemake's single dedicated escape hatch from its static-per-run DAG, whereas Nextflow's channel model treats laziness as the default rather than the exception. Neither asymmetry is a verdict. Jackson and colleagues make the case that choosing between workflow managers should rest on prototyping the real analysis rather than comparing feature lists, because the fit depends on the workflow's shape and the team's context [1]. The mapping table is best used for translation during migration and code review, not for adjudicating the choice. For teams standardizing on Snakemake, the transferable skill is the one this article teaches: locate the logic that turns user-supplied data into concrete file names, isolate it in pure, testable functions, and let the rest of the workflow consume the results. ## Common misconceptions, corrected Several misunderstandings recur in code review of Snakefiles. Each is stated as heard, then corrected. **"Input functions run at parse time, like expand()."** No. The mental model at the top of the article makes the split explicit: `expand()` and literals evaluate at parse time, input functions evaluate at DAG build time per job. The observable proof is that an input function can depend on `wildcards.sample`, which has no value at parse time. **"The function is called once per workflow."** It is called once per job per DAG build. With three samples there are three calls in the walkthrough; with a rerun evaluation, a dry run, and a real run, there are many more. This is why the purity and caching guidance exists. **"Using expand() inside an input function is a category error."** It is not. The checkpoint pattern depends on exactly this combination: the function discovers chunk identifiers at DAG build time and uses `expand()` to convert them into concrete paths. The correct division of labor is not "expand outside, functions inside" but "expand for pattern-to-paths conversion, functions for deciding what values feed the pattern." **"The input function should validate that every input file exists."** Only for leaf inputs that no rule produces. As Mistake 3 explains, existence checks on intermediates fail the very first dry run. The refined rule: a function may assert that paths it returns are either on disk as true external data or producible by some rule in the DAG, and Snakemake itself enforces the second half through `MissingInputException`. **"Input functions can read params if I pass them as extra arguments."** Extra arguments are not part of the portable calling convention, and params do not exist yet at DAG build time regardless. The closure pattern is the sanctioned route. **"Dynamic inputs mean dynamic DAGs."** The DAG is static per run. Input functions make the construction of that static DAG flexible, and checkpoints allow one bounded form of mid-run rebuild, but "decide the next step after inspecting this result" beyond the checkpoint pattern requires staged invocations, as Limitation 1 states. **"Input functions are a Snakemake weakness compared to channels."** The Nextflow comparison shows the honest framing: both systems resolve data-dependent file sets well, and the difference lies in when and where resolution happens, not in whether it is possible. Workflows like MAYA, which branch between FASTQ and FASTA entry points on a config value, are expressible in either system with comparable effort [3]. ## A design checklist for the input layer of a production pipeline Before a pipeline leaves the bench and enters shared use, its input layer should satisfy every item below. The list compresses the practices demonstrated in this article into an auditable form. 1. All sample metadata lives in one machine-readable sheet, loaded once at parse time with dtype consistency. 2. Parse-time validation rejects missing columns, duplicate sample names, and malformed file names with messages that name the offending row. 3. Every per-sample lookup goes through an input function or a parse-time comprehension, never through a naming convention assumption. 4. Input functions are pure: no writes, no network, no randomness, no reliance on files that change during a run. 5. Existence checks appear only on true external inputs such as raw reads and reference databases [2]. 6. Every rule family writes to a distinct directory subtree, and `wildcard_constraints` are declared wherever file names share structure across families. 7. Unknown output sets are consumed only through the checkpoint pattern, never through a plain rule producing a variable number of files. 8. Input functions live in an importable module with pytest coverage, including a test that resolves every sample in the sheet. 9. Configuration branches that change the shape of the DAG, such as optional data types or alternative input modes, are expressed in input functions with explicit `WorkflowError` messages for unsupported combinations. 10. Swappable sources are supported where useful: a computed file or a user-provided equivalent can serve as the same input, the pattern uORF-Tools demonstrates by accepting either a workflow-computed annotation file or a provided one [8]. Items 1 through 5 prevent the majority of failures observed in practice. Items 6 and 7 prevent the structural errors, ambiguity and missing inputs, that cost the most debugging time. Items 8 through 10 are what separate a pipeline that works on the author's data from one that works on a collaborator's data, which is the actual threshold for a bioinformatics workflow manager earning its keep in a laboratory [1]. ## Frequently Asked Questions ### What is a Snakemake input function? A Snakemake input function is a Python callable passed as a rule's input, which Snakemake calls at DAG build time with the job's bound wildcards. Whatever it returns (a string, a list of strings, or a dict of named inputs) becomes the concrete input paths for that job. ### When should I use an input function instead of expand()? Use `expand()` when the full set of files is known at parse time from a sample list, and an input function when each job's input depends on per-sample data from a sheet, on config branches, or on values only available per wildcard binding. Most real pipelines use both, `expand()` for aggregate targets and input functions for per-sample leaf lookups. ### Can an input function access params, threads, or resources? No. An input function receives only the wildcards object. To use configuration values, read the global `config` or capture values in a closure created by a factory function. Params, threads, and resources are evaluated later, per job, after the DAG is already resolved. ### Why does Snakemake report missing input for files another rule creates? Snakemake must resolve every input to a concrete path before execution. If a producing rule is a normal `rule` rather than a `checkpoint`, its outputs do not exist during DAG building, so the downstream resolution fails. Convert the producing rule to a `checkpoint` and have the consuming input function call `checkpoints..get(**wildcards)` before globbing the results. ### How many arguments can an input function take? Design for exactly one, the wildcards object, which is the portable contract across Snakemake versions. If the function needs more context, use a closure to capture it rather than adding positional arguments. ### Do input functions run on every Snakemake invocation? Yes. They run during every DAG build, including dry runs and rerun evaluations, once per job that needs resolution. This is why they must be deterministic, side-effect free, and reasonably fast, and why heavy file scans are better done once at parse time. ### How does Nextflow handle the same problem differently? Nextflow resolves data through channels, so a process consumes values emitted by [upstream processes](/knowledge/molecular-biology/upstream-processes) asynchronously rather than calling a resolution function against a prebuilt DAG. Snakemake resolves a declared graph up front with lazy input functions and checkpoints. The right choice depends on prototyping your own workflow in both, as recommended in the workflow-management literature. ### Can an input function return directories or named inputs? It can return directory paths, typically combined with the `directory()` flag on the consuming rule and the checkpoint pattern for dynamic contents. Recent Snakemake versions also accept a dict from an input function to produce named inputs, which lets you write `{input.r1}` style references for function-resolved files. ## Sources 1. [Using prototyping to choose a bioinformatics workflow management system.](https://pubmed.ncbi.nlm.nih.gov/33630841/) 2. [Colora: a Snakemake workflow for complete chromosome-scale de novo genome assembly.](https://pubmed.ncbi.nlm.nih.gov/40238183/) 3. [Deep Genomic Characterization of Bacillus subtilis SNUY1 & 168 Using MAYA (Multistage Analysis of Prokaryotic Annotations): An Integrated Snakemake-Based Bioinformatics Pipeline.](https://pubmed.ncbi.nlm.nih.gov/42941958/) 4. [Tourmaline: A containerized workflow for rapid and iterable amplicon sequence analysis using QIIME 2 and Snakemake.](https://pubmed.ncbi.nlm.nih.gov/35902092/) 5. [TrAnnoScope: A Modular Snakemake Pipeline for Full-Length Transcriptome Analysis and Functional Annotation.](https://pubmed.ncbi.nlm.nih.gov/39766814/) 6. [Automated Absolute Binding Free Energy Calculation Workflow for Drug Discovery.](https://pubmed.ncbi.nlm.nih.gov/38952038/) 7. [RepairNatrix: a Snakemake workflow for processing DNA sequencing data for DNA storage.](https://pubmed.ncbi.nlm.nih.gov/38496344/) 8. [uORF-Tools-Workflow for the determination of translation-regulatory upstream open reading frames.](https://pubmed.ncbi.nlm.nih.gov/31513641/)