# How to Use Conda Environments in Nextflow and Snakemake By the end of this guide you will be able to attach a pinned conda environment to every process or rule in a Nextflow pipeline and a Snakemake workflow, point both managers at a single shared package cache, build environments offline on a cluster, and diagnose the six or seven conda failures that account for most support questions. You will need conda (Miniconda or Miniforge) installed, Snakemake 7 or later and Nextflow 22 or later available, and about 10 GB of free disk space in your home or scratch directory. Container and environment tools exist because bioinformatics pipelines fail for boring reasons. A workflow that runs today can break next week when a dependency quietly updates. Container virtualization technologies such as Docker became popular in the bioinformatics domain because they improve the portability and reproducibility of software deployment, and conda environments provide the same benefit at the level of individual tools inside a workflow [1]. A bioinformatics workflow manager such as Nextflow or Snakemake ties those isolated environments to specific steps, so each command runs against the exact tool versions its rule declares. This guide covers the practical layer where the conda environment meets the workflow manager. Everything here is written for someone who has run at least one pipeline before and now wants reproducibility rather than luck. ## Why Pin Software at the Step Level A conda environment is a directory tree containing a specific set of packages and their dependencies, isolated from the system and from other environments. When a workflow manager knows about these environments, two things change: 1. Each task declares exactly which tools and versions it needs, so a collaborator can rebuild the whole pipeline from the workflow files alone. 2. Conflicting dependencies stop being a problem. Your QC step can use Python 3.11 while your variant caller uses Python 3.9, on the same machine, in the same run. Resource profiling research points to a related lesson. A study of transcriptome quantification workflows used CWL-metrics, a tool for the CWL reference implementation cwltool, to collect runtime metrics of containers and workflow metadata, analyzing 7 transcriptome quantification workflows across 6 cloud instance types [1]. The choice of instance type delivered lower financial costs and faster execution times when it matched the measured resource requirements of the workflow [1]. Pinning your software solves the "which version" problem. It does not solve the "how much memory does this step actually need" problem, which is why profiling and pinning go together. ## Nextflow vs Snakemake for Conda Workflows Both managers treat conda as a first-class dependency mechanism, and both create one environment per rule or process by default. The differences that matter: - **Snakemake** makes the conda file a property of the rule in the Snakefile. Environment definitions sit next to the workflow code, and Snakemake hashes each environment file so an unchanged rule reuses its environment between runs. - **Nextflow** makes the conda directive a property of the process, and caches environments by content hash in a shared cache directory. Environments survive across pipeline runs and across projects that declare the same environment. - Snakemake integrates more deeply with conda itself, because Snakemake is typically installed inside a conda environment and reuses that conda installation to build job environments. Nextflow is a JVM application and shells out to whatever conda it finds. Neither choice is wrong. Teams already on Snakemake should use its conda directives, and teams already on Nextflow should use its conda directives. The rest of this guide handles both in parallel so you can implement whichever you run. ## Conda Fundamentals You Need First ### Channels and channel priority Conda installs packages from channels, which are package repositories. Bioinformatics software mostly lives on Bioconda, which itself depends heavily on conda-forge. Get channel configuration right before you touch a workflow manager: ```bash conda config --add channels defaults conda config --add channels bioconda conda config --add channels conda-forge conda config --set channel_priority strict ``` Strict priority means conda prefers packages from the first channel listed and avoids mixing package builds across channels, which is what Bioconda recommends. You can verify the result: ```bash conda config --show channels conda config --show channel_priority ``` Expected output: ```text channels: - conda-forge - bioconda - defaults channel_priority: strict ``` ### Environment definition files Both workflow managers accept YAML files that declare channels and dependencies. A minimal example, `envs/qc.yaml`: ```yaml channels: - conda-forge - bioconda dependencies: - cutadapt=4.6 - fastqc=0.12.1 - python=3.11 ``` Two conventions matter. Pin every top-level package with an exact version using a single equals sign. List channels in the order you want them honored, conda-forge before bioconda works well with strict priority. ### Mamba and the solver Classic conda dependency solving can take minutes on complex environments. The mamba project reimplements the solver in C++ and typically resolves the same environments in seconds, and recent conda releases ship the libmamba solver as the default. For workflow use, install mamba once and configure your manager to use it. This single change removes the most common complaint about conda-based pipelines, which is slow environment creation. ### Where packages are cached Conda downloads every package once into a package cache directory, then hard-links files from that cache into each environment. Check yours: ```bash conda config --show pkgs_dirs ``` This cache is why two workflow managers that each build their own environments still avoid re-downloading the same packages. Environments themselves are not shared between managers, but downloaded packages are. ## Step by Step: Conda Environments in Snakemake ### 1. Create an environment file per functional step Keep these in an `envs/` directory inside the project: `envs/trimming.yaml` ```yaml channels: - conda-forge - bioconda dependencies: - cutadapt=4.6 ``` `envs/alignment.yaml` ```yaml channels: - conda-forge - bioconda dependencies: - bwa=0.7.17 - samtools=1.19.2 ``` ### 2. Attach environments to rules ```python rule trim_reads: input: reads="data/{sample}.fastq.gz" output: "results/trimmed/{sample}.trimmed.fastq.gz" log: "logs/trim/{sample}.log" conda: "envs/trimming.yaml" shell: "cutadapt -a AGATCGGAAGAGC -o {output} {input} > {log} 2>&1" rule align_reads: input: reads="results/trimmed/{sample}.trimmed.fastq.gz" output: "results/bam/{sample}.bam" conda: "envs/alignment.yaml" shell: "bwa mem data/reference/ref.fa {input} | samtools view -b -o {output}" ``` The `conda` directive takes a path relative to the Snakefile. Snakemake resolves it relative to the workflow directory regardless of where you launch the pipeline from, so your environment paths stay stable. One environment file per rule is the limit. If a step needs tools from two files, merge them into one file or split the step into two rules. ### 3. Run with conda enabled ```bash snakemake --use-conda --cores 8 ``` On the first run you will see output like this: ```text Creating conda environment envs/trimming.yaml... Downloading and installing remote packages. Environment for envs/trimming.yaml created (location: .snakemake/conda/hash-value) Creating conda environment envs/alignment.yaml... Downloading and installing remote packages. Environment for envs/alignment.yaml created (location: .snakemake/conda/hash-value) ``` Without `-use-conda`, Snakemake ignores the `conda` directives entirely and runs your shell commands against whatever happens to be on `PATH`. This silent failure mode is the single most common way people ship a pipeline that only worked on their own laptop. ### 4. Redirect environments to a shared location By default environments live under `.snakemake/conda/` in the working directory. On a cluster, point them at a writable shared filesystem so every node and every user reuses them: ```bash snakemake --use-conda --cores 8 --conda-prefix /shared/projects/myproj/conda ``` With a custom prefix, Snakemake stores each environment under a directory named by the content hash of its definition file, plus a copy of the YAML beside it. Two pipelines that declare identical environment files share the same built environment at that prefix. ### 5. Build environments ahead of time For clusters with no internet access on compute nodes, or for image baking, create all environments on a login node first: ```bash snakemake --use-conda --conda-prefix /shared/projects/myproj/conda --conda-create-envs-only ``` Then run the actual workflow offline: ```bash snakemake --use-conda --conda-prefix /shared/projects/myproj/conda --conda-offline --cores 8 ``` ### 6. Verify what a rule will actually run Every job environment is a real conda environment, so inspect it directly: ```bash conda list -p .snakemake/conda/ | head -5 ``` Expected output: ```text # packages in environment at .snakemake/conda/0d7c2a3b: # # Name Version Build Channel bwa 0.7.17 h5bf99c6_9 bioconda samtools 1.19.2 h50ea8bc_1 bioconda ``` You can also run `snakemake, use-conda, list-rules` to confirm every rule that needs an environment has one. Rules without a `conda` directive run in the base environment, which is exactly what you usually want for pure file operations but exactly what you want to avoid for tool invocations. ## Step by Step: Conda Environments in Nextflow ### 1. Declare environments per process `main.nf`: ```groovy nextflow.enable.dsl = 2 process TRIM { conda "envs/trimming.yaml" input: tuple val(sample), path(reads) output: tuple val(sample), path("${sample}.trimmed.fastq.gz") script: """ cutadapt -a AGATCGGAAGAGC -o ${sample}.trimmed.fastq.gz ${reads} """ } process ALIGN { conda "envs/alignment.yaml" input: tuple val(sample), path(reads) output: path("${sample}.bam") script: """ bwa mem data/reference/ref.fa ${reads} | samtools view -b -o ${sample}.bam """ } ``` The conda directive accepts either a YAML file path or an inline package specification: ```groovy process ALIGN { conda "bioconda::bwa=0.7.17 bioconda::samtools=1.19.2" ``` Nextflow resolves the YAML path relative to the pipeline launch directory, and robust pipelines reference it through `projectDir` so it works no matter where the pipeline is launched from: ```groovy conda "${projectDir}/envs/alignment.yaml" ``` One conda directive per process. If a process needs tools from two files, merge the files or split the process. ### 2. Configure the cache and solver in nextflow.config ```groovy conda { cacheDir = "/shared/projects/myproj/nextflow_conda" mamba.enabled = true } ``` `cacheDir` controls where Nextflow builds and stores environments. Setting it to a shared, writable location means every pipeline you ever run reuses environments by content hash instead of rebuilding them. `mamba.enabled` tells Nextflow to invoke mamba instead of conda for environment creation, which typically cuts creation time from minutes to seconds. ### 3. Run with conda enabled ```bash nextflow run main.nf -with-conda ``` The first execution of each process creates its environment. In `.nextflow.log` you will find lines describing the conda invocation and, on success, confirmation that the environment was created and is being used for that task. On subsequent runs of the same pipeline, or any other pipeline that declares the same environment file, Nextflow reuses the cached environment and skips creation. ### 4. Offline operation ![Flowchart showing a login node build step, a shared cache, then offline runs on compute nodes for both workflow managers.](/article-images/illustrations/conda-environment-nextflow-snakemake-offline-cluster.webp) Nextflow mirrors the Snakemake pattern: ```bash nextflow run main.nf -with-conda -offline ``` For a cluster with compute nodes behind a firewall, run the pipeline once on a login node with internet access so all environments get built into the shared cache, then run with `-offline` afterward. ### 5. Verify what ran Two checks cover most cases: ```bash # Confirm which environments were used grep -i "conda" .nextflow.log | less # Inspect the environment Nextflow built conda list -p /shared/projects/myproj/nextflow_conda/ ``` If `conda list` shows the versions you pinned in the YAML, the environment is correct. If a process ran without an environment, the log will show no conda setup for that task, which means you forgot `-with-conda` on the command line. ## Key Options at a Glance | Task | Snakemake | Nextflow | |--|--|--| | Enable conda support | `-use-conda` | `-with-conda` | | Attach an env file | `conda: "envs/x.yaml"` under a rule | `conda "envs/x.yaml"` in a process | | Inline package spec | Put it in the YAML | `conda "bioconda::samtools=1.19.2"` | | Set environment cache | `-conda-prefix PATH` | `conda.cacheDir` in nextflow.config | | Use the fast solver | Recent versions prefer mamba automatically, older ones accept `-conda-frontend mamba` | `conda.mamba.enabled = true` | | Build envs without running | `-conda-create-envs-only` | Run once on a node with internet access | | Fully offline runs | `-conda-offline` | `-with-conda -offline` | ## Tips That Pay Off in Real Pipelines **Use mamba or micromamba everywhere.** Environment creation time is the main tax on conda-based workflows. Configure mamba in both managers and the tax mostly disappears. **Set a shared, writable prefix on day one.** Decide where environments live before your pipeline grows. A `-conda-prefix` or `conda.cacheDir` on shared storage turns every future run of every future pipeline into a cache hit for the package download layer. **One environment per step, small environments.** A single env with 200 packages takes far longer to solve and is far more likely to hit a conflict than ten envs with 5 packages each. Small environments also match the mental model of a workflow, where each rule or process is a tool invocation. **Lock environments for publication.** Pinning top-level versions controls direct dependencies, but the solver can still pick different builds of transitive dependencies months apart as channels update. For a published pipeline, generate a lock file with conda-lock and reference the locked specification, or pair the conda environment with a container image so the full software stack is frozen. **Treat the environment file as part of the code.** Both managers hash the environment definition, so editing the YAML produces a new environment on the next run while old runs keep using the old environment. That is the behavior you want. Commit the YAML files, review changes to them like code changes, and never edit an environment file and hope nobody notices. **Profile your steps.** Pinning software versions answers what runs. How much CPU, memory, and time each step needs is a separate question, and measuring it matters because matching compute resources to measured workflow requirements reduces both cost and runtime [1]. ## Common Mistakes and Limitations **Forgetting the enable flag.** In both managers, environment declarations are inert until you enable conda. A Snakemake run without `-use-conda`, or a Nextflow run without `-with-conda`, executes everything against the ambient environment. If a pipeline works on your machine and fails on a server, check this first. **Assuming conda solves reproducibility completely.** An environment file pins what you listed. The solver resolves the rest, and resolution depends on what the channels contain at solve time. Two builds of the same environment created six months apart can differ in transitive dependencies. For anything published, add a lock file or a container image. **Giant environments.** Combining every tool into one YAML is tempting and always ends in an unsatisfiable dependency conflict between two packages that need different versions of the same library. Split by step. **Unpinned or floating specifications.** Writing `samtools` with no version means your environment changes every time Bioconda publishes a release. Writing `samtools>=1.10` means the same thing with extra steps. Use single equals signs with exact versions. **Path confusion in Nextflow.** A relative conda path such as `envs/alignment.yaml` resolves against the launch directory, which breaks when you launch from elsewhere. Use `${projectDir}/envs/alignment.yaml` in the directive. **Concurrent environment creation on shared storage.** If twenty cluster jobs hit an empty cache at once and try to build the same environment, you can get race conditions and corruption on some filesystems. Warm the cache with a single `-conda-create-envs-only` run, or a single Nextflow run on a login node, before fanning out. **Editing a YAML and expecting old results to update.** Managers version environments by content hash. Changed file means new environment, which is correct behavior, but it also means old work directories still reference superseded environments. Rerun with the appropriate resume or rerun flags once you have changed an environment. **Mixing pip silently.** A `pip` section in a conda YAML works, but pip does not participate in conda's dependency solving or channel discipline. If you must use pip, pin exact versions there too and keep the pip section as small as possible. **Conda does not manage your hardware assumptions.** A pinned environment guarantees software versions. It says nothing about whether the step needs 8 GB or 800 GB of memory. The resource-profiling lesson from containerized workflow measurements applies here, since running workflows on instance types matched to their measured requirements produced lower costs and faster executions [1]. **The honest limitation.** Conda environments are one layer of reproducibility. They do not pin your operating system libraries, your kernel, or your reference data. Teams that need airtight reproducibility pair conda environments with container images, and use the conda layer for fast local iteration. ## Troubleshooting: Exact Failures and Fixes ![Decision path that sorts conda workflow errors into missing channel, dependency conflict, wrong path, read-only cache and missing enable flag.](/article-images/illustrations/conda-environment-nextflow-snakemake-conda-triage.webp) **PackagesNotFoundError** ```text PackagesNotFoundError: The following packages are not available from current channels: - samtools=1.99 ``` Either the version string does not exist or you are missing the bioconda channel. Search Bioconda for the real version numbers, and recheck your channel configuration with `conda config, show channels`. A missing channel in the YAML itself is a frequent cause, because the manager uses the channels declared in the file, not your global configuration. **UnsatisfiableError** ```text UnsatisfiableError: The following specifications were found to be incompatible with the existing python installation ``` Two packages in the same environment demand different versions of a shared dependency. Split the environment, or upgrade the version pin of one package until the conflict disappears. Switching to mamba does not make an unsolvable environment solvable, it just fails faster. **CondaEnvironmentException in Snakemake** ```text CondaEnvironmentException: Conda environment /path/to/env.yaml not found ``` The path in the `conda:` directive is wrong. Snakemake resolves it relative to the Snakefile, so check for typos and check that the file was committed and not just on your laptop. **Environment creation fails on a read-only filesystem** Conda writes to its installation prefix and package cache. On clusters where the central conda installation is read-only, point the manager's cache at your own space using `-conda-prefix` or `conda.cacheDir`. The conda binary can stay where the administrators put it. **Download integrity errors** ```text CondaError: Downloaded bytes did not match Content-Length ``` A flaky network or an aggressive proxy truncated a download. Clear the affected packages from the package cache and rerun. Repeated failures usually mean a proxy or mirror problem rather than a workflow problem. **Environment creation succeeds but the tool is not found in the job** The tool is not in the environment file, or the YAML pin is wrong and conda installed a different package with a similar name. Run `conda list -p ` and search for the exact binary name. Environment verification in the workflow directory is faster than debugging shell errors inside a job. ## How to Check Your Setup Is Right A correct configuration passes four checks: 1. `conda config, show channel_priority` reports `strict`. 2. Every rule with a shell command that invokes an external tool has a `conda` directive, and every process in Nextflow that runs a tool has one. 3. A dry inspection shows environments attached where you expect: `snakemake, list-rules` in Snakemake, or a grep of `.nextflow.log` for conda lines in Nextflow. 4. `conda list -p ` on one or two spot-checked environments shows your pinned versions, not floating ones. For a published pipeline, add a fifth check. Run the whole workflow from scratch in a fresh container or a fresh account with an empty conda cache. If it builds every environment and completes without manual intervention, your environment definitions are genuinely self-contained. ## Frequently Asked Questions ### Do Nextflow and Snakemake share conda environments with each other? No. Each manager builds environments in its own directory layout and uses its own hashing scheme, so a Snakemake environment is not reused by Nextflow. They do share the underlying conda package cache, so downloads are not repeated. ### Should I use containers instead of conda environments? Use containers for publication-grade reproducibility and conda for fast development. The two compose, since both managers let a process or rule carry a conda environment and a container image at the same time. ### Why is conda environment creation so slow in my workflow? The classic conda solver is the bottleneck on complex environments. Install mamba, set `conda.mamba.enabled = true` in Nextflow, and use a recent Snakemake that prefers mamba automatically. ### Can I use one environment for the whole pipeline? You can, and it will eventually produce a dependency conflict between two tools that need incompatible versions of the same library. Small per-step environments solve faster, fail less, and document each step's dependencies. ### How do I pin exact package versions? List each tool in the environment YAML with a single equals sign and an exact version, such as `samtools=1.19.2`. For full reproducibility of transitive dependencies, generate a lock file with conda-lock or add a container image. ### Where are the environments stored and how do I clean them? Snakemake puts them under `.snakemake/conda/` or your `-conda-prefix`. Nextflow puts them in the directory set by `conda.cacheDir`. Delete unused environment directories directly, and note that the conda package cache can be cleaned independently with `conda clean` while leaving environments intact. ### Does conda work on HPC clusters without internet access on compute nodes? Yes. Build all environments on a login node with internet access, using `-conda-create-envs-only` in Snakemake or a single warm-up run in Nextflow, then run the workflow offline with `-conda-offline` or `-with-conda -offline`. ### What is the difference between conda, mamba, and micromamba? Conda is the package manager that both workflow managers shell out to. Mamba is a faster drop-in replacement for conda's solver. Micromamba is a standalone, self-contained binary that does not require a conda installation, useful in containers and ephemeral environments. ## Sources 1. [Accumulating computational resource usage of genomic data analysis workflow to optimize cloud computing instance selection.](https://pubmed.ncbi.nlm.nih.gov/31222199/)