# Ncbi-Genome-Download


## Key Takeaways

- `ncbi-genome-download` is a command-line Python utility designed for the automated, batch retrieval of assembled genome sequences from NCBI FTP servers, specifically targeting prokaryotic and eukaryotic reference genomes.
- The tool facilitates systematic downloads by allowing filtering based on taxonomic groups (e.g., genus *Escherichia*), assembly levels (e.g., 'complete', 'chromosome', 'scaffold', 'contig'), and RefSeq categories ('reference', 'representative').
- It is optimized for large-scale operations, enabling the download of hundreds or thousands of genomes efficiently, and preserves the original directory structure for downstream bioinformatics pipelines.
- `ncbi-genome-download` outputs FASTA files (genomic, protein, CDS) and accession lists, but it does not process raw sequencing reads (SRA data) or perform analyses like BLAST.
- Users should verify downloaded data integrity by checking file completeness, assembly report consistency (e.g., Genome Coverage), and comparing sequence lengths against NCBI records.
- Common pitfalls include not specifying `--assembly-level` (leading to excessive download of draft assemblies) and failing to use `--retry` on unstable networks, which can result in incomplete downloads.

---

Ncbi-genome-download is a command-line Python tool that downloads genome assemblies from the NCBI FTP servers in a structured, batch-friendly way. It retrieves genomes by taxonomic group, assembly level, or RefSeq category, saving you from manual FTP browsing. Use this guide if you need to fetch prokaryotic or eukaryotic reference genomes for comparative genomics, metagenomic binning, or phylogenetic analysis. The tool is especially valuable when working with hundreds of species at once, as it automates file selection and retains the original directory structure. For authoritative background on NCBI databases, see the [NCBI Bookshelf](https://www.ncbi.nlm.nih.gov/books/).

Before you begin, understand that ncbi-genome-download does not run BLAST or process SRA data. It is purely a download manager for assembled genomes. For training on how to handle these files after download, the [EMBL-EBI Training](https://www.ebi.ac.uk/training/) portal offers courses on genome annotation and sequence analysis. This guide will walk you through core concepts, decision points, a safe workflow, common pitfalls, and the limits of what this tool can tell you.

## At a Glance

| Aspect | Detail |
|--------|--------|
| Primary purpose | Download bacterial, archaeal, fungal, and viral genome assemblies from NCBI |
| Input | Taxonomic group (e.g., bacteria), optional filter flags |
| Output | FASTA files (genomic, protein, CDS) plus accession lists |
| License | Free, open source |
| Dependencies | Python 3.6+, ftplib (standard library) |
| Typical use case | Batch retrieval of complete genomes for bioinformatics pipelines |

The table above summarizes the tool’s core function. It is not a replacement for NCBI’s web search or the Entrez API, but it excels at systematic downloads. The flexibility comes from command-line flags such as `--assembly-level` (complete, chromosome, scaffold, contig) and `--refseq-category` (reference, representative). For the most up-to-date documentation, always consult the [Galaxy Training Network](https://training.galaxyproject.org/), which includes hands-on lessons for batch genome handling.

## Decision Criteria

When should you choose ncbi-genome-download over other methods? The decision hinges on three factors: scale, automation, and specificity.

**Scale.** If you need a single genome, use the NCBI web browser or `wget` with a direct link. For 10 to 10,000 genomes, this tool saves time by resolving accessions automatically. It also creates a clean folder hierarchy grouped by assembly accession.

**Automation.** The tool integrates easily into scripts and pipelines. You can trigger downloads from continuous integration tests or nightly data refreshes. It logs output and errors predictably.

**Specificity.** You can filter by assembly level (e.g., only ‘complete’ genomes), taxonomic rank (e.g., genus Escherichia), or RefSeq quality. This is hard to replicate with generic FTP clients. For example, `ncbi-genome-download --taxid 562 bacteria` downloads only *Escherichia coli* assemblies.

Avoid the tool if you need raw reads or unassembled sequence data. For that, use the [NCBI Sequence Read Archive](https://www.ncbi.nlm.nih.gov/sra) tools like `fastq-dump`. Also avoid it if you need a very old assembly that NCBI may have retired. The tool pulls only current GenBank and RefSeq records.

## Practical Workflow or Implementation Steps

Follow these steps to safely download a large set of genomes. Execute each command in a Unix terminal (Linux or macOS). Windows users should use Windows Subsystem for Linux or a Python environment with Cygwin.

### 1. Install Python and the tool

```bash
pip install ncbi-genome-download
```

Verify installation with `ncbi-genome-download --help`. If you lack pip, install Python 3 from the official website. For guidance on setting up a bioinformatics environment, see the [Bioconductor](https://bioconductor.org/) documentation, which explains dependency management.

### 2. Choose a taxonomic group and filters

List all bacterial genera with the `--list-genera` flag:

```bash
ncbi-genome-download --list-genera bacteria
```

Select a genus, say Salmonella, and specify the assembly level. For example, to download only complete genomes:

```bash
ncbi-genome-download bacteria --genera Salmonella --assembly-level complete
```

The `--section` flag chooses between “genbank”, “refseq”, or “both”. Use “refseq” for non-redundant reference genomes. The “genbank” set includes all submitted assemblies, which can be redundant.

### 3. Run the download

Start with a dry-run to review the planned file list:

```bash
ncbi-genome-download bacteria --genera Salmonella --assembly-level complete --dry-run
```

If the list looks correct, remove `--dry-run`. The tool creates a subdirectory `refseq` or `genbank` containing species folders.

### 4. Retrieve multiple groups

You can chain commands or use a shell script. For every genus in a text file:

```bash
while read genus, do
    ncbi-genome-download bacteria --genera "$genius" --assembly-level complete || true
done < genera.txt
```

The `|| true` keeps the loop running even if a genus has no matching assemblies.

### 5. Verify the download

Check file count and size. A simple verification:

```bash
find . -name "*.fna.gz" | wc -l
```

Corrupt files can be detected by testing gzip integrity:

```bash
gzip -t *.fna.gz
```

If integrity fails, remove the file and rerun (the tool will skip already downloaded files unless you add `-F`).

## Quality Checks

After downloading, perform these checks to ensure the data are usable.

1. **File completeness.** Use `find` combined with `wc -l` to confirm each folder contains the expected files (genomic FASTA, protein FASTA, feature table, assembly report). Missing files often indicate network interruptions. Rerun the command with `--retry`.

2. **Assembly report consistency.** Each download includes a file like `*.assembly_report.txt`. Open it and confirm the “Genome Coverage” field is not empty. For a complete genome, coverage should be roughly 1.0 or higher. Low coverage suggests a draft assembly.

3. **Check against NCBI records.** Pick a random accession from the download and compare its total sequence length to the same record on the NCBI web browser. The values should match. The [NCBI Bookshelf](https://www.ncbi.nlm.nih.gov/books/) describes how to interpret assembly statistics.

4. **Species verification.** If you downloaded by taxid, verify that the assembly report lists the correct scientific name. Rarely, NCBI may reassign an assembly to a different species. Use `grep ORGANISM *assembly_report.txt` to inspect.

## Common Mistakes

- **Forgetting the `--flat-output` for simple scripts.** By default, the tool creates a nested folder structure. If you want all files in one directory, add `--flat-output`. Otherwise scripts expecting a single directory may break.

- **Using the wrong genus spelling.** NCBI uses official names (e.g., “Escherichia” not “E. coli”). The `--list-genera` flag shows only accepted names. Typing “staph” instead of “Staphylococcus” yields no downloads.

- **Omitting `--assembly-level`.** Without it, the tool downloads all levels including “contig” and “scaffold”. This can inflate data size massively. Always constrain with `--assembly-level complete` for finished genomes.

- **Mixing sections.** When you download from both “genbank” and “refseq”, you may get the same assembly twice. For most analyses, refseq is sufficient. Avoid `--section both` unless you need duplicates.

- **Not using `--retry` on unstable networks. The tool will fail silently if the connection drops mid download. Add `--retry 3` to reattempt up to three times.

- **Assuming all genomes are good for alignment.** Filter by assembly level, but also check the “assembly name” in the report. Some genomes are annotated as “Representative” but may have low contig N50. Use the assembly stats file to decide.

## Limits and Uncertainty

Ncbi-genome-download is a convenience wrapper, but it has well defined boundaries.

**It cannot download raw reads.** For high throughput sequencing data, use the [NCBI Sequence Read Archive](https://www.ncbi.nlm.nih.gov/sra) command-line tools. The tool also does not download RNA or protein datasets except those bundled in the assembly directory.

**It relies on NCBI FTP availability.** If the FTP servers are down or the directory structure changes, the tool may break. The maintainers usually update quickly, but there is a gap. Check the tool’s GitHub page for known outages.

**Taxonomic filtering is approximate.** The `--genera` flag uses the current NCBI taxonomy. If a species has been recently reclassified, it may appear under a different genus. Always verify the taxids from the NCBI Taxonomy Browser.

**Uncertainty in assembly quality.** ‘Complete’ means the assembly is closed and ordered, but it does not guarantee error free sequence. Single base errors, misassemblies, and missing genes can still occur. Always validate your own set of markers. The [EMBL-EBI Training](https://www.ebi.ac.uk/training/) resources on assembly validation provide best practices.

**Does not fetch metadata beyond assembly.** If you need BioSample attributes or submission details, use the NCBI Entrez Direct (EDirect) tool in parallel. This tool only downloads what is in the assembly directory.

## Frequently Asked Questions

**1. Can I use ncbi-genome-download to download eukaryotic genomes?**
Yes. Specify a group like “fungi” or “invertebrate”. The tool supports all major taxonomic divisions. For plants, use “plants”. The same flags apply, but eukaryotic genome files can be very large (multi-gigabyte). Consider limiting to “complete” and “chromosome” levels.

**2. How do I download genomes for a list of specific accessions?**
The tool does not directly accept a list of accessions. A workaround is to write a shell script that loops over accession numbers and uses `ncbi-genome-download --accession` for each. Note that only RefSeq and GenBank accessions (e.g., GCF\_0000001.1) work. The [Galaxy Training Network](https://training.galaxyproject.org/) includes an exercise on scripting these loops.

**3. What should I do if the download stops halfway?**
Use `--retry 5` and resume. The tool skips files that already exist. If a file is truncated, delete it and rerun. For large batches, consider running inside a `screen` session to keep the process alive even if you disconnect.

**4. Does the tool download everything again each time I run it?**
No, by default it does not overwrite existing files. This makes the tool safe for incremental updates. If you need to force a fresh download (e.g., after NCBI releases new versions), use the `-F` flag. The [Bioconductor](https://bioconductor.org/) documentation recommends versioning your downloaded data with dates to track changes.

## Related Clinical & Scientific Guides

* [Observational vs. Experimental Studies: How to Tell Them Apart](/blog/guides/observational-vs-experimental-studies-how-to-tell-them-apart)
* [Astrocyte Single Cell Rna Seq](/blog/guides/astrocyte-single-cell-rna-seq)
* [Structural Genes](/blog/guides/structural-genes)


## References and Further Reading

- [NCBI Bookshelf: Genome Assembly and Annotation](https://www.ncbi.nlm.nih.gov/books/NBK537222/)
- [EMBL-EBI Training: Genome browsing and analysis](https://www.ebi.ac.uk/training/online/courses/genome-browsing-and-analysis/)
- [Galaxy Training Network: Data download with ncbi-genome-download](https://training.galaxyproject.org/training-material/topics/genomics/tutorials/ncbi-genome-download/tutorial.html)
- [NCBI Sequence Read Archive documentation](https://www.ncbi.nlm.nih.gov/sra/docs/)
- [Bioconductor: GenomicRanges package for handling annotations](https://bioconductor.org/packages/release/bioc/html/GenomicRanges.html)
- [NCBI FTP site structure overview](https://ftp.ncbi.nlm.nih.gov/genomes/)
- [Python ncbi-genome-download GitHub repository](https://github.com/kblin/ncbi-genome-download)
- [NCBI Taxonomy Browser for checking taxids](https://www.ncbi.nlm.nih.gov/taxonomy)
- [Assembly quality metrics from NCBI](https://www.ncbi.nlm.nih.gov/assembly/help/faq/)

## Related Articles

- [Protein Synthesis](/blog/guides/protein-synthesis)
- [Incomplete Dominance Gene](/blog/guides/incomplete-dominance-gene)
- [Cell Membrane Function Biology](/blog/guides/cell-membrane-function-biology)
- [Dna Structure](/blog/guides/dna-structure)
- [Protein Structure](/blog/guides/protein-structure)