Zubair Khalid

Virologist/Molecular Biologist | Veterinarian | Bioinformatician

Conventional & Molecular Virology • Vaccine Development • Computational Biology

Dr. Zubair Khalid is a veterinarian and virologist specializing in conventional and molecular virology, vaccine development, and computational biology. Dedicated to advancing animal health through innovative research and multi-omics approaches.

Dr. Zubair Khalid - Veterinarian, Virologist, and Vaccine Development Researcher specializing in Computational Biology, Multi-omics, Animal Health, and Infectious Disease Research

Category: Guides

Ncbi-Genome-Download

Ncbi-genome-download is a command-line Python tool that downloads genome assemblies from the NCBI FTP servers in a structured, batch-friendly way. It retrieves genomes by taxonomic group, assembly level, or RefSeq category, saving you from manual FTP browsing. Use this guide if you need to fetch prokaryotic or eukaryotic reference genomes for comparative genomics, metagenomic binning, or phylogenetic analysis. The tool is especially valuable when working with hundreds of species at once, as it automates file selection and retains the original directory structure. For authoritative background on NCBI databases, see the NCBI Bookshelf.

Before you begin, understand that ncbi-genome-download does not run BLAST or process SRA data. It is purely a download manager for assembled genomes. For training on how to handle these files after download, the EMBL-EBI Training portal offers courses on genome annotation and sequence analysis. This guide will walk you through core concepts, decision points, a safe workflow, common pitfalls, and the limits of what this tool can tell you.

At a Glance

Aspect Detail
Primary purpose Download bacterial, archaeal, fungal, and viral genome assemblies from NCBI
Input Taxonomic group (e.g., bacteria), optional filter flags
Output FASTA files (genomic, protein, CDS) plus accession lists
License Free, open source
Dependencies Python 3.6+, ftplib (standard library)
Typical use case Batch retrieval of complete genomes for bioinformatics pipelines

The table above summarizes the tool’s core function. It is not a replacement for NCBI’s web search or the Entrez API, but it excels at systematic downloads. The flexibility comes from command-line flags such as --assembly-level (complete, chromosome, scaffold, contig) and --refseq-category (reference, representative). For the most up-to-date documentation, always consult the Galaxy Training Network, which includes hands-on lessons for batch genome handling.

Decision Criteria

When should you choose ncbi-genome-download over other methods? The decision hinges on three factors: scale, automation, and specificity.

Scale. If you need a single genome, use the NCBI web browser or wget with a direct link. For 10 to 10,000 genomes, this tool saves time by resolving accessions automatically. It also creates a clean folder hierarchy grouped by assembly accession.

Automation. The tool integrates easily into scripts and pipelines. You can trigger downloads from continuous integration tests or nightly data refreshes. It logs output and errors predictably.

Specificity. You can filter by assembly level (e.g., only ‘complete’ genomes), taxonomic rank (e.g., genus Escherichia), or RefSeq quality. This is hard to replicate with generic FTP clients. For example, ncbi-genome-download --taxid 562 bacteria downloads only Escherichia coli assemblies.

Avoid the tool if you need raw reads or unassembled sequence data. For that, use the NCBI Sequence Read Archive tools like fastq-dump. Also avoid it if you need a very old assembly that NCBI may have retired. The tool pulls only current GenBank and RefSeq records.

Practical Workflow or Implementation Steps

Follow these steps to safely download a large set of genomes. Execute each command in a Unix terminal (Linux or macOS). Windows users should use Windows Subsystem for Linux or a Python environment with Cygwin.

1. Install Python and the tool

pip install ncbi-genome-download

Verify installation with ncbi-genome-download --help. If you lack pip, install Python 3 from the official website. For guidance on setting up a bioinformatics environment, see the Bioconductor documentation, which explains dependency management.

2. Choose a taxonomic group and filters

List all bacterial genera with the --list-genera flag:

ncbi-genome-download --list-genera bacteria

Select a genus, say Salmonella, and specify the assembly level. For example, to download only complete genomes:

ncbi-genome-download bacteria --genera Salmonella --assembly-level complete

The --section flag chooses between “genbank”, “refseq”, or “both”. Use “refseq” for non-redundant reference genomes. The “genbank” set includes all submitted assemblies, which can be redundant.

3. Run the download

Start with a dry-run to review the planned file list:

ncbi-genome-download bacteria --genera Salmonella --assembly-level complete --dry-run

If the list looks correct, remove --dry-run. The tool creates a subdirectory refseq or genbank containing species folders.

4. Retrieve multiple groups

You can chain commands or use a shell script. For every genus in a text file:

while read genus, do
    ncbi-genome-download bacteria --genera "$genius" --assembly-level complete || true
done < genera.txt

The || true keeps the loop running even if a genus has no matching assemblies.

5. Verify the download

Check file count and size. A simple verification:

find . -name "*.fna.gz" | wc -l

Corrupt files can be detected by testing gzip integrity:

gzip -t *.fna.gz

If integrity fails, remove the file and rerun (the tool will skip already downloaded files unless you add -F).

Quality Checks

After downloading, perform these checks to ensure the data are usable.

  1. File completeness. Use find combined with wc -l to confirm each folder contains the expected files (genomic FASTA, protein FASTA, feature table, assembly report). Missing files often indicate network interruptions. Rerun the command with --retry.

  2. Assembly report consistency. Each download includes a file like *.assembly_report.txt. Open it and confirm the “Genome Coverage” field is not empty. For a complete genome, coverage should be roughly 1.0 or higher. Low coverage suggests a draft assembly.

  3. Check against NCBI records. Pick a random accession from the download and compare its total sequence length to the same record on the NCBI web browser. The values should match. The NCBI Bookshelf describes how to interpret assembly statistics.

  4. Species verification. If you downloaded by taxid, verify that the assembly report lists the correct scientific name. Rarely, NCBI may reassign an assembly to a different species. Use grep ORGANISM *assembly_report.txt to inspect.

Common Mistakes

  • Forgetting the --flat-output for simple scripts. By default, the tool creates a nested folder structure. If you want all files in one directory, add --flat-output. Otherwise scripts expecting a single directory may break.

  • Using the wrong genus spelling. NCBI uses official names (e.g., “Escherichia” not “E. coli”). The --list-genera flag shows only accepted names. Typing “staph” instead of “Staphylococcus” yields no downloads.

  • Omitting --assembly-level. Without it, the tool downloads all levels including “contig” and “scaffold”. This can inflate data size massively. Always constrain with --assembly-level complete for finished genomes.

  • Mixing sections. When you download from both “genbank” and “refseq”, you may get the same assembly twice. For most analyses, refseq is sufficient. Avoid --section both unless you need duplicates.

  • **Not using --retry on unstable networks. The tool will fail silently if the connection drops mid download. Add --retry 3 to reattempt up to three times.

  • Assuming all genomes are good for alignment. Filter by assembly level, but also check the “assembly name” in the report. Some genomes are annotated as “Representative” but may have low contig N50. Use the assembly stats file to decide.

Limits and Uncertainty

Ncbi-genome-download is a convenience wrapper, but it has well defined boundaries.

It cannot download raw reads. For high throughput sequencing data, use the NCBI Sequence Read Archive command-line tools. The tool also does not download RNA or protein datasets except those bundled in the assembly directory.

It relies on NCBI FTP availability. If the FTP servers are down or the directory structure changes, the tool may break. The maintainers usually update quickly, but there is a gap. Check the tool’s GitHub page for known outages.

Taxonomic filtering is approximate. The --genera flag uses the current NCBI taxonomy. If a species has been recently reclassified, it may appear under a different genus. Always verify the taxids from the NCBI Taxonomy Browser.

Uncertainty in assembly quality. ‘Complete’ means the assembly is closed and ordered, but it does not guarantee error free sequence. Single base errors, misassemblies, and missing genes can still occur. Always validate your own set of markers. The EMBL-EBI Training resources on assembly validation provide best practices.

Does not fetch metadata beyond assembly. If you need BioSample attributes or submission details, use the NCBI Entrez Direct (EDirect) tool in parallel. This tool only downloads what is in the assembly directory.

Frequently Asked Questions

1. Can I use ncbi-genome-download to download eukaryotic genomes? Yes. Specify a group like “fungi” or “invertebrate”. The tool supports all major taxonomic divisions. For plants, use “plants”. The same flags apply, but eukaryotic genome files can be very large (multi-gigabyte). Consider limiting to “complete” and “chromosome” levels.

2. How do I download genomes for a list of specific accessions? The tool does not directly accept a list of accessions. A workaround is to write a shell script that loops over accession numbers and uses ncbi-genome-download --accession for each. Note that only RefSeq and GenBank accessions (e.g., GCF_0000001.1) work. The Galaxy Training Network includes an exercise on scripting these loops.

3. What should I do if the download stops halfway? Use --retry 5 and resume. The tool skips files that already exist. If a file is truncated, delete it and rerun. For large batches, consider running inside a screen session to keep the process alive even if you disconnect.

4. Does the tool download everything again each time I run it? No, by default it does not overwrite existing files. This makes the tool safe for incremental updates. If you need to force a fresh download (e.g., after NCBI releases new versions), use the -F flag. The Bioconductor documentation recommends versioning your downloaded data with dates to track changes.

References and Further Reading

Related Articles