# Understanding NCBI FTP File Formats: GenBank, FASTA, GFF3, and More NCBI FTP file formats are the standardized text and binary file types that the National Center for Biotechnology Information (NCBI) distributes over its anonymous File Transfer Protocol (FTP) server, including FASTA for raw sequences, GenBank flat files for annotated records, GFF3 and .tbl tables for gene features, FASTQ for raw reads, and specialized formats such as SRA archives, ClinVar XML, and GEO SOFT files. Understanding them means knowing what each format stores, where it lives on the server, and how to parse it reliably in a reproducible workflow. The NCBI FTP site, now served over both FTP and HTTPS at ftp.ncbi.nlm.nih.gov, is the bulk download endpoint behind nearly every large-scale genomics project. The web interface at ncbi.nlm.nih.gov is designed for browsing single records, but any serious analysis, whether it is a genome annotation project, a variant database, or a reanalysis of public sequencing runs, eventually needs files rather than web pages. As sequencing throughput has grown, the raw data repositories have scaled accordingly, and the Sequence Read Archive (SRA) was built specifically as the public archive for next-generation sequencing data within the International [Nucleotide Sequence](/knowledge/molecular-biology/nucleotide-sequence) Database Collaboration, which links NCBI with the [European Bioinformatics Institute](/knowledge/bioinformatics/the-european-bioinformatics-institute-embl-ebi) and the DNA Data Bank of Japan [1]. Choosing the right file from the right directory, and knowing how to read it once downloaded, is a core skill that separates a working pipeline from hours of frustration. A reference dataset such as the Genome in a Bottle tumor-normal benchmark materials is distributed directly through an NCBI-hosted FTP directory, which shows how central this delivery mechanism remains for actively used, community-critical data [2]. ## What the NCBI FTP Site Is and How to Use It FTP, the File Transfer Protocol, is one of the oldest methods for moving files between computers over a network [3]. NCBI operates an anonymous FTP server, meaning anyone can connect without a password, and it also mirrors the same tree over HTTPS, which many users now prefer because HTTPS clients handle proxies and firewalls more gracefully. The root address is ftp.ncbi.nlm.nih.gov, and the directory layout is organized by database: genomes/refseq for reference genomes, genomes/genbank for the broader GenBank collection, sra for sequence reads, pub/clinvar for variation data, geo for [gene expression](/blog/guides/gene-expression) archives, and so on. The most common command-line tools are wget, curl, lftp, and rsync. To pull a single assembly summary file over HTTPS, you can use: ``` curl -O https://ftp.ncbi.nlm.nih.gov/genomes/refseq/vertebrate_mammalian/Homo_sapiens/annotation_results/README.txt ``` To mirror an entire directory recursively over FTP, wget works well: ``` wget -m --no-parent ftp://ftp.ncbi.nlm.nih.gov/genomes/refseq/bacteria/Escherichia_coli/reference/ ``` For large trees, rsync is often faster because it can resume interrupted transfers and only copies changed files: ``` rsync -av rsync://ftp.ncbi.nlm.nih.gov/genomes/refseq/bacteria/Escherichia_coli/reference/ ./ecoli_ref/ ``` Most files on the server are gzip-compressed, indicated by a .gz suffix, and some larger datasets are split into numbered chunks such as file.00.gz, file.01.gz, and so on, which keeps individual files within size limits that standard tools can handle. Almost every directory also contains a README file, a checksum file such as md5checksums.txt, and sometimes a manifest. Reading the README before downloading anything else is a habit worth building, because the file naming conventions within each database differ. Two complementary programmatic routes exist alongside FTP. NCBI's E-utilities API retrieves individual records on demand, and newer tooling such as the datasets command-line client packages metadata and download links for assemblies and sequences into a single call. Work that connects Entrez searches to structured file downloads has become a common pattern for building local databases from NCBI content [4]. The general rule is simple: use the web interface to explore, use E-utilities or datasets for a handful of records, and use the FTP tree for anything you would download repeatedly or at scale. ## Quick Reference Table of NCBI FTP File Formats The file formats you encounter most often on the server fall into a small number of families. Each stores a different layer of biological information, and they are frequently confused with one another because their names sound similar. | Format | What it stores | Typical location | How it is read | |--|--|--|--| | FASTA (.fa, .fna, .faa) | Nucleotide or protein sequences only | genomes/, refseq assemblies | Any text editor, Biopython, samtools faidx | | FASTQ (.fastq, .fq) | Raw sequencing reads with per-base quality scores | SRA pipelines, ENA mirrors | FastQC, cutadapt, aligners | | GenBank flat file (.gb, .gbff) | Sequence plus rich annotation and metadata | genomes/genbank/ or Entrez retrieval | Biopython, GeneMatrix, Artemis | | GFF3 (.gff, .gff3) | Gene feature coordinates, tab-delimited, 9 columns | Refseq/GFF directories | genomeTools, gffread, BED tools | | Feature table (.tbl) | Five-column tab-delimited NCBI submission annotation | Submission packages, annotation tools | NCBI pipelines, annotation checkers | | VCF (.vcf, .vcf.gz) | Variant positions and genotypes relative to a reference | Variation datasets, GIAB materials | bcftools, GATK | | ClinVar XML (.xml, VCV) | Variant-centric variation and interpretation records | pub/clinvar/xml/ | Python XML parsers, ClinVar tools | | SRA (.sra) | Compressed archive of raw sequencing data | sra/sra-instant/ | sra-tools (fasterq-dump) | | SOFT / MINiML (.soft, .txt) | Microarray and gene expression metadata plus values | geo/ series and platform directories | GEOquery in R | | Taxdump (names.dmp, nodes.dmp) | Taxonomy names and parent-child hierarchy | pub/taxonomy/ | Lineage parsers, taxid mappers | A general introduction to these common file formats, with worked examples of each, has long been part of the standard bioinformatics training literature [5]. The sections below go format by format, with enough structural detail to parse each one correctly. ## FASTA: The Universal Sequence Format FASTA is the simplest and oldest format on the server, and it stores sequence characters plus a single header line per record. The header begins with a greater-than symbol, followed by an identifier and an optional description. Every line after the header contains raw sequence characters until the next header appears. ``` >NP_000511.1 hemoglobin subunit alpha [Homo sapiens] MVLSPADKTNVKAAWGKVGAHAGEYGAEALERMFLSFPTTKTYFPHFDLSHGSAQVKGHG KKVADALTNAVAHVDDMPNALSALSDLHAHKLRVDPVNFKLLSHCLLVTLAAHLPAEFTP ``` Three conventions matter in practice. First, the identifier is the single word immediately after the greater-than symbol, and everything after the first space is a human-readable description that some parsers mistakenly treat as part of the identifier. Second, nucleotide files typically use the extensions .fna or .fa, protein files use .faa, and multi-organism collections often use .ffn for nucleotide coding sequences, though the internal format is identical in all cases. Third, ambiguity codes are legal: N means an unknown base, R means a purine, Y means a pyrimidine, and the full IUPAC code extends to other mixed positions. Tools that assume only ACGT will misbehave on genomes rich in N runs, which appear wherever a reference assembly could not resolve a gap. A typical RefSeq assembly directory contains several FASTA flavors for the same genome: a genomic FASTA with one record per contig or chromosome, a transcript FASTA, a protein FASTA, and translated coding-sequence files. Separating these files by type rather than mixing them is deliberate, and it lets alignment and annotation tools consume exactly the layer they need without filtering. FASTA files support random access after indexing. The samtools faidx command builds a companion .fai index that records byte offsets for each sequence, after which any subsequence can be extracted instantly: ``` samtools faidx genome.fna samtools faidx genome.fna NC_000001.11:1000000-1010000 ``` For most users, the practical mistake with FASTA is treating headers as unique when they are not. NCBI FASTA headers from an all-organisms dump can contain accession numbers, organism names, and titles, and duplicate accessions do occur when a file combines updates. Before building a database keyed on FASTA identifiers, deduplicate and confirm that the identifier you key on is actually unique within your file. ## FASTQ and Raw Read Data ![Table comparing FASTA and FASTQ by what each stores, its file extensions, lines per record and the tools that read it.](/article-images/illustrations/ncbi-ftp-file-formats-fasta-vs-fastq.webp) FASTQ extends FASTA by attaching a per-base quality score to each sequence. Each read occupies four lines: a header starting with the at symbol, the sequence itself, a plus sign separator, and a quality string of equal length in which each character encodes a phred-scaled probability that the base call is wrong. Raw read files are the input to essentially every modern sequencing analysis, and quality control on them is the first step of any pipeline. Integrated analysis suites now bundle FASTQ filtering, assembly, and even taxonomic assignment against NCBI databases into single applications, which reflects how standardized the FASTQ-to-analysis path has become [6]. On the NCBI FTP tree, raw reads live in the SRA rather than as plain FASTQ. The Sequence Read Archive stores submissions in its own compressed .sra format, with submission metadata kept separate from the read data, a design chosen precisely because of the explosive growth in sequencing volume that the archive has had to absorb [1]. Users convert .sra files to FASTQ locally with the fasterq-dump utility from the sra-tools package: ``` fasterq-dump SRR35776009 --split-files gzip SRR35776009_1.fastq ``` The, split-files flag matters for paired-end libraries, because without it a single interleaved FASTQ can appear, or worse, reads can be silently dropped. When planning a reanalysis project that needs many runs, download the .sra files rather than pre-converted FASTQ where possible, since the native format is smaller and the conversion is fast and deterministic. ## GenBank Flat Files The GenBank flat file is the richest nucleotide format NCBI distributes. It bundles the sequence, its coordinates, all annotated features such as genes, coding sequences, transfer RNAs, and mobile elements, plus citations, organism taxonomy, keywords, and free-text comments, all in a structured plain-text layout with sections delimited by keywords such as LOCUS, DEFINITION, ACCESSION, FEATURES, ORIGIN, and a terminating double slash. Because GenBank held approximately 4.7 billion sequences from over 580,000 species as of October 2024, these files underpin an enormous share of sequence retrieval work, and the bottleneck in many analyses has shifted from generating data to extracting and filtering it [7]. A condensed look at the anatomy of a flat file shows how features reference sequence positions: ``` LOCUS NC_000001 1000000 bp DNA linear CON DEFINITION Homo sapiens chromosome 1. ACCESSION NC_000001 FEATURES Location/Qualifiers source 1..1000000 /organism="Homo sapiens" gene 11874..14409 /gene="DDX11L1" CDS <1234..>5678 /codon_start=1 /product="example protein" ORIGIN 1 gattacaag ... // ``` Every feature carries a location, written as start..stop in 1-based coordinates, and qualifier lines beginning with a slash that attach attributes such as /gene, /product, and /db_xref. The less-than and greater-than symbols inside locations mark partial features whose boundaries fall outside the available sequence. Location strings can also express joins for exons split across positions, and compl for features on the reverse strand, so robust parsers must handle these compound expressions rather than assuming simple intervals. Users most often obtain GenBank flat files through Entrez retrieval rather than bulk FTP, for example: ``` efetch -db nuccore -id NC_000001.11 -format gb -out chr1.gb ``` Once downloaded, the format supports extraction, filtering, and aggregation of individual genes or whole small genomes such as viruses and mitochondria, and dedicated applications exist precisely to automate that selection and alignment work for non-programmers [7]. The parser ecosystem is mature: Biopython's SeqIO module reads flat files into rich feature objects, and graphics tools such as Artemis render the annotation visually. One frequent source of confusion is that custom GenBank-style files, the kind exported by desktop annotation software, are not themselves acceptable for direct database submission. Converting them requires generating NCBI's submission-oriented formats, and a web tool exists that transforms custom GenBank annotation into the direct submission format while also producing the required five-column tab-delimited feature table and a FASTA file [8]. The distinction matters because a file that looks like a GenBank record and a file that passes NCBI submission validation are not the same thing. ## GFF3 and Annotation Tables (.tbl) GFF3, the Generic Feature Format version 3, is the annotation format most often distributed alongside genome assemblies on the FTP tree. It is a tab-delimited text format with exactly nine columns per line: sequence identifier, annotation source, feature type, start coordinate, stop coordinate, score, strand, phase, and an attributes column holding key-value pairs. A comment line beginning with a hash can define sequence regions, and the directive lines beginning with ## support a controlled parent-child structure through the ID and Parent attributes. ``` NC_000001.11 RefSeq gene 11874 14409 . + . ID=gene:DDX11L1;Name=DDX11L1 NC_000001.11 RefSeq mRNA 11874 14409 . + . ID=transcript:NM_001;Parent=gene:DDX11L1 NC_000001.11 RefSeq exon 11874 12227 . + . Parent=transcript:NM_001 NC_000001.11 RefSeq CDS 11874 12227 . + 0 ID=cds:NP_001;Parent=transcript:NM_001 ``` Three details cause most parsing errors. GFF3 coordinates are 1-based and inclusive, so a feature spanning bases 11874 through 14409 reports exactly those numbers, unlike zero-based BED files. The phase column records how many bases to skip at the start of a coding feature to reach the first complete codon, and ignoring it corrupts translated sequences. And because attributes are separated by semicolons within the ninth column, naive splitting on any semicolon anywhere in a line destroys parent-child links, so real parsers must respect column boundaries. Alongside GFF3, NCBI maintains its own annotation table format, the .tbl file, which is a five-column, tab-delimited feature table used for genome submission and internal annotation exchange. Tools have been built specifically to convert a draft assembly and its annotation set into the .tbl format to facilitate submission to NCBI's genome assembly resources, precisely because many labs lack the backend infrastructure that large genome centers use for this step [9]. The .tbl format is also the only annotation source available for some model organisms, and pipelines designed around it can classify genome-wide variants by genomic region and mutation class for species whose chromosome files are annotated in .tbl form rather than in other annotation table formats [10]. A .tbl line looks like this: ``` >Feature NC_000001.11 11874 14409 gene gene DDX11L1 11874 14409 mRNA product some transcript ``` The first line names the sequence with a Feature directive, subsequent rows give start, stop, and feature type, and indented qualifier rows attach attributes. Coordinates are 1-based inclusive on the plus strand and written in descending order on the minus strand, which is a convention that trips up anyone porting code between GFF3 and .tbl. When you download a RefSeq assembly package, both the GFF3 file and its companion genomic FASTA are usually present together, and the README documents which features are included. ## Variation Formats: VCF and ClinVar XML Variant data follows a different logic from sequence and annotation data. The Variant Call Format (VCF) stores positions at which a sample differs from a reference, together with genotypes, quality metrics, and filter status. VCF is a tab-delimited, header-rich text format that is usually kept gzip-compressed and indexed as .vcf.gz with a tabix index. Community benchmark materials distributed through NCBI's FTP directories include genome-scale variant call sets in exactly this form, which laboratories then use to evaluate their own calling pipelines [2]. ClinVar, NCBI's public archive of human genetic variants and their interpreted relationships to disease, distributes its bulk content through FTP and through programmatic access, and its variant-centric XML, which organizes records around Variation ClinVar (VCV) accessions, reached production release in April 2019 with support from both the website and E-utilities [11]. The variant-centric XML files on the FTP tree live under pub/clinvar/xml/ and are large, compressed, and often chunked. Each VCV record nests the variant's sequence context, submitted interpretations from multiple laboratories, and aggregation of those submissions, so parsing requires an XML reader rather than line-based text tools: ```python import xml.etree.ElementTree as ET for event, elem in ET.iterparse("clinvar_variation.xml", events=("end",)): if elem.tag.endswith("VariationArchive"): rcv = elem.find(".//RCVList") # process record, then clear memory elem.clear() ``` Streaming iteration is essential here. Loading a full ClinVar XML release into memory on a modest workstation fails, whereas an iterparse loop that clears finished elements keeps memory flat. The interpretation logic inside these records, in which multiple submitters' assertions are aggregated into consensus clinical significance, is precisely what the redesign of ClinVar's data presentation and programmatic access was intended to make more accessible [11]. For most analyses, users pair the XML with the tab-delimited summary files in the same directory, variant_summary.txt being the flat-file view of the same content, and reserve full XML parsing for cases that need submitter-level detail. ## GEO Formats: SOFT and MINiML Gene expression and [functional genomics](/blog/guides/functional-genomics) archives use their own metadata-rich formats. The [Gene Expression Omnibus](/blog/guides/gene-expression-omnibus) (GEO) distributes its data as SOFT (Simple Omnibus Format in Text) and MINiML (MIAME Notation in Markup Language) files, which together carry sample metadata, platform descriptions, series organization, and processed data matrices [12]. SOFT files use a simple keyword grammar in which lines beginning with an exclamation mark declare entities and attributes: ``` ^SAMPLE = GSM1 !Sample_title = control replicate 1 !Sample_organism_ch1 = Homo sapiens !Sample_characteristics_ch1 = tissue: liver ``` MINiML expresses the same content as XML. Both formats exist because microarray and sequencing expression experiments carry structured metadata that plain matrices cannot hold, and the archive's standards and services were designed to make that metadata machine-readable for large-scale reanalysis [12]. Within the GEO FTP tree, series directories contain both the metadata files and processed value tables, and users running bulk expression analyses typically parse SOFT files for sample grouping while reading the processed matrices for values. ## Companion Files: Taxonomy, Metadata, Checksums Several small but critical file types appear throughout the FTP tree. The taxonomy directory contains names.dmp and nodes.dmp, pipe-delimited files that map numeric taxonomy identifiers to scientific names and to their parent-child hierarchy, and these identifiers are the glue that connects sequence records to organism lineages across all NCBI databases. Because scientific name reconciliation is a persistent task across biodiversity resources, and because multiple services including NCBI provide name resolution tools with differing scopes and outputs, using the authoritative taxdump files directly avoids a whole class of naming ambiguity [13]. Checksum files deserve attention in any serious workflow. The md5checksums.txt file present in most assembly and annotation directories lists an MD5 digest for every file, and verifying downloaded files against it catches truncated or corrupted transfers before they propagate silently into a pipeline: ``` md5sum -c md5checksums.txt ``` Finally, README and manifest files document the naming scheme in force for that directory. File naming on NCBI FTP is consistent within a database but not across databases, so a convention you learned in the genomes tree may not apply in pub/clinvar or geo, and the local README is the contract that tells you what each suffix means. ## A Practical Download and Parse Workflow ![Flow of discover, download, verify and parse phases with the tool used at each step.](/article-images/illustrations/ncbi-ftp-file-formats-download-parse-workflow.webp) A repeatable workflow for NCBI FTP data has four phases: discover, download, verify, and parse. 1. **Discover.** Use the web interface or the datasets client to identify the exact accession or assembly you need, and record it in a small metadata file. Programmatic pipelines that link Entrez searches to structured download lists make this discovery step reproducible rather than ad hoc [4]. 2. **Download.** Prefer rsync or the HTTPS mirror for large trees, and use wget with the mirror flag only for complete directory copies. Decompress with gunzip as needed, and keep the original compressed files if disk permits. 3. **Verify.** Run the checksum file against your downloads with md5sum -c. This single command prevents the most insidious failure mode in bulk data work, in which a partially transferred file parses without errors and produces quietly wrong results. 4. **Parse.** Match the tool to the format. Biopython for GenBank flat files, genomeTools or gffread for GFF3, bcftools for VCF, sra-tools for SRA archives, and streaming XML parsers for ClinVar. Integrated desktop suites can also carry a project from raw read quality control through assembly and taxonomic assignment against NCBI databases in one environment, which is a reasonable choice for teams without command-line infrastructure [6]. A minimal end-to-end example for one bacterial genome looks like this: ``` rsync -av rsync://ftp.ncbi.nlm.nih.gov/genomes/refseq/bacteria/Escherichia_coli/reference/ ./eco/ gunzip eco/GCF_000005845.2_ASM584v2_genomic.fna.gz md5sum -c eco/md5checksums.txt samtools faidx eco/GCF_000005845.2_ASM584v2_genomic.fna ``` From there, the indexed FASTA supports arbitrary subsequence extraction, the GFF3 file provides gene coordinates for intersection analysis, and the protein FASTA feeds downstream annotation searches. Keeping the rsync source URL in a shell script or workflow file makes the entire data acquisition step reproducible, which matters when a reviewer or collaborator needs to rebuild your dataset months later. The same discipline applies to proteomics and expression data. Mass spectrometry repositories capture everything from original spectra files through experimental metadata to conclusion-level results, and their acceptance of commonly used open formats with syntactic validation established the pattern that later archives followed [14]. Whenever you step outside the sequence world, check whether the FTP directory supplies a format definition or schema alongside the data, because validated, schema-described files parse far more predictably than free-text exports. ## Common Mistakes and Limitations **Confusing coordinate systems.** GFF3 and GenBank features are 1-based inclusive. BED files and many programmatic intervals are zero-based half-open. Translating coordinates between them without an offset by one error produces off-by-one gene boundaries that look plausible in a browser and are wrong in every downstream calculation. **Assuming file names mean the same thing everywhere.** A .faa is a protein FASTA in a RefSeq assembly directory, but suffix conventions in ClinVar, GEO, and SRA directories are unrelated. Always consult the README in the specific directory you are downloading from. **Ignoring the checksum files.** Silent truncation during large FTP transfers is common enough that skipping md5sum verification is the single most costly shortcut in bulk genomics. Corrupted sequence files can parse successfully while containing wrong bases. **Treating GenBank-format files as submission-ready.** Desktop annotation exports that resemble GenBank records are not accepted for direct submission, and conversion through the proper submission formats, including the five-column feature table, is a required step [8]. Planning a submission around the flat file alone wastes effort. **Parsing attribute strings naively.** Splitting GFF3 lines on every semicolon, or GenBank qualifiers on every slash, breaks legitimate values that contain those characters. Use a real parser library rather than string splitting for anything you will publish or archive. **Loading large XML into memory.** ClinVar XML releases and similar archives exceed the memory of a typical laptop when read wholesale. Streaming parsers that process and release each record keep these files tractable. **Assuming annotation completeness is uniform.** RefSeq reference genomes carry curated annotation, while many GenBank entries carry whatever the original submitter provided. Comparing gene counts between a reference assembly and a submitted assembly without checking annotation provenance produces misleading conclusions. **Relying on a single mirror snapshot.** The FTP tree changes as records are updated and files are repackaged. A pipeline that pins itself to a dated release and records checksums will reproduce its results months later, while one that pulls from the live tree whenever convenient will not. ## Quick Review - NCBI serves its bulk data at ftp.ncbi.nlm.nih.gov over FTP, HTTPS, and rsync, organized into database-specific directory trees with README and checksum files in nearly every directory. - FASTA stores sequence only, FASTQ adds per-base quality scores, and the two are frequently mistaken for interchangeable formats. - The GenBank flat file bundles sequence, features, citations, and taxonomy in one structured text record, and its feature locations use 1-based coordinates with join and complement expressions for complex features. - GFF3 is a nine-column tab-delimited annotation format with parent-child relations, and NCBI's own .tbl five-column feature table serves genome submission and species that lack other annotation tables [9]. - Variant data appears as VCF for call sets and as ClinVar variant-centric XML for aggregated clinical interpretations keyed on VCV accessions [11]. - Raw sequencing reads are stored natively in SRA format and converted to FASTQ locally with fasterq-dump [1]. - GEO distributes SOFT and MINiML files that pair expression values with structured sample metadata [12]. - Verify every bulk download against md5checksums.txt, and parse complex formats with established libraries rather than string splitting. ## Frequently Asked Questions ### What does the NCBI FTP site actually contain? It mirrors the bulk content of NCBI's databases as files: reference and GenBank genome assemblies with their FASTA and GFF3 annotations, raw sequencing data through the SRA, ClinVar variation records as XML and flat files, GEO expression archives as SOFT and MINiML, taxonomy dumps, and supporting documentation. If a database has a web interface, its underlying data almost always has an FTP or HTTPS equivalent. ### Which file should I download for a genome, FASTA or GFF3? Both, because they answer different questions. The genomic FASTA holds the sequence itself, and the GFF3 file holds the gene models that say where features sit on that sequence. A complete assembly directory usually also offers protein FASTA, transcript FASTA, and a README that explains the naming of each file. ### How do I convert an SRA file to FASTQ? Install sra-tools and run fasterq-dump with the accession number, using the, split-files flag for paired-end libraries so each mate lands in its own FASTQ. Converting locally from the native .sra file is more bandwidth-efficient than downloading preconverted FASTQ, because the native archive format is smaller and was designed for exactly this distribution pattern [1]. ### Why does my GFF3 parsing code give wrong gene coordinates? The usual cause is a coordinate-system mismatch. GFF3 is 1-based and inclusive, while many analysis formats are zero-based half-open, so coordinates translated between them need a one-position adjustment. A second common cause is splitting attribute columns on semicolons, which destroys the parent-child identifiers that link exons to transcripts and transcripts to genes. ### Are GenBank flat files the same thing as what I submit to NCBI? No. Custom GenBank-format files exported from desktop annotation tools are convenient for local work but are not accepted for direct database submission. The submission path requires NCBI's own formats, including a five-column tab-delimited feature table and a FASTA file, and converter tools exist to produce them from GenBank-style annotation automatically [8]. ### How do I handle ClinVar XML files without running out of memory? Stream the file with an incremental XML parser that processes and releases each record as it completes, rather than loading the entire document. Pair the XML with the tab-delimited variant summary files in the same FTP directory when you only need the aggregated interpretation level, and reserve full XML parsing for cases that require submitter-level detail from the variant-centric records [11]. ## Sources 1. [The sequence read archive: explosive growth of sequencing data](https://pubmed.ncbi.nlm.nih.gov/22009675/) 2. [Abstract 3551: Continued analysis of extensive data towards Genome in a Bottle benchmarks for a new tumor normal pair](https://doi.org/10.1158/1538-7445.am2024-3551) 3. [Ftp- The File Transfer Protocol](https://doi.org/10.1007/978-1-4020-6754-9_6281) 4. [Alembic: a framework for converting disparate biological data into structured resources](https://pubmed.ncbi.nlm.nih.gov/42111795/) 5. [Common File Formats](https://pubmed.ncbi.nlm.nih.gov/18428774/) 6. [NSeqVerify: An Easy-to-Use Desktop Suite for Integrated NGS Data Analysis, from Raw Reads to Taxonomic Assignment](https://doi.org/10.1101/2025.10.31.685854) 7. [User Guided Selection and Alignment of Sequence Data by GeneMatrix](https://pubmed.ncbi.nlm.nih.gov/42184002/) 8. [GB2sequin - A file converter preparing custom GenBank files for database submission.](https://pubmed.ncbi.nlm.nih.gov/29842948/) 9. [Genome Annotation Generator: a simple tool for generating and correcting WGS annotation tables for NCBI submission](https://pubmed.ncbi.nlm.nih.gov/29635297/) 10. [SNPAAMapperT2K: A genome-wide SNP downstream analysis and annotation pipeline for species annotated with NCBI.tbl data files](https://pubmed.ncbi.nlm.nih.gov/25512690/) 11. [ClinVar: improvements to accessing data](https://pubmed.ncbi.nlm.nih.gov/31777943/) 12. [NCBI GEO standards and services for microarray data](https://pubmed.ncbi.nlm.nih.gov/17160034/) 13. [Can We Standardize Name Reconciliaton via OpenRefine?](https://doi.org/10.3897/biss.8.134910) 14. [NCBI Peptidome: a new repository for mass spectrometry proteomics data](https://pubmed.ncbi.nlm.nih.gov/19942688/)