# How to Convert VCF to BCF and Back: A Practical Guide By the end of this guide you will be able to take a variant file in VCF format, compress and index it correctly, convert it to BCF in a single command, use the binary file with the tools that require random access, and convert it back to VCF without losing a single sample, genotype, or annotation field. You will also know what each of the common failure messages actually means, and you will have a repeatable check that proves the round trip worked. You need three things on hand before you start: a variant file (`.vcf` or `.vcf.gz`), a working installation of `bcftools` with `htslib`, and enough free disk space to hold a temporary copy of the data. A compressed VCF and its BCF equivalent both fit comfortably on a laptop for cohort files up to a few hundred samples. Whole-population files need a server and a plan for temporary space. The commands below are the ones you will actually type. Every one has been kept to standard `bcftools` syntax rather than shorthand that only works in one shell. ## What VCF and BCF Actually Are Both formats hold the same information. The difference is how they store it. ### VCF is text you can read Variant Call Format (VCF) is a plain-text format for describing positions in a genome where a sample differs from a reference. A VCF file has two parts. The header is a block of lines beginning with `##` that declare the file format version, the reference build, the contig list, and every INFO, FILTER, and FORMAT field used later. The column header line begins with a single `#` and names the fixed columns plus one column per sample. The fixed columns are, in order: `CHROM`, `POS`, `ID`, `REF`, `ALT`, `QUAL`, `FILTER`, `INFO`, and `FORMAT`. Everything after `FORMAT` is a per-sample value block. Genotypes live in that block under the `GT` key. A minimal record looks like this: ``` chr1 1000 rs123 A G 50 PASS AF=0.25;DP=40 GT:DP 0/1:18 ``` Text formats are easy to inspect, easy to diff, and easy to break by hand. They are also slow to parse. A parser has to convert every number from characters to binary on every read, and random access requires a separate index that maps genomic intervals to byte offsets in the compressed file. ### BCF is the same data in a binary container Binary Call Format (BCF) stores the same header and the same records, but pre-encoded as typed binary values and compressed with BGZF, a block-compressed variant of gzip that allows random access. The header is stored once in a compact dictionary, and each record references header entries by integer ID instead of repeating long INFO keys. BCF files begin with a five-byte magic number, `BCF` followed by a major and minor version byte, and then the BGZF stream. You can see those bytes if you decompress the first few: ``` zcat cohort.bcf | head -c 5 | xxd 00000000: 4243 4602 02 BCF.. ``` That reads as `BCF`, version 2.2. The version matters. A tool built against a very old copy of the HTSlib library may refuse a 2.2 file. In that case, keep a `.vcf.gz` export for that tool rather than fighting the binary. Because BCF is typed, an INFO field declared as `Integer` is stored as a 4-byte integer, not as the characters `40`. This is why BCF files load faster in `bcftools`, `GATK`, and any other tool built on HTSlib. It is also why BCF is not something you edit in a text editor. ### FASTQ and FASTA hold sequences, not variants These two formats come up in the same conversations as VCF and BCF, and confusing them wastes time. FASTA is a text format for one or more nucleotide or protein sequences. Each record starts with a `>` line holding the identifier and description, followed by the sequence on one or more lines. There are no quality scores. ``` >chr1 GATCACAGGTCTATCACCCTATTAACCACTCACGGGAGCTCTCCATGCATTTGGTATTTTCGTCTGGGGGGT ``` FASTQ is a text format for sequencing reads. Each record uses four lines: a `@` header, the sequence, a `+` separator, and a quality string of the same length as the sequence. The quality characters encode per-base error probabilities using Phred scores offset by 33. ``` @read1 GATCACAGGTCTATCACCCTATTA + IIIIIIIIIIIIIIIIIIIIIIIII ``` The difference in one sentence: FASTA carries sequence only, FASTQ carries sequence plus a quality score for every base. Neither carries variants, genotype likelihoods, or sample-level allele calls. You do not convert a FASTQ to a VCF directly. Reads are aligned to a reference first, producing SAM, BAM, or CRAM files, and variants are called from those alignments. BCF enters the picture after calling, when you want the calls in a fast, indexable binary form. ### Where each format sits | Format | Content | Encoding | Typical role | |--|--|--|--| | FASTA | Reference or consensus sequence | Text | Reference genome, contigs | | FASTQ | Reads plus per-base quality | Text | Raw sequencing output | | SAM | Alignments plus header | Text | Intermediate alignment | | BAM | Alignments plus header | Binary, BGZF | Aligned reads, sorted and indexed | | CRAM | Alignments, reference-based compression | Binary | Compressed archive of alignments | | VCF | Variants plus genotypes | Text | Exchange, review, small files | | BCF | Variants plus genotypes | Binary, BGZF | Production storage, fast random access | | BED | Genomic intervals | Text | Regions, targets, masks | | GFF/GTF | Gene and feature annotation | Text | Annotation tracks | If you remember one thing from this table, remember that FASTA and FASTQ describe sequence, while VCF and BCF describe differences from a reference. ## Why Convert VCF to BCF Converting VCF to BCF is worth doing for four practical reasons. The first is speed. Because values are pre-typed, reading a BCF file avoids repeated string parsing. Commands that touch every record, such as `bcftools stats`, `bcftools norm`, and `bcftools merge`, finish noticeably faster on BCF input. On large cohort files this is the difference between an interactive command and a lunch break. The second is size. A BCF built from an uncompressed VCF is dramatically smaller because it is BGZF-compressed and because repeated header strings are replaced by integer references. A BCF built from an already compressed VCF is usually smaller again, though the gain is modest. The third is random access. BCF files are indexed with a CSI index, which supports genomic intervals larger than the 512 megabase ceiling of the older tabix TBI index. Cohorts on large chromosomes or with many contigs benefit from this. The fourth is pipeline hygiene. Many tools either require an index or silently run much slower without one. Giving them BCF plus `.csi` removes a whole class of avoidable failures. The counterargument is interoperability. Some tools, some web portals, and some collaborators only accept text VCF. That is why you keep the round trip working in both directions. ## Install the Tools First Everything in this guide uses `bcftools` and the companion utilities `bgzip`, `tabix`, and `htsfile`, all of which ship with HTSlib. On a Debian or Ubuntu system: ``` sudo apt-get update sudo apt-get install bcftools tabix ``` On macOS with Homebrew: ``` brew install bcftools htslib ``` With Conda or Mamba, which is the most reproducible route: ``` conda create -n vcfwork -c conda-forge -c bioconda bcftools htslib conda activate vcfwork ``` Confirm what you have before running anything: ``` bcftools --version ``` Expected output looks like this, with the version and library versions on separate lines: ``` bcftools 1.19 Using htslib 1.19 Copyright (C) 2023 [Genome Research](/blog/guides/genome-research) Ltd. ``` Also confirm that `htsfile` is present. It is the quickest way to ask a file what it is: ``` htsfile cohort.vcf.gz ``` ``` cohort.vcf.gz: VCF version 4.2 ``` If your file is truncated or is plain gzip rather than BGZF, `htsfile` will often tell you before `bcftools` does. ## Convert VCF to BCF Step by Step This walkthrough starts from an uncompressed VCF named `cohort.vcf` and ends with `cohort.bcf` plus `cohort.bcf.csi`. ### Step 1: Inspect the input Read the header before you convert anything. Header problems survive conversion and become harder to spot later. ``` bcftools view -h cohort.vcf | head -20 ``` ``` ##fileformat=VCFv4.2 ##FILTER= ##fileDate=20260101 ##reference=file:///refs/GRCh38.fa ##contig= ##INFO= ##INFO= ##FORMAT= ##FORMAT= #CHROM POS ID REF ALT QUAL FILTER INFO FORMAT SAMPLE1 SAMPLE2 ``` Two things to check here. The `##fileformat` line must be the very first line of the file. The `#CHROM` line must list exactly as many sample columns as there are sample names. If those are wrong, fix them now. ### Step 2: Confirm the file is valid enough to convert ``` bcftools view -H cohort.vcf | wc -l ``` This prints a record count. It also surfaces parsing warnings to standard error. Suppress output and keep only the count if the file is large: ``` bcftools view -H cohort.vcf 2>/dev/null | wc -l ``` ``` 48213 ``` Write that number down. You will compare against it later. ### Step 3: Sort if the file is not already sorted Indexing and most interval operations require records ordered by contig and then by position. Check first: ``` bcftools index --stats cohort.vcf ``` That command will fail on an uncompressed VCF, which is expected. For sorting, use `bcftools sort` with an explicit temporary directory on a disk with room: ``` bcftools sort -T /scratch/tmp -Oz -o cohort.sorted.vcf.gz cohort.vcf ``` ``` Writing to /scratch/tmp/bcftools.XXXXXX Merging 3 temporary files Cleaning Done ``` The `-T` option matters on shared servers. Without it, `bcftools sort` writes to the system temporary directory, which is often small and sometimes read-only. If you skip sorting and your file is unsorted, indexing fails later with an unsorted positions error. ### Step 4: Compress to BGZF VCF If your file is plain text, make the compressed VCF version first. This is the archival copy most collaborators will want. ``` bcftools view -Oz --threads 4 -o cohort.vcf.gz cohort.sorted.vcf.gz ``` Alternatively, use `bgzip` directly: ``` bgzip -@ 4 -c cohort.vcf > cohort.vcf.gz ``` Do not use `gzip`. The two produce files with the same `.gz` extension and different internal structure. Plain gzip cannot be indexed, and `bcftools` will refuse to index it. This is the single most common conversion mistake. ### Step 5: Index the compressed VCF ``` tabix -p vcf cohort.vcf.gz ``` or, equivalently inside the bcftools family: ``` bcftools index -t cohort.vcf.gz ``` Both produce `cohort.vcf.gz.tbi`. The `-t` flag asks for a tabix index. Drop the flag and `bcftools index` writes a CSI index instead (`.csi`), which is the better choice for large chromosomes and is the only index type BCF accepts. ### Step 6: Convert VCF to BCF This is the core command: ``` bcftools view -Ob --threads 4 -o cohort.bcf cohort.vcf.gz ``` Read it as: `bcftools view`, output type BCF (`-Ob`), use four compression threads, write to `cohort.bcf`, read from `cohort.vcf.gz`. `bcftools` detects the input format automatically, so the same command works whether the input is `.vcf`, `.vcf.gz`, or `.bcf`. Append a digit from 0 to 9 to control compression. `-Ob0` means no compression, which is fast to write and large on disk. `-Ob9` means maximum compression, which is slow and small. The default sits in the middle and is fine for most work. ### Step 7: Index the BCF ``` bcftools index cohort.bcf ``` ``` ls -lh cohort.bcf* ``` ``` -rw-r--r-- 1 user staff 412M cohort.bcf -rw-r--r-- 1 user staff 71M cohort.bcf.csi ``` The `.csi` file is the index. Without it, interval queries and most multi-file operations fail. Note that this step needs a seekable file on disk. You cannot index a stream coming through a pipe. ### Step 8: Confirm the conversion preserved the records ``` bcftools view -H cohort.bcf | wc -l ``` ``` 48213 ``` The number must match the count from Step 2. If it does not, stop and investigate before you delete the source file. Common causes are a malformed record that the parser skipped with a warning, or a filter accidentally applied during conversion. A useful compact check is the allele count summary, which exercises the genotype fields: ``` bcftools stats cohort.vcf.gz | grep "^SN" ``` ``` SN 0 number of samples: 2 SN 0 number of records: 48213 SN 0 number of no-ALTs: 0 SN 0 number of SNPs: 40122 SN 0 number of MNPs: 14 SN 0 number of indels: 8077 ``` Run the same command on `cohort.bcf` and compare. The SNP and indel counts should be identical. ## Convert BCF Back to VCF The reverse direction uses the same subcommand with a different output flag. To produce an uncompressed VCF for inspection: ``` bcftools view -Ov --no-version -o restored.vcf cohort.bcf ``` To produce a compressed, indexable VCF for sharing: ``` bcftools view -Oz --threads 4 --no-version -o restored.vcf.gz cohort.bcf tabix -p vcf restored.vcf.gz ``` The `-no-version` flag suppresses the two header lines that `bcftools view` normally appends, `##bcftools_viewVersion` and `##bcftools_viewCommand`. Without it, the restored file's header differs from the original even when every record is identical. That is harmless for analysis and annoying for byte-level comparison. You can also pipe the whole thing without writing intermediates: ``` bcftools view -Ou cohort.bcf | bcftools norm -m -any -Oz -o split.vcf.gz ``` The `-Ou` output type is uncompressed BCF. It is the right choice for the middle of a pipeline because it avoids paying for compression twice, and the receiving `bcftools` process can parse it directly. ## The Options That Actually Matter | Option | Meaning | When to use it | |--|--|--| | `-Ov` | Output plain text VCF | Inspection, small files, tools that need text | | `-Oz` | Output BGZF-compressed VCF | Sharing, archival, tabix indexing | | `-Ob` | Output BGZF-compressed BCF | Default working format | | `-Ou` | Output uncompressed BCF | Middle of a pipe, fastest handoff | | `-o FILE` | Write to a file instead of stdout | Always, for anything you keep | | `-threads N` | Threads for compression and decompression | Files above roughly a gigabyte | | `-Ob0` to `-Ob9` | Compression level | `1` for scratch files, default for kept files | | `-no-version` | Omit `bcftools` command lines from the header | Round-trip comparison, reproducible outputs | | `-s LIST` | Subset to named samples | Before conversion, to shrink the file | | `-i EXPR` / `-e EXPR` | Include or exclude by expression | Before conversion, to shrink the file | | `-write-index` | Write the index in the same pass | Recent releases, saves a second pass | | `-T DIR` | Temporary directory (for `bcftools sort`) | Shared servers with small `/tmp` | Two behaviors of `bcftools view` catch people out. Its default output type is plain VCF, so a bare `bcftools view in.bcf` prints text to your terminal rather than making a binary file. And its default output target is standard output, so forgetting `-o` produces a screen full of records or a broken pipe when the terminal closes. ## Verify the Round Trip A conversion is only trustworthy if you check it. Here is a complete round-trip test you can run on any file. ``` # 1. Records and header from the original bcftools view -H cohort.vcf.gz > original.records.txt bcftools view -h cohort.vcf.gz > original.header.txt # 2. Convert out and back bcftools view -Ob -o cohort.bcf cohort.vcf.gz bcftools view -Ov --no-version -o restored.vcf cohort.bcf # 3. Records and header from the restored file bcftools view -H restored.vcf > restored.records.txt bcftools view -h restored.vcf > restored.header.txt # 4. Compare diff -q original.records.txt restored.records.txt diff -q original.header.txt restored.header.txt ``` Two silent `diff` commands mean the content is identical. Any output means it is not, and you should look at the first few differing lines with `diff | head`. A faster checksum approach when the files are large: ``` md5sum original.records.txt restored.records.txt ``` Do not compare the BCF files themselves with `md5sum`. Two BCF files with identical content can have different bytes because of compression block boundaries and gzip header metadata. Compare the decoded records instead. The same caution applies to `.vcf.gz` files. If a record-level diff shows differences, the usual causes are, in order: floating-point reformatting inside INFO or FORMAT fields, a change in the order of fields inside INFO because the original file was not written in header order, and a change in the representation of missing genotypes. All three are cosmetic. Changes in the number of records or in the `GT` values are not cosmetic and need investigation. ## Common Error Messages and Their Fixes Exact wording shifts between versions. The messages below are the ones you are most likely to meet, with the cause and the fix. ### `Failed to read from standard input: unknown file type` The parser received bytes it does not recognize as VCF or BCF. Common causes are an empty pipe, a file that is still being written by another process, or a path typo that produced an empty file. Check the file size with `ls -lh` and the first bytes with `zcat file.gz | head -c 200`. An empty result means the file is empty. ### `[E::bgzf_read_block] Invalid BGZF header at offset 0` The file has a `.gz` extension but was compressed with plain `gzip`, not `bgzip`. Rewrite it: ``` zcat broken.vcf.gz | bgzip -c > fixed.vcf.gz tabix -p vcf fixed.vcf.gz ``` Then convert as usual. Never use `gzip` on a VCF you intend to index. ### `Could not retrieve index file` A command that needs random access was given a file without a `.csi` or `.tbi`. Build the index: ``` bcftools index cohort.bcf ``` For a compressed VCF, use `bcftools index -t` or `tabix -p vcf`. If an index exists but the error persists, the file may have been modified after indexing. Rebuild with `bcftools index -f`. ### Unsorted positions Index creation fails when records are not in contig-then-position order. Sort the file and then index: ``` bcftools sort -T /scratch/tmp -Ob -o sorted.bcf unsorted.bcf bcftools index sorted.bcf ``` Sorting a BCF directly works. `bcftools sort` reads and writes both formats. ### `Number of columns at chr1:1000 does not match the number of samples` The `#CHROM` header line lists a different number of samples than the data rows contain. This usually happens after a manual edit, a concatenation of files with different sample sets, or a merge that dropped a sample. Recount with: ``` grep -c "" <(bcftools view -h broken.vcf | tail -1) awk -F'\t' 'NR==1{print NF}' broken.vcf ``` The first number minus 9 must equal the second number minus 9. If they differ, rebuild the file from a consistent source rather than patching rows. ### `Contig 'chr1' is not defined in the header` A warning, not an error. It appears when records use contig names that the `##contig` lines do not declare. It is safe to ignore for many analyses, but some tools will later refuse the file. To fix it properly, generate a header that includes the contigs and apply it: ``` bcftools view -h cohort.vcf > newheader.txt # add ##contig= style lines for every contig bcftools reheader -h newheader.txt -o reheadered.vcf cohort.vcf ``` Also check for the `chr1` versus `1` naming mismatch. A file that mixes both styles will silently lose sites during any overlap or merge operation. ### A BCF file that will not open in an older tool The tool was built against an older HTSlib and does not recognize the BCF version. Export to compressed VCF for that tool and keep the BCF for the rest of the pipeline: ``` bcftools view -Oz -o for_old_tool.vcf.gz cohort.bcf tabix -p vcf for_old_tool.vcf.gz ``` ### The process runs out of memory or fills the disk Conversion itself is streaming and needs little memory. Sorting is the memory-hungry step. Give it a temporary directory with space and let it spill to disk: ``` bcftools sort -T /scratch/tmp -m 2G -Ob -o sorted.bcf unsorted.vcf.gz ``` Uncompressed VCF output is the other disk hazard. A whole-genome VCF that is a few hundred megabytes compressed can expand by an order of magnitude. Prefer `-Oz` or `-Ob` for anything you keep. ## Common Mistakes and Limitations **Compressing with `gzip` instead of `bgzip`.** This is the most frequent mistake and the easiest to prevent. Plain gzip produces a single compressed stream with no block boundaries, so no index can point into it. `bgzip` produces a series of independently decompressible blocks, which is what makes random access possible. If your file cannot be indexed, check the compressor before anything else. **Trying to index a pipe.** `bcftools view -Ou in.vcf | bcftools index` cannot work because the index needs to seek backward. Write to a file, then index it. **Forgetting that BCF needs a CSI index.** Tabix TBI indexes are not valid for BCF files. `bcftools index cohort.bcf` writes `.csi` and that is correct. Do not rename a `.tbi` to `.csi`. **Deleting the source VCF too early.** Keep the original VCF.gz until the round-trip check passes on a representative subset. Binary formats are less forgiving of a truncated write, and an interrupted conversion can leave a file that opens but is missing the tail. **Editing a BCF by hand.** There is no supported way to do this. Any text editor will corrupt the file. Convert to VCF, edit, then reconvert and reindex. **Assuming BCF conversion validates the file.** `bcftools view` will pass through many malformed records with a warning. Conversion is not a validator. Run `bcftools norm -c w` or a dedicated check if correctness matters. **Splitting multi-allelic records after conversion.** Splitting changes the number and identity of records. Do it before conversion, with `bcftools norm -m -any`, so that the BCF you keep reflects the final analysis-ready state. Left-aligning indels with `bcftools norm -f reference.fa` is the same situation. **Mixing contig naming conventions.** A BCF with `chr1` and a BCF with `1` will not overlap in `bcftools isec` or `bcftools merge`. The output will be empty or wrong without an error. Standardize names before conversion, using `bcftools annotate, rename-chrs` with a two-column mapping file. **Expecting byte-identical files after a round trip.** Record content is preserved. Text layout, INFO field order, and float formatting may not be. If you need a reproducible artifact, compare decoded records, not checksums of the binary. **Assuming BCF is the right archival format.** VCF.gz is more portable and readable by more software, including simple scripts and spreadsheet imports. BCF is faster and smaller. Many groups keep VCF.gz as the archive of record and BCF as the working copy. Choose deliberately and document the choice. **Ignoring compression level for intermediates.** Scratch BCFs written at level 9 waste minutes per file. Use `-Ob1` for anything temporary and reserve the default or higher for files that will be read many times. **One more limitation worth stating plainly.** Individual files vary in quality, and a conversion cannot repair a file that was wrong to begin with. When a result looks impossible, go back to the source VCF and re-derive rather than trusting the binary. ## Practical Tips for Big Files and Repeated Work Use `-Ou` between pipeline stages. Chaining `bcftools view -Ou file.bcf | bcftools filter ... | bcftools view -Ob -o out.bcf` avoids two rounds of compression and decompression. Set `-threads` to the number of physical cores you actually have available, not the number the machine advertises. Compression is the only part that parallelizes, and oversubscribing slows everything else down. Build indexes immediately after writing any file. Add `-write-index` to the writing command if your `bcftools` version supports it, which saves a full read of the file. Do sample subsetting and filtering on the VCF side when the file is large and the subset is small. A 5,000-sample VCF reduced to 200 samples is a fifth of the conversion cost. Keep a small fixture. Extract 1,000 records with `bcftools view -r chr1:1-1000000` once, and use that fixture to test pipeline changes before running them on the full cohort. Pin your tool versions in the pipeline definition. A `bcftools` upgrade can change header handling and compression defaults, and the resulting files will not match your previous outputs. Record the exact command that produced every kept file. The `##bcftools_viewCommand` header line does this automatically. If you use `-no-version`, write the command into your [lab notebook](/knowledge/molecular-biology/lab-notebook) or pipeline log instead. ## Frequently Asked Questions ### How do I convert a VCF to BCF with one command? Run `bcftools view -Ob -o output.bcf input.vcf.gz`. The `-Ob` flag sets BCF output, `-o` sets the filename, and `bcftools` detects the input format on its own. Add `bcftools index output.bcf` immediately afterward. ### Does converting VCF to BCF lose any information? No, provided the source VCF is well formed. The header, samples, genotypes, INFO fields, and FILTER values all survive. The text layout and the order of fields inside INFO can change when you convert back. ### Why does bcftools say it could not retrieve the index file? The command needs random access and found no `.csi` or `.tbi` beside the data file. Build the index with `bcftools index` for BCF or `bcftools index -t` for compressed VCF, then rerun the original command. ### Can I convert BCF back to VCF exactly as it was? The records come back intact, but the file will not be byte-identical. Field ordering and float formatting can shift, and `bcftools` appends its own header lines unless you pass `-no-version`. ### What is the difference between FASTQ and FASTA? FASTQ stores a sequence plus a quality score for every base, in four lines per record. FASTA stores sequence only, in two lines per record, with no quality information. ### Should I store my variants as VCF or BCF long term? Keep compressed VCF as the archival copy and BCF as the working copy. VCF.gz is readable by more software and by simple scripts, while BCF is faster and smaller for repeated analysis. ### Why is my BCF file bigger than the VCF? The VCF was probably already compressed, or the BCF was written at compression level 0 or 1 with `-Ob0` or `-Ob1`. Rewrite it with the default level or `-Ob6` and compare again. ### Can bcftools read a VCF compressed with gzip instead of bgzip? It can read it, but it cannot index it. Decompress with `zcat` and recompress with `bgzip`, then build the index with `tabix -p vcf`.