# How to Compress FASTQ Files with bgzip By the end of this guide you will be able to compress a FASTQ file with bgzip, do it in a way that keeps the file compatible with every gzip-based tool in your pipeline, verify that the compressed file still contains exactly the original bytes, and build an index that lets you pull reads from a genomic region without decompressing the whole file. You will need a Linux or macOS shell, the bgzip program from the htslib package, and optionally samtools if you want indexed access. A FASTQ file of any size will do, though a small test file is the smartest place to start. ## What bgzip is and why it exists FASTQ is a four-line text format that stores a read identifier, the [nucleotide sequence](/knowledge/molecular-biology/nucleotide-sequence), a separator line, and the per-base quality scores. Sequencing runs produce these files by the hundreds of gigabytes, and general-purpose compressors struggle with them. A benchmarking study of 57 compression methods on FASTQ data found that traditional tools such as tar and gzip leave much of the redundancy in sequencing data untouched, because FASTQ has properties that generic compressors handle poorly, including a tiny alphabet, repetitive headers, and quality strings that look close to random [1]. bgzip, which ships with the htslib library behind samtools and bcftools, compresses with gzip but writes the output in BGZF, the blocked GNU zip format. A BGZF file is a series of independent gzip blocks, each holding at most 64 KB of decompressed data, with an extra field in the header that records the block length. This design has three practical consequences for your work: 1. Any tool that reads gzip can read a bgzip file, because every BGZF block is itself a legal gzip stream. Your zcat, gzip -t, aligners, and QC tools keep working. 2. The file can carry a small companion index (.gzi) that records where each block starts, which enables random access through samtools, tabix, and the htslib API without decompressing everything upstream. 3. Tools in the samtools ecosystem can seek to a block and start reading mid-file, which is what powers position-based queries in VCF, BED, and FASTQ files. The block structure is also what makes bgzip attractive for archiving and real-time compression workflows. A study of bgzip variants found that swapping the deflation backend, for example to libdeflate for higher compression or igzip for speed, improves performance while keeping interoperability with existing software intact [2]. The broader point stands even outside the bgzip family: compression choices for sequencing data involve real tradeoffs between ratio, speed, and shared compute resources, which matters on high-performance clusters where many jobs compete for the same cores and disks. One expectation to set honestly. bgzip will not shrink a FASTQ file as dramatically as specialized genomic compressors. It is the right choice when compatibility, indexing, and pipeline integration matter most. When raw ratio matters most, purpose-built tools exist, and this guide covers them later. ## Installing bgzip bgzip comes with htslib. Pick whichever route matches your system. **Conda or mamba (recommended for reproducibility):** ```bash mamba install -c bioconda htslib bgzip --version ``` **Debian or Ubuntu.** On these systems bgzip ships in the tabix package: ```bash sudo apt-get update sudo apt-get install -y tabix bgzip --version ``` **macOS with Homebrew:** ```bash brew install htslib ``` **From source** (useful on clusters where you lack root access): ```bash wget https://github.com/samtools/htslib/releases/download/1.20/htslib-1.20.tar.bz2 tar -xjf htslib-1.20.tar.bz2 cd htslib-1.20 ./configure --prefix=$HOME/.local make -j 4 make install export PATH=$HOME/.local/bin:$PATH bgzip --version ``` Replace 1.20 with whatever release is current when you do this. If `bgzip, version` prints a version string, you are ready. If you also plan to index FASTQ files for random access, install samtools the same way, since it provides the `samtools fqidx` command. ## How to compress FASTQ files with bgzip, step by step The following walkthrough uses a single-end file called `sample.fastq`. Run every step on a small file first so that mistakes cost seconds rather than hours. ### Step 1: Inspect the file before compressing A FASTQ file must have a number of lines divisible by 4. Confirm that, and look at the first records: ```bash wc -l sample.fastq head -n 8 sample.fastq ``` Expected output: ``` 4000000 sample.fastq @SRR000001.1 1 length=150 GATCGATCGATC... + FFFFFFFFF:FF... @SRR000001.2 2 length=150 ... ``` If the line count is not divisible by 4, the file is truncated or malformed. Fix that before compressing, because compression preserves corruption as faithfully as it preserves data. ### Step 2: Compress in place The simplest command compresses the file and replaces the plain-text original, exactly the way gzip behaves: ```bash bgzip sample.fastq ls -lh sample.fastq* ``` Expected output: ``` -rw-r--r-- 1 user user 900M Nov 11 09:21 sample.fastq.gz ``` The original `sample.fastq` is gone and `sample.fastq.gz` sits in its place. For a single-end 150-base read file, a result in the range of roughly a third of the original size is typical, though the exact ratio depends on read length, quality-string entropy, and how repetitive the identifiers are. Treat these example numbers as illustrative, not as a promise. If you are nervous about losing the original on your first run, use the redirect form in Step 3 instead. ### Step 3: Compress while keeping the original The `-c` flag writes the compressed output to standard output, which leaves the input untouched: ```bash bgzip -c sample.fastq > sample.fastq.gz ``` This form is the one you will use most in pipelines, because it also lets bgzip sit in the middle of a stream rather than at the end of one. ### Step 4: Use multiple threads bgzip is single-threaded by default. The `-@` flag controls the number of additional worker threads: ```bash bgzip -@ 8 -c sample.fastq > sample.fastq.gz ``` On multi-core machines this shortens wall-clock time substantially for large files. On a shared cluster, be a good neighbor and match the thread count to what your job allocation actually reserved. A study of FASTQ compression on shared high-performance systems emphasizes that balancing compression speed against CPU and memory allocation is a central operational concern, not an afterthought [1]. ### Step 5: Pick a compression level The `-l` flag sets the gzip compression level: ```bash bgzip -l 9 -@ 8 -c sample.fastq > sample.fastq.gz ``` Level 1 compresses fastest with a larger result, level 9 compresses slowest with a smaller result. Builds of htslib linked against libdeflate accept higher levels, and benchmarking of bgzip variants shows that libdeflate achieves high compression with a tolerable speed penalty, while igzip-style backends are extremely fast [2]. For most FASTQ archives, level 6 is a reasonable middle ground, and level 9 is worth the extra minutes only for files you will keep for years. ### Step 6: Verify the result ![Flowchart from gzip -t through an md5 checksum roundtrip to safe removal of the original or recompression, with htsfile confirming BGZF blocks.](/article-images/illustrations/bgzip-compress-fastq-verify-branch.webp) Verification has two parts. First, check that the compressed file is structurally sound: ```bash gzip -t sample.fastq.gz echo $? ``` A silent exit status of 0 means every block decompresses cleanly. Second, prove losslessness with a checksum roundtrip: ```bash md5sum sample.fastq > original.md5 bgzip -d -c sample.fastq.gz > roundtrip.fastq md5sum roundtrip.fastq cat original.md5 ``` The two md5 values must match. Once they do, delete `roundtrip.fastq` and, if you kept the original for safety, you can now safely delete it as well. Third, confirm the file is BGZF rather than plain gzip, which matters if you plan to index it: ```bash htsfile sample.fastq.gz ``` htsfile identifies BGZF-compressed data and will tell you whether the file carries the block structure that indexing requires. ### Step 7: Index for random access If you want to pull reads from a coordinate range later, create the .gzi index. You can do it during compression: ```bash bgzip -i -@ 8 -c sample.fastq > sample.fastq.gz ``` Or add an index to a file that is already compressed: ```bash bgzip -r sample.fastq.gz ``` You should now see a companion file named `sample.fastq.gz.gzi`. With an index in place, samtools can treat the FASTQ like an indexed reference: ```bash samtools fqidx sample.fastq.gz chr22:17000000-17001000 ``` This prints the FASTQ records of reads that overlap that region, without touching the rest of the file. The same bgzip-plus-index pattern underlies position-based queries across the samtools and tabix ecosystem, and later tools built on genomic data have treated bgzip and Tabix indexing as the baseline to improve upon [3]. ### Step 8: Decompress when needed ```bash bgzip -d sample.fastq.gz ``` or, to decompress to standard output without touching the file: ```bash bgzip -d -c sample.fastq.gz > sample.fastq ``` Both `zcat sample.fastq.gz` and `gzip -dc sample.fastq.gz` also work, because BGZF is gzip. That redundancy is a feature, not an accident. ## Key bgzip options at a glance | Option | What it does | Example | |--|--|--| | (none) | Compress a file in place, replacing the input | `bgzip sample.fastq` | | `-c` | Write to standard output, keep the input file | `bgzip -c sample.fastq > sample.fastq.gz` | | `-d` | Decompress | `bgzip -d sample.fastq.gz` | | `-@ N` | Use N additional worker threads | `bgzip -@ 8 sample.fastq` | | `-l N` | Compression level, 1 fastest through 9 smallest | `bgzip -l 9 sample.fastq` | | `-i` | Build a .gzi index while compressing | `bgzip -i sample.fastq` | | `-I FILE` | Choose a custom name for the index file | `bgzip -i -I my.gzi sample.fastq` | | `-r` | Rebuild an index on an existing BGZF file | `bgzip -r sample.fastq.gz` | | `-f` | Force overwrite of existing output | `bgzip -f sample.fastq` | | `-o FILE` | Write compressed output to a named file | `bgzip -o sample.fastq.gz sample.fastq` | ## Working with paired-end FASTQ files Paired-end sequencing produces two files, R1 and R2, where the Nth read in R1 pairs with the Nth read in R2. Compression never changes read order, but your workflow choices can. Two rules keep pairs intact: 1. Compress each mate file separately and never interleave them unless your downstream tools expect interleaved input. 2. Never sort, shuffle, or filter one mate file without doing exactly the same to the other. A parallel compression snippet that fits well in job scripts: ```bash bgzip -@ 4 -c sample_R1.fastq > sample_R1.fastq.gz & bgzip -@ 4 -c sample_R2.fastq > sample_R2.fastq.gz & wait ``` This runs both mates concurrently, each with four worker threads, on an eight-core allocation. Confirm both compressed files afterward: ```bash gzip -t sample_R1.fastq.gz sample_R2.fastq.gz ``` Then verify that the read counts still match: ```bash echo $(( $(zcat sample_R1.fastq.gz | wc -l) / 4 )) echo $(( $(zcat sample_R2.fastq.gz | wc -l) / 4 )) ``` The two numbers must be identical. If they are not, the pairs are out of sync and an aligner will reject or silently mis-pair them later. ## Using bgzip in everyday Linux commands for bioinformatics Most Linux commands for bioinformatics work by chaining small tools with pipes, and bgzip is built for exactly that style. These patterns cover the common cases. **Streaming a file through a pipeline on the fly:** ```bash zcat sample.fastq.gz | fastp -i - -o - -I - -O - 2> fastp.log | bgzip -@ 4 -c > trimmed.fastq.gz ``` Here fastp reads and writes compressed streams on standard input and standard output, and the output is recompressed immediately. Many modern tools, including fastp, accept gzipped input directly, so you often do not need an explicit decompress step at all. **Compressing over an SSH connection** so a file is compressed in transit rather than stored uncompressed on a shared filesystem: ```bash cat big_run.fastq | ssh storage-server "bgzip -@ 4 -c > /archive/big_run.fastq.gz" ``` **Merging runs by concatenation.** Because a BGZF file is a series of self-contained gzip blocks, simple concatenation produces a valid file: ```bash cat run1.fq.gz run2.fq.gz > merged.fq.gz gzip -t merged.fq.gz bgzip -r merged.fq.gz ``` Reindex after concatenation, because the block offsets recorded in any old .gzi no longer apply to the merged file. **Feeding an aligner directly.** bwa mem, bowtie2, and most contemporary aligners read gzipped FASTQ without a decompression step: ```bash bwa mem -t 8 reference.fa sample_R1.fastq.gz sample_R2.fastq.gz | samtools sort -@ 8 -o aligned.bam ``` A note on where the real bottleneck lives. Research on FASTQ parsing on modern hardware shows that file parsing, not alignment logic alone, dominates runtime for many I/O-heavy tools, and that optimized parsers deliver large speedups on both plain and gzip-compressed files [4]. This is one more reason to keep files compressed on disk and let each tool stream them, rather than maintaining sprawling directories of decompressed FASTQ files. ## How bgzip compares with other FASTQ compression options ![Decision path asking whether files must stay readable by gzip tools, then whether you control both ends, ending in bgzip or specialized compressors.](/article-images/illustrations/bgzip-compress-fastq-ratio-triage.webp) bgzip is a general-purpose compressor wearing a sequencing-aware container. Knowing where it sits among the alternatives helps you choose deliberately. **General-purpose alternatives.** A large benchmark of general-purpose methods on FASTQ data found that zpaq at high settings delivered the best compression ratio of the 57 methods tested, while pzstd at medium settings was recommended as the best all-around choice for shared or resource-limited systems, delivering high ratios at fast speeds with good CPU and memory efficiency [1]. Those tools produce plain compressed files with no block structure, so you give up indexed access in exchange for ratio or speed. **Specialized FASTQ compressors.** Purpose-built tools exploit the structure of the format itself. A benchmark of four such tools on paired-end short-read data reported that DRAGEN ORA and Genozip achieved compression ratios of approximately 1:6 when compressing already-gzipped FASTQ, while repaq and SPRING reached lower ratios of 1:2 and 1:4 respectively and took longer on both ends [5]. The repaq authors report that for some deep-sequencing datasets its ratio exceeds 25, close to four times better than gzip [6]. Genozip 15 adds a Deep method that co-compresses a FASTQ file together with the BAM file generated from it, reporting improvements of 75 to 96 percent over the already-compressed source files, which translates to 2.3 to 6.8 times better compression than compressing the two file types separately [7]. Newer format-aware general compressors continue to appear, such as NYX, which applies a reversible format-specific transform before compression and reports ratios up to 2.3 times those of the strongest general-purpose codecs while compressing up to two orders of magnitude faster on some inputs [8]. The recurring cost of all of these is interoperability: a file compressed with ORA, repaq, or SPRING needs the same tool to be read, whereas a bgzip file reads anywhere gzip does. **Reference-based compressors.** These map each read against a supplied reference genome and store compact difference descriptions instead of raw sequence. LW-FQZip splits a FASTQ into metadata, read, and quality streams, maps reads against a reference, and reached compression ratios between 0.111 and 0.201 across eight real datasets [9]. Its successor, LW-FQZip 2, adds parallelism and improved coding, handles both short-read and long-read data, and targets archival and space-sensitive applications [10]. RBFQC streams the identifier, sequence, and quality components into separate pipelines with a lightweight mapping mechanism, and its authors report better ratios and compression times than gzip-based approaches [11]. For nanopore data, RENANO improved base-sequence compression over its predecessor ENANO by 39.8 percent when both compressor and decompressor have the reference, and by 33.5 percent when the reference must travel inside the compressed file, with total file improvements of 12.7 and 10.6 percent respectively [12]. NanoSpring, a reference-free approach for nanopore reads based on approximate assembly, encodes recently basecalled high-quality data at 0.35 to 0.65 bits per base, three to six times lower than general-purpose compressors like gzip [13]. BEETL-fastq goes further and turns the archive into a searchable structure, converting 6.6 terabytes of human FASTQ reads into 1.7 terabytes of indexed files that support k-mer search, returning full FASTQ records for matching reads [14]. The tradeoff is the reference: every decompressor needs the exact reference file, which complicates sharing and long-term archiving. **Quality-aware and lossy approaches.** Quality scores consume roughly half the space of a FASTQ file in the uncompressed domain, which makes them the natural target for lossy compression. QVZ is a lossy quality-value compressor that lets the user define the distortion function to be minimized and produced data that genotyped better than comparable-rate alternatives from earlier algorithms [15]. A recent review benchmarked 12 quality-score compression tools across 14 real datasets and catalogs both lossless and lossy approaches for readers who need to choose among them [16]. Lossy quality compression is inappropriate when you must deposit raw data in public archives or when downstream callers depend on exact quality values. For day-to-day work the decision is simple. Use bgzip when files must stay readable by standard tools, when you want indexed random access, and when compatibility with samtools-based pipelines matters. Reach for specialized compressors only when storage or transfer cost is the binding constraint and you control the toolchain on both ends. ## Common Mistakes and Limitations **Double-compressing an already gzipped file.** Running bgzip on a `fastq.gz` file does not error. It treats the gzip stream as opaque bytes and compresses the compressed data again, which produces a file that is often larger than the input and always harder to reason about. Check first with `htsfile old.fastq.gz`. If the file is plain gzip and you want BGZF, convert it properly: ```bash zcat old.fastq.gz | bgzip -@ 8 -c > old.bgzf.fastq.gz gzip -t old.bgzf.fastq.gz ``` **Truncated files and the missing EOF marker.** A BGZF file must end with a final empty block that acts as an end-of-file marker. When a transfer is interrupted or a job is killed mid-write, samtools and bcftools warn with a message like "EOF marker is absent, the input BGZF is probably truncated." The fix is not clever tooling, it is a clean re-transfer or recompression from the original. The same rule applies when indexing: `bgzip -r` requires a complete file with a proper EOF block, so reindexing cannot rescue truncation. **Expecting miracle ratios from bgzip.** FASTQ data resists generic compression because of its small alphabet and high-entropy quality strings [1]. If your compressed file comes out at two thirds of the original size, nothing is wrong. If storage economics demand better, that is the moment to evaluate specialized tools rather than to keep raising the bgzip compression level and waiting. **Deleting the original before verifying.** The single most expensive mistake in this workflow is removing a plain-text FASTQ file based on the assumption that compression succeeded. Verify with `gzip -t` and a checksum roundtrip first, then delete. Disk space is cheap, a re-sequenced sample is not. **Assuming random access works without an index.** A bgzipped FASTQ file without a .gzi file must be read from the beginning like any gzip file. If `samtools fqidx` complains, the index is missing or stale. Create it with `bgzip -i` during compression or rebuild it with `bgzip -r` afterward. **Stale indexes after modifying files.** If you concatenate files, trim reads, or otherwise rewrite a compressed file, the .gzi offsets no longer match the content. Always run `bgzip -r` on the new file or re-compress with `-i`. An outdated index can return the wrong reads, which is worse than returning none. **Letting paired files drift out of sync.** Compression itself preserves order, but running different filtering, trimming, or subsetting commands on R1 and R2 desynchronizes the pairs. Verify mate counts after every operation, as shown earlier, before archiving. **Forgetting the thread count on shared systems.** `-@ 16` on a cluster node where you requested two cores will either get you killed by the scheduler or earn you an email from the administrator. The balancing act between compression speed and shared resource allocation is a documented operational concern on multi-user platforms. **One limitation to keep in view.** bgzip is lossless, which is exactly what raw sequencing data requires for archival, publication, and reproducibility. If your use case tolerates quality-score approximation, lossy methods exist and can do better, but that decision belongs to the project design, not to a compression convenience script [15]. ## Frequently Asked Questions ### Is bgzip the same as gzip? No. bgzip produces BGZF, a blocked variant of gzip in which each block is an independent gzip stream of at most 64 KB of decompressed data. The result reads perfectly in any gzip-compatible tool, but the block structure additionally allows indexes, random access, and parallel decompression that a single-stream gzip file cannot support. ### Will my aligners and QC tools read bgzipped FASTQ directly? Yes. bwa, bowtie2, fastp, cutadapt, and most contemporary tools read gzip-compressed FASTQ through standard library routines, and BGZF is gzip. You rarely need to decompress before analysis. ### How many threads should I give bgzip? Match the thread count to the cores you actually have, either locally or in your job allocation. bgzip scales well with `-@` on large files, but extra threads buy nothing beyond your allocation and hurt neighbors on shared clusters. ### Can I concatenate two bgzipped FASTQ files? Yes. Run `cat run1.fq.gz run2.fq.gz > merged.fq.gz`, since a BGZF file is a series of self-contained gzip blocks that concatenate legally. Verify with `gzip -t`, then rebuild the index with `bgzip -r merged.fq.gz`, because any old index no longer matches the merged offsets. ### How do I check that my compressed FASTQ is intact and lossless? Run `gzip -t file.fastq.gz` to confirm structural integrity, then decompress to a scratch file with `bgzip -d -c` and compare md5 checksums against the original. Delete the scratch file only after the checksums match. ### Can I pull reads from a genomic region without decompressing the whole file? Yes, if the file carries a .gzi index. Create one with `bgzip -i` during compression or `bgzip -r` afterward, then query with `samtools fqidx file.fastq.gz chr22:17000000-17001000`. This is the same bgzip-plus-index pattern that underlies Tabix-style queries across genomic formats [3]. ### Why is my bgzipped FASTQ barely smaller than the original? FASTQ files resist general-purpose compression because of their tiny alphabet, repetitive headers, and near-random quality strings, a result confirmed across a 57-method benchmark [1]. bgzip adds block structure and compatibility, not dramatic ratio gains. For better ratios, evaluate specialized compressors such as ORA, Genozip, repaq, or reference-based tools [5]. ### Can I convert an existing fastq.gz to the bgzip format without the original plain-text file? Yes. Decompress to a stream and recompress in one pipe: `zcat old.fastq.gz | bgzip -@ 8 -c > old.bgzf.fastq.gz`. The contents are identical bytes to the original decompressed data, so no quality is lost, and you can confirm with a checksum roundtrip. ## Sources 1. [FASTQ File Compression Benchmarking Using Lossless General Purpose Algorithms](https://doi.org/10.1145/3569951.3597577) 2. [7bgzf: Replacing samtools bgzip deflation for archiving and real-time compression](https://pubmed.ncbi.nlm.nih.gov/32092548/) 3. [STABIX: summary-statistic-based GWAS indexing and compression](https://pubmed.ncbi.nlm.nih.gov/40315153/) 4. [RabbitFX: Efficient Framework for FASTA/Q File Parsing on Modern Multi-Core Platforms](https://pubmed.ncbi.nlm.nih.gov/36327193/) 5. [A benchmark study of compression software for human short-read sequence data.](https://pubmed.ncbi.nlm.nih.gov/40316539/) 6. [Efficient sequencing data compression and FPGA acceleration based on a two-step framework.](https://pubmed.ncbi.nlm.nih.gov/37811144/) 7. [Deep FASTQ and BAM co-compression in Genozip 15](https://doi.org/10.1101/2023.07.07.548069) 8. [Byte-exact, format-aware compression for FASTA, FASTQ, BED and VCF files.](https://pubmed.ncbi.nlm.nih.gov/42856153/) 9. [Light-weight reference-based compression of FASTQ data.](https://pubmed.ncbi.nlm.nih.gov/26051252/) 10. [LW-FQZip 2: a parallelized reference-based compression of FASTQ files.](https://pubmed.ncbi.nlm.nih.gov/28320326/) 11. [A new efficient referential genome compression technique for FastQ files.](https://pubmed.ncbi.nlm.nih.gov/37950100/) 12. [RENANO: a REference-based compressor for NANOpore FASTQ files.](https://pubmed.ncbi.nlm.nih.gov/34128963/) 13. [Reference-free lossless compression of nanopore sequencing reads using an approximate assembly approach.](https://pubmed.ncbi.nlm.nih.gov/36747011/) 14. [BEETL-fastq: a searchable compressed archive for DNA reads.](https://pubmed.ncbi.nlm.nih.gov/24950811/) 15. [QVZ: lossy compression of quality values.](https://pubmed.ncbi.nlm.nih.gov/26026138/) 16. [Quality Scores Compression of Genomic Sequencing Data: A Comprehensive Review and Performance Evaluation](https://pubmed.ncbi.nlm.nih.gov/40811383/)