Reduce BAM File Size Compression: Common Problems and Fixes

By Dr. Zubair Khalid, DVM, MS, PhD · 2026-11-10

An oversized BAM is almost always one of two things. Either the file was written at a weak BGZF compression level, or it carries alignment records and optional tags that the analysis never reads. The first thing to check is whether the file is really a BAM at all. Run samtools quickcheck -v sample.bam. If that passes, run samtools flagstat sample.bam and samtools idxstats sample.bam. Those two commands tell you how many records exist, how many are unmapped, how many are secondary or supplementary, and how many are duplicates. In most bloated files, the answer is visible in the first thirty seconds.

This guide covers the causes behind poor BAM compression, the fixes that actually move the number, and the mistakes that make things worse. It also covers how upstream choices in bioinformatics file formats, including the difference between FASTQ and FASTA, set a floor on how small a BAM can ever become.

What a BAM File Actually Contains

SAM is a tab-delimited text format for storing sequencing alignments. BAM is the binary equivalent, wrapped in BGZF blocks. BGZF is gzip-compatible but blocked, which is what allows random access through a companion index file (.bai or .csi). Each alignment record carries a read name, a bitwise flag, a reference name, a position, a mapping quality, a CIGAR string, mate coordinates, template length, the read sequence, a per-base quality string, and zero or more optional tags.

The header holds @HD, @SQ, @RG, @PG, and @CO lines. The @SQ lines define every reference sequence and its length. The @PG lines record every program that touched the file, and they accumulate quietly over a project's lifetime.

File size is driven by three factors: the number of records, the size of each record, and the compression ratio achieved on the record stream. A 30x human whole-genome BAM commonly lands in the tens to low hundreds of gigabytes. A targeted gene panel BAM for the same sample might be a few hundred megabytes. Neither number is wrong. They simply reflect different record counts, and any attempt to "fix" size has to start from a realistic expectation for the assay.

Cause 1: The Compression Level Is Lower Than You Think

BGZF compression defaults to level 6 in most tools, including samtools and samtools sort. Level 9 produces smaller output at the cost of CPU time. Level 1 produces larger output very quickly. The gap between level 1 and level 9 is real but bounded. Moving from level 1 to level 9 might shrink a file by roughly a tenth to a fifth, not by half. Recompressing an already-compressed BAM at level 9 is therefore a modest fix, not a rescue.

The diagnostic is simple. Take a fixed subset of reads, write it twice at different levels, and compare the byte counts.

samtools view -h sample.bam | head -n 4000000 > subset.sam
samtools view -b -l 1 -o subset_l1.bam subset.sam
samtools view -b -l 9 -o subset_l9.bam subset.sam
ls -l subset_l1.bam subset_l9.bam

If the two sizes are within a few percent of each other, compression level is not your problem and you should look at record content instead. If level 9 wins by a wide margin, recompressing the full file is worth doing, ideally merged into the same pass as sorting or filtering so you pay the I/O cost only once.

The fix in practice is to set the level explicitly in every production command rather than accepting the default. For archival copies, use level 9. For scratch files that will be deleted within a day, level 1 is usually the better trade because pipeline turnaround matters more than disk.

Cause 2: Records You Do Not Need

The largest single lever on BAM size is not compression at all. It is deciding which records deserve to exist. A raw alignment BAM contains unmapped reads, secondary alignments, supplementary alignments, and PCR or optical duplicates. Many downstream analyses use none of them.

The SAM flag field encodes record status in a bitmask. The bits that matter most here are 0x4 (unmapped), 0x100 (secondary), 0x800 (supplementary), 0x400 (duplicate), and 0x200 (failed quality control). Excluding several at once is done with a combined mask. -F 0x904 excludes unmapped, secondary, and supplementary records in one pass.

samtools view -b -F 0x904 -q 20 -l 9 -o filtered.bam sample.bam
samtools index filtered.bam

The -q 20 adds a mapping-quality floor. For variant calling on well-behaved germline data this is defensible. For structural variant detection, fusion calling, or anything that relies on split or discordant reads, excluding supplementary and unmapped records will destroy the signal. Check what the downstream tool expects before you filter.

Duplicates are a separate decision. Duplicate marking should happen once, early, with samtools markdup -r or Picard MarkDuplicates. Removing duplicates permanently shrinks the file, but if a later analysis needs duplicate counts for library complexity estimation, you have thrown that information away. Marking without removing keeps the flag and the size, so the choice is between disk space and future flexibility.

The confirmation step is samtools flagstat, which prints counts for total, primary, secondary, supplementary, duplicates, mapped, and paired reads.

samtools flagstat sample.bam

If secondary and supplementary together account for a meaningful fraction of the total, filtering will move the size by a similar fraction.

Cause 3: Optional Tag Bloat

Optional tags are appended to each record after the quality string. Common tags include NM (edit distance), MD (mismatch string), AS (alignment score), XS (second-best score), RG (read group), OQ (original quality), SA (supplementary alignment), and BQ (base quality). Each tag adds bytes to every record, and a tag that costs 10 bytes per read costs several gigabytes across a billion reads.

Some aligners and preprocessing tools add verbose tags by default. Base quality recalibration tools may store original qualities in OQ. Some pipelines retain MD and NM even when the variant caller recomputes them. Custom tags from in-house scripts can be the largest offender of all.

To see what is actually present, sample a few thousand records and count tag names.

samtools view -q 1 sample.bam | head -n 10000 | cut -f 12- | tr '\t' '\n' | cut -d: -f1 | sort | uniq -c | sort -rn

Once you know the tag inventory, remove the ones nothing downstream reads. The -x option takes one tag name and can be repeated.

samtools view -b -x MD -x NM -x AS -x XS -l 9 -o lean.bam sample.bam

The risk here is real. Some variant callers require MD or NM. Read-group splitting requires RG. Methylation-aware tools may depend on tags you did not expect. Test the lean file against your actual downstream command before you delete the original.

Cause 4: The File Is Not Really a Correctly Formed BAM

A surprisingly common cause of an "oversized BAM" is that the file is not a BAM. It may be an uncompressed SAM with a .bam extension. It may be a plain gzip stream rather than a BGZF stream. It may be a BGZF file with a truncated final block, missing the 28-byte EOF marker that tells tools the file is complete.

The checks are quick and should be routine before any size investigation.

file sample.bam
htsfile sample.bam
samtools quickcheck -v sample.bam

A plain gzip file will fail indexing because the index needs BGZF block boundaries. The fix is to decompress and recompress correctly.

samtools view -b -l 9 -o sample.fixed.bam sample.sam

A truncated BGZF file will fail quickcheck and may fail samtools index. If the file is a download, re-fetch it. If it came off a failed pipeline run, treat the output as corrupt and regenerate rather than trying to repair it. Random-access formats do not tolerate missing blocks.

Cause 5: Not Using Reference-Based Compression

BAM stores each read sequence in full, aside from the clipping encoded in the CIGAR string. CRAM stores only the differences between each read and the reference it aligns to. Because a well-aligned read is mostly identical to the reference, the delta is small, and CRAM output is usually substantially smaller than the equivalent BAM. The reference is not stored inside the CRAM file. It is identified in the header by an MD5 checksum and must be supplied at read time.

samtools view -C -T reference.fa -@ 8 -o sample.cram sample.bam
samtools index sample.cram

The tradeoff is operational. A CRAM is not self-contained. You need the matching reference with the matching checksum, and you need it forever. If the reference is lost, the CRAM is unreadable for practical purposes. Some tools still have incomplete CRAM support, though the situation has improved considerably.

Schema-aware storage layouts go further than generic compression. Campagne and colleagues built a system that stores high-throughput sequencing data in a structured, evolving schema rather than a flat record stream, and they reported that spliced RNA-Seq alignments could be stored in less than 4% of the size of a BAM file with perfect data fidelity, while exome, gene expression, and DNA methylation datasets shrank by more than 40% relative to the previous compression state of the art [1]. That result illustrates the ceiling. Generic BGZF compression of a flat record stream will never reach what a layout designed around the data's structure can do.

Cause 6: Header and Index Bloat

Headers rarely dominate, but they can grow without bound in long-lived projects. Every tool that touches a file appends a @PG line. Some pipelines add full command lines, environment variables, and long @CO comment blocks. A header that started at a few kilobytes can end up at hundreds.

Check the header size directly.

samtools view -H sample.bam | wc -c

If it is large, rewrite it. samtools reheader accepts a replacement header file.

samtools view -H sample.bam > header.sam
# edit header.sam to remove redundant @PG and @CO lines
samtools reheader header.sam sample.bam > trimmed.bam

Index files are small relative to the BAM but not free. The classic .bai index is limited to chromosomes under about 512 megabases, which is why .csi exists for large plant and vertebrate genomes. Keeping both indexes, plus a .bam.bai copy inside every directory that ever touched the file, adds up across a project.

Cause 7: Upstream FASTQ and FASTA Choices

BAM size is partly decided before alignment ever runs. FASTQ is the standard raw read format. Each record has a header beginning with @, a sequence line, a + separator, and a per-base quality string encoded as ASCII characters offset by 33 (Phred+33). FASTA has a header beginning with > and a sequence line, with no quality scores at all.

That difference matters for storage. A FASTQ file carries roughly one quality character per base, which means the quality string is comparable in size to the sequence itself before compression. Compression helps because quality strings are highly skewed, but the information is still there. A FASTA file of the same reads is a fraction of the size, and the missing information cannot be recovered.

When you compress a BAM, you are compressing a record stream that inherited those quality strings from the original FASTQ. If an analysis genuinely does not need per-base qualities, dropping them is a large saving. In practice, most variant calling does need them, so the saving is usually not available. The lesson is that format decisions early in a project constrain compression options late in the project.

Cryfa addresses the raw read side of this problem. It encrypts FASTA, FASTQ, VCF, SAM, and BAM, and it reduces the storage size of FASTA and FASTQ files to about one third while running roughly four times faster than a general-purpose encryption tool and about nine times faster than its closest competitor on FASTA input, with memory use in the low megabytes [2]. That combination of encryption and compression in a single pass is useful for raw data archives that must remain confidential.

Troubleshooting Table

A Diagnostic Sequence You Can Follow

Work through these steps in order. Each one either identifies a cause or rules one out, and you should stop as soon as the cause is clear.

1. Confirm the file type. Run file and htsfile. A .bam that reports as ASCII text is a SAM file and should be converted.
2. Confirm integrity. Run samtools quickcheck -v. Any output means the file is malformed and should be regenerated.
3. Count records. Run samtools view -c. Compare the count against the expected number of reads for the assay and coverage.
4. Break down flags. Run samtools flagstat. Look at the unmapped, secondary, supplementary, and duplicate lines as a fraction of total.
5. Break down contigs. Run samtools idxstats. Unmapped reads and reads on unplaced contigs often account for a large share.
6. Inspect the header. Run samtools view -H and look for an excessive number of @PG lines.
7. Inventory tags. Run the sampling command from the tag section and rank tags by frequency.
8. Test compression level. Recompress a fixed subset at -l 1 and -l 9 and compare.
9. Test reference-based compression. Convert the same subset to CRAM and compare.
10. Only then decide. Apply the combination of fixes that gives the largest saving without breaking downstream tools.

Steps 4, 5, and 7 usually produce the answer. Compression level alone rarely explains a file that is several times larger than expected.

Worked Fixes: From Raw Alignment to Lean Archive

The fixes below are written as a chain because that is how they should run. Each stage writes a new file, and each stage should be followed by samtools quickcheck.

Stage 1: Deduplicate properly

Duplicate removal requires name sorting, mate fixing, and coordinate sorting in that order. Skipping the mate-fixing step produces incorrect duplicate flags for paired reads.

samtools sort -n -@ 8 -m 2G -o namesort.bam raw.bam
samtools fixmate -m -@ 8 namesort.bam fixmate.bam
samtools sort -@ 8 -m 2G -o positionsort.bam fixmate.bam
samtools markdup -r -@ 8 positionsort.bam dedup.bam

The -r flag removes duplicate reads rather than only marking them. Drop -r if you need to keep duplicate counts.

Stage 2: Filter records and strip tags in one pass

Combine flag filtering, mapping-quality filtering, tag removal, and level 9 output into one samtools view call. Each separate pass costs a full read and write of the file.

samtools view -b -F 0x904 -q 20 -x MD -x AS -x XS -l 9 -@ 8 -o lean.bam dedup.bam
samtools index lean.bam

Stage 3: Convert to CRAM for archival

Convert the lean BAM to CRAM against the exact reference used for alignment, then index and verify.

samtools view -C -T reference.fa -@ 8 -o archive.cram lean.bam
samtools index archive.cram
samtools quickcheck -v archive.cram
samtools view -c archive.cram

The record count should match samtools view -c lean.bam. If it does not, stop and investigate before deleting anything.

Stage 4: Verify and remove the intermediate files

Round-trip a small sample to confirm the archive is readable.

samtools view archive.cram chr1:1000000-1010000 > roundtrip.sam
samtools view lean.bam chr1:1000000-1010000 > original.sam
diff <(cut -f1-11 roundtrip.sam) <(cut -f1-11 original.sam)

Safety note: deletion is irreversible and a BAM with missing BGZF blocks cannot be repaired. Confirm checksums, confirm record counts, and keep one verified copy in a separate location before removing any intermediate file. Never delete the only copy of a raw alignment during a space-constrained cleanup.

Memory, Threads, and Throughput

Compression is CPU-bound and memory-bound at the same time. samtools sort uses -m for memory per thread and -@ for thread count. Setting -@ 8 without raising -m above the default can produce many small temporary files and slow the sort. Setting -m too high on a shared node can trigger the out-of-memory killer.

The elPrep project addressed exactly this constraint. Its developers set out to build a multithreaded framework for processing SAM and BAM files and identified memory management as a central challenge for any program that manipulates these files at scale, which is why they benchmarked C++, Go, and Java implementations before choosing Go [3]. The practical takeaway for pipeline authors is that memory pressure, not raw CPU, is usually what limits throughput on alignment-file processing.

A reasonable starting configuration on a node with 32 GB of RAM is -@ 8 -m 2G. On a laptop, use -@ 2 -m 1G. Match the thread count to the number of physical cores available to the process, not to the number visible to the operating system.

Encryption, Tiering, and Long-Term Storage

Compressed alignment files are not automatically confidential. A BAM can contain identifying variants, and in clinical settings it is protected health information. Encryption at rest is a baseline control, not an optional extra.

Cryfa provides one integrated option. It handles FASTA, FASTQ, VCF, SAM, and BAM, combines AES encryption with a shuffling mechanism to strengthen security against low-complexity attacks, and delivers the speed and size advantages described earlier [2]. The low memory footprint matters because it means encryption can run on the same modest hardware as the rest of the pipeline rather than requiring a separate appliance.

Storage tiering is the other half of the answer. Campagne and colleagues argued that a multi-tier data organization, with raw reads, alignments, and derived results stored in layouts matched to how often each tier is accessed, dramatically reduces the storage, compute, and network burden of large sequencing projects [1]. A practical version of this idea is straightforward. Keep raw FASTQ compressed and encrypted in cold storage. Keep one filtered CRAM as the analysis master copy. Keep BAM only for tools that cannot read CRAM. Delete scratch files on a schedule rather than by hand.

Common Mistakes and Limitations

Stripping tags the downstream tool needs. MD and NM are recomputed by some variant callers and required by others. RG is required for per-read-group analyses. Test the lean file against the real downstream command before deleting the original.

Filtering records that carry biological signal. Unmapped reads matter for pathogen detection, viral integration, and contamination screening. Supplementary and split reads matter for structural variants and gene fusions. Duplicate reads matter for library complexity. Never apply -F 0x904 by reflex.

Assuming level 9 is a large win. The saving from level 1 to level 9 is real but modest. If a file is three times larger than expected, compression level is not the explanation.

Using plain gzip on a BAM. A BAM recompressed with gzip is no longer BGZF and cannot be indexed. The file may appear to shrink while becoming unusable.

Deleting before verifying. A truncated CRAM or BAM may fail only when a specific region is queried much later. Always round-trip a region and compare record counts before deletion.

Comparing sizes across different aligners. Different aligners produce different CIGAR strings, tag sets, and secondary alignment policies. Two BAMs from the same reads can differ substantially in size for reasons that have nothing to do with compression.

Ignoring the reference checksum. A CRAM without its matching reference is effectively lost data. Store the reference FASTA alongside the archive and record its MD5 in the project manifest.

Keeping indexes in every directory. A .bai file is small, but hundreds of stale copies across a project add up and confuse tooling that guesses which index to use.

Confusing FASTQ and FASTA size expectations. A FASTA file is much smaller than the equivalent FASTQ because it has no quality strings. Expecting a BAM derived from FASTQ to reach FASTA-like sizes is unrealistic.

Prevention Practices

Set compression level explicitly in every production command rather than accepting defaults. Use level 9 for anything that will persist beyond a day and level 1 only for scratch files.

Standardize on one archival format per project. CRAM against a recorded reference is usually the best choice for whole-genome and whole-exome data. BAM remains necessary for tools with limited CRAM support, so keep BAM as a working format and CRAM as the archive.

Run samtools quickcheck -v after every write. It takes a second and catches truncation immediately rather than six months later.

Record every program invocation in the header. The @PG line is the audit trail for how a file was produced. Trim redundant @PG lines only when the provenance is captured elsewhere.

Maintain a manifest with file path, format, record count, checksum, reference checksum, and the command used to produce each artifact. A manifest turns a storage cleanup from a guessing game into a checklist.

Apply storage tiering deliberately. Raw reads belong in cold, encrypted storage. Alignments belong in warm storage in a reference-based format. Derived results belong in fast storage. Scratch intermediates should expire automatically.

Review tag inventory at project setup, not after the disk fills. Knowing which tags your pipeline actually consumes takes minutes and prevents the most common destructive mistake in BAM slimming.

Frequently Asked Questions

Why is my BAM file so large?

The most common reason is that it still contains unmapped, secondary, supplementary, and duplicate records that the analysis never uses, followed by retention of optional tags that add bytes to every read. Compression level is a smaller factor than most people assume.

Does samtools -l 9 reduce file size much?

Level 9 helps, but the gain over the default level 6 is modest, and the gain over level 1 is larger. It will not fix a file that is several times bigger than expected because of record content.

Is CRAM always smaller than BAM?

CRAM is usually much smaller for whole-genome and whole-exome alignments because it stores only differences from the reference. For very small targeted panels, the fixed overhead of the CRAM header and container structure can narrow or erase the advantage.

Can I delete unmapped reads to shrink a BAM?

Yes, if no downstream analysis uses them. samtools view -b -F 0x4 removes unmapped reads. Pathogen detection, contamination screening, and viral integration analyses do rely on unmapped reads, so check before filtering.

How do I safely remove optional tags?

Sample 10,000 records, count tag names, identify the tags your downstream tools actually read, and remove only the rest with samtools view -x. Test the output against the real analysis command before deleting the original.

Why did my BAM get bigger after sorting?

Sorting itself does not add records, but a sort followed by an unmarked duplicate set or a merge across lanes can increase the record count. Check samtools flagstat before and after to see where the growth came from.

What is the difference between FASTQ and FASTA, and why does it matter for BAM size?

FASTQ carries per-base quality scores in a Phred+33 ASCII string, while FASTA carries sequence only. Since BAM records inherit those quality strings, a BAM derived from FASTQ carries information a FASTA-derived reference never had. That is part of why BAMs cannot be compressed to FASTA-like sizes without losing data.

Is compression compatible with encryption?

Yes. Tools such as Cryfa combine encryption with compression in a single pass and handle FASTA, FASTQ, VCF, SAM, and BAM, which is useful when raw reads must remain confidential in long-term storage [2].

Sources

1. Compression of structured high-throughput sequencing data.
2. Cryfa: a secure encryption tool for genomic data.
3. Comparing Ease of Programming in C++, Go, and Java for Implementing a Next-Generation Sequencing Tool.