# How to Create and Index a BAI File for Efficient BAM Access A BAM file without an index is a very expensive archive. Every region query has to stream through the whole file, and a request for 10 kb of a 60 GB alignment can take minutes instead of milliseconds. By the end of this guide you will be able to take a coordinate-sorted BAM, generate a matching BAI index with one command, confirm that index is valid, measure the speedup, and diagnose the handful of errors that account for nearly every failed indexing attempt. You will also know when a BAI is the wrong index and what to use instead. You need a coordinate-sorted BAM or CRAM, a samtools binary (version 1.9 or newer is a safe baseline for modern index behavior), and free disk space roughly equal to the uncompressed size of the BAM for sorting scratch. Optional tools that appear later are chopBAI for cohort-scale index reduction and Atlantool for read-identifier retrieval. No reference genome is required to build an index. The index describes positions inside the BAM file, not positions in the genome, so it works entirely from the alignment file and its header. ## What a BAI File Actually Contains A BAM file is a SAM file compressed in BGZF, which stands for Blocked GNU Zip Format. BGZF is ordinary gzip chopped into a series of independently decompressible blocks, typically under 64 KB uncompressed each, with an end-of-file marker block. That block structure is what makes random access possible. Any byte inside the decompressed stream can be addressed by a virtual offset, a 64-bit number that packs the compressed file offset of the block in the high bits and the offset within that decompressed block in the low 16 bits. The BAI file, short for BAM index, is a binary companion file that maps genomic coordinates to those virtual offsets. For each reference sequence in the BAM header it stores two structures. The first is a binning index, a hierarchy of bins that subdivide the reference so a query can skip large empty spans. The finest bins are 16 kbp wide, and the coarsest covers the whole reference. The second is a linear index, one virtual offset for each 16 kbp window, recording the earliest alignment that overlaps that window. A region query walks down the bin hierarchy, then uses the linear index to discard chunks that end before the query start. The result is that the tool decompresses a small slice of the file rather than all of it. Because the index holds only offsets and counts, it is dramatically smaller than the BAM. It also means the index is completely dependent on the exact byte layout of the BAM it was built from. Re-sort the BAM, filter it, or merge another file into it, and the old index is worthless even though it still sits on disk. The standard binning scheme supports references up to roughly 512 Mbp (2^29 bases). That ceiling is the single most common reason to reach for a different index type. ## Before You Start: Inputs, Tools, and Sanity Checks Three properties must hold before you type the index command. 1. The file is BAM or CRAM, not SAM. Indexing a plain SAM file with samtools produces an error because SAM is not blocked or seekable in the required way. 2. The file is coordinate sorted. Every record must appear in ascending order of reference and position, and the header must declare `SO:coordinate`. 3. The file is a regular file on a seekable filesystem. You cannot index a stream arriving through a pipe. Run these checks first. ```bash # Version check (any 1.x release works, use the newest you can get) samtools version # Is the file readable and complete? samtools quickcheck -v sample.bam # What does the header say about sort order? samtools view -H sample.bam | head -n 5 ``` Expected header output for a properly sorted file: ``` @HD VN:1.6 SO:coordinate @SQ SN:chr1 LN:248956422 @SQ SN:chr2 LN:242193529 ... ``` `quickcheck -v` prints nothing and exits 0 when the file is intact. It prints a diagnostic and exits non-zero when the BAM is truncated, which happens after interrupted downloads or failed pipeline steps. Indexing a truncated BAM produces an index that appears to exist but silently loses the tail of the file. If `SO:` says `queryname` or `unsorted`, sort before indexing. If the `@HD` line is missing entirely, most tools will still index but downstream comparisons and merge operations become unreliable, so fix the header. ## Step-by-Step: From Raw Alignments to a Verified BAI ### Step 1: Sort the BAM by coordinate Alignment output from BWA, HISAT2, STAR, and most other aligners is not coordinate sorted by default. BWA emits reads in the order they were processed, which is essentially random with respect to position. The sort step is where most wall time goes for large files. ```bash samtools sort -@ 8 -m 2G -o sample.sorted.bam sample.bam ``` The `-@ 8` flag uses eight background threads for compression and decompression, and `-m 2G` caps each thread's in-memory buffer. Total RAM use is roughly `threads × memory`, so plan for about 16 GB with these settings. For a human whole-genome BAM, expect this to take tens of minutes and to need temporary disk equal to the uncompressed data. If your aligner already produced a sorted BAM, skip this step, but confirm with the header check above. Sorting an already sorted file wastes hours. ### Step 2: Index the sorted BAM ```bash samtools index -@ 8 sample.sorted.bam ``` This writes `sample.sorted.bam.bai` next to the BAM. There is no stdout on success. Confirm with: ```bash ls -lh sample.sorted.bam sample.sorted.bam.bai ``` Expected shape of the output: ``` -rw-r--r-- 1 user group 58G Nov 10 09:14 sample.sorted.bam -rw-r--r-- 1 user group 6.1M Nov 10 09:15 sample.sorted.bam.bai ``` Indexing a [human genome](/blog/guides/human-genome) BAM typically takes seconds to a couple of minutes, because the tool reads the compressed blocks and records offsets without decompressing every record fully. Note the naming. The default is `.bai`, so the full filename is `sample.sorted.bam.bai`. The older convention `sample.sorted.bai` (replacing `.bam`) is also accepted by htslib-based tools when the BAM is present. Pick one convention and use it everywhere. Mixed conventions across a project cause the "index not found" failures described later. ### Step 3: Confirm the index resolves region queries ```bash samtools view -c sample.sorted.bam chr1:1000000-2000000 ``` Expected output is a single integer, for example: ``` 18422 ``` That number is the count of alignment records overlapping the 1 Mb window. The command returns in well under a second on a local disk. To see what happens without an index, temporarily move it: ```bash mv sample.sorted.bam.bai /tmp/ samtools view -c sample.sorted.bam chr1:1000000-2000000 ``` Expected error: ``` [E::hts_idx_load3] Could not load the index [main_samview] random alignment retrieval only works for indexed BAM or CRAM files. ``` Move the index back and the query works again. That error message is the canonical signal that a region query has no usable index, and it is covered again in the troubleshooting section. ### Step 4: Pull per-reference statistics ```bash samtools idxstats sample.sorted.bam ``` Expected output, one line per reference plus a final `*` line: ``` chr1 248956422 20154321 0 chr2 242193529 19873210 0 chr3 198295559 16220114 0 ... * 0 0 1523 ``` Columns are reference name, reference length, mapped read count, and unmapped read count. The `*` row counts reads with no coordinate, which are typically unmapped reads retained in the file. `idxstats` reads the index directly, so it is fast and it doubles as an index integrity check. If it prints all zeros for mapped counts while the BAM clearly contains alignments, the index does not match the BAM. Counts from `idxstats` are also the cheapest way to spot a sample swap or a failed lane before running expensive downstream analysis. ### Step 5: Build a CSI index when BAI cannot represent the reference If any reference in the header is longer than roughly 512 Mbp, the BAI format cannot address it. Build a CSI index instead. ```bash samtools index -c -@ 8 sample.sorted.bam ``` This writes `sample.sorted.bam.csi`. Keep only one index type per BAM to avoid ambiguity about which one a tool loaded. CSI is the modern default for large genomes and for references with many contigs, and current samtools, htslib, and Picard releases all read it. ### Step 6: Test a batch of regions in one invocation ```bash samtools view -c sample.sorted.bam \ chr1:1000000-2000000 \ chr2:5000000-6000000 \ chr7:117000000-117100000 ``` Each region is resolved through the index independently, and the counts are summed. This pattern is far more efficient than running three separate commands, because the BAM header and index are loaded once. When you have hundreds of intervals, pass a BED file instead: ```bash samtools view -c -L targets.bed sample.sorted.bam ``` The `-L` flag restricts output to reads overlapping any interval in the BED file. With an index, whole-chromosome scanning is avoided. ### Step 7: Record provenance ```bash samtools view -H sample.sorted.bam > sample.header.sam sha256sum sample.sorted.bam sample.sorted.bam.bai > sample.sha256 ``` The checksums let you confirm later that a BAM and its index are the pair that were validated together. In multi-user environments and in managed research data systems, this kind of metadata-aware tracking of large sequencing assets is exactly what keeps delivery reliable across projects and storage tiers [1]. ## The Parameters That Matter | Goal | Command | What to know | |--|--|--| | Default BAI | `samtools index -b aln.bam` | `-b` is the default and can be omitted. Writes `aln.bam.bai`. | | CSI for long contigs | `samtools index -c aln.bam` | Writes `aln.bam.csi`. Use for any reference over about 512 Mbp. | | Custom filename | `samtools index -o idx/aln.bai aln.bam` | You take responsibility for the name being discoverable. | | Threads | `samtools index -@ 8 aln.bam` | Speeds up decompression, not the index math. Diminishing returns past 8. | | Stats without scanning | `samtools idxstats aln.bam` | Reads the index only. Fast integrity check. | | Region query | `samtools view -c aln.bam chr1:1-10000` | Fails with a clear error if no index is present. | | Bed-limited query | `samtools view -c -L regions.bed aln.bam` | Best pattern for panel and target analyses. | Two flags cause most of the confusion. `-b` and `-c` are mutually exclusive index types, and whichever you used must match what the consumer expects. `-o` changes only the filename, not the format, so an `-o` path ending in `.csi` while `-b` is in effect still produces a BAI payload with a misleading name. Let the tool name the file. ## Verifying the Result Is Correct A four-part check catches nearly every bad index. 1. Timestamp. The index must be newer than the BAM. `stat -c '%y %n' sample.sorted.bam sample.sorted.bam.bai` shows both. If the BAM is newer, rebuild. 2. Size sanity. A BAI for a whole-genome BAM is a small fraction of the BAM, but it is never zero bytes. A 0-byte index means the build failed silently, often because the process was killed. 3. Content sanity. `samtools idxstats` must show mapped counts that roughly match your expected yield. Compare the sum of mapped counts against `samtools view -c sample.sorted.bam`, which counts everything and does not rely on the index. The two numbers should agree closely, with differences explained by secondary and supplementary alignments. 4. Query sanity. Pick a region you know is covered and one you know is empty. The covered region returns records, the empty one returns zero. If both return zero, the index is pointing at the wrong file. ## BAI vs CSI vs Other Indexes BAI is not the only game in town, and picking the wrong index is a real workflow cost. | Index | File suffix | Built by | Best for | Key limit | |--|--|--|--|--| | BAI | `.bam.bai` | `samtools index -b` | Standard genomes, maximum tool compatibility | Around 512 Mbp per reference | | CSI | `.bam.csi` | `samtools index -c` | Large genomes, assemblies with very long contigs, many contigs | Larger index files | | CRAI | `.cram.crai` | `samtools index` on CRAM | CRAM archives, lossless compression at rest | Must match CRAM reference | | Tabix TBI/CZI | `.tbi`, `.csi` | `tabix -p vcf` | VCF, BED, GFF text files | Text formats only | | Read-name index | tool-specific | Atlantool | Retrieving reads by identifier rather than position | Separate from coordinate index | | Chopped BAI | one `.bai` per interval | chopBAI | Many samples queried at the same small regions | Requires [preprocessing](/blog/data-analysis/what-is-data-preprocessing) | The read-name case deserves emphasis. Coordinate indexes answer "what aligns here". They do not answer "where is read ABC123". Atlantool was built for exactly that gap, creating a one-time index in the same BGZF container style as BAM and then retrieving reads by identifier essentially instantaneously [2]. If your pipeline pulls specific read names out of a large BAM, a BAI will not help and a full scan is otherwise unavoidable. The cohort case matters too. chopBAI reduces a BAI into a set of small per-interval indexes that stay compatible with existing tools, and preprocessing the indexes cut I/O by more than 95 percent when analyzing 10 kb genomic regions, which made joint analysis of more than 10,000 individuals feasible [3]. The lesson generalizes: when many samples are queried at the same tiny region, the index file itself, not the alignment data, becomes the I/O bottleneck. ## A Worked Example: What Indexing Buys You Consider a 30x human whole-genome BAM. The haploid genome is about 3.1 Gbp, and paired 150 bp reads give roughly 600 million read pairs, so on the order of 1.2 billion records before any filtering. Multiply by the compressed per-record footprint and you land in the tens of gigabytes, which is why these files are shipped on external drives rather than downloaded casually. Now suppose you want a 10 kb promoter window. Without an index, samtools must decompress from the start of the file until it passes the target coordinate. For a locus near the end of chromosome 1, that is close to the entire file. With a BAI, the binning index narrows the candidate blocks and the linear index supplies a starting virtual offset within 16 kbp of the target, so samtools decompresses a handful of blocks. The difference is routinely three to four orders of magnitude in wall time and in bytes read. That scaling behavior is why indexing is not optional in production pipelines. A bulk RNA-seq protocol that goes from SRA download through FastQC, trimming, alignment with BWA or HISAT2, and samtools-based format conversion leans on indexed BAM access at every downstream counting step [4]. A pipeline that ignores indexing still finishes, but it re-reads the same terabytes for every counting job. The same principle explains why format choices carry real costs. Comparative benchmarking of nanopore signal formats found that binary SLOW5 beat POD5 on sequential and random access reads on most systems, with measurable consequences for basecalling cost and speed [5]. Random access performance is a property of the container format, not a detail you can bolt on afterward. BAM and BAI were designed around that constraint from the start. ## Where BAI Fits Among Bioinformatics File Formats File formats in sequencing fall into a rough hierarchy, and knowing where each sits prevents the most basic category errors. **FASTQ** holds raw reads. Each record carries an identifier, a [nucleotide sequence](/knowledge/molecular-biology/nucleotide-sequence), and a per-base quality string in Phred scale. It is the output of the sequencer and the input to the aligner. FASTQ is text, it is not sorted, and it is not indexable by position because the reads have no genomic coordinates yet. **FASTA** holds sequences without quality scores. It is the standard container for reference genomes, transcript sets, and protein databases. A proteomics reanalysis effort identified missing protein FASTA database files as a recurring barrier to reproducibility, which shows how central this plain format remains across disciplines [6]. Reference FASTA files are what aligners index with tools like `bwa index` or `hisat2-build`, a completely different index from a BAI. Do not confuse the two. A BWA index is derived from the reference and is reusable across samples. A BAI is derived from one BAM and is specific to that file. **SAM** is the human-readable alignment format. **BAM** is its BGZF-compressed binary form. **CRAM** is a reference-based compressed alternative that stores only differences from the reference. All three describe alignments. Only BAM and CRAM support positional indexing, and CRAM uses CRAI rather than BAI. Downstream formats sit on top of indexed alignments. Variant annotation pipelines read aligned records and classify variants by region and predicted effect [7]. Genotype storage formats have their own design tradeoffs, and a simple indexable binary format was reported to be more than 100 times faster and 3.5 times smaller than `vcf.gz` on biobank-scale whole-genome sequence data [8]. Assay-specific formats compress further, with one methylation toolkit reporting over 100x compression from its input BAM into a custom epiread representation [9]. None of these replace the BAM and its BAI. They are what you build once indexed access to the BAM makes the extraction step cheap. The practical rule: FASTQ and FASTA are inputs, SAM is an interchange format, BAM plus BAI is the analysis workhorse, and everything else is a projection built from it. Harmonized file formats across a consortium are what make pipelines reproducible and comparable between groups [10], and a BAM/BAI pair is one of the few formats nearly every tool in the ecosystem agrees on. ## Tips for Efficient BAM Access at Scale Query regions, not chromosomes. `samtools view sample.bam chr1` still decompresses a large share of the file. If you need a gene, a peak, or a panel target, pass the interval. Reuse one open file handle. Loop over intervals inside a single `samtools view -L targets.bed` call rather than a shell loop that reopens the BAM hundreds of times. Each open re-reads the header and index. Match threads to storage. `-@` helps when decompression is the bottleneck. On network filesystems, threaded decompression often just queues more I/O and does not help. Keep index and BAM on the same tier. Splitting them across a fast cache and a slow archive produces pathological latency. A delivery layer that registers assets with their metadata and hands them off consistently avoids this class of problem entirely [1]. Choose the container for its access pattern. If your reads will be queried overwhelmingly by position, BAM plus BAI is correct. If they will be queried by identifier, build a read-name index [2]. If the same small intervals are queried across thousands of samples, pre-chop the indexes [3]. Convert to a compact domain format after extraction, not before. Pull the records you need through the index, then project them into whatever representation your analysis wants [9]. Make steps modular. A pipeline assembled from independent modules that each consume and emit declared formats can be rerun piecewise when one stage changes, which is the design principle behind modular metagenomics systems [11]. Keep CRAM in mind for cold storage. CRAM with CRAI cuts disk use substantially and is fully indexable. Convert back to BAM for tools that require it. ## Common Mistakes and Limitations **Indexing an unsorted BAM.** The most frequent failure by a wide margin. samtools refuses with: ``` [E::hts_idx_push] Unsorted positions on sequence #1: 10542 followed by 10420 samtools index: failed to create index for "sample.bam" ``` Fix: run `samtools sort` first. If the file came from a collaborator, ask whether they sorted it. Never trust a filename that says "sorted". **Querying a BAM whose index lives elsewhere.** Tools look for `.bai`, then `.bai`, next to the BAM. Put the index beside the BAM or set an explicit path where the tool allows it. Moving a BAM to a new directory without its index produces the "random alignment retrieval only works for indexed BAM or CRAM files" error. **Editing a BAM after indexing.** Any operation that rewrites the file invalidates the index. That includes `samtools view` with new filters writing to the same path, `samtools merge`, `samtools markdup`, and adding read groups with Picard. Delete the stale index and rebuild. A stale index is worse than no index because queries return plausible but wrong records. **Using BAI on a long contig.** A reference over roughly 512 Mbp cannot be represented. samtools reports that the region cannot be stored in a BAI index and suggests a CSI index. Rebuild with `samtools index -c`. This affects some plant and amphibian assemblies and any custom reference with concatenated super-scaffolds. **Trying to index a stream.** `samtools index` requires a seekable file. You cannot pipe an aligner's stdout directly into it. Write the BAM to disk or a seekable volume first. **Mixing naming conventions.** A project where half the files use `sample.bam.bai` and half use `sample.bai` will fail unpredictably when scripts assume one layout. Standardize early. **Expecting a BAI to accelerate read-name lookups.** Coordinate indexes are blind to identifiers. Region queries that return zero hits send people looking for a broken index when the real problem is the query type [2]. **Ignoring index overhead in cohorts.** When you query one small interval across thousands of BAMs, the total bytes read from index files can exceed the bytes read from the alignments themselves, and index reduction addresses that specifically [3]. **Assuming the index proves file integrity.** BAI records offsets, not checksums. A truncated BAM can still produce a syntactically valid index for its intact prefix. Run `samtools quickcheck` and keep SHA-256 sums for anything archival. **Deleting the index to save space.** A BAI is a tiny fraction of the BAM. Regenerating it after a restore costs time and introduces the risk of a version mismatch. Keep it. There is one genuine limitation worth stating plainly. Indexes describe coordinates within one file. They cannot reconcile two BAMs built against different reference versions, they do not detect that a BAM was generated with a different aligner version, and they tell you nothing about whether the alignments are biologically correct. Validate the BAM content with alignment metrics and, for clinical interpretation, have an individual case reviewed by a qualified professional. ## Frequently Asked Questions ### What is a BAI file? A BAI file is the binary index for a BAM alignment file. It stores virtual file offsets organized by genomic bin so that region queries can jump directly to the relevant compressed blocks instead of scanning the whole file. ### Do I have to sort a BAM before creating its BAI? Yes. The BAI format assumes records arrive in ascending coordinate order, and samtools aborts with an "Unsorted positions" error when they do not. Sort with `samtools sort` and confirm the header shows `SO:coordinate`. ### How do I create a BAI file from a BAM? Run `samtools index sample.sorted.bam`. The tool writes `sample.sorted.bam.bai` beside the BAM with no console output. Verify with `samtools idxstats sample.sorted.bam` and a test region query. ### Why does samtools say random alignment retrieval only works for indexed BAM or CRAM files? The queried BAM has no discoverable index at the expected path. Either the index was never built, it was moved or renamed, or the BAM was modified after indexing. Rebuild it next to the BAM and rerun the query. ### What is the difference between BAI and CSI? Both index BAM files, but BAI cannot address references longer than about 512 Mbp. CSI uses a more flexible binning scheme and handles large or highly fragmented assemblies. Build CSI with `samtools index -c`. ### Can I delete the BAI file to save disk space? You can, but the savings are negligible, since the index is a small fraction of the BAM. Deleting it forces every future region query to scan the full file and breaks many tools outright. ### Does a BAI index make searching by read name faster? No. BAI maps genomic coordinates to file offsets and has no knowledge of read identifiers. Retrieving reads by name requires a dedicated read-name index, as implemented in Atlantool [2]. ### How do I check that my BAI matches my BAM? Compare modification times, confirm the index was written after the BAM, run `samtools idxstats` and check that mapped counts are plausible, then query a known covered region and a known empty one. ## Sources 1. [MetaServe: a lightweight, metadata-aware governance and delivery layer for pre-publication research omics data.](https://pubmed.ncbi.nlm.nih.gov/42323537/) 2. [Atlantool: a command line tool to retrieve DNA and RNA sequencing reads from BAM files by the read identifier.](https://pubmed.ncbi.nlm.nih.gov/41230488/) 3. [chopBAI: BAM index reduction solves I/O bottlenecks in the joint analysis of large sequencing cohorts.](https://pubmed.ncbi.nlm.nih.gov/27153590/) 4. [Streamline Protocol for Bulk-RNA Sequencing: From Data Extraction to Expression Analysis.](https://pubmed.ncbi.nlm.nih.gov/41637157/) 5. [The enduring advantages of the SLOW5 file format for raw nanopore sequencing data.](https://pubmed.ncbi.nlm.nih.gov/41100284/) 6. [Preventing Proteomics Data Tombs Through Collective Responsibility and Community Engagement.](https://pubmed.ncbi.nlm.nih.gov/41571719/) 7. [SNPAAMapper-Python: A highly efficient genome-wide SNP variant analysis pipeline for Next-Generation Sequencing data.](https://pubmed.ncbi.nlm.nih.gov/36171799/) 8. [IGD: a simple, efficient genotype data format.](https://pubmed.ncbi.nlm.nih.gov/40980555/) 9. [wgbstools: a computational suite for DNA methylation sequencing data analysis.](https://pubmed.ncbi.nlm.nih.gov/41611450/) 10. [Uniform processing and analysis of IGVF massively parallel reporter assay data with MPRAsnakeflow.](https://pubmed.ncbi.nlm.nih.gov/42581006/) 11. [CAMP: a modular metagenomics analysis system for integrated multistep data exploration.](https://pubmed.ncbi.nlm.nih.gov/41551931/)