# How to Index a BAM File By the end of this guide you will be able to take a coordinate-sorted BAM file and produce a working `.bai` or `.csi` index, confirm that region queries return the correct reads, and diagnose the errors that stop indexing. You will also know when a BAI index is enough and when a long reference forces you to use CSI. You need four things on hand: a BAM file, samtools 1.10 or newer, write permission in the directory that holds the BAM, and enough free disk space for an index file that is small relative to the BAM itself. A reference FASTA is not required to index a BAM. You need the FASTA later if you call variants, convert to CRAM, or compute coverage against the reference. The commands below work on Linux, macOS, and Windows Subsystem for Linux. They also work inside the standard samtools container images and inside a conda environment built from the bioconda channel. If samtools is not installed yet, one common installation route is: ```bash conda install -c bioconda samtools ``` ## What a BAM Index Actually Does A BAM file is the binary, compressed form of SAM (Sequence Alignment/Map). SAM stores alignments as text, one line per read. BAM stores the same records in a compact binary layout and compresses the blocks with BGZF, a blocked variant of gzip that allows random access. SAM and BAM files sit at the center of next-generation sequencing pipelines, and the software built around them has to manipulate very large data volumes [1]. An index is a lookup table stored next to the BAM. It records which byte ranges of the BAM contain reads that overlap each region of each reference sequence. Without an index, a tool that wants reads from `chr7:55019017-55211628` has to scan the file from the beginning until it reaches that chromosome and position. With an index, the tool seeks directly to the relevant compressed blocks and reads only those. That difference matters in practice. A whole-genome BAM can be tens of gigabytes. A single gene query touches a few kilobytes of alignments. Tools such as `samtools view`, `samtools depth`, `bcftools mpileup`, GATK variant callers, and genome browsers all rely on the index to make region queries fast. The index does not store reads, coverage, or variant calls. It stores coordinates and file offsets. It also does not change the BAM. Indexing creates a second file and leaves the alignment data untouched. A BAM must meet two conditions before it can be indexed: 1. It must be compressed with BGZF, which standard BAM writers do automatically. 2. It must be coordinate sorted, meaning records appear in reference order and then by position within each reference. If either condition fails, indexing either stops with an error or produces an index that gives wrong results. ## BAI Versus CSI: Choosing the Index Format Two index formats dominate BAM workflows. Both are produced by samtools, and most modern tools read either one. **BAI** is the original BAM index format. It uses a fixed binning scheme with a limited coordinate range. The practical limit is a reference sequence shorter than 512 megabases and alignment positions below 2^29 minus 1, which is about 536.9 million. Most human, mouse, and microbial genomes fit comfortably inside that range. The human chromosome 1 is about 249 megabases, well under the limit. **CSI** is the newer, more flexible format. It uses a configurable binning depth, so it can represent much longer reference sequences. CSI is the right choice for plant genomes with multi-hundred-megabase chromosomes, for some invertebrate assemblies with very long scaffolds, and for any reference that trips the BAI coordinate limit. CSI also tends to be somewhat more compact for large files, though the difference is usually minor at human-genome scale. The table below summarizes the practical differences. | Feature | BAI | CSI | |--|--|--| | File extension | `.bai` | `.csi` | | Maximum reference length | About 512 Mb | Much larger, set by binning depth | | Maximum position | About 536.9 million | Scales with binning depth | | Default in `samtools index` | Yes | No, request with `-c` | | Typical use | Human, mouse, microbial, most short genomes | Large plant and invertebrate genomes, very long contigs | | Tool support | Universal | Broad in current tools, weaker in old tools | For a human whole-genome BAM, BAI is the safe default. It is what most tutorial pipelines expect, and it is what most downstream tools test against first. Reach for CSI when a reference contig exceeds the BAI limit or when your pipeline explicitly standardizes on CSI. Some projects keep both indexes during a transition. That is fine for storage, but it can cause confusion because tools choose one index and ignore the other. If both `sample.bam.bai` and `sample.bam.csi` exist, remove the one you do not intend to use so that every tool in the pipeline reads the same index. ## Step-by-Step: How to Index a BAM File The walkthrough below starts from an unindexed BAM and ends with a verified index and a successful region query. Each step includes the command, the expected output, and the reason for the step. ### Step 1: Confirm that samtools is installed and recent ```bash samtools version ``` Expected output begins with a line like: ```text samtools 1.19 Using htslib 1.19 ``` Any version from 1.10 onward supports BAI and CSI indexing with the options used here. Versions before 1.10 still index BAM files, but some convenience features differ. ### Step 2: Check that the BAM file is intact ```bash samtools quickcheck -v sample.bam ``` Expected output is nothing at all. Silence means the file has a valid header, a valid end-of-file marker, and a readable structure. A failure prints a message such as: ```text sample.bam was not identified as an existing samtools file ``` or ```text sample.bam has no targets in header ``` A truncated BAM, a file copied during writing, or an empty file will fail this check. Re-download or re-generate the BAM before indexing. Indexing a corrupt file wastes time and produces a misleading index. ### Step 3: Read the header and confirm the sort order ```bash samtools view -H sample.bam | head -n 5 ``` Expected output looks like this: ```text @HD VN:1.6 SO:coordinate @SQ SN:chr1 LN:248956422 @SQ SN:chr2 LN:242193529 @SQ SN:chr3 LN:198295559 @PG ID:samtools PN:samtools VN:1.19 CL:samtools view -H sample.bam ``` The important line is `@HD`. The `SO:coordinate` tag means the file is coordinate sorted and ready to index. If you see `SO:queryname`, `SO:unsorted`, or no `@HD` line at all, stop and sort the file first. ### Step 4: Sort the BAM if needed ```bash samtools sort -@ 8 -m 1G -o sample.sorted.bam sample.bam ``` The parameters that matter: - `-@ 8` uses eight threads for compression and sorting. - `-m 1G` gives each thread up to 1 GB of RAM before spilling to temporary files. Total memory use is roughly the thread count multiplied by this value, plus overhead. - `-o sample.sorted.bam` names the output file. If the input is already coordinate sorted, skip this step and index the original file. Sorting a sorted BAM wastes hours on a large file. Expected output from `samtools sort` is usually silent. When it finishes, the output BAM has `SO:coordinate` in its header. Two practical notes about sorting. First, write temporary files to a fast local disk, not to a network share. Second, if disk space is tight, the `-T` prefix option lets you direct temporary files to a specific directory with more room. ### Step 5: Build the BAI index ```bash samtools index sample.sorted.bam ``` Expected output is nothing. The command writes `sample.sorted.bam.bai` in the same directory. ```bash ls -lh sample.sorted.bam* ``` Expected output resembles: ```text -rw-r--r-- 1 user group 2.1G Nov 10 10:12 sample.sorted.bam -rw-r--r-- 1 user group 4.3M Nov 10 10:13 sample.sorted.bam.bai ``` The exact sizes depend on read count, read length, and alignment density. The index is normally a small fraction of the BAM size. A BAI that is nearly as large as the BAM usually signals a problem, such as an unusually fragmented file or a failed sort. To write the index to a specific path, pass the output file as the second positional argument: ```bash samtools index sample.sorted.bam indexes/sample.sorted.bam.bai ``` The directory must already exist. samtools does not create parent directories for you. ### Step 6: Build a CSI index instead, when needed ```bash samtools index -c sample.sorted.bam ``` Expected output is nothing. The command writes `sample.sorted.bam.csi`. The `-c` flag selects CSI. The optional `-m` flag sets the minimum interval size for the CSI binning scheme, with a default of 14. You rarely need to change it. Leave the default unless a tool's documentation gives a specific reason to alter it. ```bash samtools index -c -m 14 sample.sorted.bam ``` If you are indexing a reference with contigs longer than 512 megabases and the BAI step fails, this is the command that fixes it. ### Step 7: Confirm the index with idxstats ```bash samtools idxstats sample.sorted.bam ``` Expected output has one line per reference sequence plus a final line for unmapped reads: ```text chr1 248956422 1843201 0 chr2 242193529 1756432 0 chr3 198295559 1633110 0 ... * 0 0 52311 ``` The four columns are reference name, reference length, mapped read count, and unmapped read count. The final `*` line holds reads that have no reference assignment. If every mapped count is zero, the BAM contains no alignments or the index is attached to the wrong file. ### Step 8: Test a region query ```bash samtools view -c sample.sorted.bam chr7:55019017-55211628 ``` Expected output is a single integer, the number of alignments overlapping that interval. The command should return in well under a second even on a large BAM. If it hangs or prints an error about a missing index, the index is absent, misnamed, or unreadable. A second test confirms that records come back correctly: ```bash samtools view sample.sorted.bam chr7:55019017-55211628 | head -n 2 ``` Expected output is two alignment records in SAM text format, each starting with a read name, flag, reference name, position, mapping quality, CIGAR string, and the rest of the SAM fields. ### Step 9: Inspect the index file header ```bash xxd -l 4 sample.sorted.bam.bai ``` Expected output: ```text 00000000: 4241 4901 BAI. ``` For a CSI index: ```bash xxd -l 4 sample.sorted.bam.csi ``` Expected output: ```text 00000000: 4349 4901 CSI. ``` These four bytes are the format magic. If you see something else, the file is not a BAM index, even if it has the right extension. ## Command and Option Reference The table below collects the options that matter for routine indexing. | Option or argument | Tool | Effect | Default | |--|--|--|--| | `samtools index in.bam` | samtools | Writes a BAI index next to the BAM | BAI | | `-b` | samtools | Forces BAI output | On | | `-c` | samtools | Writes a CSI index | Off | | `-m INT` | samtools | Sets CSI minimum interval size | 14 | | `-@ INT` | samtools | Threads used while writing | 1 | | `-o FILE` | samtools | Writes the index to a named file | `in.bam.bai` or `in.bam.csi` | | Second positional argument | samtools | Alternative way to name the output index | Same as above | | `sambamba index -t 8 in.bam` | sambamba | Writes a BAI index with multithreading | BAI | | `picard.jar BuildBamIndex` | Picard | Writes a BAI index | BAI | Threads help less for indexing than for sorting or compression. The index itself is small, and the work is dominated by reading the BAM once. Setting `-@` to a high value on a fast machine gives a modest gain at best. ## Other Tools That Build BAM Indexes samtools is the reference implementation for most labs, but it is not the only option. **sambamba** is a faster drop-in tool for several BAM operations. Its index command is straightforward: ```bash sambamba index -t 8 sample.sorted.bam ``` It writes a `.bai` file next to the input. sambamba is popular in pipelines that already use it for sorting and filtering because it avoids installing a second toolchain. **Picard** builds a BAI index through the `BuildBamIndex` tool: ```bash java -jar picard.jar BuildBamIndex I=sample.sorted.bam O=sample.sorted.bam.bai ``` Picard is common in GATK-oriented pipelines. It writes BAI only, so a genome that needs CSI should use samtools for the index step. **elPrep** is an extensible multithreaded framework for processing SAM and BAM files, and its authors describe memory management as a key challenge for tools in this space [1]. If your pipeline already uses elPrep for sorting or filtering, keep samtools or a similar tool for the final index step, because the index still has to be written in BAI or CSI format. Whichever tool you use, the output is interchangeable as long as the format matches. A BAI written by samtools is readable by Picard, GATK, bcftools, and genome browsers. The file magic, not the producer, defines compatibility. ## Checking That the Index Is Correct Three checks catch almost every indexing problem. **Check 1: The index exists and has a plausible size.** A BAI or CSI should be present next to the BAM and should be far smaller than the BAM. A zero-byte index means the command failed or was interrupted. **Check 2: Region queries return reads.** Pick a region you know contains alignments and run `samtools view -c`. Compare the count with a region you know is empty. The first should be positive and the second should be zero or near zero. **Check 3: Whole-file counts match indexed counts.** Count all records and compare with the sum of the idxstats columns. ```bash samtools view -c sample.sorted.bam samtools idxstats sample.sorted.bam | awk '{sum += $3 + $4} END {print sum}' ``` The two numbers should agree, though small differences can appear when a tool counts secondary or supplementary alignments differently. A large mismatch means the index does not match the BAM, which usually happens when the BAM was replaced or filtered without rebuilding the index. A fourth check is useful on shared systems where files move between machines. Compare modification times: ```bash ls -l --time-style=full-iso sample.sorted.bam sample.sorted.bam.bai ``` If the BAM is newer than the index, rebuild the index. This is the most common cause of silent wrong results in production pipelines. ## Where BAM Fits Among FASTQ, FASTA, and Other Formats Indexing makes more sense when you see BAM in the wider file format landscape. Bioinformatics file formats fall into a few families: raw reads, reference sequences, alignments, and variant calls. **FASTA** stores biological sequence with a header line and no quality information. A reference genome is distributed as FASTA. A protein set is distributed as FASTA. The header line begins with `>` and carries an identifier and optional description. FASTA files are indexed with `.fai`, built by `samtools faidx`, which lets tools jump to a specific region of the reference. **FASTQ** stores raw sequencing reads with a header, the sequence, and a per-base quality string. Each base has a Phred quality score encoded as an ASCII character. The common encoding is Phred+33, where the character `I` represents quality 40 and an error probability of 1 in 10,000. FASTQ is the input to aligners. It is not indexed because it has no coordinate system. Compressed FASTQ files use ordinary gzip, which does not support random access. **SAM** is the text alignment format. It has a header section and one line per alignment. SAM is readable and useful for debugging, but it is large and slow to parse at scale. **BAM** is the binary, BGZF-compressed form of SAM. It is the working format for alignment processing, and it is what BAI and CSI index. The compressed blocks allow a tool to seek to a genomic region without decompressing everything before it. **CRAM** is a reference-based compressed alignment format. It stores reads as differences from a reference sequence, which usually makes it smaller than BAM. CRAM uses `.crai` indexes rather than BAI or CSI. **VCF and BCF** store variant calls. VCF is text, BCF is binary. Both can be indexed with tabix or bcftools, producing `.tbi` or `.csi` files. The table below summarizes the differences that matter for indexing decisions. | Format | Content | Quality scores | Coordinate system | Index type | |--|--|--|--|--| | FASTA | Sequence only | No | Positions along the sequence | `.fai` | | FASTQ | Reads plus per-base quality | Yes | None | None | | SAM | Text alignments | Yes, carried from FASTQ | Reference and position | None by default | | BAM | Binary BGZF alignments | Yes | Reference and position | `.bai` or `.csi` | | CRAM | Reference-compressed alignments | Yes | Reference and position | `.crai` | | VCF | Text variant calls | No | Reference and position | `.tbi` or `.csi` | | BCF | Binary variant calls | No | Reference and position | `.csi` | The FASTQ versus FASTA distinction trips up many new users. FASTQ is what comes off the sequencer and what you align. FASTA is the reference you align against. FASTQ carries quality scores because each read base has a confidence value. FASTA does not carry quality scores because a reference base is treated as known. Neither format is a BAM, and neither is indexed with BAI or CSI. ## Automating Indexing in Pipelines Indexing belongs at the end of the alignment stage, after sorting and before any region-based analysis. In a shell pipeline, a simple loop handles a batch of BAMs: ```bash for bam in *.sorted.bam do samtools index "$bam" done ``` For workflow managers, the index is a declared output of the alignment rule, and downstream rules declare it as an input. That ordering prevents the classic failure where a variant caller starts before the index exists. A minimal Snakemake-style rule looks like this: ```python rule index_bam: input: "aligned/{sample}.sorted.bam" output: "aligned/{sample}.sorted.bam.bai" shell: "samtools index {input}" ``` Two pipeline practices prevent most index problems. First, treat the BAM and its index as a single unit. Copy them together, move them together, and archive them together. Many workflow failures come from a BAM that arrived without its index or with an index from a previous version of the file. Second, rebuild the index after any step that changes alignments. Filtering, duplicate marking, base quality score recalibration, and read trimming all change the BAM. If the output is written as a new BAM, it needs a new index. If the tool updates the BAM in place, delete the old index first so that a stale file cannot be mistaken for a valid one. Newer samtools releases can build the index during the sort in a single pass, which avoids reading the BAM twice. If your version supports it, use that option for large files. The result is the same BAI or CSI file, just produced with less I/O. ## Performance and Storage Tips **Index once, at the end.** Every reindex reads the whole BAM. On a 60 GB file, that is a full pass over 60 GB of compressed data. Sort, filter, and mark duplicates first, then index the final product. **Keep the BAM and index on fast local storage for region-heavy work.** Random access over a network file system is slow because each seek becomes a network round trip. If many samples are queried repeatedly, stage them locally or use a storage layer that supports byte-range reads efficiently. **Do not use CSI by default.** BAI is smaller, more portable, and sufficient for the vast majority of genomes. Switch to CSI only when a contig exceeds the BAI limit or when a downstream tool requires it. **Do not expect indexing to speed up whole-file scans.** A tool that reads every record, such as a full-depth calculation or a complete variant call over all positions, gains nothing from the index. The index helps only when the tool asks for specific regions. **Watch disk space during sorting, not indexing.** Sorting creates temporary files that can exceed the input size. Indexing itself needs only enough space for the index, which is small. **Check permissions on the directory, not just the BAM.** The index is a new file. A read-only directory containing a writable BAM still fails at the index step. ## Common Mistakes and Limitations **Indexing an unsorted BAM.** This is the most common failure. The command stops with a message like: ```text [E::hts_idx_push] Unsorted positions on sequence #1: 100001 followed by 99999 samtools index: failed to create index for "sample.bam" ``` The fix is to sort first with `samtools sort` and index the sorted output. Reads must appear in increasing coordinate order within each reference. **Indexing a name-sorted BAM.** A BAM sorted by read name has `SO:queryname` in the header. It is valid for some tools, but it cannot be indexed for coordinate queries. The error may read: ```text [E::hts_idx_push] Chromosome blocks not continuous ``` Re-sort by coordinate before indexing. **Leaving a stale index after filtering.** Suppose you filter a BAM to remove low-quality reads and write `sample.filtered.bam`. The old `sample.bam.bai` is irrelevant, but if someone renames the filtered file to `sample.bam` and keeps the old index, region queries return offsets that no longer match the data. Rebuild the index after every change to the alignment records. **Copying the BAM without the index.** The index is a separate file. A transfer that moves only the BAM leaves downstream tools with no way to seek. Always move the pair. When in doubt, reindex at the destination. **Mismatched naming.** samtools looks for `sample.bam.bai`, then `sample.bai`, then the CSI equivalents. A file named `sample.index.bai` or `sample_bam.bai` is not found automatically. Keep the default naming unless a pipeline gives you a reason to change it. **Hitting the BAI coordinate limit.** On a genome with a contig longer than 512 megabases, indexing fails with a message like: ```text [E::hts_idx_push] Region 536870913..536870914 cannot be stored in a BAI index. Try using a CSI index instead ``` The fix is `samtools index -c`. No amount of re-sorting will get around the format limit. **Trying to index from a pipe.** An index requires a seekable file. A stream from `samtools view` or a decompression process has no fixed byte offsets, so it cannot be indexed in the usual way. Write the BAM to disk first, then index it. **Indexing a SAM file.** SAM is text and is not BGZF compressed. `samtools index` expects BAM or CRAM input and will refuse a SAM file or produce an error. Convert to BAM first with `samtools view -b`. **Assuming the index validates the alignments.** An index records positions. It does not check that reads are correctly placed, that the reference matches, or that pairing is consistent. A BAM can be indexed and still be biologically wrong. Use alignment quality metrics for that question. **Assuming the index makes every tool faster.** Tools that scan the whole file ignore the index. Only region-based access benefits. **Ignoring memory pressure during the surrounding steps.** Processing BAM files at scale is memory intensive, which is why the elPrep authors treat memory management as a central design problem for SAM and BAM tools [1]. Indexing itself is light, but the sort that usually precedes it is not. Plan memory for the sort, not for the index. **Using an index across incompatible file versions.** An index is tied to the exact bytes of the BAM it was built from. Recompressing a BAM, changing its compression level, or rewriting it with a different tool invalidates the index even if the alignments are identical. Rebuild after any rewrite. **Forgetting that CRAM needs a different index.** A `.bai` next to a `.cram` is not useful. CRAM uses `.crai`. The same principle applies, but the file format and the tooling differ. ## Frequently Asked Questions ### Do I need to sort a BAM file before indexing it? Yes. A BAM must be coordinate sorted before it can be indexed. The header should show `SO:coordinate`, and reads must appear in increasing position order within each reference. If the file is unsorted or name sorted, run `samtools sort` first. ### What is the difference between a BAI and a CSI index? BAI is the original BAM index format with a fixed coordinate range of about 512 megabases per reference. CSI is a newer format with a configurable binning depth that supports much longer references. BAI is the default and is sufficient for human and most other genomes. CSI is needed when a contig exceeds the BAI limit. ### How can I tell whether a BAM file is already indexed? Look for a file named `sample.bam.bai` or `sample.bam.csi` next to the BAM. You can also run `samtools idxstats sample.bam`. If the index is missing, the command prints an error about retrieving the index file. A region query such as `samtools view -c sample.bam chr1:1-10000` is a quick functional test. ### Why do I get "Could not retrieve index file"? The message means samtools could not find or open an index for that BAM. Common causes are a missing index, a nonstandard filename, an index in a different directory, or an index that was deleted during cleanup. Rebuild it with `samtools index` and confirm the filename matches the BAM. ### Can I index a BAM file that is being streamed through a pipe? No. Indexing needs random access to the file to record byte offsets, and a pipe is not seekable. Write the BAM to disk first, then index the file. If you are sorting on the fly, use the sort option that writes the index during the sort, or index the finished file afterward. ### Does indexing a BAM file modify or compress the BAM? No. Indexing reads the BAM and writes a separate index file. The BAM is unchanged in content and size. The index adds storage overhead, usually a small fraction of the BAM size, and can be deleted and rebuilt at any time. ### Is BAI enough for a human genome? Yes. Human chromosomes are all well under the BAI coordinate limit, with the longest around 249 megabases. BAI is the usual choice for human whole-genome and exome BAMs. CSI is only needed for references with much longer contigs. ### Do I need to rebuild the index after I filter or mark duplicates? Yes. Any step that adds, removes, or moves alignment records changes the byte layout of the BAM, which invalidates the old index. Rebuild the index on the final BAM after filtering, duplicate marking, or base quality score recalibration. ## Sources 1. [Comparing Ease of Programming in C++, Go, and Java for Implementing a Next-Generation Sequencing Tool.](https://pubmed.ncbi.nlm.nih.gov/31452597/)