How to Fix 'BAM File Not Sorted' Errors: A Troubleshooting Guide

By Dr. Zubair Khalid, DVM, MS, PhD · 2026-11-10

The most likely cause of a BAM file not sorted error is that the BAM was never coordinate sorted after alignment, or it was sorted by read name and then passed to a tool that requires coordinate order. The first thing to check is the BAM header. Run samtools view -H sample.bam | head and look for the @HD line. If it says SO:unsorted, if SO:queryname is present, or if the @HD line is missing, sort the file by coordinate before you do anything else. If the header says SO:coordinate and the error persists, the header is not telling the whole story. The records may be out of order, the index may be missing or stale, or the contig order may not match the reference used by the downstream tool.

This guide explains how to fix 'BAM file not sorted' errors in real workflows. It covers the common causes, the diagnostic commands that separate a header problem from a record-order problem, and the practical solutions for SAMtools, Picard, GATK, and similar tools. It also explains why BAM, SAM, CRAM, FASTQ, and FASTA play different roles, because format confusion is a frequent reason a pipeline sends the wrong file to the wrong step.

What the Error Means and Why Sort Order Matters

A BAM file is the binary, compressed form of a SAM file. SAM is a tab-delimited text format for aligned reads. It has a header made of lines that begin with @, followed by alignment records. BAM stores the same information in a compact binary format that tools can read quickly. CRAM is another alignment format that stores reads relative to a reference sequence and can be much smaller than BAM, but it depends on the correct reference being available.

Sort order is a property of the alignment records and the header. The SAM/BAM specification uses the @HD header line to record the version and the sort order. Common values are SO:unsorted, SO:queryname, and SO:coordinate. Coordinate sort means records are ordered by reference sequence name and then by the leftmost alignment position. Queryname sort means records are ordered by read name, which keeps paired reads near each other. The order of reference sequences is defined by the @SQ header lines.

Many tools require coordinate-sorted BAM because they walk along the genome. samtools index requires coordinate sort. Region queries such as samtools view sample.bam chr1:1000-2000 require an index. samtools mpileup, bcftools mpileup, GATK variant calling, and most genome browsers expect coordinate-sorted, indexed BAM. Some tools are stricter than others. A viewer may display an unsorted BAM without complaint, but indexing and variant calling will fail.

The error can appear in several forms. samtools index may report unsorted positions. A tool may stop with a message that says the BAM is not sorted. Picard may report a sort-order validation error. A region query may fail because the index cannot be loaded. The wording differs, but the root question is the same: does the file's actual record order match the order that the tool and the index expect?

Header sort order and actual record order are different things

The @HD SO tag is metadata. It records what the producing tool intended. It does not prove that every record is in the correct place. A BAM can have SO:coordinate in the header and still contain out-of-order records after a merge, a manual edit, a buggy writer, or a truncated and restarted job. This is why samtools index is such a useful test. It reads the records and fails when their positions go backward within a reference.

The opposite can also happen. A BAM can be in coordinate order by accident while its header says SO:unsorted. Some tools trust the header and refuse the file even though the records happen to be sorted. The clean fix is to sort the file with a tool that rewrites the header sort order, not to hand-edit the tag.

Coordinate sort, queryname sort, and the index

Coordinate sort orders reads by reference and position. Unmapped reads are placed at the end. This order supports indexing, region queries, pileup, and variant calling. Queryname sort orders reads by read name. It supports tools that need to compare mates, such as mate fixing and some duplicate-marking preparation steps. A name-sorted BAM is not coordinate sorted, even if the reads look tidy in a viewer.

An index stores offsets into a coordinate-sorted BAM so a tool can jump to a genomic region without reading the whole file. For BAM, the common index formats are .bai and .csi. The .bai format has size limits that can matter for very large chromosomes or many contigs. The .csi format is more flexible. If samtools index fails, do not assume the index file is corrupt. The more common reason is that the BAM is not coordinate sorted or the BAM was modified after the index was built.

FASTQ vs FASTA: why neither file is a BAM

FASTQ and FASTA are sequence files, not alignment files. A FASTA file contains one or more sequences with header lines that begin with >. It has no per-base quality scores. Reference genomes are usually distributed as FASTA because the reference sequence does not need quality scores. A FASTQ file contains reads with header lines that begin with @, a sequence line, a separator line, and a quality line. The quality line encodes Phred-scaled base qualities.

A BAM contains aligned reads with coordinates, mapping information, flags, mate information, and optional tags. You cannot sort a FASTQ into coordinate order because reads in a FASTQ do not have genomic coordinates until they are aligned. You can sort FASTQ by read name, but that is a different operation. This distinction matters in pipelines. A file named sample.fastq.gz is not a BAM, and a BAM cannot be validated with a FASTQ tool. If a pipeline error says a BAM is not sorted, confirm that the file is actually a BAM and not a renamed FASTQ, SAM, or CRAM file. samtools quickcheck and samtools view -H will expose that mistake quickly.

First Diagnostic Sequence

Work through these checks in order. Each step narrows the problem and avoids unnecessary re-sorting.

1. Confirm the file is readable and not truncated. Run samtools quickcheck -v sample.bam. A clean file produces no output. A nonzero exit status means the file is missing, truncated, or not a valid BAM. Re-download or regenerate the file before sorting.

1. Inspect the header. Run samtools view -H sample.bam | head. Look for the @HD line and the first few @SQ lines. Record the SO value and the reference names. If SO:queryname or SO:unsorted appears, coordinate sorting is required.

1. Check whether an index exists and whether it is current. Run ls -l sample.bam sample.bam.bai sample.bam.csi and compare timestamps. If the BAM is newer than the index, rebuild the index.

1. Try to build the index. Run samtools index sample.bam. If this succeeds, the BAM is coordinate sorted enough for indexing. If it fails with an unsorted-positions message, the records are out of order or the header order does not match the records.

1. Test a region query. Run samtools view sample.bam chr1:1000-2000 | head. If the index is missing or stale, this fails even when the BAM itself is sorted.

1. Validate with Picard when the error is ambiguous. Run picard ValidateSamFile I=sample.bam M=SUMMARY. This reports sort-order problems, missing read groups, and other SAM/BAM issues. A summary is enough for triage.

1. Re-sort and re-index only after the checks above identify the problem. Sorting is cheap compared with rerunning an alignment, but it is not a substitute for finding the cause. A BAM that fails samtools quickcheck will not be fixed by sorting.

Troubleshooting Table

Common Causes and Solutions

Cause 1: The alignment output was never sorted

This is the most common cause. Many aligners emit SAM or BAM in the order reads were processed. That order is not coordinate order. If the output is written directly to a BAM without a sorting step, the file will fail indexing and variant calling.

The mechanism is simple. The aligner assigns each read a reference and position, but it does not necessarily reorder the output. A pipeline that runs alignment and then immediately calls variants has skipped a required state transition. The fix is to place coordinate sorting directly after alignment and before indexing.

A typical paired-end alignment command can pipe into sorting:

bwa mem -t 8 ref.fa reads_1.fq.gz reads_2.fq.gz \
  | samtools sort -@ 4 -m 2G -o sample.sorted.bam -
samtools index sample.sorted.bam

The -@ 4 option asks for four sorting threads. The -m 2G option sets the memory per thread, so check the total memory before using it on a shared server. The -o option writes the final BAM. If the aligner has a native sorted-output mode, such as STAR's sorted BAM output, confirm that the mode is actually enabled.

Cause 2: The BAM is name sorted

samtools sort -n produces a queryname-sorted BAM. This is useful for samtools fixmate and other mate-aware steps. It is not coordinate sorted. A later tool that expects coordinate order will reject it.

The header usually reveals this with SO:queryname. The fix depends on the tool. If the tool needs name order, keep a name-sorted copy. If the tool needs coordinate order, run a coordinate sort. A robust paired-end workflow often alternates between the two:

samtools sort -n -o sample.qname.bam sample.bam
samtools fixmate -m sample.qname.bam sample.fixmate.bam
samtools sort -o sample.coord.bam sample.fixmate.bam
samtools markdup sample.coord.bam sample.md.bam
samtools index sample.md.bam

Each step has a required order. Using one BAM for every step without checking the sort requirement is a common source of confusing errors.

Cause 3: The header claims coordinate sort but records are out of order

This cause is common after merging, concatenating, or manually editing BAMs. The header may be copied from a coordinate-sorted file while the records come from multiple sources. A merge of one coordinate-sorted BAM and one name-sorted BAM can produce a file that claims coordinate order and fails indexing.

Another version occurs when a pipeline writes records in chunks. If the chunks are concatenated without a final sort, the file may be locally sorted within each chunk but out of order across chunk boundaries. samtools index catches this because it sees positions go backward.

The fix is to sort the final merged or concatenated file, not only the individual inputs. If you merge BAMs, sort after the merge and then index the merged BAM.

samtools merge -o merged.bam lane1.bam lane2.bam
samtools sort -@ 4 -m 2G -o merged.sorted.bam merged.bam
samtools index merged.sorted.bam

Cause 4: The index is missing, stale, or built from another file

A sorted BAM without an index will fail region queries even though the BAM itself is fine. A stale index is worse because it exists and looks valid. If the BAM was re-sorted, filtered, or renamed after indexing, the old index no longer describes the current file.

The fix is to rebuild the index from the final BAM. Do not copy an index from one BAM to another. Do not rename a BAM without renaming or rebuilding its index. The index name must match the BAM name, such as sample.bam.bai or sample.bam.csi.

samtools index sample.sorted.bam

For a very large reference, samtools index -c sample.sorted.bam creates a CSI index. Use the index format supported by your downstream tools. Some older tools expect .bai and do not read .csi.

Cause 5: Contig order or reference naming does not match

The @SQ header lines define the order of reference sequences. If two BAMs use 1, 2, 3 in one file and chr1, chr2, chr3 in another, merging or comparing them can break coordinate order. The same problem appears with mitochondrial names such as MT versus chrM, and with alternate contigs that are present in one reference but absent in another.

GATK and Picard also expect a sequence dictionary, usually a .dict file, that matches the reference FASTA. A BAM aligned to one reference build and sorted against a different dictionary can fail validation or produce incorrect region queries.

The fix is to use one reference naming scheme from alignment through variant calling. Build the FASTA index and dictionary from the same FASTA file.

samtools faidx ref.fa
samtools dict -o ref.dict ref.fa

If the BAM was aligned to a different reference, re-align or remap rather than forcing a sort against the wrong dictionary. Sorting can reorder records within the existing header, but it cannot change the biological coordinates to a different reference build.

Cause 6: Merging, adding reads, or streaming broke the order

Any operation that adds records to a BAM can break coordinate order. This includes merging lanes, adding unmapped reads, combining ChIP-seq and input samples, or appending reads from a second alignment run. Streaming operations are especially prone to this problem because each stream may be sorted internally while the combined output is not.

The fix is to sort after the last operation that changes the record set. If the workflow has multiple merge steps, sort once at the end, then index. If downstream tools need sorted intermediate files, sort each intermediate as well. The cost of one extra sort is small compared with debugging a failed variant call.

Cause 7: The tool actually needs name-sorted input

Not every not-sorted error means coordinate sort. Some tools require queryname order because they need to compare read pairs. samtools fixmate is a common example. If you give such a tool a coordinate-sorted BAM, it may report that the file is not sorted or produce incorrect mate information.

Read the tool's error message and documentation together. If the tool says it needs name-sorted input, use samtools sort -n. Keep the name-sorted BAM as an intermediate and coordinate sort the final output. Do not convert the whole pipeline to name sort just because one step needs it.

Cause 8: CRAM and missing reference

CRAM stores reads relative to a reference. If the reference is missing or does not match the CRAM header, tools may fail while reading the file. The error can look like a sort or format problem even when the real issue is reference resolution. Check the CRAM header and make sure the matching FASTA and its index are available.

The safest path in a mixed workflow is to convert CRAM to BAM with the correct reference, sort the BAM if needed, and index it. Confirm that the reference used for conversion matches the reference used for alignment and downstream analysis.

Cause 9: File corruption or truncation

A truncated BAM can fail in ways that resemble sorting errors. A transfer that stopped early, a full disk, or an interrupted writer can leave a file with a valid header and an incomplete record stream. samtools quickcheck -v is the fastest check. It will report a truncated file or a missing end-of-file marker.

Sorting will not repair a truncated BAM. Regenerate the file from the original data or re-download it from the repository. If the file came from a sequencing center, compare its checksum with the delivered manifest before spending time on downstream fixes.

Practical Fixes You Can Run

Fix 1: Coordinate sort and index

Use this when the header says SO:unsorted, SO:queryname, or the header is missing. Sort with enough memory and threads for the file size, then index the output.

samtools sort -@ 4 -m 2G -T sample.sorted.tmp -o sample.sorted.bam sample.bam
samtools index sample.sorted.bam

The -T option sets a temporary file prefix. Put that prefix on a disk with enough free space. Sorting a large BAM creates temporary files that can be several times the size of the input.

Safety note: check free disk space before sorting and keep a backup of the original BAM. Do not overwrite the only copy in a shared project directory. A full disk during sorting can leave a partial output file and a misleading error later.

Fix 2: Rebuild a missing or stale index

Use this when the BAM is already coordinate sorted but region queries fail. Confirm the BAM is sorted, then rebuild the index.

samtools index sample.sorted.bam
samtools idxstats sample.sorted.bam | head

idxstats prints one line per reference with the reference length and the number of mapped and unmapped reads. If it fails, the index or the BAM is still inconsistent. If it succeeds but a region query fails, check the contig name in the query against the @SQ names.

Fix 3: Re-sort after merge or annotation

Use this when a BAM was assembled from multiple files or modified after its last sort. Sort the final file and rebuild the index.

samtools sort -@ 4 -m 2G -o final.sorted.bam final.bam
samtools index final.sorted.bam

If the merge involved BAMs with different reference dictionaries, inspect the @SQ lines first. Sorting cannot reconcile incompatible reference names.

Fix 4: Use the right reference dictionary

Use this when GATK or Picard complains about the sequence dictionary or when contig names do not match. Build the .fai and .dict files from the same FASTA used for alignment.

samtools faidx ref.fa
samtools dict -o ref.dict ref.fa

Then re-sort and index the BAM with that reference naming scheme. If the BAM was aligned to a different build, re-align. Sorting alone is not a lifeline to a different genome build.

Fix 5: Validate before downstream analysis

Use Picard validation as a final gate for production pipelines.

picard ValidateSamFile I=sample.sorted.bam M=SUMMARY

A clean summary does not replace a tool-specific check, but it catches common sort-order, header, and read-group problems before they waste compute time. For a quick shell check, also run samtools quickcheck -v sample.sorted.bam and confirm the index exists.

Verification Checklist

After any fix, confirm the following before running variant calling or another expensive step.

• samtools quickcheck -v sample.sorted.bam produces no output.
• samtools view -H sample.sorted.bam | grep '^@HD' shows SO:coordinate.
• samtools index sample.sorted.bam succeeds without an unsorted-positions error.
• samtools idxstats sample.sorted.bam | head lists the expected contigs.
• samtools view sample.sorted.bam chr1:1000-2000 | head returns records for a known region.
• The index timestamp is newer than the BAM timestamp.
• The contig names in the BAM match the reference FASTA and sequence dictionary used downstream.

If any check fails, return to the diagnostic sequence rather than layering more fixes. One correct sort and one correct index are better than repeated attempts with stale files.

Common Mistakes and Limitations

Sorting the wrong file. A common mistake is sorting a FASTQ or a SAM file that was renamed with a .bam extension. Confirm the format with samtools view -H before sorting.

Trusting the header alone. SO:coordinate is metadata. Always test with samtools index or a validation tool, especially after a merge.

Sorting each input but not the merged output. Merging sorted BAMs does not guarantee a sorted merged BAM. Sort the final file after the merge.

Using name sort for the whole pipeline. Name sort is correct for mate-aware steps, not for variant calling or region queries. Keep name-sorted intermediates separate from coordinate-sorted analysis files.

Forgetting to rebuild the index. A new BAM with an old index will fail or return wrong regions. Rebuild the index whenever the BAM changes.

Copying an index between files. Index files contain offsets for one specific BAM. Copying a .bai to another BAM is not a valid shortcut.

Hand-editing the @HD tag. Changing SO:unsorted to SO:coordinate in a text header does not reorder records. Use a sorting tool that rewrites both records and header.

Assuming sorting repairs corruption. A truncated BAM will still be truncated after sorting. Check samtools quickcheck first.

Ignoring reference naming. Sorting against chr1 while the downstream tool expects 1 will create new failures. Match contig names across alignment, sorting, indexing, and variant calling.

Overlooking tool-specific sort requirements. Some tools want coordinate order, some want queryname order, and some can accept either. Read the tool documentation before converting an entire workflow.

Using too little memory or temp space. Sorting large BAMs can fail with cryptic errors when temporary storage fills. Set -m and -T deliberately, and monitor disk space.

Treating every not-sorted error as a BAM defect. Sometimes the BAM is fine and the index path, reference path, or tool input is wrong. The diagnostic sequence separates these cases.

Limitations matter here. A guide cannot know the exact reference build, contig naming scheme, or tool version in your pipeline. Those details determine the correct fix. Individual production failures need inspection of the actual header, index, and tool log.

Prevention Practices

Coordinate sort immediately after alignment, then index, then validate. Make this a fixed stage in the workflow rather than an afterthought.

Record the expected sort order in pipeline configuration and check it at every boundary. A workflow manager can run samtools view -H and fail fast if SO:coordinate is missing.

Use one reference FASTA, one .fai, and one .dict for the entire project. Lock the contig naming scheme before alignment. Do not mix RefSeq-style names with UCSC-style names in the same analysis.

Keep BAM and index files together. Treat an index as part of the BAM artifact, not as an optional side file. Rebuild the index whenever the BAM is rewritten, filtered, or renamed.

Use samtools quickcheck after transfers and after long-running steps. A quick integrity check catches truncated files before they reach a sorter.

Document the sort order needed by each tool in the pipeline. If a step needs name-sorted input, create a clearly named intermediate such as sample.qname.bam and keep the coordinate-sorted analysis file separate.

Frequently Asked Questions

What does a BAM file not sorted error mean?

It means the BAM records are not in the order the tool expects, usually coordinate order. The fix is to check the header and sort the BAM by coordinate before indexing or downstream analysis.

How do I check if a BAM is coordinate sorted?

Run samtools view -H sample.bam and look for SO:coordinate. Then run samtools index sample.bam. A successful index is strong evidence that the records are coordinate sorted.

Does samtools sort fix every not sorted error?

No. It fixes sort-order problems in a valid BAM. It does not repair a truncated file, a wrong reference, a missing index, or a tool that actually requires name-sorted input.

Why does samtools index still report unsorted positions after I sorted?

The sort may have been applied to a different file, the output may be stale, or the BAM may have been modified after sorting. Re-sort the exact file you plan to index and check the header again.

Should I sort by read name or coordinate?

Use coordinate sort for indexing, region queries, pileup, and variant calling. Use name sort only for tools that explicitly require read-name order, such as mate-aware preparation steps.

Do I need to sort FASTQ files?

No. FASTQ reads do not have genomic coordinates until they are aligned. You can sort FASTQ by read name, but coordinate sorting applies to alignment files such as SAM, BAM, and CRAM.

Is BAM the same as SAM or CRAM?

BAM is the binary compressed form of SAM. CRAM is a reference-based compressed alignment format. All three can hold aligned reads, but BAM and CRAM are the common formats for large sequencing projects.

What is the difference between FASTQ and FASTA?

FASTQ stores reads with per-base quality scores. FASTA stores sequences, often reference genomes, without quality scores. Neither is an alignment file, so neither can be coordinate sorted like a BAM.