How to Sort a BAM File: Coordinate vs Read Name Sorting

By Dr. Zubair Khalid, DVM, MS, PhD · 2026-11-10

Coordinate sorting orders alignments by genomic position and is the normal layout for visualization and most downstream analysis, while read-name sorting groups all records from the same read or read pair and is used for mate-aware processing, pairing checks, and a few legacy workflows. The rule of thumb is simple: sort by coordinate unless a specific tool or step requires name order, and treat a name-sorted BAM as an intermediate that you convert back to coordinate order before indexing, visualization, or variant calling.

A BAM file is the compressed binary form of SAM, the text format for aligned sequencing reads. Each BAM record stores a read name, also called QNAME, plus flags, reference name, position, mapping quality, CIGAR string, mate information, sequence, base qualities, and optional tags. BAM sorting changes the order of records in the file. It does not change the alignments themselves. SAM/BAM processing tools must handle large volumes of data, so memory management is a central engineering concern in this part of a pipeline [1].

The question of how to sort a BAM file comes up at nearly every stage of a sequencing workflow. An aligner may emit reads in alignment order. A duplicate marker may want coordinate order. A mate-fixing step may want names grouped together. A genome browser needs a coordinate-sorted and indexed file. A collaborator may send a name-sorted BAM because an older pipeline expected it. The right answer depends on the next consumer of the file, not on a universal ranking of sort orders.

The short answer: choose by the next tool

Use coordinate sorting when the next step needs genomic access, regional queries, or a standard BAM index. This covers variant calling, depth calculation, read visualization, most duplicate marking, and many RNA-seq counting workflows. Use read-name sorting when the next step needs all records for a read or read pair to sit next to each other. This covers mate fixing, some paired-end quality control steps, and a few older tools that read BAM sequentially and expect queryname order. The best sort order is the one the next tool documents as valid.

A useful decision rule has three parts. First, check the tool documentation for the required sort order. Second, check whether the workflow needs random access by genomic region. Third, check whether the step needs mates grouped by read name. If the answer to the second question is yes, the file must be coordinate sorted and indexed. If the answer to the third question is yes, use name sorting for that step, then return to coordinate order.

Read-name sorting is not a general replacement for coordinate sorting. A name-sorted BAM is valid BAM, but it is not suitable for a standard BAM index. A coordinate-sorted BAM without an index is also incomplete for many region-based tools. Sort order and indexing are separate requirements. Coordinate order is usually the prerequisite for indexing.

Coordinate vs read-name sorting at a glance

The table captures the pros and cons that decide most real choices. Coordinate sorting wins on compatibility with genomic analysis. Read-name sorting wins when the read pair, not the genome position, is the unit of work. The sections below explain each option on its own terms before returning to the decision points.

What BAM sorting changes and what it does not

Sorting changes record order and the sort-order field in the BAM header. The SAM header has an @HD line that can carry an SO tag, such as SO:coordinate or SO:queryname. Some tools read that tag, and some infer order from the file. A file named sample.sorted.bam is not proof of anything. Check the header with samtools view -H sample.bam and look at the @HD line. If the SO tag is missing or says unsorted, do not assume the file is analysis-ready.

Sorting does not repair alignments. It does not fix mate flags, add read groups, mark duplicates, recalibrate base qualities, trim adapters, or remove low-quality reads. It does not convert FASTQ to FASTA. It does not turn a BAM into a different file format. It only rearranges records and updates the header. That narrow scope is why sorting is a reliable step. It is also why sorting cannot rescue a BAM that has malformed records or missing metadata.

Where BAM sits among bioinformatics file formats

A practical view of bioinformatics file formats helps prevent sort-order mistakes. FASTQ stores unaligned reads with per-base quality scores. FASTA stores sequences, often reference sequences, without per-base quality scores. SAM stores aligned reads in text form. BAM stores the same aligned-read information in compressed binary form. CRAM is another compressed aligned-read format that uses a reference sequence to reduce size. The fastq vs fasta distinction is about whether per-base quality scores are present. The BAM question is different. BAM is for reads that have been aligned, and sorting is about the order of those aligned records.

A BAM file can be converted back to FASTQ if the records still contain the sequence and quality fields, but sorting itself does not perform that conversion. A BAM file is not a FASTA file, even when it contains reference-aligned reads. A coordinate-sorted BAM is not automatically a variant file, a count matrix, or a genome browser track. Each downstream format has its own requirements.

Sort order is metadata plus physical order

The header tag and the physical record order should agree. A BAM header that says SO:coordinate while the records are in name order is worse than an honest unsorted file because it can fool a tool that trusts the header. After any sort, confirm both the header and the intended index state. For coordinate-sorted output, run samtools quickcheck and then build an index. For name-sorted output, confirm SO:queryname and do not attempt a standard genomic index.

Coordinate sorting for genomic analysis

Coordinate sorting, also called position sorting, places records in the order of the reference sequences declared in the BAM header. Within each reference, records are ordered by the leftmost aligned position. Reads that are unmapped or lack a coordinate generally go after the mapped records. The exact tie-breaking among records at the same position can differ between tools and versions. That detail matters for reproducibility checks, but it rarely matters for variant calling or visualization.

Coordinate order is the layout that makes regional access possible. A BAM index stores offsets that let a tool jump to a genomic interval instead of reading the entire file. A standard BAI or CSI index depends on coordinate order. If the BAM is not coordinate sorted, the index cannot answer region queries correctly. This is why genome browsers, region-based pileup tools, and many variant callers require coordinate-sorted and indexed input.

How to sort a BAM file by coordinate with samtools

The common command uses samtools sort with threads, a per-thread memory limit, a temporary-file prefix, and an explicit output file. The example below uses 8 threads and 2 GB per thread. That can approach roughly 16 GB of sort buffer memory before overhead, compression buffers, and other process memory are counted.

samtools quickcheck sample.bam
samtools view -H sample.bam | head -n 20
samtools sort -@ 8 -m 2G -T tmp/sample.coord -o sample.coord.bam sample.bam
samtools index sample.coord.bam
samtools quickcheck sample.coord.bam

The -@ option sets worker threads. The -m option sets the maximum memory per thread. The -T option sets the temporary-file prefix. The -o option writes the output BAM. The samtools index step creates a genomic index for the coordinate-sorted BAM. The final samtools quickcheck checks for truncation and basic file integrity. It does not prove that the sort order is correct, so also inspect the header.

samtools view -H sample.coord.bam | grep '^@HD'

The @HD line should show SO:coordinate. If it does not, stop and find out why. A common cause is writing output through a command that does not preserve the header as expected, or sorting a file that already had a misleading header and then failing to regenerate it. Use the sorter's normal output path rather than ad hoc redirection when possible.

Coordinate sorting with Picard

Picard SortSam uses a different command style but the same concept. The SORT_ORDER=coordinate setting requests coordinate order. CREATE_INDEX=true asks Picard to build an index for coordinate-sorted output. The TMP_DIR setting points temporary files to a scratch location with enough space.

java -jar picard.jar SortSam \
  I=sample.bam \
  O=sample.coord.bam \
  SORT_ORDER=coordinate \
  CREATE_INDEX=true \
  TMP_DIR=/scratch/picard_tmp

Picard and samtools can both produce coordinate-sorted BAMs. They may differ in tie-breaking for records at identical positions. That is usually acceptable for analysis. If a workflow compares BAMs record by record, use one sorter consistently or normalize the comparison rather than assuming byte-for-byte identity.

When coordinate sorting is the right choice

Coordinate sorting is the right choice for variant calling because variant callers traverse the genome and expect reads near the positions they evaluate. It is the right choice for genome browser review because the browser needs to fetch a region quickly. It is the right choice for depth and coverage calculations over intervals. It is the right choice for many RNA-seq counting tools because they assign reads to features in genomic order. It is also the usual final order for duplicate marking and base quality score recalibration pipelines, even when an earlier mate-fixing step used name sorting.

Coordinate sorting is not only about convenience. It is a contract with downstream tools. A tool that requests coordinate-sorted input may fail, warn, or silently produce wrong results if given name-sorted input. Silent wrong results are the dangerous case, so check the header and the index before trusting a BAM.

The trade-offs of coordinate sorting

The main cost is that mates are usually far apart. A read pair from a fragment of a few hundred bases may have one mate on the forward strand and one on the reverse strand, with both alignments near each other in coordinate space. That sounds convenient, but records from the same read name can still be separated by other reads, secondary alignments, and supplementary alignments. Coordinate order is not a pairing layout. Any step that needs all records for one QNAME together must not rely on coordinate order alone.

Coordinate sorting also needs memory and temporary disk. Both samtools and Picard spill to temporary files when the data exceed the configured in-memory buffer. SAM/BAM processing frameworks are built for large files, but memory management remains a key challenge [1]. A sort that runs out of temporary disk can fail late, after hours of work. Point the temporary prefix or temporary directory at a filesystem with sufficient free space.

Read-name sorting for mate grouping

Read-name sorting places records in order by QNAME. In a paired-end library, both mates share the same QNAME, so name sorting brings them together, along with secondary and supplementary records that carry the same name. Picard calls this order queryname. The SAM header tag is usually SO:queryname. Read-name order is useful when the read pair is the unit of work and the genomic position is not the primary concern.

Name sorting is not the same as name collation. Full name sorting produces a globally ordered file by QNAME. Collation groups records with the same name into adjacent blocks without guaranteeing a global name order. samtools collate is a common collation tool. It can be faster or less memory-hungry than a full name sort for some workloads, but its output should not be described as a fully name-sorted BAM unless the tool explicitly says so. If a downstream tool requires queryname order, use a real name sort.

How to sort a BAM file by read name with samtools

The samtools command is close to the coordinate command, with -n added to select name sorting. The output cannot be indexed for genomic region lookup. Do not run samtools index on this file and expect a usable genomic index.

samtools sort -n -@ 8 -m 2G -T tmp/sample.qname -o sample.qname.bam sample.bam
samtools view -H sample.qname.bam | grep '^@HD'

The header should show SO:queryname. If you need coordinate order later, run a second sort.

samtools sort -@ 8 -m 2G -T tmp/sample.coord2 -o sample.coord.bam sample.qname.bam
samtools index sample.coord.bam

That two-step pattern is common. Name sorting is often a temporary state, not the final deliverable.

How to sort a BAM file by read name with Picard

Picard uses SORT_ORDER=queryname. Set CREATE_INDEX=false because a standard genomic index does not apply to queryname order.

java -jar picard.jar SortSam \
  I=sample.bam \
  O=sample.qname.bam \
  SORT_ORDER=queryname \
  CREATE_INDEX=false \
  TMP_DIR=/scratch/picard_tmp

The output header should show queryname order. If a later step needs coordinate order, run Picard SortSam again with SORT_ORDER=coordinate and CREATE_INDEX=true, or use samtools for the conversion.

When read-name sorting is the right choice

Read-name sorting is the right choice when a tool needs to process both mates of a pair together. Mate fixing is the classic example. In a common samtools duplicate-marking pipeline, the input is name sorted so that samtools fixmate can add or correct mate information and tags. The file is then coordinate sorted before duplicate marking. The name sort is an intermediate, not the final analysis layout.

Read-name sorting can also help with paired-end quality control, custom scripts that iterate over read names, and some older tools that expect queryname input. It is useful when you need to compare the two mates directly, check insert sizes by name, or extract paired reads into two FASTQ files. Some BAM-to-FASTQ workflows are easier when mates are adjacent, though the exact requirement depends on the tool.

The trade-offs of read-name sorting

The main cost is loss of genomic access. A name-sorted BAM cannot support standard region queries. A genome browser cannot use it as a normal BAM track. A variant caller that expects coordinate order may reject it. The file is still valid BAM, but it is not analysis-ready for position-based work.

Name order also has a subtle compatibility issue. Different tools may use different rules for ordering names that contain numbers. Lexicographic order places read10 before read2 because the character 1 sorts before 2. Natural order places read2 before read10 because it compares the numeric parts. A file sorted with one rule may not match the exact name order expected by a tool that uses the other rule. Check the tool documentation when exact name order matters. For most workflows, using one tool consistently is safer than mixing sorters without checking.

Options that decide speed, memory, and safety

The sort order is the main decision, but the options determine whether the job finishes cleanly. The most important samtools options are threads, per-thread memory, temporary prefix, compression level, and output path. The most important Picard options are sort order, temporary directory, maximum records in RAM, and index creation. These settings do not change the biological meaning of the file. They change runtime, memory use, temporary disk use, and output size.

Threads and memory

samtools sort -@ increases worker threads. More threads can speed up compression, decompression, and sorting, but each thread can add memory pressure. samtools sort -m sets the maximum memory per thread. If you set -@ 8 and -m 2G, plan for roughly 16 GB of sort buffer before overhead. That is not a hard cap on total process memory. Compression buffers, file handles, the operating system page cache, and other allocations also matter. SAM/BAM tools manipulate large data sets, and memory management is a key design concern in these programs [1]. Plan the job for the machine, not for an ideal machine.

When memory is tight, reduce -m and accept more temporary-file spilling. When temporary disk is tight, increase -m within the memory budget or move -T to a larger filesystem. Do not set both memory and temporary space too low. A sort can fail after reading most of the input because it cannot spill any more data.

Temporary files and output paths

Use an explicit temporary prefix with samtools sort -T. On shared clusters, the default temporary location may be small, slow, or private to a job in an unexpected way. Point -T at a scratch directory with enough free space and enough inodes. Use a unique prefix per sample to avoid collisions when multiple samples sort at the same time. Picard users should set TMP_DIR for the same reason.

Write the output to a new path. Do not sort a BAM onto itself. Even if a tool appears to handle the overwrite, a failed job can leave a truncated input and no recoverable output. Keep the original until the sorted file passes quick checks and the downstream step succeeds.

Compression and indexing

Compression level changes file size and CPU cost. A lower compression level can make the sort finish faster and produce a larger BAM. A higher compression level can reduce storage at the cost of CPU time. The right setting depends on whether the bottleneck is compute, network transfer, or long-term storage. For a temporary name-sorted file that will be deleted after conversion, faster compression is often the practical choice.

Coordinate-sorted output should be indexed before region-based use. samtools index sample.coord.bam creates a standard index for coordinate-sorted BAM. A CSI index can be requested when reference lengths exceed the limits of the default index format. The details depend on the tool version, so check the local documentation. The critical point is that the index is tied to coordinate order. Name-sorted output should not be indexed for genomic access.

Validation and header checks

samtools quickcheck is a fast integrity check. It can catch truncation and some malformed files. It does not prove sort order, reference compatibility, or biological correctness. Check the @HD line for the expected SO tag. Check that the @SQ reference names and lengths match the reference used by downstream tools. A coordinate-sorted BAM built against one reference dictionary can be useless or dangerous when used with a different dictionary. Sorting does not fix a reference mismatch.

Worked workflows

The best way to choose a sort order is to follow the file through a real workflow. The examples below show when coordinate sorting is the final layout and when read-name sorting is a temporary step. The commands are standard patterns. Adjust thread counts and memory limits to the machine and dataset.

Variant calling from raw alignments

A variant-calling workflow usually starts with a aligned BAM that may be unsorted or only partially sorted. The first goal is a coordinate-sorted, indexed BAM with the required read groups and duplicate handling. A minimal sorting step looks like this.

samtools quickcheck sample.raw.bam
samtools sort -@ 8 -m 2G -T tmp/sample.coord -o sample.coord.bam sample.raw.bam
samtools index sample.coord.bam
samtools view -H sample.coord.bam | grep '^@HD'

After sorting and indexing, run the duplicate-marking and base-quality steps required by the variant caller. Sorting itself does not mark duplicates. It does not add read groups. If the BAM lacks @RG records, fix that before variant calling. A coordinate-sorted BAM with no read groups may pass a quick check and still fail a downstream tool.

A variant caller may also require a reference dictionary that matches the BAM header. Check the @SQ lines. Coordinate sorting uses the header's reference order, so a coordinate-sorted BAM with the wrong dictionary is not equivalent to a correctly built file. The sort order and the reference dictionary are separate contracts.

Duplicate marking with a name-sort intermediate

A common samtools duplicate-marking pipeline uses name sorting to group mates before mate fixing, then returns to coordinate order for duplicate marking. The name-sorted BAM is an intermediate that can be deleted after the coordinate-sorted file is built.

samtools sort -n -@ 8 -m 2G -T tmp/sample.names -o sample.names.bam sample.raw.bam
samtools fixmate -m sample.names.bam sample.fixmate.bam
samtools sort -@ 8 -m 2G -T tmp/sample.coord -o sample.coord.bam sample.fixmate.bam
samtools markdup sample.coord.bam sample.markdup.bam
samtools index sample.markdup.bam

The first sort is by read name because fixmate needs mates grouped. The second sort is by coordinate because markdup and downstream analysis need genomic order. If you stop after the first sort and treat the name-sorted file as the final BAM, the later steps will not behave as expected.

Picard duplicate marking commonly works from coordinate-sorted input. Check the version's documentation for the exact requirement. The practical point is the same. Use the sort order the tool declares, and do not assume that a name-sorted file is a universal input.

Visual inspection in a genome browser

A genome browser loads only the region being viewed. It needs a coordinate-sorted BAM and an index. A name-sorted BAM is the wrong input even if the alignments are correct. The sorting step is usually a one-time preparation.

samtools sort -@ 8 -m 2G -T tmp/sample.view -o sample.view.bam sample.raw.bam
samtools index sample.view.bam

After indexing, open the BAM in the browser with the matching reference. If the track loads but reads appear in the wrong places, check the reference dictionary and the alignment coordinates before re-sorting. Sorting cannot correct wrong coordinates.

Converting a name-sorted BAM from a collaborator

A collaborator may send a name-sorted BAM because their pipeline used queryname order for mate fixing or for an older tool. Do not index it and do not pass it to a region-based tool. Convert it to coordinate order first.

samtools quickcheck collaborator.qname.bam
samtools view -H collaborator.qname.bam | grep '^@HD'
samtools sort -@ 8 -m 2G -T tmp/convert.coord -o collaborator.coord.bam collaborator.qname.bam
samtools index collaborator.coord.bam

Check the header before and after conversion. If the original header says SO:queryname, the converted file should say SO:coordinate. If the original header is missing or wrong, the sorter may still produce coordinate order, but downstream tools that trust the header may remain confused. Regenerate the header through the sorter rather than editing tags by hand.

Read counting and RNA-seq preparation

Many RNA-seq counting tools expect coordinate-sorted BAM because they assign reads to genomic features. Some tools can read unsorted BAM but run more slowly or require extra memory. Coordinate sorting is the safer default. If an upstream step used name sorting to fix mates or check pairing, convert back to coordinate order before counting. Keep the name-sorted intermediate only if it is needed for a specific diagnostic step.

SAM/BAM processing frameworks are designed to stream and transform large files, but they still face memory pressure when reads, qualities, and tags are held for sorting [1]. A counting workflow that starts from a coordinate-sorted, indexed BAM is easier to troubleshoot than one that depends on implicit sort order.

Common mistakes and limitations

The most common mistake is treating a name-sorted BAM as if it were coordinate sorted. The file is valid BAM, but it cannot support standard region queries. A tool may fail clearly, or it may produce suspicious results. Check the @HD SO tag and avoid indexing queryname output for genomic access.

The second common mistake is forgetting to index a coordinate-sorted BAM. Coordinate order alone does not create random access. A genome browser or region-based tool needs the index. Run samtools index after the sort and confirm that the index file exists and is newer than the BAM. If the BAM is replaced later, rebuild the index.

The third mistake is overwriting the input. samtools sort -o sample.bam sample.bam is a risky pattern. A failure can leave no usable input. Write to a new file, validate it, and only then remove the original if disk space requires it.

The fourth mistake is underestimating memory and temporary disk. -@ increases parallelism, and -m sets per-thread memory. The product of those settings is not the whole process memory, but it is a useful planning number. SAM/BAM tools handle large data volumes, so memory management is a real constraint rather than an afterthought [1]. Point temporary files at a filesystem with enough space and inodes.

The fifth mistake is confusing name sorting with name collation. samtools collate groups records with the same name into adjacent blocks. It does not guarantee a globally name-sorted file. A tool that requires queryname order may not accept a collated file as equivalent. Use samtools sort -n when full name order is required.

The sixth mistake is assuming that all name sorters use the same comparison rule. Names with numeric parts can sort differently under lexicographic and natural ordering. A BAM sorted by one tool may not match the exact order expected by another. Use one tool consistently when exact name order matters, and check the tool documentation.

The seventh mistake is expecting sorting to fix metadata. Sorting does not add read groups, repair mate flags, mark duplicates, or validate the reference dictionary. It also does not convert FASTQ to FASTA or BAM to FASTA. Sorting changes order. Everything else needs its own step.

A limitation of any sort order is that it is a file-layout decision, not a biological quality check. A coordinate-sorted BAM can still contain adapter contamination, low mapping quality, PCR duplicates, or wrong reference coordinates. A name-sorted BAM can still have broken mate information. Sorting makes downstream processing possible. It does not make the data correct.

Frequently Asked Questions

Is coordinate sorting always better than read-name sorting?

No. Coordinate sorting is the better default for genomic analysis, but read-name sorting is necessary when a tool needs mates grouped by read name. Use coordinate order for the final analysis-ready BAM unless a specific step requires otherwise.

Do I need to sort a BAM before indexing it?

Yes for a standard BAM index. Indexing requires coordinate order, and the index supports genomic region lookup. A name-sorted BAM is not a valid input for a standard genomic BAM index.

Can I index a read-name-sorted BAM?

No. Standard BAM indexes are built for coordinate order and genomic intervals. A read-name-sorted BAM can be read sequentially, but it cannot support region queries through a BAI or CSI index.

Does samtools sort -n produce the same order as Picard SORT_ORDER=queryname?

Not necessarily. Both sort by read name, but the exact name comparison rules can differ, especially for names with numeric parts. Check the header and use one sorter consistently when exact order matters.

Should I use samtools collate or samtools sort -n?

Use collate when you only need mates grouped into adjacent blocks and want a lighter-weight grouping step. Use sort -n when you need a fully name-ordered BAM. Collation is not the same as a complete name sort.

Does sorting a BAM convert FASTQ to FASTA or change the file format?

No. Sorting changes record order and the header sort-order tag. It does not convert FASTQ to FASTA, BAM to FASTA, or BAM to any other file format.

Sources

1. Comparing Ease of Programming in C++, Go, and Java for Implementing a Next-Generation Sequencing Tool.