Zubair Khalid

Virologist/Molecular Biologist | Veterinarian | Bioinformatician

Conventional & Molecular Virology • Vaccine Development • Computational Biology

Dr. Zubair Khalid is a veterinarian and virologist specializing in conventional and molecular virology, vaccine development, and computational biology. Dedicated to advancing animal health through innovative research and multi-omics approaches.

Dr. Zubair Khalid - Veterinarian, Virologist, and Vaccine Development Researcher specializing in Computational Biology, Multi-omics, Animal Health, and Infectious Disease Research

Category: Guides

Read Duplication Sequencing

If you work with high throughput sequencing data, you need to understand read duplication. This guide explains what read duplication is, how it arises, when to remove duplicate reads, and how to handle them in different experimental contexts. It is written for bioinformaticians, genomic analysts, and researchers who process next generation sequencing (NGS) data and need a practical framework for making duplication related decisions. Read duplication refers to multiple sequencing reads that map to the same genomic start position (or nearly the same position) and have identical sequences. These duplicates can stem from library amplification steps (PCR duplicates), optical clustering errors during sequencing (optical duplicates), or genuine biological events such as high copy number regions or repetitive elements. Understanding the source of duplication is critical because each type has different implications for downstream analysis. The National Center for Biotechnology Information (NCBI) Bookshelf provides an authoritative overview of sequencing artifacts, including duplication, in its technical reference materials NCBI Bookshelf. The EMBL European Bioinformatics Institute (EMBL EBI) training resources also cover duplication detection and removal in their NGS analysis modules EMBL EBI Training. This guide synthesizes information from these sources and from published studies to give you a rigorous yet practical approach.

At a Glance

Aspect Key Point
What is read duplication? Identical or nearly identical reads mapping to the same genomic coordinates, often from PCR or optical artifacts, but also from biological copy number variation.
Why it matters Artifactual duplicates inflate coverage estimates, bias variant allele frequencies, and reduce effective sample size for variant calling and copy number analysis.
How to detect duplicates Tools like Picard MarkDuplicates, samtools markdup, and sambamba markdup compute duplication metrics from aligned BAM files.
When to remove duplicates For most DNA sequencing (whole genome, exome, targeted panels) removing PCR and optical duplicates is standard. For RNA seq, ChIP seq, and single cell data, removal depends on library design and use of unique molecular identifiers (UMIs).
Consequences of ignoring duplicates Overestimation of sequencing depth, false positive variant calls in high coverage regions, and incorrect copy number profiles.

Core Concepts

Read duplication in sequencing arises from two main processes. PCR duplication occurs during library preparation when the same original DNA fragment is amplified multiple times, producing identical reads after sequencing. Optical duplication occurs when the sequencer misidentifies a single cluster as two adjacent clusters, generating two reads with the same start and end positions. In PCR free libraries, optical duplicates are the primary artifact. Biological duplication, by contrast, originates from genuine genomic duplications, including copy number variants, segmental duplications, and repetitive elements. Distinguishing between technical and biological duplicates is impossible without additional information such as UMIs or paired end mapping properties. The Galaxy Training Network provides a comprehensive workflow for duplicate marking and analysis Galaxy Training Network. Their tutorials emphasize that the decision to remove duplicates should be based on the experimental design and the question being asked.

The level of read duplication is often expressed as a duplication percentage. In a typical whole genome sequencing library, PCR free protocols yield duplication rates below 5 percent, while PCR based exome libraries can reach 30 percent or more. Extremely high duplication rates (above 50 percent) indicate problems in library preparation or sequencing and may require a new library. The NCBI Sequence Read Archive stores raw sequencing data and associated duplication metrics for thousands of public experiments, serving as a reference for typical duplication levels across different protocols NCBI Sequence Read Archive. For example, a study on clonal evolution in grapevine reported duplication metrics in their whole genome bisulfite sequencing data, showing the need to account for duplicates when analyzing methylome data A dual genome methylome map.

Decision Points

You must decide whether to remove duplicate reads based on several factors.

Library type is the primary consideration. PCR amplified libraries almost always require duplicate removal to mitigate PCR bias. PCR free libraries (e.g., Illumina TruSeq PCR Free) have minimal PCR duplicates, so only optical duplicates need to be marked and possibly removed. For single cell RNA seq libraries that use UMIs, duplicates are expected, UMIs allow you to collapse reads from the same original molecule, but standard duplicate marking tools will incorrectly remove these biological duplicates if used without UMI aware algorithms.

Sequencing depth matters. In deep targeted sequencing (greater than 100x coverage), duplicates can inflate coverage non uniformly and bias variant allele frequency estimates. For low coverage whole genome sequencing (less than 5x), duplicate removal may reduce coverage too much, instead, you can mark duplicates and ignore them for certain analyses.

Downstream analysis dictates your approach. For copy number variant (CNV) calling, removing PCR duplicates is essential because duplicates artificially inflate read depth signals. A recent study on copy number variants in strabismus used duplicate removed data for accurate CNV detection Genomic Copy Number Variants. For structural variant (SV) calling, duplication handling varies by tool, some SV callers recommend keeping duplicates for breakpoint detection, while others prefer duplicate marking. You should consult tool specific documentation.

Use of UMIs changes everything. If your library incorporates UMIs, do not use traditional duplicate marking tools because they will discard unique biological reads. Instead, use UMI aware pipelines such as fgbio or PICARD's UmiAwareMarkDuplicatesWithMateCigar. The Bioconductor project provides packages like DupliCheckr and UMIanalyze for UMI based duplicate handling Bioconductor.

Practical Workflow

  1. Assess raw sequencing quality with FastQC or MultiQC. Look for high duplication levels in the raw read level report. If duplication is above 50 percent, investigate library preparation.

  2. Align reads to the reference genome using a splice aware aligner (for RNA seq) or a standard aligner (for DNA seq). For DNA seq, BWA MEM or Bowtie2 are typical.

  3. Sort and index the BAM file using samtools sort and samtools index.

  4. Mark or remove duplicates. Use Picard MarkDuplicates for DNA seq. Run with REMOVE_DUPLICATES=false to mark duplicates and keep them in the file. This allows you to optionally filter them later. For RNA seq without UMIs, use MarkDuplicates with VALIDATION_STRINGENCY=SILENT and TAGGING_POLICY=All.

    picard MarkDuplicates I=aligned.bam O=marked.bam M=metrics.txt

    For large files, sambamba markdup is faster. For UMI data, use UMI aware tools.

  5. Evaluate duplication metrics from the MarkDuplicates output. Key columns: PERCENT_DUPLICATION, UNMAPPED_READS, UNPAIRED_READS.

  6. Filter duplicates if needed for downstream analysis. You can use samtools view with flag 0x400 to exclude duplicates, or grep out reads with "duplicate" tag.

  7. Perform quality checks after removal. Re run FastQC and compare coverage histograms. Use IGV to visually inspect duplicate marking in high coverage regions.

Quality Checks

After duplicate marking, verify that the process worked correctly. Check that the percentage of duplicates in the metrics file matches expectations for your library type. A PCR free library should show less than 2 percent optical duplicates. For PCR based libraries, duplication rate should roughly equal (average fragment length / mean insert size) per target, but rates above 30 percent warrant attention.

Plot the duplication histogram from MarkDuplicates to see the distribution of duplicate read groups. A strong peak at low numbers (one or two copies) indicates normal amplification, a long tail of high copy numbers suggests overamplification.

Use Bioconductor packages like dupRadar to assess duplication across genomic regions Bioconductor. dupRadar plots duplication rate against expression level for RNA seq and can flag problematic genes.

For structural variant callers that rely on read pair signatures, check that duplicate marking does not remove discordant pairs needed for SV detection. Some callers, such as LUMPY, recommend not removing duplicates prior to calling. The study on resolving complex structural variants used multimodal genomics and likely accounted for duplication in their pipeline Resolving Complex Structural Variants.

Common Mistakes

Removing duplicates in UMI data without adjusting tools. Using standard MarkDuplicates on UMI libraries will discard genuine unique reads because the tool treats UMIs as part of the read sequence or ignores them. Always use UMI aware software.

Removing duplicates in RNA seq without considering isoform abundance. In RNA seq, the same transcript can generate multiple fragments that map to the same genomic position but originate from different mRNA molecules. Removing these duplicates destroys isoform quantification. Only remove duplicates in RNA seq when you have stranded libraries and use a tool that accounts for transcript expression (e.g., RSEM).

Assuming that duplication metrics in FastQC are accurate. FastQC's duplication level module uses only the first 50 bases of each read. It does not consider mapping position. This metric is a coarse screen, not a substitute for BAM level duplication analysis.

Overfiltering in low coverage studies. If your mean coverage is below 10x, removing duplicates might drop coverage below usable thresholds. In such cases, keep duplicates but consider using duplicate aware variant callers like GATK's HaplotypeCaller after marking duplicates (it ignores marked reads by default).

Ignoring optical duplicates when using patterned flow cells. Newer Illumina platforms have lower optical duplicate rates, but they still occur. Always run duplicate marking even on PCR free libraries.

Limits of Interpretation

Read duplication analysis cannot distinguish technical from biological duplicates without UMIs. If you observe high duplication in a single genomic region, it could be a true copy number gain, a repetitive element, or a PCR artifact. Without orthogonal evidence (e.g., array CGH, long read sequencing), you cannot conclude the cause. In a study on copy number variants in syndromic orofacial clefts, the authors used case parent trios and duplicate consensus detection to differentiate artifacts from genuine CNVs Consensus based detection.

Duplication metrics depend heavily on library complexity and sequencing depth. A low duplication rate does not guarantee a high quality library if the overall coverage is low. Conversely, a high duplication rate may be acceptable in targeted panels where the target region is small and heavily amplified (e.g., amplicon sequencing). The clinical genetics study on the SET gene deletion used targeted sequencing and likely encountered high duplication but acceptable results Elucidating the Role of SET.

Finally, duplicate removal is not reversible. Once you physically remove duplicates from the BAM file, you lose the ability to re evaluate duplication. Therefore, always mark duplicates but keep them in the BAM for future reanalysis. The evolutionary innovation study on fusion sequences noted the importance of maintaining read level records for reproducibility Evolutionary innovation through fusion.

Frequently Asked Questions

What is a typical read duplication rate for whole genome sequencing? For PCR free libraries, duplication rates are usually below 5 percent. For PCR based libraries, rates range from 10 to 30 percent depending on input DNA amount and number of PCR cycles. Rates above 40 percent often indicate low library complexity or overamplification.

Should I remove duplicates in whole exome sequencing? Yes, for whole exome sequencing using PCR amplified capture protocols, removing PCR duplicates is standard practice. This reduces false positive variant calls and improves coverage uniformity across the targeted regions.

How do unique molecular identifiers (UMIs) help with duplication? UMIs are short random sequences attached to each original molecule before amplification. They allow you to collapse reads from the same molecule into a consensus read, thereby removing PCR duplicates while preserving genuine biological duplicates (e.g., from multiple transcripts in RNA seq). UMI aware tools are essential for accurate duplicate handling.

Can I use duplicate marking on single cell RNA seq data? Not with standard tools. Single cell RNA seq libraries typically include UMIs, and duplicates result from capturing the same transcript in the same cell. Standard MarkDuplicates would incorrectly remove these. Use UMI aware pipelines designed for single cell data, such as those in the Cell Ranger or STARsolo workflows.

References and Further Reading

Related Articles