# Reducing Genomics Storage Costs with Data Deduplication By the end of this guide you will be able to inventory a sequencing directory and fingerprint every file, find and consolidate byte-identical copies, catch duplicate sample records before they are ever sequenced, mark or remove PCR duplicates in BAM alignments, collapse UMI families, repack alignments as CRAM, and write a deduplication policy that keeps raw data immutable. Each step includes the exact commands, the output you should expect, the errors you are most likely to hit, and how to verify the result. You need a Linux or macOS shell with a current release of samtools and htslib, Python 3 with pandas for the record-level check, and the standard coreutils md5sum. Picard, UMI-tools, and TrieDedup are optional but cover cases the samtools-only path cannot. Write access to the storage you manage and a maintenance window for the first full sweep are the only other prerequisites. One framing point before the commands. Genomics data deduplication is three different operations that get confused constantly. Record-level deduplication removes duplicate patients, families, or samples from a cohort. Read-level deduplication flags or drops PCR duplicates inside an alignment. File-level deduplication removes duplicate copies of files on disk. Reducing genomics storage costs with data deduplication works best when you attack the layer where the redundancy actually lives. Record-level dedup saves the most because a dataset that is never generated costs nothing. Read-level dedup improves quantification accuracy and can shrink files dramatically for low-input libraries. File-level dedup is pure waste removal with zero analytical risk. A cost-conscious genomics data storage strategy uses all three, in that order of impact. ## Where Duplicates Hide in a Genomics Workflow Deduplication means any operation that identifies redundant entries and either collapses them to a single representative or flags them so downstream steps can ignore them. In a sequencing operation, redundant entries appear at three layers. At the record layer, the same person or family can enter a cohort more than once. Multicenter studies are especially exposed because relatives from one family may enroll at different sites, and deidentification removes the personal identifiers that would make the match obvious. Duplicates of this kind waste a sequencing lane and also bias family-based genetic analyses, which is why pedigree-aware deduplication is treated as a required pre-sequencing step in cancer genetics cohorts [1]. At the read layer, PCR duplicates are identical or near-identical reads that descend from one starting DNA or cDNA fragment copied many times during library amplification. They carry no extra biological information, but they inflate read counts and file sizes. Optical duplicates are a related artifact in which the same cluster on a flow cell is imaged as two nearby clusters. Unique molecular identifiers, UMIs, are random barcodes attached to each starting molecule during library prep, and they let a pipeline tell true biological copies apart from amplified copies of the same molecule. At the file layer, the same FASTQ (raw reads with base qualities), BAM (binary alignment of reads to a reference), CRAM (the reference-based compressed form of BAM), or VCF (variant calls) file tends to accumulate in project, scratch, backup, and collaborator transfer directories. These copies are byte-identical and detectable with certainty. The table below summarizes the layers, the detection method, and the storage action that follows. | Duplicate type | Typical example | How it is detected | Storage action | |--|--|--|--| | Record | Same family enrolled at two sites in a deidentified cohort | Pedigree, graph features, and machine-learning matching [1] | Merge records before library prep | | PCR duplicate | Two reads from one starting fragment | Identical alignment coordinates, or a shared UMI | Mark with the 0x400 flag, optionally drop | | Optical duplicate | Neighboring flow-cell clusters read as two | Picard proximity metrics | Mark and monitor the rate | | UMI family | Dozens of reads sharing one UMI at one position | UMI grouping and clustering tools | Count one molecule per family | | File copy | Same FASTQ in scratch and project trees | md5 checksums | Keep one copy, hardlink the rest | The walkthrough below starts with the zero-risk file layer, moves to the record layer, then handles reads, and closes with verification and policy. ## Genomics Data Deduplication Step by Step The walkthrough uses a directory called /data/seq/run_2026_05. Replace paths with your own. Every command runs on one machine with standard tools, and every command is safe to re-run. ### Step 1. Inventory and checksum every file A checksum is a fixed-length fingerprint computed from file contents. If a single byte changes, the fingerprint changes, so identical fingerprints mean byte-identical files. md5 is fast and sufficient for duplicate detection. Pair it with sha256 for archival integrity if your policy requires a second hash. Build the inventory: ```bash cd /data/seq/run_2026_05 find . -type f \( -name "*.fastq.gz" -o -name "*.bam" -o -name "*.cram" -o -name "*.vcf.gz" \) -print0 | xargs -0 md5sum > inventory.md5 wc -l inventory.md5 ``` Expected output: ``` 638 inventory.md5 ``` On multi-terabyte trees this step is disk-bound, so run it inside tmux or nohup and expect hours rather than minutes. The -print0 and xargs -0 pair keeps filenames containing spaces intact. ### Step 2. Find and consolidate byte-identical files Extract hashes that appear more than once, then list the matching paths: ```bash cut -d' ' -f1 inventory.md5 | sort | uniq -d > duplicate_hashes.txt grep -Ff duplicate_hashes.txt inventory.md5 | sort ``` Expected output, illustrative: ``` 3b5d2f1a9c8e47d0b6f2a1e5d9c8b7a4 raw/SampleA_R1.fastq.gz 3b5d2f1a9c8e47d0b6f2a1e5d9c8b7a4 projects/AML_2024/raw/SampleA_R1.fastq.gz ``` Every pair of lines with the same hash is a guaranteed byte-identical copy. Confirm the pair directly, then keep one canonical path and replace the other with a hardlink, a directory entry that points at the same inode, the filesystem object holding the data, so no additional blocks are consumed: ```bash md5sum raw/SampleA_R1.fastq.gz projects/AML_2024/raw/SampleA_R1.fastq.gz rm projects/AML_2024/raw/SampleA_R1.fastq.gz ln raw/SampleA_R1.fastq.gz projects/AML_2024/raw/SampleA_R1.fastq.gz ls -l projects/AML_2024/raw/SampleA_R1.fastq.gz ``` Expected output: ``` -rw-r--r-- 2 bio bio 2467803125 May 14 09:12 projects/AML_2024/raw/SampleA_R1.fastq.gz ``` The 2 in the link-count column confirms both paths share one inode. Two rules keep this safe. Hardlinks only work within a single filesystem, so check mount points with df before linking across them. Never hardlink a file that any pipeline may rewrite in place, because an edit through one path appears through every path. ### Step 3. Deduplicate sample records before they become data This is the highest-impact step and the one labs skip most often. In the LiFT UP study of families with TP53 mutations, the investigators note that multicenter enrollment means relatives from one family can appear at several sites, that deidentified data blocks linkage through personal identifiers, and that these redundancies bias family-based genetic studies [1]. Their answer combines heuristic labeling from mutation variant diversity and family characteristics, graph-based features, and a [random forest](/blog/data-analysis/what-is-random-forest) classifier refined by active learning, and it reached 99.95 percent automated processing of pedigree pairs in that cohort [1]. A practical version for any lab runs before library ordering. Match enrollment records on stable, non-identifier keys and review every collision: ```python import pandas as pd ped = pd.read_csv("enrollment.csv", dtype=str) ped["family_key"] = ped["family_id"].str.strip().str.upper() ped["pair_key"] = ped["family_key"] + "|" + ped["relation"].str.strip().str.upper() dupes = ped[ped.duplicated("pair_key", keep=False)].sort_values("pair_key") print(dupes[["site", "family_id", "relation", "subject_id"]].to_string(index=False)) ``` Expected output, illustrative: ``` site family_id relation subject_id BOST F1022 proband S-4471 MAYO F1022 proband S-8812 ``` Reconcile with the coordinating center before sequencing either record. The cheapest byte to store is the one you never generate. The same logic applies downstream of sequencing. Opfi, for example, filters redundant candidate gene clusters during annotation of assembled genomic and metagenomic data, which keeps cluster catalogs from filling storage with near-identical entries [2]. ### Step 4. Mark PCR duplicates in aligned reads ![Four-step flow from name sort through fixmate to coordinate sort and markdup, with the flags each step needs.](/article-images/illustrations/genomics-data-deduplication-markdup-chain.webp) For a standard alignment, samtools needs the file name-sorted so mates sit together, then mate information filled in, then coordinate order restored, then duplicates flagged: ```bash samtools sort -n -o namesorted.bam aligned.bam samtools fixmate -m namesorted.bam fixmate.bam samtools sort -o positionsorted.bam fixmate.bam samtools markdup -s positionsorted.bam marked.bam ``` The -m flag on fixmate fills the MC tag (the mate CIGAR string) and the ms tag (a mate score) that markdup requires. The -s flag on markdup prints a summary while it sets the duplicate bit, flag 0x400, on reads judged to duplicate an earlier read pair. Expected summary output, abbreviated, with exact key names varying slightly between samtools versions: ``` READ: 94658528 WRITTEN: 94658528 EXCLUDED: 612204 PAIRED: 90331420 SINGLE: 998454 DUP PAIR: 5820417 DUP SINGLE: 41032 ``` Confirm with flagstat: ```bash samtools flagstat marked.bam ``` Expected output, abbreviated: ``` 94658528 + 0 in total (primary) 5861449 + 0 duplicates ``` Two failures account for most support tickets at this step. Running markdup on a file that never went through fixmate -m stops with an error about a missing MC tag, and the fix is to run fixmate with -m and re-sort. Running markdup on a name-sorted file fails because the algorithm needs coordinate order. Both are fixed by following the four-command chain in order. ### Step 5. Decide whether to mark or remove ![A short decision path asking whether the downstream tool needs duplicate flags, leading to keep marked or remove reads.](/article-images/illustrations/genomics-data-deduplication-mark-or-remove.webp) Marking writes a flag and changes file size very little. Removal physically drops the reads: ```bash samtools markdup -r -s positionsorted.bam deduped.bam ``` You can also filter duplicates out of an already-marked file, since 1024 is the decimal value of 0x400: ```bash samtools view -b -F 1024 -o deduped_from_marked.bam marked.bam samtools view -c deduped_from_marked.bam ``` The count should equal WRITTEN minus the duplicate reads. Compare sizes: ```bash ls -lh marked.bam deduped.bam ``` Illustrative output: ``` -rw-r--r-- 1 bio bio 41G marked.bam -rw-r--r-- 1 bio bio 33G deduped.bam ``` A useful approximation is expected reduction in gigabytes ≈ duplicate fraction × file size, because duplicate reads compress about as well as unique ones. Keep the marked file when downstream tools consume duplicate information, as some variant callers do, and store the deduplicated file when the pipeline is count based. Whichever you choose, save the metrics output and the exact command line in a provenance log next to the data, because a deduplicated BAM without its metrics cannot be audited later. The Picard equivalent writes a rich metrics file: ```bash picard MarkDuplicates I=positionsorted.bam O=marked.bam M=markdup_metrics.txt ``` The metrics separate optical from PCR duplicates and report the duplicate fraction per library, which makes the metrics file the artifact to archive even when you delete nothing. ### Step 6. Collapse UMI families when the library has them When the library carries UMIs, deduplication should group by UMI rather than by coordinates alone: ```bash umi_tools dedup -I aligned_umi.bam -S umi_deduped.bam ``` The default method, directional, builds an adjacency graph of UMIs observed at the same alignment position, connects each UMI to a better-supported neighbor within one substitution, and counts one molecule per family. Alternative methods include unique, percentile, adjacency, and cluster. Two refinements matter for hard cases. UMI-nea compares UMIs with Levenshtein distance, an edit distance that counts insertions and deletions as well as substitutions, where Hamming distance counts substitutions only. It pairs that comparison with a clustering approach and an adaptive UMI filter, giving more accurate groupings on error-prone long-read data and ultra-high-depth short-read data [3]. For single-cell data, let the quantifier own deduplication. Alevin performs cell barcode detection, UMI deduplication, and gene count estimation end to end, considers transcript-level constraints and reads that multimap between genes, and runs roughly eight times faster than existing gene quantification approaches with less memory [4]. scruff covers CEL-Seq and CEL-Seq2 [preprocessing](/blog/data-analysis/what-is-data-preprocessing) in R with UMI deduplication and extensive quality visualizations [5]. Pre-collapsing single-cell reads yourself discards information the quantifier needs. ### Step 7. Repack alignments as CRAM CRAM, developed by the team behind the European Nucleotide Archive at EMBL-EBI, stores each read as differences from a reference sequence rather than as full sequence text, which is why CRAM files are consistently smaller than the equivalent BAM. Conversion is one command: ```bash samtools view -C -T GRCh38.fa -o marked.cram marked.bam samtools quickcheck marked.cram && echo "CRAM OK" ``` Expected output: ``` CRAM OK ``` Three operational rules apply. Readers need the reference, so keep it at a stable path or configure the REF_PATH and REF_CACHE environment variables that CRAM clients use to locate reference sequences. Avoid lossy CRAM options on anything archival. Verify that read counts match the BAM before you delete the BAM, using flagstat on both files. ### Step 8. Verify, log, and schedule Close every deduplication pass with verification and a written record: ```bash samtools flagstat deduped.bam > deduped.flagstat.txt md5sum deduped.bam > deduped.bam.md5 md5sum -c deduped.bam.md5 ``` Expected output: ``` deduped.bam: OK ``` A failed integrity check looks like this: ``` sample_R1.fastq.gz: FAILED md5sum: WARNING: 1 computed checksum did NOT match ``` The fix is to re-transfer the file from the canonical copy and recheck. Write a dedup.log per run containing the samtools version string, every command line, the metrics files, and the checksum inventory. Schedule the file sweep quarterly and after every project closeout, because duplicate file trees regrow as fast as labs ship data to collaborators. ## How Much Storage Can Genomics Data Deduplication Save? The honest answer depends on the layer, and the numbers differ by an order of magnitude. At the file layer, savings equal the size of every byte-identical copy, and detection is exact, so this is risk-free space back. Labs that have never swept their trees routinely find that scratch, backup, and project directories hold the same FASTQs two or three times. At the read layer, savings track the duplicate fraction, which is a property of the library, not of the storage system. In a four-platform study spanning the Illumina NovaSeq 6000, Illumina NovaSeq X, Element Biosciences AVITI, and Singular Genomics G4, RNA input below 125 ng produced libraries in which 34 to 96 percent of reads were discarded by deduplication, with the discarded share rising as input fell, and the pattern held on all four platforms [6]. A low-input library can therefore shed a third to nearly all of its aligned reads when duplicates are dropped, while a high-input library with a low duplicate rate shrinks only modestly. Those percentages describe reads, and file size follows approximately. At the record layer, the largest single saving is a dataset never generated. One duplicate enrollment caught before library prep saves the full per-sample footprint, on the order of tens of gigabytes for an exome and roughly a hundred gigabytes for a 30-fold whole genome once aligned, with raw reads adding more on top. One framing caution. Read-level deduplication exists primarily for accuracy, and storage savings are a co-benefit. The two goals conflict when a downstream tool needs marked duplicates present, so plan storage around the scientific decision rather than letting file size make the scientific decision for you. ## Deduplication Choices That Change Your Results Duplicate handling is an analysis parameter, and the evidence for how much it matters is direct. In paired hepatocellular carcinoma and non-tumor liver RNA-seq samples analyzed under four protocols (no deduplication, SAMtools-style deduplication, and deduplication using one or two molecular indices), deduplication without molecular indices shifted estimated expression values, most strongly for highly expressed genes, and produced a high proportion of false negatives in differential expression analysis. Leaving duplicates in place biased results toward false positives. Unique molecular indices improved accuracy, especially for highly expressed genes, and where a protocol could not carry them the study recommended a paired-end sequencing layout [7]. The library-prep angle matters as much as the pipeline angle. Duplicate rate is driven by the combined effect of RNA input amount and PCR cycle number, so a duplicate rate that climbs across runs is telling you about input material and amplification before it tells you anything about storage [6]. Treat a rising duplicate rate as a library-prep signal first. Practical rules follow from this. For RNA-seq with UMIs, deduplicate by UMI. For RNA-seq without UMIs, choose and record a deduplication policy deliberately rather than accepting a pipeline default. For variant calling, mark duplicates and follow the recommendations of the caller you use. For single-cell quantification, use the quantifier's built-in UMI handling. ## Tool Selection at a Glance | Tool | Deduplication level | Input it needs | Where it fits | |--|--|--|--| | samtools markdup | Read | Name-sorted BAM through fixmate -m, then coordinate sort | Routine marking or removal in WGS, WES, and bulk RNA-seq | | Picard MarkDuplicates | Read | Coordinate-sorted BAM | Per-library metrics, optical duplicate separation, GATK pipelines | | UMI-tools dedup | UMI | BAM carrying UMI tags | Directional UMI collapsing for tagged libraries | | UMI-nea | UMI | UMI sequences, reference-free | Indel-aware grouping for long reads and ultra-high depth [3] | | TrieDedup | Raw sequence | FASTA or FASTQ sequences | Prefix-tree deduplication that handles ambiguous N bases at raw-sequence level [8] | | Alevin | UMI, single-cell | FASTQ plus transcriptome reference | Droplet scRNA-seq quantification with deduplication built in [4] | | scruff | UMI, single-cell | CEL-Seq or CEL-Seq2 data in R | Preprocessing with UMI deduplication and QC visualizations [5] | | Opfi | Gene cluster | Annotated metagenomic assemblies | Filtering redundant candidate clusters during annotation [2] | A trie is a prefix tree, a structure that stores shared prefixes once so identical and near-identical sequences are compared without scanning every pair. That design is where TrieDedup's speed comes from, up to 270-fold faster than pairwise comparison at the cost of 32-fold higher memory use, with roughly 20 percent memory savings from a restricted dictionary implementation [8]. ## Genomics Data Deduplication Tips 1. Checksum before you delete anything. A duplicate pair you found by filename is a guess, a duplicate pair you found by md5 is a fact. 2. Keep raw reads immutable. Apply deduplication to derived files and to redundant copies, never to the only copy of the original FASTQ. 3. Archive metrics with every deduplicated file. The metrics file is the evidence the step happened and the record of the duplicate fraction on that date. 4. Fix duplicate rates at the bench. Input amount and PCR cycles drive the rate [6], so a protocol change often beats any storage optimization. 5. Design libraries to carry UMIs when quantification accuracy matters, because UMI-based deduplication is the only form that reliably distinguishes true molecular copies from amplification artifacts [7]. 6. Name intermediates by content hash. A file named by its own checksum lets pipelines skip recomputation and makes accidental duplication visible at a glance. 7. Sweep for duplicate files on a schedule, not on a grievance. Quarterly sweeps keep scratch trees from silently doubling your footprint. 8. Never deduplicate reads across sample boundaries by sequence identity. Two samples can share identical reads by chance or by design, and identity alone never proves the same source molecule. ## Storage-Layer Deduplication and Bioinformatics Data Management Filesystem-level deduplication exists, and it is worth understanding before you enable it. ZFS can deduplicate at the block level: ```bash zfs set dedup=on tank/genomics zfs get dedup tank/genomics ``` The catch is that sequencing formats are already compressed, and compressed bytes rarely repeat in identical blocks, so block-level deduplication typically returns little space on BAM, CRAM, and gzip streams while the deduplication table consumes memory. The reliable wins in genomics storage come from the file and record layers plus plain compression discipline, not from filesystem tricks. Policy is where the savings stick. A workable genomics storage policy has five clauses. First, exactly one canonical copy of raw data exists, at a recorded path, with a checksum file beside it. Second, derived files are regenerable and may be deleted once final outputs are verified. Third, every deduplication step writes its command line and metrics into a provenance log. Fourth, retention periods follow the consent and funder terms attached to each dataset. Fifth, only named roles may delete canonical copies, and every deletion is logged. These clauses are infrastructure decisions, and the biobanking literature treats them that way. A scoping review of digital infrastructure in biobanking and genomic research describes a persistent disconnect between the physical infrastructure for biospecimens and the digital frameworks that manage their data, and proposes the DIGS model, standing for Digital Infrastructure, Interoperability, Governance, and Skills, as a way to plan biobanks as integrated digital enterprises [9]. The same review emphasizes that infrastructure decisions can widen or narrow inequities between well-resourced centers and partners in low- and middle-income countries [9]. A deduplication policy is exactly this kind of decision, because it determines who can merge records, who can delete copies, and whose storage budget absorbs the waste. ## Common Mistakes and Limitations 1. Confusing marking with removal. Marking sets a flag and leaves the file essentially the same size. If the goal is space, you must remove reads or repack as CRAM. 2. Deduplicating deep RNA-seq without UMIs and trusting the counts. Coordinate-only deduplication shifts expression estimates and biases differential expression toward false negatives, while no removal biases toward false positives [7]. 3. Removing duplicates that a downstream tool needs. Some variant callers use duplicate information. Removing reads to save space and then running such a tool silently changes results. 4. Ignoring ambiguous bases in raw-sequence deduplication. Two identical reads can differ at an N, and tools that compare sequences naively treat them as distinct. TrieDedup was built to handle Ns correctly at the raw-sequence level [8]. 5. Grouping UMIs with substitution-only distance when the protocol or platform produces indel errors. Hamming-based grouping mis-merges or mis-splits families in that setting, which is the problem UMI-nea's Levenshtein-based approach addresses [3]. 6. Treating a duplicate-rate spike as a storage problem. The rate is set by input amount and PCR cycles [6], so the fix is usually at library prep. 7. Deleting the second copy before verifying the checksums. Verify, then delete, then verify the surviving copy. 8. Hardlinking across filesystems. The link either fails or silently becomes a full copy depending on the tool, so check mount points with df first. 9. Enabling block-level filesystem deduplication on compressed genomics formats and expecting large savings. The formats are already compressed, so gains are small while memory overhead is real. 10. Merging deidentified records on weak matches without review. Record-level deduplication in deidentified cohorts is a hard matching problem, as the pedigree work in cancer genetics shows [1], and false merges corrupt family structure in ways that are expensive to unwind. Limitations. There is no universal duplicate-rate threshold that declares a library good or bad, because the expected rate depends on input material, PCR cycles, read length, and layout. Validate any change in duplicate handling on a pilot pair of samples before applying it to a production pipeline, compare outputs, and keep the comparison artifacts. Individual datasets, especially clinical or multicenter ones, need review by the analyst who owns the downstream analysis, and when a deduplication decision could affect published results, get that decision signed off by the analysis lead. ## Frequently Asked Questions ### Does marking duplicates make a BAM file smaller? No. Marking only sets the 0x400 flag on duplicate reads and leaves the reads in place, so the file stays essentially the same size. Space savings require removing the flagged reads with samtools markdup -r or samtools view -F 1024, or repacking the file as CRAM. ### Can I deduplicate FASTQ files before alignment? Only in specific contexts. Raw-sequence deduplication suits amplicon, UMI, and repertoire workflows where identical sequences genuinely represent the same molecule, and trie-based tools handle ambiguous N bases efficiently at that level [8]. For genome resequencing, deduplicate after alignment, because mapping context is what defines a duplicate. ### What duplicate rate should worry me? There is no universal cutoff. The expected rate depends on input amount and PCR cycle number, so track the rate per library over time and investigate jumps relative to your own baseline. Treat a rising trend as a library-prep signal before treating it as a storage problem. ### Will duplicate removal change my differential expression results? It can. The choice between coordinate-based and UMI-based deduplication is an analysis parameter that shifts expression estimates, as covered in the section on deduplication choices. Use UMIs when the protocol supports them, and record whichever policy you adopt. ### Is filesystem-level deduplication worth enabling for BAM and CRAM files? Usually not as a primary strategy. Those formats are already compressed, so identical blocks are rare, while the deduplication table adds memory overhead. Checksum-based file sweeps, record-level deduplication before sequencing, and CRAM repacking return more space. ### How do I find duplicate patients across sites without personal identifiers? Match on stable non-identifier keys such as family structure and shared variant characteristics, then review collisions. For pedigree data, graph-based features with machine-learning classification automate most of the matching [1]. Treat borderline pairs as a human review queue, not an automatic merge. ### Do single-cell pipelines already handle deduplication? Yes. Droplet and plate-based quantifiers perform UMI deduplication as part of counting, and tools such as Alevin and scruff build it into their preprocessing [4], [5]. Do not pre-collapse single-cell reads yourself, because the quantifier needs the full family information. ### Should I delete the original BAM after converting to CRAM? Only after verification. Confirm the CRAM passes quickcheck, compare read counts with flagstat on both files, and confirm your long-term readers can access the reference. Keep the raw FASTQ immutable regardless, and record the conversion command in the provenance log. ## Sources 1. [Interpretable Active Learning for Pedigree Data Deduplication in Cancer Genetics.](https://pubmed.ncbi.nlm.nih.gov/41838968/) 2. [Opfi: A Python package for identifying gene clusters in large genomics and metagenomics data sets.](https://pubmed.ncbi.nlm.nih.gov/35445164/) 3. [UMI-nea: a fast, robust tool for reference-free UMI deduplication and accurate quantification.](https://pubmed.ncbi.nlm.nih.gov/40973030/) 4. [Alevin efficiently estimates accurate gene abundances from dscRNA-seq data.](https://pubmed.ncbi.nlm.nih.gov/30917859/) 5. [scruff: an R/Bioconductor package for preprocessing single-cell RNA-sequencing data.](https://pubmed.ncbi.nlm.nih.gov/31046658/) 6. [The impact of PCR duplication on RNAseq data generated using NovaSeq 6000, NovaSeq X, AVITI, and G4 sequencers.](https://pubmed.ncbi.nlm.nih.gov/40437619/) 7. [Effect of method of deduplication on estimation of differential gene expression using RNA-seq.](https://pubmed.ncbi.nlm.nih.gov/28321364/) 8. [TrieDedup: a fast trie-based deduplication algorithm to handle ambiguous bases in high-throughput sequencing.](https://pubmed.ncbi.nlm.nih.gov/38637756/) 9. [Connecting biospecimens and data: a scoping review-informed conceptual framework for digital infrastructure in biobanking and genomic research in low- and middle-income countries.](https://pubmed.ncbi.nlm.nih.gov/42254649/)