Genomic Data Processing: From Raw Sequencing to Analysis-Ready Files

Raw sequencing data requires a structured processing pipeline before any biological interpretation can begin. This article defines the core workflow for converting FASTQ files into analysis-ready formats, covering quality control, trimming, alignment, and post-alignment assessment. The intended readers are students, researchers, analysts, and life-science professionals who need practical decision criteria for each processing stage. The workflow described here applies to whole-genome, whole-exome, RNA-seq, and targeted sequencing projects, with assay-specific adjustments noted where relevant.

At a Glance

The table below summarizes the primary processing stages, the key decisions at each stage, and the outputs that should be retained for downstream analysis.

Scope of Genomic Data Processing

Genomic data processing transforms raw instrument output into structured files that support variant calling, expression quantification, metagenomic profiling, and other downstream analyses. The raw data typically arrives as FASTQ files containing nucleotide sequences and per-base quality scores. The processing pipeline must address sequencing artifacts, mapping errors, and technical variation before biological signals can be interpreted reliably.

RNA sequencing data analysis follows the same foundational steps of quality control, preprocessing, and alignment, with additional considerations for splice-aware mapping and transcript quantification [6]. Shotgun metagenomics data requires quality control, assembly, and binning steps that differ from single-genome workflows [7]. Single-cell RNA-seq adds cell-level quality control and dimensionality reduction after the standard alignment steps [20]. The core principles described here apply across these assay types, with specific adjustments noted in the relevant sections.

Core Principles of Data Processing

Data Integrity and Format Standards

The SAM and BAM formats serve as the standard for storing alignment information. SAMtools and BCFtools provide a collection of tools for file format conversion, sorting, querying, statistics, and variant calling, and these packages have been installed more than one million times via Bioconda [5]. The widespread adoption of these tools reflects the importance of standardized file formats for reproducible analysis.

File format conversion from SAM to BAM reduces storage requirements and speeds up subsequent processing steps. The sam2bam framework demonstrated that parallel processing components can reduce preprocessing runtime substantially on multi-core systems, with duplicate marking completing in about one minute for a whole-exome dataset on a 16-core node [13]. This performance gain matters for laboratories processing large cohorts where preprocessing time directly affects project timelines.

Reproducibility and Workflow Management

Reproducible analysis requires documented software versions, reference genome versions, and parameter settings. Workflow management systems such as Snakemake provide structured pipelines that can be rerun with consistent parameters. The miND pipeline for small RNA-seq analysis uses Snakemake and includes a separate workflow for downloading and preparing reference databases, which supports reproducibility by allowing database updates while preserving the ability to recreate prior analyses [14].

The Curare workflow builder for RNA-seq data analysis addresses reproducibility by subdividing the analysis into preprocessing, quality control, mapping, and downstream analysis stages, with multiple options at each step [19]. This modular approach allows researchers to document exactly which tools and parameters were used for each dataset.

FAIR Data Principles

The FAIR Guiding Principles describe the characteristics that make data findable, accessible, interoperable, and reusable [4]. Applying these principles to genomic data processing means maintaining clear file naming conventions, documenting processing steps in machine-readable formats, and storing both raw and processed data in accessible repositories. The NCBI provides data resources that support deposition and retrieval of genomic datasets [2].

Quality Control of Raw Sequencing Data

Per-Base Quality Assessment

Quality control begins with examining the per-base quality scores across all reads in a sample. Low-quality bases at read ends are common and may result from sequencing chemistry degradation. The quality control step is the foundation for all subsequent analysis, as poor-quality input data will produce unreliable alignment and variant calls [6].

The FASTQ format stores a quality score for each base, typically encoded as a Phred score. A Phred score of 20 corresponds to a 1 in 100 error rate, while a Phred score of 30 corresponds to a 1 in 1000 error rate. The distribution of quality scores across read positions should be examined for each sample to identify systematic issues.

Adapter Contamination Detection

Adapter sequences are added during library preparation and may remain in the reads when the insert size is shorter than the read length. Adapter contamination appears as a distinctive pattern in the quality control report, with quality scores dropping at the position where the read transitions from genomic sequence to adapter sequence. Detection of adapter contamination requires comparison of read sequences against known adapter sequences.

Duplication Assessment

PCR duplicates arise during library amplification and can bias downstream analyses by overrepresenting certain genomic regions. The duplication rate should be assessed during quality control, with high duplication rates indicating potential issues with library preparation or sequencing depth. Duplicate marking during post-alignment processing allows downstream tools to account for this artifact.

Sequence Composition Analysis

The base composition across read positions should be relatively uniform for genomic DNA libraries. Deviations from expected composition may indicate contamination, amplification bias, or library preparation issues. GC bias can be assessed by comparing the GC content of sequenced reads to the expected GC content of the reference genome.

Trimming and Filtering Strategies

Adapter Trimming

Adapter sequences must be removed before alignment to prevent mismatches at read ends. Trimming tools identify adapter sequences by comparing read ends against known adapter sequences and removing the adapter portion. The choice of trimming parameters depends on the library preparation kit and the sequencing platform.

Quality Trimming

Quality trimming removes low-quality bases from read ends. The trimming threshold should be set based on the downstream analysis requirements. Variant calling may tolerate lower quality bases than expression quantification, where mismapped reads can distort transcript abundance estimates.

Minimum Read Length Filtering

After trimming, reads that fall below a minimum length should be removed from the dataset. Very short reads are unlikely to map uniquely to the reference genome and may introduce noise into downstream analyses. The minimum length threshold depends on the read length distribution and the analysis requirements.

Poly-G and Poly-A Trimming

Sequencing platforms that use two-color chemistry may produce reads with long stretches of guanine bases at the end. These poly-G tracts arise from the loss of signal in the second channel and should be removed during trimming. RNA-seq data may require poly-A trimming to remove untemplated adenine bases.

Alignment to Reference Genome

Reference Genome Selection

The choice of reference genome version affects alignment results and downstream variant calls. Reference genomes are updated periodically, and the version used for alignment should be documented in the analysis records. The NCBI provides reference genome resources and maintains versioned assemblies [2].

Aligner Selection

The aligner should be selected based on the assay type and the analysis goals. RNA-seq data requires splice-aware aligners that can map reads spanning exon junctions [6]. Whole-genome sequencing data may use general-purpose aligners optimized for speed or sensitivity. Targeted sequencing data may benefit from aligners that handle off-target reads efficiently.

Mapping Quality Thresholds

Mapping quality scores indicate the confidence that a read is placed correctly. Reads with low mapping quality may map to multiple locations in the genome and should be filtered or flagged for downstream analysis. The mapping quality threshold should be set based on the analysis requirements, with more stringent thresholds for variant calling than for coverage estimation.

Reference Genome Indexing

The reference genome must be indexed before alignment. The indexing process creates data structures that allow the aligner to locate read matches efficiently. Index files are specific to the aligner and the reference genome version, and they should be stored alongside the reference genome for reproducibility.

Post-Alignment Processing

SAM to BAM Conversion and Sorting

The alignment output is typically stored in SAM format, which is a text-based format that can be large and slow to process. Conversion to BAM format reduces file size and speeds up subsequent operations [5]. The BAM file should be sorted by genomic position to support efficient access to specific regions.

The sam2bam framework demonstrated that parallel file format conversion and duplicate marking can reduce preprocessing runtime from about two hours to about one minute for a whole-exome dataset on a 16-core system [13]. This performance gain is relevant for laboratories processing large cohorts where preprocessing time directly affects project timelines.

Duplicate Marking

PCR duplicates should be marked in the BAM file before variant calling or expression quantification. Duplicate marking identifies reads that originate from the same DNA fragment and flags them so that downstream tools can avoid counting them multiple times. The duplicate marking algorithm considers the alignment position and the read sequence to identify duplicates.

Base Quality Recalibration

Base quality recalibration adjusts the quality scores assigned by the sequencing instrument based on empirical error patterns. The recalibration process uses known variant sites to estimate the relationship between reported quality scores and actual error rates. Recalibrated quality scores improve the accuracy of variant calling.

Indel Realignment

Insertions and deletions in the sequenced sample can cause misalignment at the edges of the indel. Indel realignment locally realigns reads to minimize mismatches around known indel sites. This step improves the accuracy of variant calls in regions containing indels.

Quality Assessment of Processed Data

Mapping Statistics

The proportion of reads that map to the reference genome provides an initial assessment of data quality. Low mapping rates may indicate contamination, adapter contamination, or reference genome mismatches. The mapping statistics should be reviewed for each sample and compared across samples in the same batch.

Coverage Metrics

Coverage describes the average number of reads that align to each genomic position. The coverage distribution should be examined to identify regions with unusually high or low coverage. For whole-genome sequencing, the coverage should be relatively uniform across the genome. For targeted sequencing, the coverage should be assessed for the target regions specifically.

Contamination Assessment

Cross-sample contamination can occur during library preparation or sequencing. Contamination can be detected by examining allele frequencies at known variant sites or by comparing the sample genotype to expected patterns. The GRAPE pipeline for genomic relatedness detection includes data preprocessing steps that support contamination assessment [10].

Capture Efficiency for Targeted Sequencing

For targeted enrichment sequencing, the capture efficiency describes the proportion of reads that map to the target regions. The Castanet pipeline for multi-pathogen enrichment data reports method-specific metrics that enable quantification of capture efficiency and estimation of pathogen load [11]. These metrics help differentiate low-level positives from contamination.

RNA-Seq Specific Processing Considerations

Splice-Aware Alignment

RNA-seq reads that span exon junctions require splice-aware alignment to map correctly. The aligner must identify the exon boundaries and allow reads to map across introns. The choice of splice-aware aligner affects the sensitivity and specificity of transcript detection [6].

Transcript Quantification

After alignment, RNA-seq data can be quantified at the gene or transcript level. Quantification methods may use the aligned reads directly or may use pseudo-alignment approaches that bypass full alignment. The quantification method should be selected based on the analysis goals and the reference annotation quality.

Differential Expression Analysis

Differential expression analysis identifies genes that exhibit varying expression levels across different conditions or sample groups [6]. The analysis requires a count matrix derived from the quantification step and a statistical model that accounts for biological and technical variation. The results should be interpreted in the context of the experimental design and the sample size.

Small RNA-Seq Considerations

Small RNA-seq data requires additional processing steps to handle the short read lengths and the diverse RNA species present in the sample. The miND pipeline for small RNA-seq analysis includes preprocessing, mapping, visualization, and quantification of reads, with mapping statistics available for multiple RNA species including miRNAs, tRNAs, piRNAs, snRNAs, and snoRNAs [14]. The pipeline generates a comprehensive report containing essential qualitative and quantitative results.

Metagenomics Specific Processing Considerations

Quality Control for Metagenomic Data

Shotgun metagenomics data requires quality control to remove host contamination and low-quality reads before assembly [7]. The quality control step may include filtering against the host genome to remove reads originating from the host organism.

Metagenomic Assembly

Metagenomic assembly reconstructs genomic sequences from the mixed microbial community present in the sample. The assembly process is computationally intensive and requires careful parameter selection. The quality of the assembly affects all downstream analyses, including taxonomic assignment and functional profiling [7].

Binning and Taxonomic Assignment

Binning groups assembled contigs into bins that represent individual genomes from the metagenome [7]. Taxonomic assignment identifies the organism of origin for each contig or read. The binning and taxonomic assignment steps require reference databases and classification tools.

Single-Cell RNA-Seq Specific Processing Considerations

Cell-Level Quality Control

Single-cell RNA-seq data requires quality control at the level of individual cells, in addition to the read-level quality control used for bulk sequencing [20]. Cells with low total read counts, high mitochondrial read fractions, or unusual gene expression patterns may represent damaged cells or doublets that should be removed.

Dimensionality Reduction and Clustering

After quality control, single-cell RNA-seq data undergoes feature selection, dimensionality reduction, and cell clustering [20]. These steps group cells into populations that can be annotated based on marker gene expression. The clustering resolution affects the granularity of the cell populations identified.

Batch Correction

Single-cell RNA-seq experiments often include multiple batches, and technical variation between batches can obscure biological signals. Batch correction methods adjust the data to remove batch effects while preserving biological variation [20]. The choice of batch correction method depends on the experimental design and the expected biological variation.

Pipeline Implementation Options

Command-Line Pipelines

Command-line pipelines provide the greatest flexibility and are the standard approach for processing large datasets. The SAMtools and BCFtools packages provide command-line tools for file format conversion, sorting, querying, and statistics [5]. Command-line pipelines can be scripted and automated for batch processing.

Workflow Management Systems

Workflow management systems such as Snakemake provide structured pipelines that track dependencies between steps and support resumable execution. The miND pipeline uses Snakemake and incorporates the advantages of a flexible workflow management system [14]. Workflow systems support reproducibility by documenting the exact steps and parameters used.

Graphical User Interfaces

Graphical user interfaces can make genomic data processing accessible to researchers without command-line experience. The AgrOmicSo software provides a client-server interface that allows researchers to manage and execute complex NGS data analysis pipelines on a remote server from a graphical user interface [17]. The software integrates tools for quality control, read mapping, variant calling, and annotation, and supports three variant calling algorithms.

Cloud-Based Pipelines

Cloud computing provides scalable resources for genomic data processing. The Scalable pathogen pipeline platform (SP3) enables unified genomic data analysis with elastic cloud computing [22]. Cloud-based pipelines can scale to accommodate large datasets without requiring local high-performance computing infrastructure.

Performance Optimization

Parallel Processing

Genomic data processing can be parallelized across multiple processors to reduce runtime. The sam2bam framework demonstrated that parallel processing components can reduce preprocessing runtime substantially, with whole-genome sequencing preprocessing reduced from about 20 hours to about nine minutes on a 16-core system [13]. Parallel processing requires careful management of memory and storage resources.

Hardware Acceleration

Hardware compression accelerators and high-bandwidth storage can improve processing performance. The sam2bam framework can utilize hardware compression accelerators when available [13]. The performance gains from hardware acceleration depend on the specific hardware configuration.

Memory Management

Large genomic datasets require substantial memory for processing. The sam2bam framework used up to 711 GB of memory for whole-genome sequencing preprocessing on a single node [13]. Memory requirements should be assessed before processing large datasets to avoid out-of-memory errors.

Records and Documentation

Processing Logs

Each processing step should generate a log that records the tool version, parameters, input files, and output files. The logs should be stored with the processed data to support reproducibility and troubleshooting. The processing logs should be reviewed for errors or warnings that may indicate issues with the data.

Quality Control Reports

Quality control reports should be generated for each sample at each processing stage. The reports should include the metrics described in the quality assessment sections, including per-base quality, adapter contamination, mapping statistics, and coverage metrics. The miND pipeline automatically generates a comprehensive report containing essential qualitative and quantitative results [14].

Version Documentation

The versions of all software tools and reference databases should be documented for each processing run. The reference databases should be stored with the version information to support reproducibility [14]. The documentation should be stored in a format that can be accessed by all members of the research team.

Common Failure Patterns

Adapter Contamination Missed During Trimming

Incomplete adapter trimming can cause alignment errors and false variant calls. The quality control report should be examined after trimming to confirm that adapter sequences have been removed. If adapter contamination persists, the trimming parameters should be adjusted.

Reference Genome Version Mismatch

Using different reference genome versions for different samples in the same study can introduce false variants. The reference genome version should be documented for each sample and verified before downstream analysis.

Insufficient Coverage for Variant Calling

Low coverage in specific genomic regions can cause false negative variant calls. The coverage metrics should be reviewed for each sample, and regions with insufficient coverage should be flagged for follow-up.

Batch Effects in Multi-Batch Studies

Technical variation between sequencing batches can obscure biological signals. The batch structure should be documented and accounted for in downstream analyses. Batch correction methods may be needed for studies that include multiple batches.

Contamination From Other Samples

Cross-sample contamination can cause false variant calls and inaccurate expression estimates. Contamination should be assessed using the methods described in the quality assessment section, and contaminated samples should be flagged or reprocessed.

Limitations of Genomic Data Processing

Reference Bias

Alignment to a reference genome introduces bias toward the reference sequence. Reads from regions that differ substantially from the reference may fail to map or map with low quality. This bias can affect variant calling and expression quantification in diverse populations.

Computational Resource Requirements

Genomic data processing requires substantial computational resources, including CPU time, memory, and storage. The resource requirements scale with the size of the dataset and the complexity of the analysis. Researchers without access to high-performance computing may need to use cloud-based solutions or reduce the scope of their analysis.

Tool Selection Complexity

The large number of available tools for each processing step can make tool selection challenging. The performance of different tools varies by dataset and analysis goal. The GRAPE pipeline was developed because no open source end-to-end solution for relatedness detection existed that was fast, reliable, and accurate for both close and distant degrees of kinship [10]. Similar gaps exist for other analysis types.

Reproducibility Challenges

Reproducing genomic data processing results requires careful documentation of all software versions, parameters, and reference databases. The rapid evolution of bioinformatics tools means that pipelines may produce different results when run with different tool versions. Workflow management systems and containerization can help address these challenges.

Safety and Regulatory Context

Data Sharing Policies

Genomic data may be subject to data sharing policies that govern how the data can be stored, processed, and shared. The NIH Genomic Data Sharing Policy describes expectations for data sharing in NIH-funded research [3]. Researchers should review the applicable policies before processing genomic data.

Data Privacy and Security

Genomic data can identify individuals and may be subject to privacy protections. The data should be stored securely and access should be restricted to authorized personnel. The EMBL-EBI Training provides resources on responsible data management [1].

Ethical Considerations

Genomic data processing may raise ethical considerations related to consent, privacy, and the return of results. Researchers should consider these issues when designing their processing workflows and data management plans.

Professional Escalation Criteria

When to Consult a Bioinformatics Specialist

Researchers should consult a bioinformatics specialist when they encounter issues that cannot be resolved with standard troubleshooting. Examples include persistent quality control failures, unexpected mapping patterns, or computational resource limitations that prevent processing.

When to Seek Additional Training

The EMBL-EBI Training provides courses and resources for researchers who need to develop their bioinformatics skills [1]. Researchers who are new to genomic data processing should consider completing formal training before processing large datasets.

When to Report Data Quality Issues

Data quality issues that may affect the validity of downstream analyses should be reported to the appropriate personnel. This includes issues such as sample contamination, library preparation failures, or sequencing instrument problems.

Frequently Asked Questions

What is the difference between raw sequencing data and analysis-ready data?

Raw sequencing data consists of FASTQ files containing nucleotide sequences and quality scores directly from the sequencing instrument. Analysis-ready data has undergone quality control, trimming, alignment, and post-alignment processing to produce files that can be used for variant calling, expression quantification, or other downstream analyses. The processing steps remove technical artifacts and format the data for analysis.

How do I choose between different aligners for my data?

The aligner choice depends on the assay type and the analysis goals. RNA-seq data requires splice-aware aligners that can map reads spanning exon junctions [6]. Whole-genome sequencing data may use general-purpose aligners optimized for speed or sensitivity. The aligner performance should be evaluated on a subset of the data before processing the full dataset.

What quality control metrics should I report for each sample?

The essential quality control metrics include per-base quality scores, adapter contamination levels, duplication rates, mapping statistics, and coverage metrics. The miND pipeline for small RNA-seq data generates a report containing essential qualitative and quantitative results [14]. The specific metrics should be selected based on the assay type and the analysis requirements.

How do I handle samples with low mapping rates?

Low mapping rates may indicate contamination, adapter contamination, or reference genome mismatches. The quality control report should be examined to identify the cause. If contamination is suspected, the sample should be checked for cross-sample contamination. If the reference genome is mismatched, the data should be aligned to the correct reference.

What is the role of duplicate marking in genomic data processing?

Duplicate marking identifies reads that originate from the same DNA fragment and flags them so that downstream tools can avoid counting them multiple times. PCR duplicates can bias variant calling and expression quantification by overrepresenting certain genomic regions. The sam2bam framework demonstrated that duplicate marking can be performed efficiently with parallel processing [13].

How do I ensure that my processing pipeline is reproducible?

Reproducibility requires documenting all software versions, parameters, and reference databases used in the processing. Workflow management systems such as Snakemake provide structured pipelines that can be rerun with consistent parameters [14]. The documentation should be stored with the processed data to support verification and troubleshooting.

What are the computational requirements for processing whole-genome sequencing data?

Whole-genome sequencing data requires substantial computational resources. The sam2bam framework used up to 711 GB of memory for whole-genome sequencing preprocessing on a single node [13]. The resource requirements depend on the coverage, the number of samples, and the processing steps included.

When should I use a graphical user interface instead of a command-line pipeline?

Graphical user interfaces can make genomic data processing accessible to researchers without command-line experience. The AgrOmicSo software provides a client-server interface that allows researchers to manage and execute complex NGS data analysis pipelines from a graphical user interface [17]. Command-line pipelines provide greater flexibility and are the standard approach for processing large datasets.

Related Bioinformatics Guides

• Predicting AMR from Genomic Data
• Data Sharing and Privacy in Genomic Research
• Alternative Splicing Analysis from RNA-Seq Data
• Spatial Transcriptomics Alignment and Cellular Neighborhood Analysis
• FASTQ File Format: Decoding Phred Quality Scores and Quality Control Workflows

References and Further Reading

• EMBL-EBI Training. European Bioinformatics Institute.
• NCBI Data Resources. National Center for Biotechnology Information.
• Genomic Data Sharing Policy. National Institutes of Health.
• The FAIR Guiding Principles. Scientific Data.
• Twelve years of SAMtools and BCFtools.. GigaScience, 2021.
• RNA-Seq Data Analysis.. Methods in molecular biology (Clifton, N.J.), 2024.
• Metagenomics Bioinformatic Pipeline.. Methods in molecular biology (Clifton, N.J.), 2022.
• Data Processing and Germline Variant Calling with the Sentieon Pipeline.. Methods in molecular biology (Clifton, N.J.), 2022.
• A data processing pipeline for the AACR project GENIE biopharma collaborative data with the {genieBPC} R package.. Bioinformatics (Oxford, England), 2023.
• GRAPE: genomic relatedness detection pipeline.. F1000Research, 2022.
• Castanet: a pipeline for rapid analysis of targeted multi-pathogen genomic data.. Bioinformatics (Oxford, England), 2024.
• Protocol for post-processing of bacterial pangenome data using Pagoo pipeline.. STAR protocols, 2021.
• Sam2bam: High-Performance Framework for NGS Data Preprocessing Tools.. 2016.
• miND (miRNA NGS Discovery pipeline): a small RNA-seq analysis pipeline and report generator for microRNA biomarker discovery studies. 2026.
• Random Survival Forest Versus Elastic-Net Regularized Cox Regression for Survival Prediction in Acute Myeloid Leukemia at Distinct Treatment Time Points: Model Performance Comparison Study.. 2026.
• Endoscopic ultrasound-guided tissue acquisition as a gateway to precision oncology in pancreatic cancer: current evidence and evolving clinical utility.. 2026.
• AgrOmicSo: A client-server interface for accessible large-scale analysis of next-generation sequencing data.. 2026.
• CellHeap: A Workflow for Optimizing COVID-19 Single-Cell RNA-Seq Data Processing in the Santos Dumont Supercomputer. Brazilian Symposium on Bioinformatics, 2021.
• Curare and GenExVis: a versatile toolkit for analyzing and visualizing RNA-Seq data. BMC Bioinformatics, 2024.
• Practical bioinformatics pipelines for single-cell RNA-seq data analysis. Biophysics Reports, 2022.
• Optimizing a whole-genome sequencing data processing pipeline for precision surveillance of health care-associated infections. Microorganisms, 2019.
• Scalable pathogen pipeline platform (SP3): Enabling unified genomic data analysis with elastic cloud computing. IEEE International Conference on Cloud Computing Cloud, 2019.
• HiCUP: Pipeline for mapping and processing Hi-C data. F1000research, 2015.
• Howdah - A flexible pipeline framework for analyzing genomic data. Proceedings 2nd IEEE International Conference on Cloud Computing Technology and Science Cloudcom 2010, 2010.
• An effective processing pipeline for harmonizing DNA methylation data from Illumina’s 450K and EPIC platforms for epidemiological studies. BMC Research Notes, 2021.

This article is educational and does not replace validated analysis plans, institutional policy, clinical interpretation, or specialist review.