# A Practical Guide to FASTQ and FASTA Formats for Long-Read Sequencing: Handling Quality Scores and Base Modifications

Long-read sequencing platforms from Pacific Biosciences (PacBio) and Oxford Nanopore Technologies (ONT) produce data in FASTQ and FASTA formats that differ substantially from short-read equivalents. The critical distinctions involve how quality scores are encoded, how base modification information is stored, and how read order can influence downstream variant calling. Researchers who treat long-read FASTQ files as if they were short-read files risk parsing errors, silent data loss, and irreproducible structural variant calls. This article explains the format specifications, practical handling strategies, quality control approaches, and common failure patterns for long-read sequencing data.

## Understanding the Core Differences Between Short-Read and Long-Read FASTQ Files

FASTQ format has four lines per read: a header line beginning with the at sign, the nucleotide sequence, a plus sign separator, and a quality score line. The format appears simple, but long-read platforms introduce complications that short-read users rarely encounter.

### Quality Score Encoding Systems

Short-read platforms from Illumina use Phred quality scores encoded as ASCII characters. The Phred score represents the probability that a base call is incorrect, calculated as negative ten times the logarithm base ten of the error probability. A Phred score of 20 means a one in one hundred chance of error, while a Phred score of 30 means a one in one thousand chance.

Long-read platforms use the same Phred scoring concept but with different encoding offsets. PacBio HiFi reads and ONT basecalled reads typically use Phred+33 encoding, where the ASCII character value minus 33 gives the Phred score. This matches the Sanger and Illumina 1.8+ encoding standard. However, older PacBio continuous long read (CLR) data used a different quality value system that did not map directly to Phred scores.

The practical consequence is that quality score distributions differ dramatically between platforms. Short-read data typically shows quality scores clustered in a narrow high range. Long-read data shows greater variability, with quality scores fluctuating along the length of each read. ONT reads often have lower median quality scores than PacBio HiFi reads, but the per-read quality varies by basecalling model and chemistry version.

### Read Length and File Size Implications

Long reads routinely exceed ten kilobases, and ultra-long ONT reads can exceed one million bases. A single FASTQ file from a long-read run can contain hundreds of gigabytes of data. The [NCBI Data Resources](https://www.ncbi.nlm.nih.gov/) documentation describes how sequence read archives store these large files and the importance of proper format validation before submission.

File size affects every downstream step. Parsing a large FASTQ file with memory-inefficient code causes crashes or excessive runtime. Compression choices matter because quality score lines compress differently than sequence lines. Tools that stream FASTQ files line by line handle long-read data more gracefully than tools that load entire files into memory.

### Base Modification Information in Long-Read Data

Base modifications such as methylation are not represented in standard FASTQ format. Both PacBio and ONT can detect modified bases during sequencing, but the information travels in companion files or in BAM format instead of in FASTQ.

PacBio stores base modification information in BAM files using the MM and ML tags defined by the SAM specification. ONT stores methylation calls in BAM files using similar tags after basecalling with modified base models. When researchers convert BAM files to FASTQ for downstream analysis, the base modification tags are typically lost unless explicitly preserved.

The [LongReadSum](https://pubmed.ncbi.nlm.nih.gov/39981293) quality control tool documentation describes how different long-read file formats carry different information layers. POD5 and FAST5 files contain raw signal data used for basecalling. Unaligned BAM files contain basecalled sequences with quality scores and potentially base modification tags. Aligned BAM files add alignment information. FASTQ files contain only sequence and quality, making them the most information-poor format in the long-read ecosystem.

## FASTA Format in Long-Read Sequencing Contexts

FASTA format contains only sequence information without quality scores. Each record has a header line starting with the greater-than symbol followed by the sequence on subsequent lines. FASTA files appear in long-read workflows primarily for reference genomes, simulated data, and assembly outputs.

### Reference Genome Storage

Reference genomes are distributed in FASTA format. The [NCBI Data Resources](https://www.ncbi.nlm.nih.gov/) provide reference genome FASTA files for download, and these files follow specific naming conventions that encode chromosome or contig identifiers. Long-read aligners require reference FASTA files with consistent sequence naming because alignment output uses these names.

### Simulated Long-Read Data

Simulation frameworks generate FASTA files as intermediate outputs before producing FASTQ files. The [soMaCX generative genome modeling framework](https://pubmed.ncbi.nlm.nih.gov/41023610) produces FASTA format output that feeds into read simulators, which then generate platform-specific FASTQ files for Illumina, PacBio, ONT, 10X Genomics, and Bionano platforms. This pipeline demonstrates that FASTA serves as the common sequence-only substrate from which simulated reads are derived.

### Assembly Output

Genome assemblies from long-read data are typically output in FASTA format. The assembly contigs represent consensus sequences without per-base quality information. Some assemblers produce companion files with quality information, but the primary assembly file remains FASTA.

## At a Glance: Long-Read Format Decision Table

| Data Format | Quality Scores | Base Modifications | Primary Use | Common Pitfall |
|-------------|---------------|-------------------|-------------|----------------|
| FASTQ | Phred+33 encoded ASCII | Not represented | Basecalled reads for alignment and analysis | Parsing errors when quality line wraps or contains unexpected characters |
| FASTA | None | Not represented | References, assemblies, simulated genomes | Losing quality information when converting from FASTQ |
| Unaligned BAM | Phred scores in QUAL field | MM and ML tags | Primary output from PacBio and ONT basecallers | Dropping modification tags during BAM to FASTQ conversion |
| Aligned BAM | Phred scores in QUAL field | MM and ML tags with alignment context | Variant calling and methylation analysis | Read order effects on downstream variant callers |
| POD5 or FAST5 | Raw signal only | Raw signal contains modification evidence | Basecalling input | Attempting to extract sequences without basecalling software |

## Practical Workflow for Handling Long-Read FASTQ Files

### Step 1: Verify Format Integrity Before Analysis

Run format validation tools on every FASTQ file before starting analysis. Validation checks should confirm that every read has four lines, that sequence lines contain only valid nucleotide characters, and that quality score lines match sequence lengths. The [Galaxy Training Network](https://training.galaxyproject.org/) provides accessible tutorials on FASTQ format validation and quality assessment that apply to long-read data.

Validation becomes critical with long-read data because file sizes make manual inspection impossible. Automated validation tools catch truncation errors, encoding mismatches, and line wrapping issues that would otherwise cause downstream failures.

### Step 2: Assess Quality Score Distributions

Generate per-read quality summaries before alignment. Long-read quality profiles differ from short-read profiles in important ways. ONT reads show quality variation along their length, with quality often declining toward read ends. PacBio HiFi reads show more uniform quality because the circular consensus sequencing process averages multiple passes.

The [LongReadSum](https://pubmed.ncbi.nlm.nih.gov/39981293) tool generates summary quality control reports for multiple long-read formats including ONT POD5, ONT FAST5, ONT basecall summary files, PacBio unaligned BAM, and Illumina Complete Long Read FASTQ. This tool addresses the gap in long-read specific quality control that generic short-read tools cannot fill.

### Step 3: Decide Whether to Preserve Base Modification Information

If the research question involves methylation or other base modifications, do not convert BAM files to FASTQ. The conversion discards modification tags. Instead, work with BAM files throughout the analysis pipeline. If the research question does not involve modifications, FASTQ conversion is acceptable, but document that modification information was intentionally discarded.

### Step 4: Control Read Order for Reproducible Variant Calling

Read order in FASTQ files affects structural variant calling results. A [2024 study in PeerJ](https://pubmed.ncbi.nlm.nih.gov/38500526) demonstrated that permuting the order of reads in FASTQ files changed the structural variants predicted by different callers. The study used PacBio data from 15 Caenorhabditis elegans strains and four Arabidopsis thaliana ecotypes and found that the pbsv caller was highly sensitive to read order, with over 70 percent of structural variant calls disagreeing between differently ordered files at the highest sequencing depths.

This finding has direct practical implications. Researchers should standardize read order across replicates and across samples within a study. Document the sorting method used. If reads are sorted by alignment position, record that fact. If reads remain in original sequencing order, record that too. The [nf-core documentation](https://nf-co.re/docs) emphasizes reproducibility standards for bioinformatics pipelines, and read order control belongs in that category.

### Step 5: Use Appropriate Tools for Long-Read Data

Generic short-read tools often fail on long-read data because they assume uniform read lengths or specific quality distributions. The [Bioconductor project](https://bioconductor.org/) hosts packages designed for long-read analysis, and the [EMBL-EBI Training](https://www.ebi.ac.uk/training) resources provide learning pathways for long-read data analysis.

The [scywalker workflow](https://pubmed.ncbi.nlm.nih.gov/39254601) demonstrates the need for purpose-built tools in long-read analysis. This package processes nanopore single-cell data from FASTQ format through demultiplexing, isoform calling, and quantification in a single command. Existing tools designed for short-read single-cell data could not handle the data sizes produced by nanopore sequencing, which motivated the development of this scalable alternative.

## Quality Score Handling in Long-Read Analysis

### Understanding Platform-Specific Quality Characteristics

PacBio HiFi reads achieve high accuracy through circular consensus sequencing. Each pass of the sequencing enzyme produces a subread, and the consensus of multiple subreads produces the HiFi read. Quality scores reflect the consensus accuracy, which typically exceeds 99 percent. The quality encoding follows standard Phred+33 conventions.

ONT reads undergo basecalling from raw signal data. The basecaller converts electrical signal measurements into nucleotide sequences with associated quality scores. Different basecalling models produce different quality distributions, and newer models generally improve accuracy. ONT quality scores are informative but should not be compared directly to PacBio quality scores because the underlying error profiles differ.

### Quality Score Trimming Decisions

Trimming decisions for long-read data differ from short-read data. Short-read workflows often trim low-quality bases from read ends because adapter contamination and quality decay are common. Long-read workflows may skip trimming entirely because the error profiles are different and trimming can remove useful sequence.

When trimming is necessary, use long-read aware trimming tools instead of short-read trimmers. The [Galaxy Training Network](https://training.galaxyproject.org/) provides guidance on quality control workflows that can be adapted for long-read data.

### Quality Score Encoding Verification

Verify the quality score encoding before analysis. Most modern long-read data uses Phred+33 encoding, but older data may use different encodings. Misinterpreting the encoding offset shifts all quality scores and can cause incorrect filtering decisions.

A simple verification method involves checking the range of ASCII characters in the quality line. Phred+33 encoding produces characters in the ASCII range from 33 (Phred score zero) to 73 or higher (Phred score 40 or above). If the quality line contains characters below ASCII 33, the encoding is not Phred+33.

## Base Modification Handling in Long-Read Data

### How PacBio Represents Base Modifications

PacBio sequencing detects base modifications through kinetic signatures. The polymerase kinetics during sequencing differ when modified bases are present, and these differences can be detected and interpreted. The primary output format for PacBio data is BAM, which can carry modification information in the MM and ML tags defined by the SAM specification.

The MM tag describes which bases are modified and what type of modification is present. The ML tag provides likelihood scores for each modified base call. These tags survive BAM to BAM operations but are lost in BAM to FASTQ conversion.

### How ONT Represents Base Modifications

ONT basecallers can detect modified bases including 5-methylcytosine and 4-methylcytosine when using appropriate basecalling models. The modification calls are stored in BAM files using the same MM and ML tag conventions. Direct RNA sequencing can also detect RNA modifications.

The [LongReadSum](https://pubmed.ncbi.nlm.nih.gov/39981293) tool documentation notes that aligned BAM files may contain base modification information, while FASTQ files cannot carry this data. Researchers studying methylation must therefore work with BAM files or extract modification information before any FASTQ conversion.

### Practical Strategies for Preserving Modification Information

Plan the analysis pipeline around BAM files when base modifications matter. Convert to FASTQ only for specific purposes such as assembly input or read mapping with tools that require FASTQ. Document every format conversion and what information was lost at each step.

For methylation analysis, use tools that accept BAM input directly. The [Bioconductor project](https://bioconductor.org/) hosts packages for methylation analysis that work with aligned BAM files containing modification tags.

## Structural Variant Calling and Read Order Sensitivity

### The Read Order Problem

The [PeerJ study on FASTQ read order](https://pubmed.ncbi.nlm.nih.gov/38500526) identified a previously unrecognized source of variability in long-read structural variant calling. The study found that the order of reads in FASTQ files affected structural variant predictions, with the pbsv caller showing particular sensitivity. At the highest sequencing depths tested, over 70 percent of structural variant calls disagreed between pairs of differently ordered FASTQ files.

The study also identified the SAMtools alignment sorting algorithm as a source of variability following read order randomization. This finding means that even when reads are aligned and sorted, the sorting process itself can introduce variability depending on the input order.

### Practical Implications for Study Design

Standardize read order across all samples in a study. If reads are sorted, use the same sorting method for every sample. If reads remain in original sequencing order, document that decision. Record the specific versions of all tools used, because different versions may handle read order differently.

The [nf-core documentation](https://nf-co.re/docs) provides guidance on reproducible pipeline configuration. Read order standardization belongs in the pipeline configuration alongside reference genome version and parameter settings.

### Structural Variant Calling Complexity

Structural variant calling from long-read data remains challenging for multiple reasons. The [soMaCX framework paper](https://pubmed.ncbi.nlm.nih.gov/41023610) describes how somatic structural variations in cancer tissue are difficult to discover because tumor heterogeneity and technical sequencing factors limit detection. Only structural variants with sufficient read support spanning the event are detectable, and complex events like chromothripsis complicate interpretation.

Simulation frameworks like soMaCX generate realistic test data by using biological distributions to control variant placement. The FASTA output from such frameworks feeds into read simulators that produce platform-specific FASTQ files. This approach allows researchers to measure detection limits and validate calling pipelines under controlled conditions.

## Quality Control Tools for Long-Read Data

### Limitations of Short-Read Quality Control Tools

Short-read quality control tools assume uniform read lengths, consistent quality distributions, and specific error patterns. Long-read data violates these assumptions. Read lengths vary from hundreds to millions of bases. Quality varies along read length. Error patterns include insertions and deletions that short-read tools do not model.

The [LongReadSum](https://pubmed.ncbi.nlm.nih.gov/39981293) publication notes the general paucity of computational tools that efficiently deliver comprehensive metrics across long-read data formats. This gap motivated the development of the tool, which handles ONT POD5, ONT FAST5, ONT basecall summary, PacBio unaligned BAM, and Illumina Complete Long Read FASTQ formats.

### What Long-Read Quality Control Should Measure

Long-read quality control should measure read length distribution, quality score distribution, read N50, and base modification content when applicable. Read N50 represents the length at which half of all bases in the dataset are in reads of that length or longer. This metric matters more than mean read length for long-read applications because the longest reads contribute disproportionately to assembly contiguity.

Quality control should also assess whether basecalling models match the data. ONT data basecalled with different models will show different quality distributions, and mixing data from different basecalling models in one analysis can introduce batch effects.

### Integrating Quality Control into Workflows

Quality control should occur at multiple points in the analysis pipeline. Initial quality control on raw FASTQ files catches sequencing problems early. Post-alignment quality control catches alignment problems. Final quality control on variant calls catches analysis problems.

The [Galaxy Training Network](https://training.galaxyproject.org/) provides workflow training that emphasizes quality control at each analysis stage. The [EMBL-EBI Training](https://www.ebi.ac.uk/training) resources similarly emphasize quality assessment as an ongoing process instead of a single step.

## Common Failure Patterns in Long-Read FASTQ Handling

### Parsing Failures from Unexpected Characters

Long-read FASTQ files occasionally contain characters that break naive parsers. Sequence lines should contain only A, C, G, T, and N characters, but some files contain IUPAC ambiguity codes or unexpected characters. Quality lines can contain any ASCII character, and parsers that assume a limited character range will fail.

The [NCBI Data Resources](https://www.ncbi.nlm.nih.gov/) documentation describes sequence data validation requirements for database submission. Following these validation standards prevents parsing failures when data is shared or submitted to public databases.

### Memory Exhaustion from Large Files

Loading entire FASTQ files into memory causes crashes with long-read data. A single ONT run can produce hundreds of gigabytes of FASTQ data. Tools must stream reads one at a time or in small batches.

The [scywalker](https://pubmed.ncbi.nlm.nih.gov/39254601) workflow was developed specifically because existing nanopore single-cell analysis tools showed severe limitations in handling current data sizes. The package processes data in a streaming fashion suitable for server or cluster execution.

### Silent Data Loss During Format Conversion

Converting BAM to FASTQ loses base modification information. Converting FASTQ to FASTA loses quality scores. These losses are silent because the conversion succeeds without error messages. Researchers must know what information is lost at each conversion step.

The [LongReadSum](https://pubmed.ncbi.nlm.nih.gov/39981293) documentation explicitly describes what information each long-read format carries. POD5 files contain raw signal. BAM files contain sequences with quality and potentially modifications. FASTQ files contain only sequence and quality. FASTA files contain only sequence.

### Read Splitting Artifacts

The [BulkVis publication](https://pubmed.ncbi.nlm.nih.gov/30462145) describes a specific failure mode where long reads are incorrectly divided by MinKNOW software, resulting in single DNA molecules being split into two or more reads. The longest example observed was 2,272,580 bases reported in eleven consecutive reads. This splitting appears to vary by sample type and is more common in ultra-long read preparations.

BulkVis provides helper scripts that identify and reconstruct split reads using a sequencing summary file and alignment to a reference. Researchers working with ultra-long read data should check for this artifact pattern.

## Records and Documentation for Long-Read Sequencing Projects

### What to Record for Each Dataset

Maintain records of sequencing platform, chemistry version, basecalling model, basecalling software version, and date of basecalling. These factors affect quality score distributions and error profiles. Record whether base modification calling was performed and which modification types were detected.

Record all format conversions including the tools used and the information lost at each step. Record read order handling including whether reads were sorted and by what method. The [nf-core documentation](https://nf-co.re/docs) provides guidance on pipeline parameter documentation that applies to these records.

### Reproducibility Documentation

The read order sensitivity findings from the [PeerJ study](https://pubmed.ncbi.nlm.nih.gov/38500526) demonstrate that reproducibility requires more than sharing code and data. The order of reads in input files must be documented or controlled. The specific versions of alignment sorting tools must be recorded because the sorting algorithm itself was identified as a source of variability.

The [The Carpentries lessons](https://carpentries.org/lessons) teach reproducible computing practices including version control and documentation. These practices apply directly to long-read sequencing analysis where subtle differences in input handling produce different results.

## Limitations of Long-Read FASTQ Analysis

### Quality Score Interpretation Limits

Quality scores from different platforms and basecalling models are not directly comparable. A Phred score of 20 from PacBio HiFi data does not mean the same thing as a Phred score of 20 from ONT data. The error profiles differ, with PacBio HiFi errors being primarily substitutions and ONT errors including insertions and deletions.

Filtering thresholds based on quality scores should be validated for each platform and basecalling model. A threshold that works for one dataset may not work for another.

### Base Modification Detection Limits

Base modification detection from long-read data has sensitivity limits. Modifications present at low frequency may not be detected. The [soMaCX paper](https://pubmed.ncbi.nlm.nih.gov/41023610) describes how detection requires sufficient read support, and this principle applies to modification calling as well as structural variant calling.

Modification calling accuracy depends on coverage depth and modification frequency. Researchers should validate modification calls with orthogonal methods when possible.

### Structural Variant Detection Limits

Structural variant detection from long-read data has documented limitations. The [soMaCX paper](https://pubmed.ncbi.nlm.nih.gov/41023610) notes that only structural variants with a sufficient fraction of reads spanning the event will be detectable. Tumor heterogeneity and complex events like chromothripsis further complicate detection.

Simulation frameworks provide a way to measure detection limits for specific variant types and coverage levels. Researchers should use simulations to understand what their pipeline can and cannot detect.

## Professional Escalation Criteria

### When to Seek Expert Assistance

Seek expert assistance when quality control metrics indicate systematic problems instead of random variation. Sudden drops in quality scores across many reads may indicate instrument problems. Unexpected base modification patterns may indicate sample preparation issues. Consistent read splitting may indicate software configuration problems.

The [BulkVis publication](https://pubmed.ncbi.nlm.nih.gov/30462145) provides an example of how expert analysis identified a systematic read splitting problem. The tool developers recognized the pattern and developed reconstruction scripts. Researchers encountering similar patterns should consult with sequencing facility staff or bioinformatics support.

### When to Reconsider Data Inclusion

Consider excluding data from analysis when quality metrics fall below thresholds validated for the specific analysis type. Document exclusion criteria before analysis to avoid bias. The [Galaxy Training Network](https://training.galaxyproject.org/) provides guidance on establishing quality thresholds for sequencing data.

### When to Consult Platform Documentation

Consult platform-specific documentation when encountering unexpected format behavior. PacBio and ONT documentation describes format specifications and known issues. The [NCBI Data Resources](https://www.ncbi.nlm.nih.gov/) also provide format documentation for sequence data submission.

## Establishing a Read Order and Format Conversion Record System for Long-Read Projects

The read order sensitivity findings from the [PeerJ study on FASTQ read order](https://pubmed.ncbi.nlm.nih.gov/38500526) demonstrate that undocumented file handling decisions can invalidate otherwise sound analyses. That study showed that permuting read order changed structural variant predictions, with the pbsv caller showing disagreement in over 70 percent of calls at high sequencing depths. The study also identified the SAMtools alignment sorting algorithm as an independent source of variability. These findings mean that read order is not a cosmetic detail but a measurable experimental variable that must be tracked with the same rigor as sequencing chemistry or reference genome version.

### Why Standard Record Keeping Fails for Long-Read Data

Most laboratory record systems track sample metadata, sequencing dates, and instrument settings. These records capture what happened at the sequencing machine but not what happened to the files afterward. For short-read data, this gap rarely matters because read order has minimal effect on downstream analysis. For long-read data, the gap is consequential. The [PeerJ study](https://pubmed.ncbi.nlm.nih.gov/38500526) demonstrated that the same FASTQ file processed through the same pipeline can produce different structural variant calls depending on read order. Without a record of how reads were ordered, sorted, or shuffled, another researcher cannot reproduce the analysis even with identical input files and software versions.

The problem is compounded by the fact that many common operations silently change read order. Downloading files from a repository may reorder reads. Concatenating multiple FASTQ files changes the order of reads from each file. Running a quality trimming step may sort reads by length or quality. Even the act of copying files between storage systems can alter order if the copy operation uses parallel processes. A record system must capture beyond the final read order but every operation that could have changed it.

### Core Components of a Read Order and Conversion Log

A practical record system for long-read projects should track five categories of information for every FASTQ or FASTA file in the analysis pipeline.

**File provenance records** capture where each file came from and how it was generated. For basecalled data, this includes the basecaller name and version, the basecalling model, and the date of basecalling. For converted files, this includes the source format and the conversion tool. The [LongReadSum publication](https://pubmed.ncbi.nlm.nih.gov/39981293) notes that different long-read formats carry different information layers, and the provenance record should document what information was present in the source format and what was lost in conversion.

**Read order records** document the ordering principle for each file. Common ordering principles include original sequencing order, sorted by genomic position, sorted by read length, sorted by quality score, or randomized. The record should include the specific command or tool used to establish the order and the tool version. The [nf-core documentation](https://nf-co.re/docs) emphasizes that pipeline parameters must be recorded for reproducibility, and read order handling belongs in this category.

**Conversion event logs** track every format transformation. Each log entry should include the source file name, destination file name, conversion tool and version, date, and a note about what information was lost. For example, a BAM to FASTQ conversion loses base modification tags. A FASTQ to FASTA conversion loses quality scores. These losses are silent because the conversion succeeds without error, so the log must document them explicitly.

**Sorting and shuffling records** capture any operation that changes read order. This includes alignment sorting with tools like SAMtools, random shuffling for downsampling, and concatenation of multiple files. Each record should include the tool, version, parameters, and the ordering principle applied.

**Validation checkpoints** document that file integrity was verified at each stage. This includes checksum verification, format validation, and read count confirmation. The [Galaxy Training Network](https://training.galaxyproject.org/) provides tutorials on format validation that can be adapted for long-read data.

### Implementing the Record System in Practice

The record system should be implemented as a structured log file that travels with the data. A tab-separated text file or a simple spreadsheet works well for most projects. Each row represents one file or one operation, and columns capture the relevant metadata. The log should be stored alongside the data files and version controlled using Git or a similar system. The [The Carpentries lessons](https://carpentries.org/lessons) teach version control practices that apply directly to this use case.

For each FASTQ file in the analysis, the log should record the following fields:

| Field | Example Value | Purpose |
|-------|---------------|---------|
| File name | sample_01.fastq.gz | Identifies the file |
| Source | ONT PromethION run 2024-11-15 | Tracks origin |
| Basecaller | Dorado v0.7.0 | Records basecalling software |
| Basecalling model | sup v5.0.0 | Records model version |
| Read order | Original sequencing order | Documents ordering principle |
| Sorting tool | None | Records if sorting was applied |
| Conversion history | POD5 to BAM to FASTQ | Tracks format changes |
| Information lost | Base modification tags | Documents silent data loss |
| Checksum | SHA256 value | Verifies file integrity |
| Validation date | 2024-11-20 | Records when validation occurred |

The [EMBL-EBI Training](https://www.ebi.ac.uk/training) resources emphasize that data management plans should be established before analysis begins. The read order log should be part of the data management plan for any long-read project, and the log format should be established before the first file is processed.

### Standardizing Read Order Across Samples

The [PeerJ study](https://pubmed.ncbi.nlm.nih.gov/38500526) found that read order sensitivity varied by aligner, structural variant caller, and sequencing depth. This means that the magnitude of the problem depends on the specific analysis pipeline. A pipeline that is insensitive to read order for one caller may be sensitive for another. The only safe approach is to standardize read order across all samples in a study.

The standardization decision should be made before analysis begins and documented in the study protocol. Two common approaches exist. The first approach keeps reads in original sequencing order for all samples. This approach preserves the natural order produced by the sequencing instrument and requires no additional processing. The second approach sorts reads by genomic position after alignment. This approach requires an alignment step and a sorting step, and the sorting tool version must be recorded because the [PeerJ study](https://pubmed.ncbi.nlm.nih.gov/38500526) identified the SAMtools sorting algorithm as a source of variability.

Neither approach is inherently better. The critical requirement is consistency. Mixing samples with different read orders within a single study introduces a confound that cannot be corrected downstream. If one sample has reads in original sequencing order and another has reads sorted by position, any difference in structural variant calls between the samples could be caused by read order instead of biology.

### Handling Multi-File Datasets

Long-read runs often produce multiple FASTQ files. A single ONT run may produce one file per barcode or per flow cell channel. A PacBio run may produce separate files for different wells or movie files. When these files are combined for analysis, the concatenation order becomes part of the read order record.

The concatenation order should be deterministic and documented. Alphabetical sorting of file names is a common approach, but the sort order depends on the file naming convention. A file named sample_10.fastq sorts before sample_2.fastq in alphabetical order but after it in numerical order. The record should specify the exact sorting method used for concatenation.

The [scywalker workflow](https://pubmed.ncbi.nlm.nih.gov/39254601) demonstrates how a purpose-built pipeline handles multi-file long-read data. The workflow processes sequenced fragments in FASTQ format through demultiplexing and quantification in a single command. The pipeline documentation specifies how input files are handled, and this specification serves as part of the read order record.

### Recording Conversion Tool Versions

Format conversion tools change over time, and different versions may handle read order differently. The [Bioconductor project](https://bioconductor.org/) hosts packages for sequence data manipulation, and package versions should be recorded for every conversion step. The [nf-core documentation](https://nf-co.re/docs) emphasizes that pipeline reproducibility requires recording software versions, and this principle applies to conversion tools as well as analysis tools.

The version record should include the tool name, the version number, and the date the tool was used. If a tool was updated between processing batches, the record should reflect which version processed which files. This level of detail allows another researcher to reproduce the exact conversion steps even if the tool has since been updated.

### Common Failure Patterns in Read Order Documentation

Several failure patterns recur in long-read projects that lack a structured record system.

**Missing provenance information** occurs when files are shared between collaborators without documentation of their origin. A FASTQ file that arrives without basecaller information cannot be properly interpreted because quality score distributions depend on the basecalling model. The [LongReadSum publication](https://pubmed.ncbi.nlm.nih.gov/39981293) notes that different basecalling models produce different quality distributions, and mixing data from different models introduces batch effects.

**Undocumented concatenation** occurs when multiple FASTQ files are combined without recording the concatenation order. This failure is common because concatenation seems like a trivial operation. The [PeerJ study](https://pubmed.ncbi.nlm.nih.gov/38500526) demonstrated that read order affects structural variant calling, so undocumented concatenation can introduce irreproducible results.

**Silent information loss** occurs when files are converted between formats without documenting what was lost. A BAM to FASTQ conversion loses base modification tags. A FASTQ to FASTA conversion loses quality scores. These losses are silent because the conversion succeeds without error messages. The [LongReadSum documentation](https://pubmed.ncbi.nlm.nih.gov/39981293) explicitly describes what information each long-read format carries, and this information should be referenced when documenting conversions.

**Sorting tool version drift** occurs when different versions of sorting tools are used for different samples. The [PeerJ study](https://pubmed.ncbi.nlm.nih.gov/38500526) identified the SAMtools alignment sorting algorithm as a source of variability following read order randomization. If one sample is sorted with SAMtools version 1.17 and another with version 1.20, the sorting algorithms may handle ties differently, introducing a subtle difference in read order.

### Validation Checkpoints for Read Order Integrity

Validation checkpoints should be built into the analysis pipeline to confirm that read order has not been unintentionally changed. The [Galaxy Training Network](https://training.galaxyproject.org/) provides workflow training that emphasizes quality control at each analysis stage, and read order validation belongs in this category.

A simple validation approach involves recording the first and last read identifiers for each file at each pipeline stage. If the first read identifier changes between stages, the read order was altered. This check catches unintentional reordering from file copying, parallel processing, or tool behavior.

A more robust approach involves computing a hash of the read order. This can be done by extracting read identifiers in order and computing a checksum. The checksum should be recorded at each pipeline stage and compared to detect any reordering. This approach is more sensitive than checking only the first and last reads because it detects any change in order.

### Integrating the Record System with Existing Laboratory Practices

The read order record system should integrate with existing laboratory information management systems instead of replace them. Most laboratories already track sample metadata and sequencing information. The read order log adds file-level tracking that complements existing sample-level tracking.

The [NCBI Data Resources](https://www.ncbi.nlm.nih.gov/) provide guidance on data submission standards that include file-level metadata. When submitting long-read data to public repositories, the read order information should be included in the submission metadata. This allows other researchers to understand the read order context of the data.

The [EMBL-EBI Training](https://www.ebi.ac.uk/training) resources emphasize that data management is an ongoing process instead of a single step. The read order log should be updated throughout the analysis pipeline, not created once at the beginning. Each time a file is converted, sorted, or concatenated, the log should be updated to reflect the new state.

### Professional Escalation Criteria for Read Order Problems

Researchers should escalate read order concerns to bioinformatics support or sequencing facility staff when specific conditions are met. If a pipeline produces different results when run twice on the same input file, this indicates a nondeterministic process that requires expert investigation. If structural variant calls differ between samples that should be comparable, read order should be investigated as a potential cause before biological interpretation is attempted.

The [BulkVis publication](https://pubmed.ncbi.nlm.nih.gov/30462145) provides an example of how expert analysis identified a systematic problem in long-read data processing. The tool developers recognized that reads were being incorrectly divided by MinKNOW software and developed reconstruction scripts. Researchers encountering unexpected read order effects should similarly consult with experts who understand the specific platform and software being used.

The [nf-core documentation](https://nf-co.re/docs) provides guidance on reporting pipeline issues and seeking community support. When read order problems are suspected, the documentation should include the read order log so that experts can diagnose whether the problem stems from file handling or from the analysis tools themselves.

### Records That Support Publication and Data Sharing

Journals increasingly require data availability statements that describe how data can be accessed and reproduced. The read order log should be included in the supplementary materials or referenced in the data availability statement. This allows reviewers and readers to understand the file handling decisions that were made during the analysis.

The [PeerJ study](https://pubmed.ncbi.nlm.nih.gov/38500526) concluded that their findings have implications for the replication of structural variant studies and the development of consistent calling protocols. The read order log is a practical tool for implementing these recommendations. Without such a log, replication attempts cannot verify that the same read order was used.

The [The Carpentries lessons](https://carpentries.org/lessons) teach that reproducible research requires documenting beyond the analysis code but also the data handling steps. The read order log is a data handling document that should be treated with the same rigor as analysis code. It should be version controlled, reviewed, and shared alongside the analysis pipeline.

### Practical Implementation Timeline

The read order record system should be implemented before the first file is processed. The log format should be established, the fields should be defined, and the validation checkpoints should be built into the pipeline. Implementing the system after analysis has begun creates gaps in the record that cannot be filled retroactively.

The initial implementation should include a test run with a small dataset to verify that the log captures all necessary information. The test run should include at least one format conversion, one sorting operation, and one concatenation to ensure that all event types are recorded correctly. The [Galaxy Training Network](https://training.galaxyproject.org/) provides test datasets and workflows that can be used for this purpose.

After the test run, the system should be applied to the full dataset. The log should be reviewed at each pipeline stage to confirm that it is being updated correctly. Any gaps in the log should be addressed immediately instead of deferred to the end of the project.

### Limitations of the Record System

The read order record system documents what was done to files but cannot recover information that was already lost. If base modification tags were discarded during a BAM to FASTQ conversion, the log can document that the loss occurred, but the tags cannot be recovered. The log should therefore be established before any format conversions are performed.

The record system also cannot detect read order changes that occur within a single file without external validation. If a tool silently reorders reads within a file, the log will not detect this unless validation checkpoints are in place. The validation checkpoints described above are therefore an essential component of the system instead of an optional addition.

The [LongReadSum publication](https://pubmed.ncbi.nlm.nih.gov/39981293) notes that there is currently no single quality control tool capable of summarizing all features of all long-read formats. Similarly, there is no single tool that automates the read order record system. The system requires manual implementation and discipline to maintain. Laboratories that process large numbers of long-read samples may benefit from developing automated scripts that generate log entries from file metadata, but the core record keeping remains a human responsibility.

## Frequently Asked Questions

### Why does my long-read FASTQ file fail to parse with standard tools?

Long-read FASTQ files often exceed the size limits that short-read tools assume. Some tools load entire files into memory and crash on multi-gigabyte inputs. Other tools assume uniform read lengths and fail when reads vary from hundreds to millions of bases. Use tools designed for long-read data or stream the file line by line instead of loading it entirely.

### How do I know which quality score encoding my long-read data uses?

Check the ASCII character range in the quality score lines. Phred+33 encoding produces characters from ASCII 33 upward. If the quality line contains characters below ASCII 33, the encoding is different. Most modern PacBio and ONT data uses Phred+33 encoding, but verify this before applying quality filters.

### Can I convert my BAM file to FASTQ without losing base modification information?

No. FASTQ format cannot represent base modification information. The MM and ML tags in BAM files carry modification calls and likelihood scores, and these tags are discarded during BAM to FASTQ conversion. If you need modification information, work with BAM files throughout your analysis.

### Why do my structural variant calls change when I reorder reads in my FASTQ file?

Read order affects structural variant calling in ways that are not fully understood. A [2024 study](https://pubmed.ncbi.nlm.nih.gov/38500526) found that the pbsv caller was highly sensitive to read order, with over 70 percent of calls disagreeing between differently ordered files at high sequencing depths. The SAMtools alignment sorting algorithm was also identified as a source of variability. Standardize read order across all samples and document your sorting method.

### What is read N50 and why does it matter for long-read data?

Read N50 is the length at which half of all bases in the dataset are in reads of that length or longer. This metric matters more than mean read length for long-read applications because the longest reads contribute disproportionately to assembly contiguity and structural variant detection. Report read N50 alongside mean read length in quality control summaries.

### How do I check whether my ONT reads were incorrectly split?

The [BulkVis tool](https://pubmed.ncbi.nlm.nih.gov/30462145) can identify reads that were incorrectly divided by MinKNOW software. The tool provides helper scripts that reconstruct split reads using a sequencing summary file and alignment to a reference. Incorrect read splitting appears more common in ultra-long read preparations and varies by sample type.

### What information is contained in POD5 and FAST5 files that is not in FASTQ files?

POD5 and FAST5 files contain raw signal data from nanopore sequencing. This signal data is used for basecalling and cannot be recovered from FASTQ files. The [LongReadSum](https://pubmed.ncbi.nlm.nih.gov/39981293) tool documentation describes how POD5 files contain raw signal information used for base calling, while FASTQ files contain only the basecalled sequence and quality scores.

### How should I document read order for reproducible long-read analysis?

Record whether reads were sorted and by what method. Record the specific version of the sorting tool. Record whether reads remain in original sequencing order. The [nf-core documentation](https://nf-co.re/docs) provides guidance on pipeline parameter documentation, and read order handling should be documented as a pipeline parameter alongside reference genome version and caller settings.

## Related Bioinformatics Guides

- [Short-Read vs Long-Read Sequencing: Pros, Cons, and Selection Criteria](/knowledge/bioinformatics/short-read-vs-long-read-sequencing-pros-cons-and-selection-criteria)
- [Evaluating Metagenomic Assembly Tools: A Benchmarking Framework for Short-Read and Long-Read Data](/knowledge/bioinformatics/evaluating-metagenomic-assembly-tools-a-benchmarking-framework-for-short-read-and-long-read-data)
- [Detecting Structural Variants with Long-Read Sequencing: Methods and Considerations](/knowledge/bioinformatics/detecting-structural-variants-with-long-read-sequencing-methods-and-considerations)
- [Long-Read Sequencing Cost and Market: What to Expect](/knowledge/bioinformatics/long-read-sequencing-cost-and-market-what-to-expect)
- [Long-Read Sequencing for Isoform Quantification: Challenges and Solutions](/knowledge/bioinformatics/long-read-sequencing-for-isoform-quantification-challenges-and-solutions)

## References and Further Reading

- [NCBI Data Resources](https://www.ncbi.nlm.nih.gov/). National Center for Biotechnology Information.
- [EMBL-EBI Training](https://www.ebi.ac.uk/training). European Bioinformatics Institute.
- [Bioconductor](https://bioconductor.org/). Bioconductor Project.
- [Galaxy Training Network](https://training.galaxyproject.org/). Galaxy Project.
- [nf-core Documentation](https://nf-co.re/docs). nf-core.
- [The Carpentries Lessons](https://carpentries.org/lessons). The Carpentries.
- [The impact of FASTQ and alignment read order on structural variant calling from long-read sequencing data.](https://pubmed.ncbi.nlm.nih.gov/38500526). PeerJ, 2024.
- [Scywalker: scalable end-to-end data analysis workflow for long-read single-cell transcriptome sequencing.](https://pubmed.ncbi.nlm.nih.gov/39254601). Bioinformatics (Oxford, England), 2024.
- [LongReadSum: A fast and flexible quality control and signal summarization tool for long-read sequencing data.](https://pubmed.ncbi.nlm.nih.gov/39981293). Computational and structural biotechnology journal, 2025.
- [SoMaCX: a complex generative genome modeling framework.](https://pubmed.ncbi.nlm.nih.gov/41023610). BMC genomics, 2025.
- [BulkVis: a graphical viewer for Oxford nanopore bulk FAST5 files.](https://pubmed.ncbi.nlm.nih.gov/30462145). Bioinformatics (Oxford, England), 2019.

> This article is educational and does not replace validated analysis plans, institutional policy, clinical interpretation, or specialist review.