How to Validate Your FASTA, FASTQ, BAM, and VCF Files

By Dr. Zubair Khalid, DVM, MS, PhD · 2026-11-10

The single most common reason a bioinformatics file fails validation is that it is not the format its filename claims to be. A file named sample.bam is often a plain gzip stream, a truncated transfer, or a SAM text file someone renamed. The first thing to check is not the biology. It is the container. Run file on it, run htsfile on it, run gzip -t on it, and run samtools quickcheck on it. Those four commands take under five seconds and resolve the majority of "corrupt file" tickets before anyone opens a genome browser.

This matters more than most people assume. Genomics file formats frequently lack formal specifications, which makes it hard for tool authors to test how their software handles unusual input, and that gap produces silent errors in results rather than loud crashes [1]. A verification study that tested input validation across 80 Bioconda packages parsing the Browser Extensible Data format found that 75 of the 80 achieved less than 70 percent correctness on an edge-case test suite [1]. The lesson is blunt. Your tools are not validating for you, and a pipeline that "ran to completion" is not evidence that the input was well formed.

This guide covers the four formats that carry nearly all sequence-analysis work: FASTA, FASTQ, BAM (with CRAM), and VCF. For each one it explains what the format promises, what actually breaks, the exact command sequence to confirm the failure, and the fix. It also covers where validation belongs in a production workflow, which is at the boundaries, before compute is spent and before results are trusted [2].

The first five minutes: a triage sequence

Work through these steps in order. Stop at the first one that fails.

1. Identify the real container. file sample.bam and htsfile sample.bam. htsfile understands BGZF, BAM, CRAM, VCF, and tabix indexes. A BAM that reports as ordinary gzip is not a BAM that any indexing tool can random-access.
2. Test archive integrity. gzip -t sample.fastq.gz returns silently on success and errors on a bad CRC. For BGZF files, bgzip -t sample.vcf.gz is the correct test.
3. Test the header and terminator. samtools quickcheck -v sample.bam. This checks for a readable header, at least one target sequence, and the BGZF end-of-file marker. A truncated BAM almost always fails here.
4. Test the structure. samtools view -c sample.bam forces the parser through every record. A file that passes quickcheck but dies mid-parse has internal corruption, not a truncation.
5. Test the semantics. Does the declared reference build match the data? Does the read group exist? Does REF in the VCF match the FASTA? These are the failures that survive structural checks and quietly produce wrong answers.

Record the output of each step. A validation report that consists of "it worked on my laptop" is not reproducible, and reproducibility of the analysis environment is itself a documented source of error in bioinformatics work [3].

Three layers of file validation

Validation is not one check. It is three, and each layer catches a different class of defect.

Structural validation asks whether the file parses. Line counts are divisible by four, headers are present, the BGZF terminator exists, the tab counts match the column count. This layer catches truncation, transfer corruption, and malformed writing.

Semantic validation asks whether the content is internally consistent. Every read in a BAM has a CIGAR whose length matches the sequence. Every REF allele in a VCF matches the reference base at that position. Every ##contig length matches the .fai index. This layer catches build mismatches, normalization differences, and hand-edited files.

Contextual validation asks whether the file matches the experiment. Read one and read two are in sync. Sample names in the VCF match the sample sheet. The reference FASTA is the one the BAM header declares. The annotation GFF uses the same contig naming convention as the genome. This layer catches the errors that produce plausible but wrong biology.

Tools exist for each layer. PipeVal is a light-weight, extensible validator designed to be inserted into existing pipelines to automatically verify inputs and outputs, reducing wasted compute on corrupt files or invalid paths [2]. AssemblyQC embeds file-format validation alongside contaminant screening and completeness metrics so that assembly quality reporting starts from a verified file [4]. Neither replaces a human check of the third layer, which depends on knowing what the experiment was supposed to be.

Bioinformatics file formats: the contract each file makes

A file format is a contract between the writer and the reader. When the contract holds, tools compose. When it breaks, the failure can be loud or silent, and silent is far more expensive [1].

FASTA: sequence without quality

FASTA stores biological sequence as records. Each record begins with a header line starting with > and continues with sequence lines until the next header. There is no quality information, no per-base confidence, and no defined line width. That last point is the source of most FASTA pain.

FASTA is used for reference genomes, transcript sets, protein sequences, assemblies, and any derived sequence. Because it carries no quality data, it is also the format into which people convert FASTQ when they want to strip quality, which is exactly the conversion that loses the information you need for a re-analysis.

The critical, non-obvious rule is that indexed FASTA requires uniform line length. Every sequence line in a record must be the same length except the last one. Tools rely on this to compute byte offsets instead of reading the whole file. A FASTA that wraps at 70 characters for one contig and 60 for the next is legal as text and unusable as an index.

FASTQ: sequence with per-base quality

FASTQ stores one record per sequencing read in four lines: an identifier line beginning with @, the sequence, a separator line beginning with +, and a quality string of exactly the same length as the sequence. Each quality character encodes a Phred score, which is the estimated probability that the corresponding base call is wrong, expressed as a log scale.

FASTQ is the raw output of Illumina, PacBio, and Oxford Nanopore instruments after base calling. It is the input to aligners and assemblers. It is large, it is usually gzipped, and it is the format most often damaged in transit because the files are big enough that transfer failures are common.

fastq vs fasta: the swap that breaks pipelines

The distinction readers ask about most often is fastq vs fasta, and the practical answer is simple. FASTA has sequence only. FASTQ has sequence plus per-base quality. Everything else follows.

A FASTA file fed to a read aligner fails, because the aligner needs quality scores to run its mismatch model. A FASTQ file fed to a BLAST database builder is accepted by some wrappers and then produces nonsense, because the quality characters get interpreted as amino acids or nucleotides. A single-end FASTQ used where paired-end was expected produces a run that completes and gives lower-quality calls with no error message.

The rule to internalize: if the downstream step does variant calling, adapter trimming, or quality filtering, it needs FASTQ. If the downstream step is reference indexing, annotation, or database construction, it needs FASTA. Convert only in the direction that discards information, and keep the original.

SAM, BAM, and CRAM

SAM is the human-readable text alignment format. BAM is its binary equivalent, compressed with BGZF, which is gzip-compatible but block-structured so that readers can seek. CRAM is a reference-based compressed format that stores reads as differences against a reference genome.

BAM is the default alignment format in most pipelines. Its contract includes a header with @HD, @SQ, and @RG lines, records sorted by coordinate for indexed access, an optional companion .bai or .csi index, and BGZF block structure throughout. Break any of those and index-based tools fail.

BAM files also carry artifacts forward from earlier steps, which is why variant refinement approaches routinely re-interrogate BAM files to remove sequencing and alignment artifacts before downstream interpretation [5]. A structurally valid BAM can still be biologically noisy, and validation of the container does not validate the content.

VCF

VCF (Variant Call Format) stores variant calls. It has a header of ## meta-information lines, a mandatory column header line beginning with #CHROM, and then one line per variant. The fixed columns are CHROM, POS, ID, REF, ALT, QUAL, FILTER, and INFO. FORMAT and per-sample columns follow when genotypes are present.

VCF is the interchange format for clinical and research genomics. Converting VCF into other standards requires knowing the genome build explicitly, and production utilities take GRCh37 or GRCh38 as a required input parameter precisely because guessing is unsafe [6]. Clinical consensus recommendations go further and call for standardized file formats and strict version control across the production pipeline, with hg38 as the adopted reference build [7].

Validating FASTA files

FASTA failures cluster into four causes: wrong alphabet, inconsistent line width, duplicate or malformed identifiers, and compression that is not what the extension says.

Step 1: confirm it is FASTA and not something else

head -c 200 genome.fa
file genome.fa

The first character must be >. If it is @, you have a FASTQ. If it is 1 or a number, you have a GFF or BED. If the first bytes are non-printable, run file and check for gzip.

Step 2: check the alphabet

grep -v '^>' genome.fa | tr -d '\n\r' | fold -w1 | LC_ALL=C sort -u | tr -d '\n'
echo

A nucleotide FASTA should return a subset of ACGTN plus IUPAC ambiguity codes (R Y S W K M B D H V). A protein FASTA returns the 20 amino acids plus X, B, Z, U, O, and sometimes *. If the output contains digits, punctuation, or whitespace characters, the file is malformed or is not sequence at all. A lowercase-only output is fine, most tools are case-insensitive, but keep capitalization consistent if your downstream tool requires soft-masking.

Step 3: catch empty records, whitespace, and duplicate identifiers

awk '/^>/{if(NR>1 && n==0) print "empty record ends at line " NR; n=0; next} {if($0 ~ /[ \t]/) print "whitespace at line " NR; n+=length($0)}' genome.fa
grep '^>' genome.fa | cut -d' ' -f1 | sort | uniq -d

Duplicate identifiers break samtools faidx lookups silently, because the index keeps one of them. Whitespace inside sequence lines breaks strict parsers. Empty records pass human inspection and fail database construction.

Step 4: build the index and dictionary

samtools faidx genome.fa
samtools dict -o genome.dict genome.fa

If the FASTA has inconsistent line lengths, samtools faidx stops with [E::fai_build_core] Different line length in sequence 'chr1'. The fix is to rewrap the file uniformly:

seqkit seq -w 60 genome.fa > genome.rewrapped.fa

Use the same width for every contig. Sixty is conventional and works everywhere.

Step 5: check size and record counts

seqkit stats -a genome.fa

This reports record count, total length, min, max, and average sequence length, and GC content. Cross-check the total against the expected genome size. A human GRCh38 primary assembly is roughly 3.1 Gb across the primary chromosomes. A file that reports 200 Mb is truncated or is a subset.

Safety note

samtools faidx genome.fa writes genome.fa.fai next to the input and will overwrite an existing index without prompting. Never run indexing commands inside a directory holding the only copy of a reference you cannot re-download.

Validating FASTQ files

FASTQ failures cluster into five causes: line-count misalignment from truncation, sequence and quality length disagreement, wrong Phred encoding, corrupted gzip, and broken pairing.

Step 1: test the archive before you parse it

gzip -t reads_R1.fastq.gz && echo OK

A gzip CRC failure means the file is incomplete or was concatenated incorrectly. Re-download. Do not attempt to parse past it.

Step 2: check the four-line structure and length agreement

zcat reads_R1.fastq.gz | awk 'NR%4==1 && $0!~/^@/{print "bad header line " NR} NR%4==2{s=length($0)} NR%4==3 && $0!~/^\+/{print "bad plus line " NR} NR%4==0 && length($0)!=s{print "length mismatch line " NR}' | head

Any output is a defect. A length mismatch between sequence and quality is the classic signature of a write that was interrupted mid-record, and no trimmer will recover it cleanly.

Step 3: count records correctly

zcat reads_R1.fastq.gz | awk 'END{print NR/4}'

Do not use grep -c '^@'. In Phred+33 encoding the @ character represents quality score 31, which is a common value in high-quality reads, so the quality line of a record often begins with @ and gets counted as a header. The same trap applies to grep -c '^+', since + is quality score 10 and appears routinely in low-quality tails.

If the count from awk is fractional, the file has a partial record at the end.

Step 4: determine the quality encoding

zcat reads_R1.fastq.gz | awk 'NR%4==0' | tr -d '\n' | od -An -tu1 -v | tr ' ' '\n' | grep -v '^$' | sort -n | head -1

Valid Phred+33 quality characters fall in the byte range 33 (!) to 126 (~). If the minimum byte value is below 33, the file is corrupt or the encoding is wrong. If the minimum is below 64 (@), the file cannot be Phred+64 and is almost certainly Phred+33, which is the encoding used by Illumina 1.5 and later and by essentially all current instruments. If the minimum is 64 or above, the data are ambiguous between encodings and you should let FastQC's encoding inference decide. Mixing the two encodings across samples in one batch inflates some quality estimates by 31 points and deflates others, which distorts every downstream filter.

fastqc --outdir qc reads_R1.fastq.gz

FastQC reports the inferred encoding in its basic statistics module and flags per-base quality that would be implausible under any single encoding.

Step 5: verify pairing

seqkit stats -T reads_R1.fastq.gz reads_R2.fastq.gz
paste <(seqkit seq -n -i reads_R1.fastq.gz) <(seqkit seq -n -i reads_R2.fastq.gz) | awk '$1!=$2{print; n++} END{print n+0, "mismatched identifiers"}'

Read counts must match, and read identifiers must correspond in order. If identifiers disagree but counts match, the files were reordered independently, which is common after separate trimming runs. Re-derive both mates from the same source rather than attempting to repair by matching.

seqkit pair -1 reads_R1.fastq.gz -2 reads_R2.fastq.gz -O paired/

This writes only reads present as a matched pair, and it reports how many were dropped.

Step 6: confirm the content is what the label says

seqkit stats -a reads_R1.fastq.gz

Check the read length distribution against the instrument run. A 150 bp paired-end run that reports an average read length of 75 bp after quality trimming is normal. A 150 bp run that reports 36 bp on an untrimmed file means you were handed the wrong file.

Validating BAM and CRAM files

BAM failures cluster into six causes: truncation from an interrupted write or transfer, plain gzip instead of BGZF, unsorted records, a stale index, a header that does not match the reference, and records whose CIGAR disagrees with their sequence.

Step 1: quickcheck first

samtools quickcheck -v sample.bam

Silence means pass. Output names the specific failure. The three messages you will see most often are a missing target sequence in the header, a missing EOF block, and a file that is not recognized as any supported format. A missing EOF block is the signature of a killed writer or an interrupted copy, and there is no safe repair. Re-run or re-copy.

Step 2: confirm BGZF, not gzip

htsfile sample.bam

A healthy BAM reports as BAM compressed sequence data. If htsfile reports a generic gzip file, someone compressed a BAM with gzip instead of bgzip. The file may still decompress, but no index-based tool can seek in it. The fix requires re-reading the whole file:

samtools view -h plain.bam | bgzip -c > fixed.bam
samtools index fixed.bam

Step 3: force a full parse

samtools view -c sample.bam

This reads every record and returns the alignment count. A crash here that quickcheck did not catch indicates internal block corruption. Compare the count to samtools flagstat sample.bam output, which breaks the total into mapped, properly paired, duplicate, and secondary categories.

Step 4: run a real validator

gatk ValidateSamFile I=sample.bam MODE=SUMMARY

ValidateSamFile reports errors and warnings by category. Use MODE=VERBOSE to see every offending record and IGNORE=WARN when you only want errors. The categories worth acting on immediately are INVALID_CIGAR, INVALID_FLAG_*, MATE_NOT_FOUND, MISSING_READ_GROUP, RECORD_OUT_OF_ORDER, and INVALID_INDEXING_BIN. A BAM with no read group will fail most variant callers that expect one, and the failure message is usually unhelpful.

Do not silence a warning class just because your pipeline tolerated it. A RECORD_OUT_OF_ORDER warning means the index is unreliable even if samtools index succeeded.

Step 5: check header against reference

samtools view -H sample.bam | grep '^@SQ' | head -3
cut -f1,2 genome.fa.fai | head -3

The SN: and LN: values in @SQ must match the names and lengths in the .fai. A chromosome named 1 in the BAM and chr1 in the FASTA will fail with a "no targets" or "unable to find sequence" error at the first tool that needs both. Fix by renaming consistently, not by editing headers in place, and re-derive the BAM if alignment already happened against the wrong naming convention.

Step 6: check index freshness

samtools idxstats sample.bam > /dev/null

htslib warns when the index is older than the data file. That warning means the BAM was modified after indexing, and every region query will return wrong results. Rebuild with samtools index.

Step 7: CRAM needs its reference

samtools view -T genome.fa -c sample.cram

CRAM stores reads as differences from a reference and carries M5 checksums in its header. Without the correct reference, the file cannot be decoded. If REF_PATH or REF_CACHE is set in the environment, a mismatched cached reference produces a checksum error rather than a silent wrong decode, which is the good outcome. Set REF_PATH explicitly in pipeline configuration so that CRAM decoding does not depend on whatever happens to be cached on a given node.

Safety note

samtools sort -o sample.bam sample.bam truncates the input before the sort reads it, and the result is an empty file. Always write to a new path, then move.

Validating VCF files

VCF failures cluster into five causes: a missing or malformed header, a genome build or contig naming mismatch, REF alleles that do not match the reference, non-normalized representation of indels, and invalid compression or indexing.

Step 1: header sanity

head -1 calls.vcf | grep -q '^##fileformat=VCFv4' && echo OK
grep -m1 '^#CHROM' calls.vcf | awk -F'\t' '{print NF " columns"}'

The ##fileformat line must be first. The #CHROM line must be tab-separated with exactly the expected number of columns: 8 fixed plus FORMAT plus one per sample. Space-separated column headers are a common artifact of hand-edited files and break every parser.

Step 2: confirm the build

grep '^##contig' calls.vcf | head -5
grep '^##reference' calls.vcf

Contig names reveal the build in most cases. chr1 through chr22 with a length of 248,956,422 for chr1 indicates GRCh38. A length of 249,250,621 for chr1 indicates GRCh37. Names without the chr prefix indicate a Ensembl-style or a lifted file and require explicit checking.

Clinical consensus recommendations for production bioinformatics specifically call for adopting hg38 as the reference build and standardizing file formats with strict version control [7]. If your laboratory has not fixed a build, the mismatch will eventually surface as a batch of false homozygous reference calls in a region where the two builds differ.

Step 3: check REF alleles against the reference

bcftools norm -f genome.fa -c e -m -any calls.vcf -Ou > /dev/null

The -c e option makes a REF mismatch a hard error, -c w warns, -c x excludes offending sites, and -c s sets REF to the reference base. Use e in validation and never use s in production, because it silently rewrites alleles instead of telling you the file is wrong.

A REF mismatch almost always means one of three things. The VCF was called against a different build. The VCF uses a different contig naming convention. Or the site is a multi-allelic record that was split in a way that changed the anchor base.

Step 4: normalize and split

bcftools norm -f genome.fa -m -any calls.vcf -Oz -o calls.norm.vcf.gz
bcftools index --tbi calls.norm.vcf.gz

Normalization left-aligns and trims indels so that equivalent variants are represented identically. Without it, the same deletion in a homopolymer run appears at two different positions in two files and gets counted as two variants in a concordance comparison. Splitting multi-allelic records with -m -any produces one record per ALT allele, which is required by many downstream tools and makes allele-specific filtering possible.

Step 5: structural and cardinality checks

vcf-validator calls.vcf

This checks header definitions against field usage, verifies that INFO and FORMAT fields with Number=A, Number=R, and Number=G declare values consistent with the allele count, and flags undeclared tags. An undeclared INFO field will be dropped silently by some tools and rejected outright by others.

gatk ValidateVariants -V calls.vcf -R genome.fa --validation-stringency STRICT

Strict stringency turns warnings into errors and is the right setting for validation runs. It checks REF concordance, header correctness, and the consistency of genotype fields where possible.

Step 6: compression and index

htsfile calls.vcf.gz
tabix -p vcf calls.vcf.gz

Tabix indexing requires BGZF compression. A VCF compressed with plain gzip cannot be indexed, and tools that appear to succeed will produce empty region queries. Check the index by querying a known interval and confirming a nonzero record count.

Step 7: sample-level checks

bcftools query -l calls.vcf
grep -m1 '^#CHROM' calls.vcf | tr '\t' '\n' | tail -n +10 | sort | uniq -d

Duplicate sample names are rejected by some tools and merged silently by others. bcftools query -l lists samples in order, which you should compare against the sample sheet. A swapped column produces a VCF where the genotypes are assigned to the wrong person, and no format validator will catch it.

A further genotype-level check catches a frequent real problem: sites where the genotype calls a heterozygote but every sample's allele depth is zero for the alternate allele. Those records survive validation and fail review.

Putting validation into a pipeline

Manual checks scale to about ten files. Production needs automation at three points: on ingest, between pipeline stages, and on final delivery.

PipeVal is built for exactly this and is designed to be inserted into existing pipelines to automatically validate inputs and outputs, which reduces wasted compute on corrupt files and invalid paths [2]. AssemblyQC demonstrates the pattern inside a single nf-core-style workflow, running file-format validation as the first step before contaminant checking, contiguity, and completeness metrics [4].

Ingest validation should confirm: the container type, the checksum against the manifest, the declared build against the sample sheet, and the pairing or mate structure. Stage-boundary validation should confirm that each intermediate has a header consistent with the previous stage and that counts have not collapsed unexpectedly. Delivery validation should confirm that outputs are in standardized, versioned formats, which is the same requirement public health bioinformatics implementations impose on their own workflows [8].

Two operational practices make the whole system reproducible. First, pin tool versions. CREDO exists because reproducing an analysis depends on reproducing the environment, and it stores package information and dependencies in a version-controlled format so that images can be rebuilt from scratch [3]. Second, run validation continuously rather than once. The Metabolomics Workbench File Status website validates every publicly available dataset on a weekly schedule and publishes the resulting inconsistencies, which demonstrates that validation is a maintenance activity and not a one-time gate [9].

Troubleshooting table

Common Mistakes and Limitations

Counting FASTQ records with grep. The @ character appears in quality strings at Phred score 31. The + character appears at score 10. Neither is a reliable header marker. Use the line count.

Trusting the extension. .fa, .fastq, .bam, and .vcf are naming conventions, not guarantees. Every validation run should begin with container identification.

Assuming a successful run means valid input. A tool that silently skips unparseable records will finish and produce results. The 80-package BED validation study found that most tested tools failed edge-case handling [1], which means the tool's silence carries little information.

Skipping REF validation because it is slow. It is the single highest-value semantic check on a VCF. Build mismatches are otherwise invisible until a reviewer asks why a known variant is missing.

Editing BAM headers in place to fix chromosome names. This produces a file whose index, binning, and downstream region queries all disagree. Re-derive the alignment.

Treating ValidateSamFile warnings as noise. MISSING_READ_GROUP and RECORD_OUT_OF_ORDER cause real downstream failures. Suppress a category only after you have confirmed the specific consequence is acceptable for your pipeline.

Validating once at delivery. Public repositories that validate continuously find new inconsistencies over time as schemas, parsers, and datasets change [9]. Add validation to continuous integration and re-run it when tool versions change.

Comparing VCFs without normalizing both sides. Representation differences produce false discordance that looks like a pipeline bug. Normalize both files with the same reference and the same splitting rule before any comparison.

Limitations worth stating plainly. No validator proves that a file is biologically correct. A perfectly formed BAM can come from a failed library, and a perfectly formed VCF can encode a sample swap. Format validation catches mechanical defects, semantic validation catches reference and representation defects, and only review of the experiment design catches the rest. Coordinate liftover between builds can also introduce errors that survive every format check. Individual cases need a bioinformatician who knows the experiment, and clinical interpretation needs a qualified reviewer.

Prevention practices

Write a manifest at ingest. For every file, record the checksum (sha256sum), the byte size, the source, the date, and the declared build. Re-verify the checksum before every stage that reads the file.

Keep raw data immutable. Set read-only permissions on raw FASTQ and on delivered VCFs. All cleaning happens on derived copies.

Make the reference explicit. Carry the reference path, its checksum, and its contig naming convention in pipeline configuration rather than in a shell variable that someone exports differently on a different node.

Standardize naming and formats at the boundary. Public health implementations that produce reproducible, auditable results do so by using containerized software and workflow languages with standardized output formats and version-controlled code [8]. Standardizing at the boundary is what makes files from different runs comparable.

Validate small fixtures in continuous integration. Keep a handful of deliberately broken files, a truncated BAM, a FASTQ with mismatched quality lengths, a VCF with a single REF mismatch, and assert that your validation step rejects each one. This is the practical version of the fuzzing approach that caught errors the hand-written test suite missed [1].

Pin containers and record versions. Rebuild images from version-controlled specifications so that a validation result from last year can be reproduced this year [3].

Safety note

VCF and BAM files from clinical samples contain protected health information in headers, sample names, and occasionally free-text fields. Do not upload them to public validation services, paste them into issue trackers, or run third-party container images with network access against them unless the data use agreement permits it. De-identify or use synthetic fixtures for any external validation call.

Frequently Asked Questions

What is the difference between FASTA and FASTQ?

FASTA stores sequence only, while FASTQ stores sequence plus a per-base quality score for every call. Use FASTA for references, assemblies, and databases, and FASTQ for raw reads that need alignment or quality filtering.

Why does samtools say my BAM is truncated?

Almost always because the writing process was killed or the file transfer ended early, which removes the BGZF end-of-file block. There is no safe repair. Re-copy the file and compare checksums against the source.

Can I validate a VCF without a reference genome?

You can run structural checks and header checks, but you cannot verify REF alleles or normalize indels. Those are the two checks that catch the defects most likely to cause wrong results, so a reference is effectively required.

How do I count reads in a FASTQ file correctly?

Divide the total line count by four. awk 'END{print NR/4}' on the decompressed stream is correct. Grepping for header characters overcounts because quality strings routinely contain @ and +.

What does "was not identified as format BAM" mean?

It means the file header does not match any format htslib recognizes, usually because the file is corrupt at the start, is plain gzip rather than BGZF, or is a renamed SAM or FASTQ. Check with htsfile and file.

Why does bcftools norm report a REF mismatch?

The VCF was called against a different reference build or a different contig naming convention than the FASTA you supplied. Confirm the build from ##contig lengths before deciding whether to re-call or lift over.

Do I need to validate files if my pipeline already ran successfully?

Yes. Many tools skip or coerce malformed records without failing, so a completed run is not evidence of valid input. Validate at ingest, at stage boundaries, and on delivery.

How often should I re-validate existing files?

On every tool version change and at every delivery. Validation is maintenance, not a one-time gate, and continuous repository-scale validation shows that new inconsistencies appear over time.

Sources

1. Assessing and assuring interoperability of a genomics file format.
2. PipeVal: light-weight extensible tool for file validation.
3. CREDO: a friendly Customizable, REproducible, DOcker file generator for bioinformatics applications.
4. AssemblyQC: a Nextflow pipeline for reproducible reporting of assembly quality.
5. FIREVAT: finding reliable variants without artifacts in human cancer samples using etiologically relevant mutational signatures.
6. vcf2fhir: a utility to convert VCF files into HL7 FHIR format for genomics-EHR integration.
7. Recommendations for bioinformatics in clinical practice.
8. Accelerating bioinformatics implementation in public health.
9. The metabolomics workbench file status website: a metadata repository promoting FAIR principles of metabolomics data.