How to Recover a Truncated VCF File

By Dr. Zubair Khalid, DVM, MS, PhD ·

How to Recover a Truncated VCF File

The most common reason a VCF file is truncated is that the process writing it stopped before it finished. The writer was killed by the job scheduler when the wall clock ran out, evicted by the out-of-memory killer, unplugged by a dropped SSH session, or it simply ran out of disk. The second most common reason is that the compressed stream itself was cut, which is a different failure with a different fix.

The first thing to check is not the data. It is the container. Run bgzip -t file.vcf.gz and gzip -t file.vcf.gz and look at the exit status. If the file is compressed and those tests fail, the damage is at the compression layer and you can usually recover every complete record plus the header. If those tests pass but the file still looks wrong, the damage is inside the text and you need to inspect columns and header lines instead.

Everything below works from the compression layer inward, then from the most common cause to the rarest. Every command is standard and runs on a Linux or macOS workstation with samtools, bcftools, htslib and tabix installed. The Variant Call Format itself is specified by the Global Alliance for Genomics and Health, and the compressed container used for VCF files is defined in the SAM/BAM family of specifications maintained by hts-specs, so the behavior of bgzip, tabix and bcftools is predictable rather than vendor-specific.

Safety note. Before you run anything, copy the damaged file to a scratch location. Recovery commands edit text, and a single mistyped redirection over the original destroys the only copy. Use cp -a file.vcf.gz file.vcf.gz.orig and work on the copy.

What a VCF File Is and Why Truncation Breaks It

A VCF file, short for Variant Call Format, is a tab-delimited plain text file that stores genetic variants, meaning positions in a reference genome where one or more samples differ from that reference. It was created for the 1000 Genomes Project and is now the standard exchange format for variant calls across sequencing pipelines, genotyping arrays and population databases. The current specifications in wide use are version 4.2 and version 4.3.

Every VCF has three layers of text.

The first layer is the meta-information block. These lines begin with two hash characters and declare the file format version, the reference build, the contig names and lengths, the definitions of every INFO field, every FORMAT field and every FILTER value, and often the command line that produced the file. Example lines include ##fileformat=VCFv4.2, ##contig=<ID=chr1,length=248956422> and ##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">.

The second layer is the single header line beginning with #CHROM. It names the eight fixed columns (CHROM, POS, ID, REF, ALT, QUAL, FILTER, INFO), then the FORMAT column, then one column per sample. If the file holds 1,000 samples, that header line has 1,009 tab-separated fields.

The third layer is the data block. Each line is one variant record in one genomic position. A genotype field such as 0/1:35 means the sample carries one reference allele and one alternate allele with a depth of 35 reads. Missing genotypes are encoded as ./., never as a zero. This matters enormously later, because padding a broken record with zeros writes a false homozygous reference call into the file while padding it with dots writes an honest missing call.

Plain text has one redeeming property for recovery. Each variant record is independent of every other record and is terminated by a newline. A file cut in the middle of a write loses at most one complete record plus a partial fragment. Everything before that point is intact at the byte level. The header, however, cannot be inferred from the data block, and the compression layer adds a failure mode that has nothing to do with the text.

Why Plain Text Bioinformatics Formats Fail in Predictable Ways

VCF files belong to a family of plain text bioinformatics file formats that share the same vulnerability. FASTA and FASTQ are the two best-known examples, and comparing them explains why some truncations are cheap to fix and others are not.

A FASTA file stores sequences. It has a header line beginning with a greater-than symbol, followed by one or more lines of nucleotide or amino acid sequence. The line wrapping is cosmetic, and a parser that concatenates the sequence lines tolerates it. A truncated FASTA loses the tail of one sequence, which is a real loss of data but does not corrupt anything earlier.

A FASTQ file stores sequences together with per-base quality scores. It uses exactly four lines per record. The first line begins with an at sign and holds the read identifier. The second line is the sequence. The third line begins with a plus sign. The fourth line is the quality string, one character per base, encoded as an ASCII character offset by 33 in modern Illumina data (Phred+33). Because the record boundary is defined by counting lines and not by matching the quality length to the sequence length in most parsers, losing or gaining a single line near the truncation point shifts every subsequent record by one line. Read identifiers end up where sequences should be, and quality scores attach to the wrong bases. This is called a frame shift, and it can silently corrupt millions of reads.

VCF sits closer to FASTA on this spectrum. A newline always ends a record, and there is no fixed line count per record, so a truncation damages the last line and nothing else. Neither FASTA nor FASTQ nor VCF carries a built-in per-record checksum. Only the compression container or an external manifest can tell you whether the file is complete. That is why the diagnostic sequence starts with the container.

The Diagnostic Sequence

Run these nine checks in order. Stop when one fails, because that failure identifies the cause.

  1. Identify the container. Run file file.vcf.gz. The output tells you whether the file is plain text, ordinary gzip or BGZF, which is the blocked gzip variant used for VCF. BGZF is the only compression that tabix can index.
  2. Test the compression stream. Run gzip -t file.vcf.gz and bgzip -t file.vcf.gz and print the exit status with echo $? after each. A nonzero status means the stream is incomplete or damaged.
  3. Look for the missing end-of-stream marker. htslib writes a fixed 28-byte end-of-file block when it closes a BGZF file, and its readers use that block to confirm completeness. If the marker is absent, htslib prints a warning that the EOF marker is absent and that the input may be truncated. Treat that warning as a hard failure, not a nuisance.
  4. Decompress to a working file. Run gzip -dc file.vcf.gz > work.vcf and ignore the nonzero exit status. You now have all the bytes that survived.
  5. Inspect the tail. Run tail -n 3 work.vcf and tail -c 1 work.vcf | xxd. A final byte that is not 0a (the newline character in hexadecimal) confirms a record was cut mid-write.
  6. Check the header. Run grep -c '^#CHROM' work.vcf. The answer must be exactly 1. Run head -n 5 work.vcf to confirm the ##fileformat line is present.
  7. Check column counts. Run awk -F'\t' '/^#CHROM/{n=NF} !/^#/ && NF!=n {print NR": "NF" fields, expected "n}' work.vcf. Every data line should have the same field count as the header.
  8. Check the last data line's fixed columns. Run awk -F'\t' '!/^#/{last=$0} END{print last}' work.vcf | cut -f1-8. A POS field that is not an integer, or an INFO field with an unbalanced square bracket, means the line is incomplete.
  9. Check the index. If a .tbi or .csi file exists, run bcftools index -n file.vcf.gz and compare it with bcftools view -H file.vcf.gz | wc -l. A mismatch means the index was built before the file was damaged or vice versa.

Troubleshooting Table

SymptomLikely causeHow to confirmFix
gzip -t reports unexpected end of file and bcftools refuses to readCompressed stream cut mid-writegzip -t exits nonzeroDecompress, strip the partial tail, recompress with bgzip, rebuild the index
File opens, but the final data line has fewer fields than the headerWrite interrupted inside a recordStep 7 above reports one short lineDelete the final line with sed '$d', keep everything else
#CHROM header line missing or header block absentHeader stripped by grep -v '^#' or lost in a copygrep -c '^#CHROM' returns 0Rebuild a header from ##contig lines in the reference .fai and the known sample list, then attach with bcftools reheader -h
bgzip -t fails but gzip -t passesFile is ordinary gzip, not BGZF, or the BGZF EOF block is missingfile reports gzip rather than BGZF`gzip -dcbgzip -c to rewrite as BGZF, then tabix -p vcf`
Tabix lookups return wrong records or failStale index built on a truncated or reordered filebcftools index -n differs from the record countDelete the index and rebuild with tabix -p vcf or bcftools index -t
Data look incomplete in the middle, file parses cleanlyTwo VCFs concatenated with different sample sets, or overlapping regionsField counts vary by line, or duplicate CHROM and POS pairs existSplit by sample set, then bcftools sort and bcftools merge or bcftools concat -a
File size matches a quota almost exactlyFilesystem full, inode exhausted, or interrupted multipart uploaddf -h, df -i, scheduler logsFree space, rerun from the last checkpoint, adopt atomic writes
Every line ends with a carriage return or contains NUL bytesASCII-mode FTP transfer, editor round trip, interrupted copyfile reports CRLF line terminators, grep -c $'\r' returns the line counttr -d '\r' or re-copy in binary mode
Duplicate records at the endA retried job appended to an existing output fileDuplicate CHROM and POS at the tailDeduplicate with bcftools norm -d exact, then re-sort

Cause 1: The Write Was Interrupted Before It Finished

This is the most common cause by a wide margin. A variant calling job writes a VCF for six hours, and at hour five the cluster scheduler sends SIGTERM because the wall clock expired. Or the process grows past the memory limit and the kernel out-of-memory killer sends SIGKILL. Or you ran the pipeline over SSH without nohup or a terminal multiplexer, the connection dropped, and the shell sent SIGHUP to everything in the foreground. Or the pipeline ended in | head -n 20 and the reader closed the pipe, which delivers SIGPIPE to the writer when it tries to flush its next buffer.

The mechanism is identical in every case. The process died before it flushed its final buffer and closed the stream. What is on disk is every byte that had already reached the filesystem plus whatever partial record was in the buffer at the moment of death.

There is a nasty variant involving pipes. In bash, a pipeline's exit status is the exit status of the last command. If bcftools view -Oz | head exits zero because head finished normally, your shell reports success even though the writer was killed. Add set -euo pipefail to the top of every script that writes data through a pipe, and the shell will propagate the failure.

Confirm it with three checks. First, the header is present and the data block is short relative to what you expected from the sample count and coverage. Second, the last line fails the column count check from step 7. Third, the scheduler or kernel log shows a kill. On a Slurm cluster, sacct -j JOBID shows state TIMEOUT, OOM or CANCELLED. On a workstation, dmesg | tail -n 50 shows an out-of-memory kill naming the process.

The fix is to keep everything that was written and discard only the fragment.

cp -a truncated.vcf.gz salvage.vcf.gz
gzip -dc salvage.vcf.gz > salvage.vcf
sed '$d' salvage.vcf > salvage.clean.vcf
tail -n 2 salvage.clean.vcf

The sed '$d' deletes the final line unconditionally. That is safe here because you have already confirmed the final line is the damaged one. If the final line is complete and the file merely lacks the compression trailer, skip that step. Then validate the header and column counts, recompress with bgzip -c salvage.clean.vcf > recovered.vcf.gz, and rebuild the index with tabix -p vcf recovered.vcf.gz.

If the job is still available, rerun it with checkpointing enabled and a longer time limit rather than trying to stitch output together.

Cause 2: The Filesystem Ran Out of Space

When the disk fills, the write system call returns an error code or a short count, meaning fewer bytes were written than requested. Not every program checks that return value on every call. A writer that ignores it produces a file that ends abruptly with no error message in the log at all. This is why a truncated VCF sometimes appears alongside a perfectly clean stdout log.

Two constraints can trigger it. The first is space in bytes. The second is inodes, which are the fixed-size records that a Unix filesystem allocates for each file. A directory holding millions of tiny intermediate files can hit the inode limit while terabytes of byte space remain free.

Confirm it with df -h on the output filesystem and df -i on the same mount. Compare the file size with the limit. On shared clusters, quotas are often enforced per user and reported by a command such as quota -s or lfs quota, so check the quota rather than the mount total. If the failure happened hours ago, the space may have been released and the current df will look healthy. In that case look at scheduler accounting or the pipeline log timestamps instead.

The fix is to free space and rerun, or to salvage the partial output exactly as in Cause 1 and then resume from an earlier checkpoint. Do not delete the damaged file until you have verified that a rerun will succeed, because the rerun may take longer than the deadline you are working against.

Cause 3: The gzip or BGZF Stream Was Cut

This cause deserves its own section because the recovery is different and because many people misdiagnose it as data loss when it is only container loss.

An ordinary gzip file is a single stream with a header and a trailer. The trailer contains a CRC32 checksum and the uncompressed length. When gzip -t reaches the end of a truncated stream, it reports an unexpected end of file and exits nonzero. The decompressed data that preceded the cut is perfectly valid, and gzip -dc will write it to stdout before failing. That is why step 4 of the diagnostic sequence tells you to ignore the exit status and inspect the output.

BGZF is a blocked variant of gzip defined in the hts-specs family of specifications and used by bgzip, tabix and bcftools. It breaks the input into independently compressed blocks of up to 64 kilobytes and adds an index that lets readers seek to any block. At the end of a complete BGZF file, htslib writes a fixed 28-byte empty block as an end-of-file marker. Readers use that marker to decide whether the file is complete. When it is missing, htslib emits a warning that the EOF marker is absent and that the input may be truncated, but it will often still return records. Do not treat that warning as cosmetic.

A third variant appears in long pipelines. Plain gzip supports concatenated members, meaning you can glue two gzip files together with cat and decompress the result as one stream. This works for ordinary gzip and produces a file that zcat reads completely. It does not produce BGZF, so tabix cannot index it, and bcftools may read it but will complain. People who concatenate per-chromosome gzip outputs hit this constantly.

Confirm the container type first with file, then test both tools separately.

file truncated.vcf.gz
gzip -t truncated.vcf.gz ; echo "gzip status: $?"
bgzip -t truncated.vcf.gz ; echo "bgzip status: $?"

Note that the semicolons shown above are shell separators in the command line, not part of the data. Run each test on its own line if you prefer.

The fix is a full decompress and recompress cycle. zcat or gzip -dc reads every member of a multi-member gzip file, so a concatenated gzip becomes a single clean text stream. Then bgzip -c writes a proper BGZF file with a valid end-of-file marker, and tabix -p vcf builds a working index. This repair costs one pass over the data and requires temporary disk space equal to the uncompressed file, which for a human genome VCF can be tens of gigabytes. Point TMPDIR at a scratch filesystem before you start.

Cause 4: The Header Line Was Lost

The header is the part of a VCF that cannot be reconstructed from the data alone. Contig lines declare which reference sequence each CHROM value refers to and how long it is. INFO and FORMAT lines declare the type, number and meaning of every field. The #CHROM line declares the sample names and their order. Without those declarations, a parser has no way to interpret 0/1 or AF=0.02.

Headers go missing in three common ways. Someone runs grep -v '^#' to get a data-only file and later forgets that the header was left behind. Someone runs tail -n +2 on the wrong file. Or a tool writes records to a stream and the header was never emitted because the tool was configured to append to an existing file that it thought already had one.

Confirm with grep -c '^#CHROM' work.vcf. A count of zero means there is no header line. A count greater than one means headers were concatenated, which is Cause 6.

Recovery depends on whether you know the sample names and reference build. If you do, build a header file and attach it. The contig lines can be generated directly from the reference FASTA index, which lists each sequence name and its length. The FORMAT line for genotypes is fixed by the specification.

printf '##fileformat=VCFv4.2\n' > header.txt
awk '{print "##contig=<ID="$1",length="$2">"}' reference.fasta.fai >> header.txt
printf '##FORMAT=<ID=GT,Number=1,Type=String,Description="Genotype">\n' >> header.txt
printf '#CHROM\tPOS\tID\tREF\tALT\tQUAL\tFILTER\tINFO\tFORMAT\tSAMPLE1\tSAMPLE2\n' >> header.txt
bcftools reheader -h header.txt -o recovered.vcf.gz body.vcf.gz

The bcftools reheader command replaces the header of an existing file without rewriting the data, which is fast. It requires a bgzipped input. If your body is plain text, compress it with bgzip first.

Two things can go wrong. If the contig naming style in your header does not match the data (chr1 versus 1), every downstream tool that checks contig consistency will fail, and some will silently drop records. Match the style used in the data block exactly, which you can read from the first data line. If you do not know the sample names and their order, you cannot recover them from the genotype columns, because the file records no per-sample identifier anywhere in the data block. Get the sample list from the original BAM read groups, from the pipeline manifest, or from a sibling VCF produced by the same run.

Cause 5: A Record Was Cut Mid-Line

This is the data-layer version of Cause 1 and the most common outcome after the compression layer has been repaired. The final record stops partway through, so its INFO field may end in the middle of a key-value pair, its FORMAT field may list fewer colon-separated values than the sample columns contain, or one sample column may simply be absent.

Confirm it by comparing field counts across all lines, as in step 7 of the diagnostic sequence. A single short line at the very end is the signature. A more scattered pattern of short lines points to line-ending damage instead, which is Cause 7.

awk -F'\t' '/^#CHROM/{n=NF; print "header fields: "n}
             !/^#/ && NF!=n {print "line "NR" has "NF" fields"}' work.vcf | head -n 20

The fix is to drop the damaged record. There is no principled way to guess what the missing genotype values were, and guessing is worse than losing one variant, because a fabricated call enters downstream analysis indistinguishable from a real one. Drop the line, keep the rest, and recompress.

A subtler case is a record whose field count is correct but whose INFO string is truncated, which shows up as an unbalanced angle bracket or a key with no value. A parser may accept it or may abort on it depending on strictness. If you see only one such line at the end, drop it. If you see many, the file was damaged by something other than a single interrupted write and you should go back to the container-level checks.

One special case is worth naming. If the truncation removed the entire final record and left a clean newline in its place, nothing is detectable at the text level. The file is valid, complete-looking and missing its last variant. Only an external record count or a checksum from before the damage can reveal that. This is the argument for keeping a manifest of record counts alongside every VCF you publish.

Cause 6: Concatenation and Merge Errors

Not every broken VCF was truncated by a crash. Some were assembled incorrectly from valid pieces, and the result looks like truncation because a parser stops at the first inconsistency.

The first pattern is cat a.vcf b.vcf > combined.vcf. This produces a file with two #CHROM lines, one in the middle, and duplicate meta-information blocks. Strict parsers treat the second header as an error or as a new file. The column count check will flag it immediately because the header line reappears where a data line is expected.

The second pattern is cat a.vcf.gz b.vcf.gz > combined.vcf.gz. This is valid for ordinary gzip and produces a readable multi-member stream, but it is not BGZF and cannot be indexed. The fix is the decompress and recompress cycle from Cause 3.

The third pattern is concatenating VCFs with different sample sets. The header from file A declares 100 samples and the records from file B carry 120 sample columns. Every B record fails the field count check. The fix depends on intent. If the samples are genuinely different cohorts, use bcftools merge to combine them on shared sites. If they are the same cohort split by chromosome, use bcftools concat -a, which allows chromosome blocks to be joined without re-sorting, after stripping all but the first header.

The fourth pattern is overlapping regions, where the same variant appears twice because two intervals were called and concatenated. Duplicates are not truncation, but they break downstream counting and can make a file look shorter than expected after deduplication. Remove exact duplicates with bcftools norm -d exact, then sort with bcftools sort.

Cause 7: Text Mode Transfers and Line Ending Damage

A VCF that was downloaded over FTP in ASCII mode, opened in a text editor and saved, or passed through a Windows tool can come back with carriage return characters before every newline. Some tools tolerate the stray carriage return because they strip whitespace. Others do not. The last sample field in a genotype column is a string, so a trailing \r may survive into a numeric comparison and cause a type error far from the actual damage.

Confirm with file work.vcf, which reports CRLF line terminators when they are present, or with grep -c $'\r' work.vcf, which counts affected lines. If the count equals the total line count, the whole file was converted.

The fix is a single pass with tr -d '\r' or sed 's/\r$//' before recompressing. Run the conversion before any other repair, because a stray carriage return can masquerade as a truncated field in the column count check.

NUL bytes are a different problem. They appear when a copy was interrupted, when a sparse file was created on a filesystem that does not support holes, or when a disk returned zeros for a block it could not read. tr -d '\000' will remove them, but the underlying data may already be gone. Re-copy from the source of truth if one exists.

Cause 8: Storage and Hardware Corruption

The rarest cause is silent corruption at the storage layer. A failing disk returns a bad sector, a network filesystem client caches a partial write, an object storage multipart upload never completes, or an rsync transfer is interrupted without partial-file retention and leaves a short file behind. These look exactly like an interrupted write but do not correlate with any job failure.

Confirm with an external checksum. If you recorded sha256sum of the file when it was produced, recompute it now and compare. If you did not, check the storage layer directly. On local disks, smartctl -a reports reallocated sector counts and pending sector warnings. On network filesystems, check the server logs. For object storage, list the multipart upload and confirm the uploaded part count matches the expected size.

The fix is re-transfer or regenerate from the source of truth. If no source exists, salvage what parses, record in the file provenance that a region is missing, and do not present the recovered file as complete.

Order of Operations When the File Is the Only Copy

Follow this sequence and do not skip steps.

  1. Copy the file. Use cp -a to preserve timestamps and permissions. Never repair in place.
  2. Identify the container with file.
  3. Test both gzip -t and bgzip -t and record both exit statuses.
  4. Decompress to a working text file, ignoring the nonzero exit status.
  5. Check the header. Rebuild it if #CHROM is absent.
  6. Check column counts. Drop only the incomplete final line.
  7. Check for carriage returns and remove them.
  8. Recompress with bgzip -c.
  9. Build a fresh index with tabix -p vcf and confirm the record count matches.
  10. Record a new checksum and a note describing exactly what was removed.

Safety note. Every step up to the recompress writes new files and reads old ones. Never redirect > onto the original path. If you must delete intermediates to free space, delete only files you have named as intermediates, never a file whose name matches the original.

Verifying the Recovered VCF

A recovered file is not finished until you have checked it from three directions.

Check the header and sample list first. bcftools view -h recovered.vcf.gz prints the header, and bcftools query -l recovered.vcf.gz | wc -l counts the samples. Compare that count with the field count on the first data line minus nine, which is the number of fixed columns plus the FORMAT column. A mismatch means the header rebuild was wrong.

Check the record count second. bcftools view -H recovered.vcf.gz | wc -l gives the number of data lines. bcftools index -n recovered.vcf.gz gives the number of records the index believes exist. These must agree. If they do not, delete the index and rebuild it.

Check the content third. bcftools stats recovered.vcf.gz reports counts of SNPs, indels, multiallelic sites and missing genotypes. You are not looking for an absolute number, you are looking for a comparison against the same metric from the same pipeline on a previous run. A recovered file whose transition to transversion ratio or heterozygous to homozygous ratio has moved dramatically is a sign that something other than the last line was lost.

Finally, spot-check random regions with tabix recovered.vcf.gz chr1:1000000-1010000 and confirm the records you expect are present. A region that returns nothing is not the same as a region that was called and found to contain no variants, so do not confuse the two when you audit the recovered file.

Prevention

The cheapest recovery is the one you never need. These practices eliminate nearly every cause above.

Write to a temporary name and rename when finished. mv within a single filesystem is atomic, so a reader never sees a half-written file and a crash leaves an obviously incomplete temporary name rather than a plausible-looking output.

Use set -euo pipefail in every shell script that produces data. The pipefail option is the single most valuable line in a bioinformatics script, because it converts silent SIGPIPE failures into loud ones.

Compress with bgzip or bcftools view -Oz from the start, never with plain gzip. A BGZF file carries its own completeness marker and its own index, and every tool in the samtools family reads it natively.

Keep the upstream data. A VCF can always be regenerated from the BAM and the reference, so treat the BAM as the source of truth and the VCF as a derived artifact. This single decision turns an unrecoverable data loss into a rerun.

Record a manifest. For every VCF you publish, write a small text file holding the file name, byte size, record count, sample count, sha256sum of the compressed file and the exact command line that produced it. The record count is what catches the invisible case where the final newline is present and the last record is silently gone.

Check disk space before you start a long job. A single df -h at job start, written into the log, tells you a week later whether the filesystem was near a quota when the job failed.

Use a scheduler with checkpointing, or a workflow manager that publishes outputs only on successful completion. Snakemake and Nextflow both write outputs to a working directory and publish them atomically when the task succeeds, which prevents downstream tools from consuming partial files.

Transfer with checksums. Use rsync -c for local copies and a checksum-verified mode for object storage. Verify after transfer, not before.

Common Mistakes and Limitations

Padding a truncated record with zeros. Writing 0/0 into a missing genotype column creates a homozygous reference call. That is a positive assertion that the sample matches the reference at that position, supported by evidence that does not exist. Use ./. if you must write anything, and prefer dropping the record entirely.

Assuming a passing bcftools view means the file is complete. htslib reads past a missing EOF marker and prints a warning rather than aborting, so a command can exit zero on a damaged file. Always check the stderr stream, or run bgzip -t explicitly.

Recompressing with plain gzip and then wondering why tabix fails. Plain gzip produces a file that tabix cannot index. The error is at the container level, not the data level.

Using cat on bgzipped files. Concatenating BGZF files does not produce a valid BGZF file with a single end-of-file marker in the expected place. Decompress and recompress instead.

Rebuilding the header with the wrong contig naming style. A header that says chr1 over data that says 1 passes a naive field count check and fails every consistency check downstream, sometimes by silently dropping records. Read the first data line before you write the contig lines.

Treating a truncated file as an empty region. If a VCF was cut before chromosome 22, an analysis that reads the file will report no variants on chromosome 22. That is not a biological result, it is missing data. Flag the region in the sample manifest before any downstream interpretation.

Forgetting that the index can outlive the data. An index built on a complete file and left next to a truncated replacement will return records that no longer exist or fail with confusing errors. Delete and rebuild the index as the last step of every repair.

Limitations. A recovery can restore everything that was written. It cannot restore what was never written. If the truncation removed the last hours of a variant calling run, the missing variants are gone until the run is repeated, and no text-level technique will bring them back. Determining whether a recovered file is fit for a specific downstream analysis, especially one with regulatory or clinical implications, is a decision for the analyst and the study's data management plan, not something a command line can settle.

Frequently Asked Questions

Can a truncated VCF file be repaired without the original data?

Yes for the file, no for the missing variants. Every complete record written before the truncation is intact and can be recovered. Whatever the writer never wrote cannot be reconstructed from the surviving text.

How do I know if my VCF is truncated?

Run bgzip -t and gzip -t and check the exit status, then check that the last line has the same number of tab-separated fields as the header. A missing BGZF end-of-file marker is the single most reliable signal.

Does bcftools fix a truncated VCF automatically?

No. bcftools reads what is present and often emits a warning about an absent end-of-file marker, but it does not repair the stream. You must decompress, remove the incomplete tail, and recompress with bgzip.

Why does gzip say unexpected end of file on my VCF?

The compressed stream lacks its trailer, which holds the checksum and the uncompressed length. This means the file was cut before the writer closed it. The decompressed data before the cut is still valid.

Can I just delete the last line of a truncated VCF?

Yes, in the specific case where the last line is the incomplete one. Confirm that first with a field count check, because deleting a complete final record loses a real variant for no reason.

What is the difference between a truncated VCF and a corrupted VCF?

A truncated file is missing its tail, so everything before the cut is byte-for-byte correct. A corrupted file has damage in the middle, such as flipped bits or NUL bytes. Truncation is recoverable, corruption usually requires going back to the source.

How do I reindex a recovered VCF?

Recompress it with bgzip, then run tabix -p vcf recovered.vcf.gz or bcftools index -t recovered.vcf.gz. Delete any old index first, and confirm that the index record count matches the number of data lines.

Is a truncated FASTQ harder to recover than a VCF?

Yes. FASTQ uses fixed four-line records, so a single lost line near the cut shifts every subsequent record and corrupts read identifiers and quality scores. VCF records are newline-delimited and independent, so only the final record is at risk.