# How to Check the Integrity of Compressed Bioinformatics Files Nearly every file in a sequencing run, a public repository download, or a pipeline working directory arrives compressed. FASTQ reads ship as `.fastq.gz`, alignments as BAM or CRAM, reference genomes as `.tar.gz`, and structural data in formats designed around compression. A single truncated or corrupted archive can waste hours of compute, produce silently wrong results, or break a pipeline at 3 a.m. when no one is watching. The most common cause of a compressed-file failure in bioinformatics is simple truncation, usually from an interrupted download or file transfer, and the first thing to check is a full-stream test with `gzip -t`. That command reads every compressed block and verifies the checksum embedded at the end of the file, which catches truncation and most bit-level corruption in seconds to minutes depending on file size. This article is a practical guide to checking the integrity of compressed bioinformatics files on Linux, ordered from the most common cause of failure to the rarest, with a diagnostic sequence you can follow, a troubleshooting table, concrete fixes, and prevention practices for pipelines. ## Why compression integrity matters in bioinformatics Compression is not optional in this field. A 30x human whole-genome FASTQ easily exceeds 100 GB uncompressed, and repositories could not store or transfer data at this scale without it. Compression is so central to how structural biology data is managed that purpose-built formats now compete directly with gzip. BinaryCIF, a serialization of the Crystallographic Information File format used for macromolecular structures, was introduced specifically because data size and complexity demanded more efficient storage and transfer than gzip-compressed text alone [1]. It reduces file sizes by more than a factor of two compared with gzip-compressed CIF, and for the largest structures it achieves compression factors of ten versus plain CIF and four versus gzipped CIF [1]. When a compressed file loses integrity, two things can go wrong, and they fail differently: - **Detection failure.** The decompressor stops mid-stream with an error. You know the file is bad, and the failure is usually loud. - **Silent corruption.** The file decompresses "successfully" but contains altered or reordered bytes. This is rarer with gzip because of its internal checksum, but it is the more dangerous failure because downstream tools may accept the file and produce quietly wrong results. Gzip's defenses are built into the format itself. A gzip file consists of a short header, a stream of DEFLATE-compressed blocks, and a trailer containing a 32-bit cyclic redundancy check (CRC32, a checksum computed over all decompressed bytes) and the uncompressed size modulo 4 GB (the ISIZE field). When `gzip` finishes decompressing, it recomputes the CRC32 from the output and compares it with the stored value. Any mismatch means at least one byte changed somewhere between compression and verification. The ISIZE field also lets you confirm that you got every byte the compressor produced, which is what makes truncation detectable. Two caveats about this built-in protection shape everything else in this guide. The CRC32 covers decompressed content, not the file on disk, so an attacker or a bad transfer that reconstructs a valid-looking stream can still pass, which is why external checksums matter. And the ISIZE field is stored modulo 2 to the power of 32, so `gzip -l` reports misleading sizes for any file larger than about 4 GB uncompressed, which describes most FASTQ files you will ever touch. ## The first check: gzip -t ![Decision path from the gzip -t exit code to truncation, bit corruption, trailing garbage or a clean pass, with the next step for each.](/article-images/illustrations/check-gzip-file-integrity-bioinformatics-gzip-triage.webp) The fastest way to check gzip file integrity in bioinformatics is the test flag. It reads the entire file, verifies the DEFLATE stream, and validates the CRC32 and size trailer, all without writing a single decompressed byte to disk: ```bash gzip -t sample_R1.fastq.gz echo $? ``` Exit code 0 means the file decompressed cleanly. Exit code 1 means a hard error, almost always truncation or corruption. Exit code 2 means a warning, most commonly trailing garbage after the last gzip member. The messages you will see most often are: - **unexpected end of file.** The stream ends before the trailer. The file was cut off, almost always by a failed transfer. - **an invalid compressed data error.** The DEFLATE bitstream contains impossible values or the CRC does not match. Bytes changed somewhere. - **decompression OK, trailing garbage ignored.** The gzip stream itself is fine, but extra bytes follow it. This happens when two `.gz` files were concatenated or something was appended to one. For large archives, run the test verbosely and time it so you know what a healthy check costs: ```bash time gzip -tvv big_dataset.tar.gz ``` Two refinements matter at bioinformatics scale. First, `gzip -t` is single-threaded, and on a 40 GB FASTQ it can take several minutes. The parallel implementation `pigz` (parallel gzip) supports the same test flag and saturates multiple cores: ```bash pigz -t sample_R1.fastq.gz ``` Second, `gzip -t` works on any conforming gzip stream, including block-compressed files produced by `bgzip` (Block GZ, the_bgzip_ tool from the HTSlib library used throughout the SAMtools ecosystem). It is a valid check, just slower than format-aware alternatives discussed later. ## What "integrity" actually means: three separate questions Before running diagnostics, separate three questions that get conflated constantly in practice. Answering the wrong one wastes the most time. **Question 1: Is the compressed stream intact?** This is what `gzip -t` answers. The DEFLATE stream is complete and the CRC32 matches. **Question 2: Is this file identical to the file at the source?** This is what checksums answer. An MD5 or SHA-256 digest (fixed-length fingerprints computed over every byte of the file) lets you compare your copy against a value published by the data provider or recorded when the file was first created. ENA and NCBI sequence read archives publish MD5 values alongside files precisely so you can do this. A checksum pass means the transfer did not change the file. A checksum fail means it did, and no amount of clever recovery will tell you which bytes changed. ```bash sha256sum sample_R1.fastq.gz md5sum sample_R1.fastq.gz ``` Compare against the manifest. Note that the checksum of the `.gz` file and the checksum of the decompressed content are different numbers, and providers usually publish the checksum of the compressed file. Always confirm which one a manifest refers to before assuming a mismatch means corruption. **Question 3: Is the content actually valid for its format?** A gzip file can pass every integrity check and still be useless, because gzip guarantees nothing about what was compressed. A truncated FASTQ can be gzip-recompressed afterward, or a file can be corrupted before compression ever happened. Conversely, integrity is necessary but not sufficient. Validation tools that parse the full format, discussed below, close this gap. A useful mental model: gzip integrity says the container is sound, checksums say the container traveled without change, and format validation says what is inside is real data. ## Common causes of gzip file corruption, most common first ### 1. Truncation from an interrupted transfer This is the dominant cause by a wide margin. A download drops at 80 percent, an Aspera or FTP session times out, a laptop sleeps, a batch copy loop hits a full disk, or an `scp` session dies when a VPN reconnects. The resulting file looks normal, has a plausible size, and often decompresses partially before reporting **unexpected end of file**. Because the DEFLATE trailer with the CRC32 sits at the very end of the file, any missing tail is detectable, and `gzip -t` catches it immediately. The mechanism is worth understanding. DEFLATE encodes data as a sequence of blocks, and gzip needs both the final block marker and the trailer to declare success. A truncated file ends mid-block. The decompressor hits the physical end of the input while expecting more bits, reports the unexpected end, and stops. The bytes it did manage to decode before truncation are usually intact, which is why partial recovery is often possible. ### 2. Disk full during writing A close second in pipeline settings. A job writes `sample_R2.fastq.gz` while the filesystem silently fills, the write fails partway, and depending on the software the file is left truncated, zero-padded, or missing entirely. Compression makes this worse because the compressed output is smaller than the input, which tempts people to budget space for the compressed size only, forgetting that the decompressor, temporary files, and indices all need room too. Always check with `df -h` before large transfers and confirm the target filesystem for the decompressed output, not just the compressed input. ### 3. Binary files damaged by text-mode transfers Old FTP clients default to ASCII mode, which translates line endings on the fly. On a binary file this is destructive, because it inserts or removes bytes according to a text convention, shifting everything downstream and guaranteeing a CRC failure. Modern `scp`, `rsync`, `sftp` in binary mode, and `wget` or `curl` do not do this, so this cause shows up mostly when older instruments, shared instrument computers, or legacy Windows tools are involved in the transfer chain. If a file fails with an invalid compressed data error immediately after coming from an instrument workstation or an old FTP path, check the transfer mode first. ### 4. Bit rot and failing storage Long-term archival disks, especially external drives and aging RAID members, develop unreadable sectors. A read that returns substituted bytes produces a CRC mismatch on decompression even though the file was perfect when it was written. The telltale sign is that one file on one disk fails while everything else passes, and re-reading produces the same failure. Check for I/O errors in the kernel log: ```bash dmesg | grep -i "i/o error" ``` For a drive you suspect, examine its self-monitoring data with `smartctl -a /dev/sdX` (from the smartmontools package). Handle suspected failing drives gently, copy what you can with `ddrescue`, and never write new data to a disk that has started returning errors. ### 5. Bad memory Faulty RAM corrupts bytes in transit through the machine. It is rarer than the causes above but distinctive: multiple unrelated files fail checksums, checksums of the same file differ between runs, and errors appear on different files each time. If corruption follows the machine rather than the file, run a memory test before blaming the network or the storage. ### 6. Concatenation and accidental modification Gzip supports multiple members, meaning several gzip streams can be concatenated and decompressed as one file, which some tools use deliberately. But an accidental `cat a.fastq.gz b.fastq.gz > ab.fastq.gz` produces a file that decompresses with trailing garbage warnings, and any file that was opened, edited, or re-saved by a text-mode editor or an email client arrives with altered bytes and fails outright. A related bioinformatics-specific mistake is recompressing a partially decompressed file and treating it as the original, which changes the checksum while keeping the content valid, and quietly loses data. ## A diagnostic sequence you can follow Work through this order. Each step either identifies the cause or rules one out. 1. **Record the exact error and exit code.** Run `gzip -t file.fastq.gz`, then `echo $?`. The distinction between exit 1 (error) and exit 2 (warning) sends you down completely different paths. Truncation says unexpected end of file. Bit corruption says invalid compressed data. Trailing garbage means the stream is fine but something follows it. 2. **Compare the file size against the source.** Check the byte size on the repository page, in the provider's file listing, or in a manifest. `ls -l file.fastq.gz` gives bytes. If your copy is smaller than the published size, you have truncation and you can skip straight to re-downloading. Remember the `gzip -l` caveat for files over 4 GB uncompressed, so use it on the compressed file listing only for orientation. 3. **Verify against the published checksum.** Run `md5sum` or `sha256sum` and compare with the manifest. A checksum pass combined with a `gzip -t` failure is a strange and telling combination: the bytes match the source, so the problem predates the transfer, such as a corrupted original on the server or a file that was recompressed after partial recovery. Report it to the data provider. 4. **Count the decompressed bytes or records.** For a sanity check that the readable portion matches expectations: ```bash zcat sample_R1.fastq.gz | wc -c # total decompressed bytes zcat sample_R1.fastq.gz | wc -l # total lines, divide by 4 for FASTQ reads ``` For paired FASTQ files, R1 and R2 must have identical line counts and record counts. A mismatch means one file is truncated or one was filtered independently. 5. **Inspect the tail of the compressed file.** `xxd file.fastq.gz | tail` shows the final bytes. A healthy gzip file ends with the CRC32 and size trailer. A truncated file often ends in compressed gibberish or, in bad disk-full cases, in a run of zero bytes. 6. **Check the storage and system.** `df -h` for space, `dmesg` for I/O errors. If several files on one volume fail, suspect the volume. 7. **Re-download or re-copy with verification built in.** `rsync -c` compares checksums before and during transfer, `wget -c` and `curl -C -` resume partial downloads, and most repository clients report checksums on completion. For data from the NCBI Sequence Read Archive, the `vdb-validate` tool that ships with the SRA Toolkit checks internal SRA structure. ## Troubleshooting table | Symptom | Likely cause | How to confirm | Fix | |--|--|--|--| | gzip reports unexpected end of file, exit code 1 | Truncated file from interrupted download or transfer | Compare `ls -l` byte count with the source listing or manifest | Re-download with a resumable client, then re-run `gzip -t` and the checksum | | Invalid compressed data error, CRC mismatch | Bit rot, failing RAM, or a text-mode transfer altered bytes | Re-run `sha256sum` and compare with manifest, check `dmesg` for I/O errors, check transfer mode | Re-download from source, replace failing hardware, force binary transfer mode | | Decompression OK, trailing garbage ignored, exit code 2 | Two gzip streams concatenated, or bytes appended after the stream | `zcat file.gz \| wc -c` against expected size, inspect tail with `xxd` | If intentional concatenation, nothing to fix. If accidental, split at the first stream boundary and keep the verified member | | gunzip succeeds but FASTQ parser or aligner fails immediately | File was corrupted before compression, or has wrong line endings | `zcat file.gz \| head` and check the four-line FASTQ structure, validate with a full parser such as FastQC or seqkit | Fix at the source, regenerate the file, correct line endings | | samtools reports the BAM EOF marker is absent | Truncated BGZF copy of a BAM or CRAM | `samtools quickcheck -v file.bam` | Re-transfer the file, or reconvert from the original if the copy on the source machine is intact | | tar archive stops extracting midway | Truncated `.tar.gz` or damaged member | `tar -tzf archive.tar.gz` lists entries and fails at the damage point | Re-download the complete archive, extract the readable prefix only if partial content is acceptable | | md5 or SHA-256 mismatch against manifest | File changed in transit or on disk | Recompute locally, confirm which file the manifest describes (compressed or decompressed) | Re-transfer with `rsync -c`, verify again, escalate to the provider if the source copy also fails | | File passes all tests but the pipeline fails intermittently | Disk filled mid-write, or a concurrent job overwrote the file | `df -h`, job logs, check for overlapping write jobs with `lsof` | Add disk-space checks to the pipeline, write to a temporary file and rename atomically on completion | ## How to fix a corrupted gzip file ![Branching repair path from a failed gzip test to re-download, prefix recovery, upstream rebuild, garbage stripping or hardware replacement.](/article-images/illustrations/check-gzip-file-integrity-bioinformatics-fix-path.webp) Fixes follow the diagnosis. Work from the cheapest option to the most invasive. **Fix 1: Re-download with resumable, verified transfer.** This is the correct fix for truncation, which is the majority of cases. Delete nothing until the replacement is verified: ```bash wget -c https://example.org/data/sample_R1.fastq.gz gzip -t sample_R1.fastq.gz md5sum sample_R1.fastq.gz # compare with the manifest ``` For large repositories, prefer a client that verifies automatically. `rsync -c` from a mirror that supports it compares checksums block by block and retransmits only the differences, which is dramatically faster than a full re-download when only a tail region is missing. **Fix 2: Recover the readable prefix.** When the original is gone and a re-download is impossible, a truncated gzip file often yields intact data up to the break. Decompress to a new file rather than overwriting anything: ```bash zcat sample_R1.fastq.gz > recovered.fastq 2> recovery.log ``` Then trim the output to complete records. A truncated FASTQ almost always ends mid-record, so the last one to three lines are partial and must be dropped before the file will parse. Count lines, and cut back to the largest multiple of four: ```bash total=$(wc -l < recovered.fastq) usable=$(( total / 4 * 4 )) head -n "$usable" recovered.fastq > recovered_clean.fastq ``` If R1 recovered cleanly and R2 did not, the surviving pairs must be matched before alignment. Keep only read IDs present in both files, using a tool such as `seqkit pair` or your pipeline's paired-read filter. Document that the file is a partial recovery, rename it so nobody mistakes it for a complete file, and record how many reads were lost. **Fix 3: Reconstruct from upstream.** In a pipeline, the corrupted file often has a parent. A damaged BAM may be re-derived from the intact FASTQs, a `.tar.gz` reference bundle from its component files, or a compressed result from an intermediate directory. Rebuilding from a known-good upstream stage is usually faster and safer than any forensic recovery, provided the parent was itself verified. This is the strongest argument for keeping checksums on every artifact a pipeline produces. **Fix 4: Strip trailing garbage when the stream itself is fine.** If `gzip -t` reports trailing garbage but the warning exit code 2 rather than an error, the decompressed content may be exactly what you need. Verify by comparing `zcat | wc -c` against the expected decompressed size, and if the numbers agree, truncate the file after the last valid gzip member or simply proceed while noting the warning. Do not do this blindly. Confirm the byte count first. **Fix 5: Replace failing hardware.** If corruption follows a disk, a cable, or a machine across multiple files, no per-file fix helps. Stop using the failing component, image what is readable with `ddrescue`, and re-derive everything else. Files recovered from failing media should be treated as suspect until they pass `gzip -t` and a checksum comparison. One caution that prevents data loss during recovery: never decompress over the only copy you have. `gunzip file.fastq.gz` deletes the `.gz` after writing the output, and if the disk fills during that write you can lose both. Keep the original compressed file until the recovered output has been verified, and check `df -h` before any large decompression. ## Format-aware checks beyond plain gzip Bioinformatics uses several compressed formats that plain gzip does not fully describe, and each has a faster or stricter check. **BGZF, BAM, and CRAM.** `bgzip` produces block-compressed gzip files that allow random access, and BAM files are BGZF with a binary alignment structure. The block structure means truncation can be detected at the block level. The standard check is: ```bash samtools quickcheck -v file.bam ``` This verifies the header and the BGZF end-of-file marker, which catches truncation in seconds even on terabyte-scale files. It is deliberately shallow, so for full validation, stream the file through a reader and confirm the record count against expectations. For CRAM, `samtools quickcheck` also works, and full validation additionally requires the reference against which the CRAM was encoded, so keep the reference FASTQ and its checksum together with CRAM files. CRAM files that cannot find their reference are indistinguishable from corrupted ones to a careless observer, and this causes a remarkable number of false alarms. **tar archives.** Test without extracting: ```bash tar -tzf reference_bundle.tar.gz > /dev/null && echo OK ``` This decompresses the whole stream and verifies the tar structure, catching both gzip truncation and internal damage. **zip.** `unzip -t archive.zip` tests every member's CRC. Zip is uncommon for large genomics data but common for small supplementary files from journals and collaborators. **zstd and other modern compressors.** Newer formats follow the same pattern of embedded checksums and test flags: `zstd -t file.zst`. If you adopt alternative compressors, build their test flags into your pipeline exactly as you would `gzip -t`. **Sequence Read Archive files.** The SRA Toolkit includes `vdb-validate`, which checks the internal structure of `.sra` files. Run it after every download before spending compute on the data. **Format-level validation.** After the container passes, validate the contents with a parser that reads every record. FastQC and seqkit both read complete FASTQ files and will fail on structural problems such as quality lines shorter than sequence lines. seqkit's `pair` command additionally verifies that R1 and R2 read IDs match in order. Run these once on ingest, record the results, and skip revalidation downstream. ## A Linux command reference for integrity checks The following Linux commands cover nearly every integrity question in a bioinformatics workflow. This is the core toolkit for keeping daily work reproducible. | Command | What it does | Practical note | |--|--|--| | `gzip -t file.gz` | Full-stream test, verifies CRC32 and trailer, writes nothing | Exit 0 pass, 1 error, 2 warning | | `gzip -tvv file.gz` | Same test with verbose output | Use on first ingest of any file | | `pigz -t file.gz` | Parallel version of the test | Use for files over a few gigabytes | | `zcat file.gz \| wc -c` | Decompressed byte count | Compare against expected size or manifest | | `gzip -l file.gz` | Reports compressed and uncompressed sizes | Uncompressed size is modulo 4 GB, misleading for big files | | `md5sum file.gz` | 128-bit digest of the file on disk | Match against provider manifests | | `sha256sum file.gz` | Stronger 256-bit digest | Prefer this for long-term archival verification | | `samtools quickcheck file.bam` | Header and EOF-marker check for BAM and CRAM | Fast first-pass check, not a full validation | | `tar -tzf archive.tar.gz` | Lists archive contents, verifying the full stream | Redirect to `/dev/null` for a quiet pass or fail | | `unzip -t archive.zip` | Tests every member in a zip archive | Standard for supplementary file bundles | | `zstd -t file.zst` | Full test for zstandard files | Same pattern for other modern compressors | | `df -h` | Free space per filesystem | Check before transfers and decompressions | | `dmesg \| grep -i error` | Kernel messages, reveals disk I/O failures | First stop when many files fail at once | | `rsync -c src dst` | Checksum-verified transfer | Also retransmits only differing blocks on retry | | `wget -c` / `curl -C -` | Resumable downloads | Avoids repeated full re-downloads after interruptions | ## Verification and prevention in pipelines Checking files by hand works for one download, but reproducible pipelines need verification that runs automatically at defined points. The principles are simple and the payoff is large. **Verify at ingest, once.** Every file that enters the system gets a `gzip -t` (or format equivalent) and a checksum comparison against the manifest. Record the checksums you compute in a manifest of your own. From that point forward, any file whose checksum matches its recorded value is known-good, and downstream failures point at software, not data. **Write outputs atomically.** The disk-full scenario, where a pipeline leaves a truncated output that later looks like input corruption, is preventable. Write each output to a temporary name, run the integrity test on it, and rename it to the final name only after the test passes. A rename within the same filesystem is atomic, so a file either appears complete or does not appear at all. Workflow engines such as Snakemake and Nextflow encourage exactly this pattern, and adding an explicit test step costs one line. **Keep checksums with data.** Store a checksum file next to every data directory, and update it whenever contents change. This is the practice that makes archive integrity verifiable years later, and it converts the question "is this file still good?" from an expensive decompression into a one-second digest comparison. **Use parallel testing at scale.** Testing thousands of files serially wastes cores. `pigz -t` parallelizes each file, and `find` with `parallel` parallelizes across files: ```bash find fastq/ -name '*.fastq.gz' | parallel pigz -t {} ``` **Log test results, not just failures.** A pipeline log that records the exit code of every integrity test lets you distinguish "never tested" from "tested and passed" months later, which matters enormously when auditing results or chasing an intermittent discrepancy. Compression choices themselves benefit from the same discipline. The structural biology community's move toward purpose-built compressed formats shows where the field is heading, and the CIFTools libraries provide generic and typed handling of CIF and BinaryCIF files in Java and TypeScript, giving developers a supported path for reading and writing these formats [1]. Whatever format a pipeline adopts, the verification pattern is identical: test the container, checksum the file, and validate the content. ## Common Mistakes and Limitations - **Trusting file size alone.** A truncated file can still be within a few percent of the expected size and look completely normal in a directory listing. Only a full-stream test or checksum comparison is definitive. - **Reading only the head of the file.** `zcat file.gz | head` proves the file starts correctly and nothing more. The trailer sits at the end, and truncation hides at the end. Any check that stops early is not a check. - **Using `gzip -l` on large files.** The uncompressed size it reports is stored modulo 4 GB, so a 10 GB decompressed FASTQ can report an absurdly small size. Use `zcat | wc -c` instead. - **Confusing the compressed and decompressed checksums.** Providers publish checksums for the compressed file. Comparing your `md5sum` of the decompressed output against a compressed-file manifest will always mismatch, and this false alarm sends people hunting for corruption that does not exist. - **Assuming a gzip pass means valid data.** gzip verifies its own stream, not the semantics of what was compressed. A recompressed, truncated FASTQ can pass `gzip -t` and still be missing reads. Format-level validation is a separate and necessary step. - **Overwriting the only copy during recovery.** Decompression that replaces the original compressed file, combined with a mid-write failure, can destroy both versions. Verify recovered output before discarding the original, always. - **Ignoring exit code 2.** Trailing garbage warnings mean the stream decompressed but something unexpected follows it. Treating this as a clean pass, without checking the decompressed byte count against expectations, risks working with a file that was modified after creation. - **Blaming the file when the disk or memory is the problem.** Repeated checksum failures across unrelated files on one machine indicate hardware. Fixing files one by one on failing hardware is a treadmill. Diagnose the machine first. - **A note on limits of file-level checks.** Integrity checking cannot detect errors that were present in the original data before compression, such as barcode cross-contamination introduced at the sequencer or mislabeled samples. Checksums and gzip tests verify that bytes traveled intact, not that the experiment behind the bytes was sound, and interpreting biological results requires domain-appropriate quality control beyond file integrity. ## Frequently Asked Questions ### How do I test a gzip file without decompressing it to disk? Run `gzip -t file.fastq.gz`. The test flag reads and verifies the entire stream, including the CRC32 checksum and size trailer, without writing any decompressed output. Check the exit code with `echo $?`, where 0 means the file is intact, 1 indicates a hard error such as truncation, and 2 indicates a warning such as trailing garbage. ### What does the "unexpected end of file" error in gzip mean? It means the compressed stream ends before the gzip trailer, which is the signature of a truncated file. This almost always results from an interrupted download, transfer, or disk write. Confirm by comparing your file's byte count with the source listing, then re-download with a resumable client such as `wget -c` and re-run the test. ### Can I recover reads from a truncated FASTQ.gz file? Usually yes. `zcat truncated.fastq.gz > recovered.fastq` extracts everything up to the break point, then trim the output to a whole number of four-line FASTQ records with `head -n` after counting lines. If one mate of a pair survived and the other did not, filter both files to matching read IDs before alignment, and document that the file is a partial recovery. ### Why does gzip -t pass but my downstream tool still fail? `gzip -t` verifies the compression container, not the content's format. The file may have been corrupted before compression, may have the wrong line endings, or may have been recompressed after partial recovery. Run a format-aware validator such as FastQC or seqkit on FASTQ files, and `samtools quickcheck` on BAM and CRAM files. ### How do I check the integrity of a BAM or CRAM file? Run `samtools quickcheck -v file.bam`, which verifies the header and the BGZF end-of-file marker in seconds and catches truncation. For thorough validation, stream the entire file through samtools and confirm record counts. CRAM files additionally require their reference genome to be available, so keep the reference and its checksum alongside the CRAM. ### Why do md5sum and gzip -t disagree about my file? They measure different things. `md5sum` compares your file, byte for byte, against the value recorded at the source, while `gzip -t` verifies that the internal stream is self-consistent. A file that matches its checksum but fails `gzip -t` was likely corrupted before it was compressed or on the server itself. A file that passes `gzip -t` but fails its checksum was modified after compression, for example by recompression. ### What is the fastest way to test thousands of large FASTQ files? Use `pigz -t`, the parallel gzip implementation, which uses all available cores per file, and run it across files with GNU parallel: `find fastq/ -name '*.fastq.gz' | parallel pigz -t {}` This combination tests many large files in a fraction of the time serial `gzip -t` requires, which makes it practical to verify every file at ingest rather than only the ones that fail. ### How do I tell a bgzip file from a regular gzip file, and does it matter? The `htsfile` command from HTSlib identifies BGZF files and distinguishes them from plain gzip. A bgzip file is a valid gzip file, so `gzip -t` works on it, but bgzip files contain indexed blocks that enable random access, and tools such as samtools and tabix rely on that structure. It matters because truncation in a bgzip file is best detected with `samtools quickcheck`, and because re-compressing a bgzip file with plain gzip destroys the block structure that indexed access depends on. ## Sources 1. [BinaryCIF and CIFTools-Lightweight, efficient and extensible macromolecular data management.](https://pubmed.ncbi.nlm.nih.gov/33075050/)