# Troubleshooting Slow or Interrupted Genomics Data Transfers Slow or interrupted genomics data transfers usually come down to one of two things: a single TCP stream that cannot fill a long-distance network path, or a storage system at one end that cannot feed the network in the first place. The first thing to check is also the simplest. Measure the actual transfer rate, compare it to the slowest link in the path, and find out which side is the [limiting factor](/blog/guides/limiting-factor-definition-types-and-examples) before you change anything. This guide walks through genomics data transfer slow troubleshooting causes in order, from the most common to the rarest. Each section explains the mechanism behind the failure, how to confirm it, and how to fix it. The core of the article is a troubleshooting table you can scan in under a minute, followed by detailed sections on every cause, a complete diagnostic sequence, and prevention practices for bioinformatics [data management](/blog/guides/data-management-basics-principles-processes-and-best-practices) that keep the problem from coming back. ## Why Genomics Transfers Are Different From Ordinary File Moves Genomics workloads stress networks and storage in ways that everyday file copying does not. Three properties of genomics data drive most of the trouble. First, the files are enormous. A single 30x whole-genome sequence for one human produces FASTQ files in the tens of gigabytes and BAM or CRAM files around 50 to 100 GB. A joint cohort VCF for a few thousand genomes can reach multiple terabytes. Moving these files at even 100 MB per second takes ten to twenty minutes each, and a typical sequencing run involves many of them. Second, the files often come in large numbers. Demultiplexing a flowcell produces hundreds of individual FASTQ files. Downloading raw reads from the NCBI Sequence Read Archive (SRA) or from a European Nucleotide Archive mirror often means fetching thousands of small objects spread across many accession directories. Small-file workloads punish high-latency networks because every file pays the round-trip cost of opening a connection. Third, the transfers are usually long-distance. A collaborator in another country, an SRA endpoint on the opposite coast, or a cloud object storage bucket in a different region means round-trip latencies of 50 to 150 milliseconds. That latency interacts badly with the default behavior of TCP, the protocol underneath SSH, HTTPS, and most transfer tools. Understanding these three properties turns most troubleshooting from guesswork into a short checklist. The rest of this article applies them. ## Start Here: The First Fifteen Minutes Before touching any configuration, gather four facts. They take minutes and they eliminate half the possible causes. 1. Measure the observed rate. Watch the transfer tool's progress output, or sample the file size in the destination directory twice a minute and divide the difference. Write down the number in MB per second. 2. Compare it to the local link speed. A 1 Gbps Ethernet link tops out near 110 to 118 MB per second in practice. A 10 Gbps link tops out near 1.1 GB per second. If your observed rate already matches one of these ceilings, the network is not your problem and you can stop blaming it. 3. Compare it to the smallest disk. A single spinning hard drive sustains roughly 150 to 200 MB per second for large sequential reads. If your transfer rate is below that and both endpoints use spinning disks, look at the storage before the network. 4. Check whether the transfer is one big file or many small files. A rate of 5 MB per second on a single 90 GB FASTQ points to a network or TCP problem. A rate of 5 MB per second across 4,000 tiny files points to per-file overhead and has a completely different fix. These four facts tell you whether to work on the network path, the TCP configuration, the storage system, or the file layout. The rest of the sections assume you have this baseline. ## The Core Troubleshooting Table Use this table to jump from what you observe to the most likely cause. The detailed sections below explain each row. | Symptom | Likely cause | How to confirm | Fix | |--|--|--|--| | Single large file crawls at a fixed low rate over a long distance, faster on short paths | Single TCP stream limited by bandwidth-delay product | Rate scales with shorter distance, iperf3 single-stream test shows same low rate | Use a parallel or UDP-based transfer tool (Globus, Aspera, or parallel rsync streams) | | Thousands of small files transfer at a tiny fraction of link speed | Per-file protocol overhead | Rate improves dramatically when files are bundled with tar first | Bundle small files into an archive, or transfer a directory with many parallel workers | | Transfer rate far below disk speed on both ends, high iowait | Storage backend cannot feed the network | iostat or iotop shows the disk at 100 percent, network interface below capacity | Fix storage: stop competing jobs, stripe files across filesystems, use local scratch | | scp or sftp stuck below 20 MB per second even on a fast local network | SSH encryption bottleneck on one CPU core | Check one sshd process at 100 percent CPU during transfer | Use rsync over SSH with a faster cipher, or use a different transfer tool | | Rate drops over time, then the session dies after a while | Idle or session timeout on a firewall or NAT device | Transfers die at a similar elapsed time repeatedly | Keep-alive settings in SSH, or a transfer tool designed for long sessions | | Transfer dies mid-file with a connection reset or timeout | Packet loss or an unstable network path | mtr or ping shows loss along the route | Fix the path, reduce concurrency, or use a tool that retransmits gracefully | | Transfer dies at the same file every time | Corrupt source file, permission problem, or quota | Try that file alone, check destination free space and permissions | Remove or replace the bad file, free space, fix permissions | | Rate fine at first, then collapses after several hours | Sustained I/O exhaustion, cache depletion, or throttling by the remote service | Pattern repeats across days and endpoints | Reduce concurrency, schedule transfers, contact the remote provider | | Destination disk fills and the transfer aborts | Insufficient space or quota at destination | df -h at the destination | Free space or transfer to a larger volume, verify quota | | Intermittent authentication failures kill long jobs | Credential or token expiry mid-transfer | Transfer dies at a token lifetime boundary, not at a data boundary | Use a transfer service with managed credentials or refresh tokens | ## Cause 1: Single-Stream TCP Throughput Limits ![Worked example: a 1 Gbps path with 80 ms round-trip time needs about 10 MB in flight, and a 1 MB buffer caps the stream near 12.5 MB per second.](/article-images/illustrations/genomics-data-transfer-slow-troubleshooting-tcp-window-calc.webp) This is the most common cause of a slow single large file over a long distance, and the mechanism is worth understanding because it explains why the same transfer is fast on a local network. TCP controls how much unacknowledged data can be in flight at once. Throughput is bounded by the amount in flight divided by the round-trip time. On a path with 80 milliseconds of round-trip time, a single TCP stream needs several megabytes of data in flight continuously to sustain 1 Gbps. Default TCP buffer sizes on many systems limit a single stream to far less, so the stream spends much of its time waiting for acknowledgments instead of sending data. The result is a transfer rate that looks fine on a local network with sub-millisecond latency and collapses to a small fraction of link speed over a continental or intercontinental path. You can see the math in a worked example. A path with 1 Gbps of capacity and 80 ms of round-trip time requires roughly 10 MB of data in flight to keep the pipe full (1 Gbps times 0.08 seconds divided by 8 gives 10 MB). If the sender's buffer only allows 1 MB in flight, the ceiling is about 100 Mbps, or roughly 12.5 MB per second, no matter how idle the network is. **How to confirm it.** Run a single-stream test with iperf3 between the two endpoints. If the single-stream rate is low while a multi-stream test (iperf3 -P 8) is several times higher, you have confirmed the diagnosis. The rate also tends to change dramatically with distance, being fast to a nearby host and slow to a distant one. **How to fix it.** The practical fixes, in order of effort: - Use a transfer tool designed for long fat paths. Globus, the service built for research data, manages parallelism, tuning, and recovery for you and is the standard recommendation for moving large scientific datasets between institutions. Commercial UDP-based tools such as IBM Aspera (the FASP protocol) achieve similar results where licensed. - Run several rsync streams in parallel over different subsets of files. Even four to eight concurrent streams often multiply throughput, because each stream has its own buffer. - Tune TCP buffer sizes on the endpoints (net.ipv4.tcp_rmem and net.ipv4.tcp_wmem on Linux, plus net.core.rmem_max and wmem_max). Modern Linux kernels auto-tune reasonably well, so do this only after measuring. ## Cause 2: Many Small Files Instead of Few Big Files The second most common cause looks identical from the outside, a slow transfer, but the mechanism is completely different and the fix is different too. Every file transferred over SSH, SFTP, or HTTPS pays a fixed cost for setup: connection establishment, authentication, directory traversal, and file creation. That cost is measured in round trips. Move one 10 GB file and the setup cost is invisible. Move 10,000 files of one megabyte each and the setup cost dominates, especially over a path with 100 ms of latency. Transfer tools that iterate a directory sequentially can drop to a rate measured in kilobytes per second even on excellent networks. Genomics hits this constantly. A demultiplexed run directory, a downloaded SRA accession with per-lane files, or a single-cell analysis output tree with tens of thousands of fragments all trigger it. **How to confirm it.** Check the file count and size distribution in the directory you are transferring. If the median file is under 50 MB and the count is in the thousands, per-file overhead is almost certainly the problem. Another tell: the transfer progresses in bursts, with long pauses between short spurts of activity. **How to fix it.** Bundle before you send. A single tar archive transfers at network speed because the per-file cost is paid once at each end: ``` tar -cf - run_dir/ | pigz -p 4 > run_dir.tar.gz ``` Then transfer that single file, and extract it at the destination. Compression helps here because FASTQ and text formats compress well, typically to a third of their raw size, though already-compressed formats like BAM and bgzipped FASTQ gain little. For recurring directory transfers, modern rsync versions handle small files better than scp, and tools like Globus handle large directory trees with parallelism and batching automatically. On object storage, upload a bundled archive or use a tool that uploads objects concurrently rather than one at a time. ## Cause 3: The Storage System at Either End A network transfer is a chain: source disk, source filesystem, source network stack, destination network stack, destination filesystem, destination disk. The chain runs at the speed of its slowest link, and in genomics the slowest link is very often a disk, not the network. Several storage failure modes matter in practice. **A single spinning disk cannot keep up.** At 150 to 200 MB per second sequential, one hard drive saturates a 1 Gbps link and is the bottleneck for anything faster. If the source or destination is a single large RAID volume under heavy use, the transfer inherits whatever I/O pressure the disk already has. **Metadata operations throttle small files.** Network filesystems such as NFS pay a round trip for metadata operations like stat and open. Copying tens of thousands of small files onto NFS can take far longer than copying the same bytes as one archive. High-performance parallel filesystems such as Lustre or GPFS avoid this, but they require that files be striped across many storage servers. A single striped file on a Lustre filesystem with one stripe writes at the speed of one server. Increasing the stripe count for large files can multiply write throughput, and this is a standard practice on institutional HPC systems. **Object storage has per-request costs and rate limits.** Cloud object storage charges latency per request and throttles bursts. A single-stream upload of one huge file may use only a fraction of the bucket's capacity, while properly concurrent multipart uploads use far more of it. **How to confirm it.** Run iostat -x 5 on the endpoint in question during a transfer. If the storage device shows sustained 100 percent utilization while the network interface sits idle, storage is the bottleneck. Compare the transfer rate to a simple local benchmark such as writing a large file with dd to the destination filesystem. **How to fix it.** Stop competing I/O-heavy jobs during critical transfers, use local scratch space as a staging area before moving data to slower tiers, stripe large files appropriately on parallel filesystems, and bundle small files before sending them to NFS or object storage. ## Cause 4: SSH Encryption Capping Your Speed When you copy with scp or sftp, every byte passes through a single-threaded encryption engine inside one SSH process. On older CPUs, or with certain cipher choices, that single core tops out around 20 to 80 MB per second. The result looks like a network problem but is a CPU problem on one end. **How to confirm it.** While a transfer is running, check the sshd process on the server side. One process pinned at 100 percent of a single core during a transfer that is well below link speed points at the encryption bottleneck. If the same pair of hosts moves data at full speed with a non-SSH tool, the diagnosis is certain. **How to fix it.** Three options, in order of preference: - Choose a faster cipher. On hardware with AES acceleration, aes128-gcm or aes128-ctr is typically much faster than older defaults. Test with the command ssh -c aes128-gcm@openssh.com followed by your usual rsync or scp invocation. - Use rsync over SSH rather than scp. rsync reduces the bytes transferred for incremental updates and, unlike old scp, avoids protocol-level inefficiencies that matter on large files. - For pure bulk movement, step outside SSH entirely with Globus or a parallel HTTP transfer where the destination supports it. A note on compression: enabling SSH compression helps only when the data is compressible and the network is the bottleneck. On fast local networks, compressing incompressible data like BAM wastes CPU and slows the transfer. ## Cause 5: Network Path Problems Beyond capacity and latency, the path itself can damage transfers in three ways. **Packet loss.** TCP interprets loss as congestion and cuts its sending rate sharply, then recovers slowly. A path with even a small amount of chronic loss behaves far below its rated capacity for a single stream. Genomics transfers are sensitive to this because they are long-lived. Run mtr to the destination during a slow transfer and look for loss that persists at intermediate or final hops. Persistent loss inside the destination network is often not yours to fix, which is another reason to prefer a transfer tool that handles recovery itself. **MTU and fragmentation problems.** The maximum transmission unit is the largest packet a link will carry. A mismatched MTU somewhere along a path, common with tunnels and VPNs, causes large packets to be dropped while small ones pass. The classic symptom is a connection that opens and authenticates fine but hangs or crawls when moving bulk data, because authentication packets are small and data packets are large. Test by pinging with the don't-fragment flag set at successively smaller packet sizes. **Unstable links.** Wi-Fi, VPN tunnels, and congested shared connections drop long-lived sessions. Genomics transfers that take hours are exposed to all of them. The fix is structural: use a wired connection for bulk moves, and use a transfer tool that resumes rather than restarts (see the interrupted transfer section below). ## Cause 6: Shared or Saturated Links and Scheduled Traffic A link can be healthy and still be full. On institutional networks, backups, replication jobs, and other researchers' transfers compete for the same uplink. Your transfer may be fast at night and crawl at midday, which is a strong sign of contention rather than a configuration fault. **How to confirm it.** Repeat the same single-stream iperf3 test at two different times of day. If the rate swings several-fold, contention is the answer. Ask your institution's network group whether a science DMZ, a network architecture that routes bulk research traffic around general-purpose firewalls and shared links, exists on your campus. Many research institutions deploy one precisely to avoid this problem, and it is the standard recommended architecture for moving large scientific datasets. **How to fix it.** Schedule bulk transfers for off-peak hours, coordinate with the network group before increasing parallelism (aggressive parallelism from many users at once makes shared congestion worse), and prefer managed transfer services that queue and schedule work rather than raw ad hoc copies. ## Cause 7: Interrupted Transfers and Why They Happen ![Decision path for interrupted transfers: same file every time, same elapsed time, or random point, each leading to a different cause and fix.](/article-images/illustrations/genomics-data-transfer-slow-troubleshooting-interruption-triage.webp) Interruptions have their own set of causes, mostly distinct from slowness, and identifying the specific cause determines the fix. **Idle timeouts.** Firewalls and NAT devices drop connections they consider idle. A long transfer over a slow link can have gaps in activity long enough to trigger a timeout, especially during the handshake and metadata phases of many-small-file transfers. In SSH, set ServerAliveInterval to send keepalives, for example ssh -o ServerAliveInterval=60 -o ServerAliveCountMax=10 at the client. **Credential expiry.** Cloud session tokens and temporary credentials expire on a timer. A multi-hour upload that dies at a consistent elapsed time, regardless of how much data has moved, is almost always a token problem. The fix is to use a transfer service that refreshes credentials automatically, or to split the work into chunks shorter than the token lifetime. **Destination problems.** A full disk, an exceeded quota, a permission error, or a corrupted source file stops a transfer at the same file every time. Check destination free space with df -h and attempt the failing file on its own. **System events.** Laptops sleeping, Ethernet power management, DHCP lease renewals on some networks, and reboots all kill long sessions. Nothing on the network tier fixes a sleeping laptop. Run long transfers from a server, not a laptop that will close its lid. ## Fixing Interrupted Transfers Without Losing Work The worst part of an interrupted 80 GB transfer used to be starting over. That is avoidable, and the fix belongs in your standard workflow rather than in emergency response. rsync is the primary tool, because a rerun resumes where the last attempt stopped and verifies what it already copied: ``` rsync -avP source_file remote_host:/destination/ ``` The -P flag combines resume support with a progress display. Rerunning the same command after any interruption continues from the partial file rather than restarting. Note that rsync checks size and modification time by default, so for a strict integrity check of completed data you should run an explicit checksum pass (described in the next section). On object storage, use multipart upload tools, which split a large object into independent parts that can be retried individually. The AWS CLI, for example, uploads large files in parts and retries failed parts, and multipart behavior is the reason a single interrupted part does not doom an entire multi-terabyte object. For recurring institutional transfers, a managed service such as Globus provides automatic checkpointing and restart as a built-in feature, along with notifications when a transfer finishes or fails. This is the recommended approach for scheduled, unattended, or overnight moves, because no human is present to rerun a command when a session dies at 3 a.m. The one habit to avoid: deleting the partial destination file after a failure. Almost every resumable tool needs that partial file as its starting point. ## Verifying Integrity After a Transfer Slow transfers are annoying. Silent corruption is far worse, because a single flipped byte in a FASTQ or BAM can invalidate downstream analysis in ways that are difficult to trace. Every completed transfer of genomics data should end with a verification step. Checksums are the standard method. Generate a checksum on the source before or during transfer, then generate it again on the destination and compare: ``` md5sum file.fq.gz > file.fq.gz.md5 # on the source md5sum -c file.fq.gz.md5 # on the destination, after transfer ``` SHA-256 (sha256sum) is an equally valid and more modern choice. Many repositories, including the NCBI SRA, publish MD5 checksums alongside downloadable files precisely so that downloaders can verify integrity, and this is standard practice across archival repositories. Two practical points. First, checksumming a 100 GB file takes minutes, so for very large or very frequent transfers, checksum verification adds real time and should be planned into the workflow rather than skipped under pressure. Second, checksums verify that the destination matches the source. They do not detect corruption that existed at the source before the transfer, so fix problems at the origin before propagating them. As a functional check, run the first downstream step, such as fastqc or samtools quickcheck, on a freshly transferred file before deleting the source copy. samtools quickcheck is designed to catch truncated BAM files quickly. ## A Complete Diagnostic Sequence When a transfer is slow or failing and the table has not pinpointed the cause, work through this sequence in order. Each step either identifies the culprit or eliminates a layer. 1. Record the observed rate and compare it to the local link speed and the disk speed at both ends. If it matches one of those ceilings, the bottleneck is that component, not the network path. 2. Determine the file profile. One big file or many small ones. This splits the problem space in half. 3. Test the raw path with iperf3, single stream first, then with multiple parallel streams. A large gap between the two confirms the single-stream TCP limit. A low rate in both confirms a path or contention problem. 4. Check the path with mtr during a transfer. Persistent loss or a huge latency jump at a specific hop localizes the fault. 5. Check CPU on both endpoints during a transfer. One saturated sshd or encryption process indicates the SSH bottleneck. 6. Check I/O with iostat -x 5 at both endpoints. A device at 100 percent utilization while the network idles moves the investigation to storage. 7. Check destination health: df -h for space, permissions on the target directory, and any quota systems in force. 8. If the transfer dies rather than runs slow, note where and when it dies. The same file every time means data or permission trouble. A consistent elapsed time means timeouts or credential expiry. A random point means an unstable path or system event. 9. Apply one fix at a time and remeasure. Changing three things at once makes it impossible to know which one mattered. 10. Once a fix works, write it into your lab's transfer runbook, because genomics data transfer problems recur on the same paths with the same causes. ## Common Mistakes and Limitations Avoiding a handful of recurring errors will save more time than any single advanced tuning trick. **Blaming the network before measuring.** The most common mistake in troubleshooting slow or interrupted genomics data transfers is assuming the network is slow when the disk, the SSH cipher, or the file layout is the real limiter. Measure before you change anything. **Using scp for bulk genomics data.** scp is fine for a small config file and poor for a 90 GB BAM. It lacks resume support, uses a single stream, and cannot verify incrementally. rsync with -P or a managed transfer tool belongs in every genomics workflow. **Deleting partial files after a failure.** Rerunning a resumable transfer without the partial file wastes everything already moved. Keep partials until the completed transfer is checksum-verified. **Confusing compression wins with compression losses.** FASTQ and VCF text compress dramatically, so compressing them in transit is worthwhile. BAM, CRAM, and bgzipped files are already compressed, and in-flight compression just burns CPU. Match the technique to the format. **Maxing out parallelism.** Pushing dozens of concurrent streams across a shared institutional link degrades everyone's performance, including your own, and some remote providers throttle or block aggressive clients. Increase concurrency until the rate stops improving, then stop. **Trusting a completed transfer without verification.** A transfer that returns without an error is not the same as a transfer that arrived intact. Checksum before deleting the source copy. **A limitation of general guidance.** Every environment differs in hardware, filesystems, network policies, and security controls, and the values in this article are typical rather than universal. Measure your own endpoints, and when a fix requires infrastructure changes such as TCP tuning or a science DMZ, work with your institution's system administrators and network group rather than changing shared systems alone. ## Prevention: Bioinformatics Data Management Practices That Keep Transfers Fast Most slow transfer problems are easier to prevent than to fix. The practices below form the backbone of sound bioinformatics data management and genomics data storage planning. **Design file layouts for transport, not just for analysis.** Keep demultiplexed outputs grouped into one archive per sample or per run before they leave the sequencer host. Analyses can read from archives or extract on demand, and a layout that bundles well also transfers well. **Stage through fast local scratch.** For pipelines that read remotely or write remotely, copy data once to fast local storage, process there, and copy results back in bulk. Repeated small reads and writes across a network are the slowest possible pattern. **Standardize on one managed transfer service.** A single institutionally supported path, whether Globus or an equivalent, gives you automatic resume, notifications, checksum options, and shared credentials for collaborators. Ad hoc tool choices scattered across a lab guarantee that every interruption is handled differently and some are handled badly. **Keep a transfer runbook.** Record, for each frequent collaborator or repository, the endpoints, the tested command, the achievable rate, and the verification step. When a transfer underperforms, the runbook tells you in one glance what normal looks like for that path. **Plan capacity on the destination before large arrivals.** A 200 GB arrival that fills a volume and aborts at 190 GB is a storage planning failure, not a network failure. Check free space and quotas as part of scheduling any large inbound transfer. **Verify and then retire.** Make checksum verification an explicit, recorded step, and delete source or staging copies only after verification passes. This closes the loop and prevents both silent corruption and unbounded storage growth from never-deleted staging data. **Monitor the recurring paths.** A monthly scheduled test transfer to each frequent endpoint, logged with its rate, turns a silent degradation into a visible trend and gives you evidence to take to your network group before it becomes a crisis. ## Frequently Asked Questions ### Why is my genomics data transfer so slow? The most common cause is a single TCP stream that cannot fill a long-distance network path because of latency and buffer limits. The second most common is transferring thousands of small files, where per-file overhead dominates. Measure the actual rate against your link speed and disk speed to find which layer is limiting you. ### How do I fix a slow transfer of one large BAM or FASTQ file? Switch from scp to a tool built for long paths. Globus or a UDP-based commercial transfer tool handles parallelism and tuning automatically. Alternatively, run several rsync streams in parallel or tune TCP buffer sizes on both endpoints. Confirm the diagnosis first with a single-stream versus multi-stream iperf3 test. ### Why do thousands of small files transfer so much slower than one big file? Every file pays a fixed round-trip cost for connection setup, authentication, and creation, and that cost multiplies across thousands of files. Bundle the directory into a single tar archive, transfer the archive, and extract it at the destination, or use a transfer tool that batches and parallelizes directory copies. ### How can I resume an interrupted transfer without starting over? Rerun the same rsync command with -P, which resumes from the partial file at the destination. Keep the partial file in place, because resumable tools need it. For object storage, multipart upload tools retry only the failed parts. Managed services checkpoint transfers automatically. ### How do I verify that my transferred genomics files are intact? Generate an MD5 or SHA-256 checksum on the source, transfer the checksum file alongside the data, and verify with md5sum -c or sha256sum -c at the destination. For alignment files, samtools quickcheck catches truncated BAMs quickly. Verify before deleting any source copy. ### Should I compress genomics files before transferring them? Compress plain-text formats such as FASTQ and VCF, which shrink dramatically and speed up the transfer. Do not bother compressing BAM, CRAM, or bgzipped files, which are already compressed, because in-flight compression wastes CPU on both ends and can slow the transfer. ### Why does my long transfer die at roughly the same time every night? A consistent elapsed time to failure points to a timeout or credential expiry rather than a data problem. Idle firewalls kill quiet sessions, which keepalives fix in SSH, and cloud tokens expire on a timer, which calls for a tool that refreshes credentials or a managed transfer service. ### When should I contact my institution's network team about slow transfers? Contact them when repeated measurements show a persistent bottleneck on the institutional path, when contention at certain times of day limits your work, or when a fix would require infrastructure changes such as TCP tuning, firewall exceptions, or a science DMZ. Bring your measurements, including iperf3 and mtr results, because they turn a complaint into a workable ticket.