# Data Transfer Best Practices for Large-Scale Genomic Datasets By the end of this guide you will be able to move a multi-terabyte sequencing dataset from a sequencer, a lab server, or a public archive into a cloud storage bucket, verify that every file arrived byte-for-byte intact, attach the metadata that makes the dataset findable and reusable, and keep the transfer inside your security and budget boundaries. The workflow uses standard command-line tools: AWS CLI, rclone, md5sum, and the SRA Toolkit. To follow the walkthrough you need a Linux or macOS shell (or Windows Subsystem for Linux), roughly 200 GB of free scratch space for testing, an account on your target cloud platform with permission to create storage buckets, and the AWS CLI v2 or rclone installed. If your data originates from the Sequence Read Archive (SRA), the SRA Toolkit lets you pull samples programmatically, which is how pipelines such as GeniePool continuously retrieve raw human next-generation sequencing data from SRA and preprocess them before loading a cloud data lake [1]. ## Why Genomic Transfers Fail at Scale A single 30x human whole genome produces roughly 100 GB of FASTQ output. A 2,000-sample study therefore moves around 200 TB, and population-scale projects operate well beyond that. Celeste, the cloud infrastructure behind the All of Us sequencing program, processes fluctuating workloads of up to 9,000 whole-genome samples per month inside a single AWS Region, coordinating roughly 200 million serverless function invocations and over 20 million messages per year of production operation [2]. At that scale, a transfer strategy based on dragging folders through a browser window collapses. Three properties of genomic data make transfer planning different from ordinary file backups. First, files are enormous and immutable, so multipart upload and parallel streams matter more than incremental syncing. Second, every byte is scientifically load-bearing, which means silent corruption during transfer destroys the value of the entire downstream analysis. Third, human genomic data is legally and ethically sensitive, and the legal questions around cloud genomics concentrate on data control, data security, confidentiality and transfer, and accountability, points that researchers should settle by reading a provider's Terms of Service before uploading anything [3]. Data transfer best practices for large-scale genomic datasets address all three properties at once. ## What You Need On Hand Before starting, confirm the following: 1. A complete inventory of source files with sizes and formats (FASTQ, BAM/CRAM, VCF, plus metadata tables). 2. Write access to a destination bucket or container, and credentials stored securely on the sending machine. 3. A checksum manifest for the source data, generated before anything moves. 4. A named person or role responsible for the data agreement with the cloud provider, which is the accountability point raised in published guidance on genomic cloud computing [3]. 5. Roughly two hours of uninterrupted network time for a pilot transfer of 10 to 20 files. Do not begin a full transfer until the pilot passes verification end to end. ## Step-by-Step: Moving a Genomic Dataset to the Cloud ### Step 1. Inventory and Checksum the Source Data Start by measuring what you actually have. Run: ```bash du -sh /data/run_2024/ # expected output: # 4.2T /data/run_2024/ find /data/run_2024/ -name "*.fastq.gz" -type f | wc -l # expected output: # 96 ``` The `du` command reports total disk usage, and the `find ... | wc -l` pattern counts files matching your naming scheme. Record both numbers. If the file count is lower than the sequencer's sample sheet implies, stop and reconcile the discrepancy now, because a missing FASTQ discovered after upload costs a full re-transfer. Next, generate a checksum manifest. A checksum is a fixed-length fingerprint computed from file contents, and any change to the file, even a single flipped bit, changes the fingerprint. MD5 is the traditional choice in genomics and is still the de facto standard in sequencing core facility manifests: ```bash cd /data/run_2024/ md5sum *.fastq.gz > manifest.md5 head -3 manifest.md5 # expected output: # d41d8cd98f00b204e9800998ecf8427e sample01_R1.fastq.gz # 5d41402abc4b2a76b9719d911017c592 sample01_R2.fastq.gz # 6512bd43d9caa6e02c990b0a826520d7 sample02_R1.fastq.gz ``` Keep this manifest on the sending machine and upload a copy alongside the data. It is the reference for every integrity check that follows. For datasets where MD5 is discouraged by institutional policy, SHA-256 works identically through `sha256sum`. ### Step 2. Choose the Transfer Method ![Decision path asking about cloud target, dataset size and link speed, ending in AWS CLI, rclone, Globus or physical shipment.](/article-images/illustrations/large-genomic-data-transfer-to-cloud-best-practices-transfer-method-choice.webp) Not every transfer method suits every dataset. The table below summarizes the main options. | Method | Best for | Parallelism | Resumable | Notes | |--|--|--|--|--| | `aws s3 cp` / `aws s3 sync` | One-time bulk load into S3 | Yes, configurable | Yes (sync) | Default choice for AWS genomics work | | rclone | Any cloud target, heterogeneous storage | Yes, configurable | Yes | Single tool covers S3, Google Cloud, Azure, SFTP | | Globus Transfer | Institutional to institutional moves, recurring pipelines | High | Yes | Used by cloud workflow platforms to move large data reliably and was integrated into Galaxy-based cloud pipelines precisely for this purpose [4] | | SFTP / desktop clients | Smaller uploads, restricted networks | No | Partial | GenoVault, an OpenStack-based genomics repository, supports SFTP for upload and download and provides a desktop client specifically because large NGS files strain browser-based uploads [5] | | Physical shipment | Sites with slow or metered internet | N/A | N/A | Providers offer encrypted physical appliances for petabyte-scale one-time loads | A useful planning calculation: a 10 TB dataset over a 1 Gbps link at a realistic 100 MB/s sustained throughput takes about 28 hours of continuous transfer. Any interruption restarts the affected file, which is why resumable, checksum-aware tools dominate this list. If your link sustains only 10 MB/s, the same dataset needs roughly 12 days, and physical shipment or a staged sync over multiple nights becomes the rational choice. For teams feeding shared analysis platforms, matching the transfer tool to the analysis platform avoids double handling. Platforms such as VDJServer operate as cloud-based data commons where users upload immune repertoire data and then share it publicly or privately through the platform itself [6], and forensic databases such as diversID deliberately simplified their upload and download processes to improve convenience while applying stringent quality filtering on ingest [7]. If your destination is one of these portals, use its native upload path rather than uploading to raw storage first. ### Step 3. Prepare the Destination Create a dedicated bucket with a naming convention that encodes project, data class, and date. Object storage systems, which store data as addressable objects rather than files in folders, are the standard substrate for cloud genomics, and object-based architectures allow faster retrieval from distributed nodes [5]. ```bash aws s3 mb s3://zubairlab-genomics-prod # expected output: # make_bucket: zubairlab-genomics-prod ``` Key decisions at this stage: - **Naming.** Use a hierarchical key scheme such as `raw/run_2024/sample01/sample01_R1.fastq.gz`. Downstream tools and metadata systems parse these keys, and a consistent scheme lets you grant access by prefix. - **Encryption at rest.** Apply server-side encryption on upload with `-sse AES256`, or rely on a bucket-level default encryption policy. Published guidance on genomic cloud computing emphasizes that diligent practice means leveraging security standards and evaluation processes, the same practices you would apply to local infrastructure [3]. - **Access control.** Create a dedicated IAM role with permissions scoped to this bucket only. Avoid long-lived access keys on shared servers where possible. For multi-cloud or cross-institutional settings, federated identity is the modern answer. A Galaxy-based cloud-bursting implementation combined OpenID Connect and OAuth2 so users could access data distributed across multiple cloud providers without sharing permanent credentials such as passwords or API keys [8]. If your institution runs single sign-on, prefer federated access over static keys for every transfer job. - **Storage class.** For raw data you will reprocess occasionally, Standard-Infrequent Access or a similar tiered class reduces cost. For data you will analyze within days, keep Standard. Do not archive data that an active pipeline still needs, because retrieval from archive tiers adds hours and cost. - **Versioning and lifecycle rules.** Enable bucket versioning before the first upload so an accidental overwrite is recoverable, and set a lifecycle rule that transitions or expires intermediate files on a schedule you choose deliberately rather than by default. ### Step 4. Run the Transfer Tune the AWS CLI before the first large run. The defaults are conservative for genomic workloads: ```bash aws configure set default.s3.max_concurrent_requests 20 aws configure set default.s3.multipart_chunksize 64MB aws configure set default.s3.multipart_threshold 64MB ``` `max_concurrent_requests` controls how many parallel streams the CLI opens. Multipart upload splits each large file into chunks uploaded in parallel, which is mandatory for genomic files, because S3 rejects any single non-multipart PUT above 5 GB with an `EntityTooLarge` error. A 64 MB chunk size is a reasonable balance between per-part retry cost and overhead. For the bulk load from a directory: ```bash aws s3 cp /data/run_2024/ s3://zubairlab-genomics-prod/raw/run_2024/ \ --recursive \ --sse AES256 \ --storage-class STANDARD_IA \ --exclude "thumbnail*" \ --exclude "*.log" # expected output (one line per file): # upload: data/run_2024/sample01_R1.fastq.gz -> s3://zubairlab-genomics-prod/raw/run_2024/sample01_R1.fastq.gz # ... # Completed 96 part(s) with 96 file(s) sent ``` Parameter notes: - `-recursive` walks the directory tree. - `-exclude` and `-include` patterns let you filter, for example `-include "*.bam", include "*.bai"` to move alignments and indexes only. - `-storage-class` sets the tier at write time, avoiding a later migration. - `-sse AES256` forces server-side encryption per object. If the transfer is interrupted, resume with `sync` instead of repeating `cp`: ```bash aws s3 sync /data/run_2024/ s3://zubairlab-genomics-prod/raw/run_2024/ --size-only ``` By default `sync` compares timestamps and sizes, which behaves unpredictably when the source filesystem loses timestamps. The `-size-only` flag compares sizes only, which is the correct mode when source files are immutable and checksums will be verified separately. `sync` copies only objects missing or changed at the destination, so rerunning it is cheap and idempotent. For non-AWS targets or mixed environments, rclone covers the same ground with one configuration: ```bash rclone copy /data/run_2024/ aws-genomics:raw/run_2024 \ --transfers 8 --checkers 16 --fast-list --log-file transfer.log # expected tail of transfer.log: # 2026/11/11 09:14:22 INFO : sample01_R1.fastq.gz: Copied (new) # 2026/11/11 09:14:25 INFO : Transferred: 12.4 GiB / 4.2 TiB, 0.3%, 108 MiB/s, ETA 11h2m ``` `-transfers` sets parallel file streams, `-checkers` sets parallel integrity checks, and `-fast-list` reduces API listing calls on large buckets, which matters when a bucket holds hundreds of thousands of objects. Keep the log file. It is your transfer audit trail. For data arriving from public archives rather than your own sequencer, script the retrieval. The SRA Toolkit retrieves raw reads by accession: ```bash prefetch SRR24994256 fasterq-dump SRR24994256 --split-files --progress # expected output: # spots read : 12,345,678 # reads read : 24,691,356 # bases read : 3,703,703,400 ``` Continuous scripted ingestion of this kind is how GeniePool keeps a cloud data lake populated with preprocessed SRA samples queryable through a REST API [1]. ### Step 5. Verify Integrity After Upload Verification is not optional. A transfer that completes quickly but silently corrupts one BAM index will fail weeks later in a variant caller, far from the cause. First, confirm the object count and total size match your source inventory: ```bash aws s3 ls s3://zubairlab-genomics-prod/raw/run_2024/ \ --recursive --summarize | tail -3 # expected output: # 2026-11-11 09:12:04 48213754892 raw/run_2024/sample96_R1.fastq.gz # ... # Total Objects: 96 # Total Size: 4613311547392 ``` `Total Objects` must equal your file count from Step 1, and `Total Size` must match your `du` figure (note that `du` reports allocated blocks, so a few percent of difference is normal, while missing gigabytes are not). Second, spot-check or fully verify checksums. To verify a subset, download those objects and compare against the manifest: ```bash aws s3 cp s3://zubairlab-genomics-prod/raw/run_2024/sample01_R1.fastq.gz /tmp/verify/ cd /tmp/verify md5sum sample01_R1.fastq.gz # compare the output hash against the matching line in manifest.md5 ``` For a full verification, run checksums server-side or pull every object, which costs egress and time, so most teams verify a random 5 to 10 percent sample per transfer batch plus every index file, since small index files are disproportionately likely to be damaged by interrupted transfers. Third, check that encryption is actually applied: ```bash aws s3api head-object \ --bucket zubairlab-genomics-prod \ --key raw/run_2024/sample01_R1.fastq.gz \ --query "ServerSideEncryption" # expected output: # "AES256" ``` A verification step that many teams skip is a functional smoke test: load the transferred VCF or BAM into your analysis environment and confirm the tool opens it. Automated validation on ingest is a feature of mature cloud platforms. GWASHub validates and harmonizes uploaded GWAS summary statistics automatically before meta-analysis [9], and the California IGED system applies standardization, validation, and deduplication protocols to genomic surveillance data arriving from heterogeneous laboratories precisely to preserve data quality and interoperability [10]. Build the equivalent check for your own data types, even if it is a single `bcftools view` or `samtools quickcheck` call per file class. ### Step 6. Attach Metadata and Enforce Governance Files in a bucket without metadata are a liability. Follow the FAIR principles, making data findable, accessible, interoperable, and reusable, an approach explicitly adopted in cloud metagenomics platforms [11]. In practice this means: 1. Upload a machine-readable metadata table (CSV or JSON) alongside every data batch, containing sample identifiers, library preparation, sequencing platform, read length, and consent or access restrictions. 2. Tag objects at upload time where the platform supports it, for example `-tagging "project=all_of_us_pilot&consent=IRB-2024-118"` on the AWS CLI. 3. Encode access tier in your key structure so restricted and open data never share a prefix with a single blanket permission. 4. Record provenance: which pipeline version, which reference genome, which transfer job produced each object. Governance questions should be settled before the first byte moves. Published analysis of genomic cloud computing identifies three points to consider when negotiating arrangements with commercial providers: data control, data security, confidentiality and transfer, and accountability [3]. Concretely, read the provider's Terms of Service for clauses on data use, subcontractor access, and deletion, and name the individual accountable for the data agreement. For distributed multi-cloud datasets, federated identity protocols such as OpenID Connect and OAuth2 allow access management without circulating permanent credentials, an approach demonstrated in production for Galaxy-based cloud analysis [8]. ### Step 7. Control Cost and Connect Analysis Transfer and storage costs behave differently, and confusing them is expensive. Data ingress to major clouds is typically free, storage accrues monthly, inter-region and egress traffic accrues per GB, and retrieval from archive tiers carries its own charges. The strongest argument for doing analysis where the data lands is empirical: a cloud variant-calling pipeline for 2,535 human genomes completed joint calling in 5 days at a compute cost of $7.33 per sample, about $18,590 for completed jobs, demonstrating near-linear scalability on cloud infrastructure [12]. Paying to move that data out for local analysis would have added cost and days of transfer time for no scientific gain. So the final best practice is architectural: upload once, analyze in place. Serverless and container-orchestrated architectures scale analysis against cloud-stored data without moving it again, the pattern behind Celeste's serverless pipeline for population-scale WGS [2] and behind CloudASM, a turn-key pipeline that pairs a pipeline manager with a cloud data warehouse for allele-specific methylation analysis [13]. When downstream work involves thousands of heterogeneous experiments, query engines that operate over cloud-hosted genomic data with distribution transparency, such as PyGMQL on Apache Spark, avoid the transfer entirely by pushing the computation to the data [14]. ## Common Errors and Exact Fixes | Error message | Cause | Fix | |--|--|--| | `Unable to locate credentials. You can configure credentials by running 'aws configure'` | No credentials on the sending machine | Run `aws configure` or attach an IAM role to the compute instance | | `AccessDenied` on upload | IAM policy lacks `s3:PutObject` on the target prefix | Scope a policy to `arn:aws:s3:::bucket/prefix/*` and attach it | | `NoSuchBucket` / `The specified bucket does not exist` | Typo in bucket name or wrong Region | Verify with `aws s3 ls` and add `-region` explicitly | | `EntityTooLarge` | Single PUT above the 5 GB S3 limit | Raise `multipart_threshold` and `multipart_chunksize` so the CLI uses multipart upload | | Transfer crawls at a fraction of link speed | Default 10 concurrent requests, single stream per file | Raise `max_concurrent_requests`, use multipart, and run rclone or Globus for high-throughput paths | | `WARNING: 1 computed checksum did NOT match` during verification | Corrupted or truncated object | Delete the object, re-run `aws s3 sync`, re-verify that file specifically | | `md5sum: manifest.md5: no properly formatted checksum lines found` | Manifest edited on Windows with CRLF line endings | Convert with `dos2unix manifest.md5` | | rclone `directory not found` | Remote name or path typo | Run `rclone listremotes` and `rclone lsd remote:` to confirm paths | ## Common Mistakes and Limitations **Skipping the checksum manifest.** The single most common failure in genomic data transfer is treating integrity as someone else's problem. Without a pre-transfer manifest, a corrupted object is undetectable until a pipeline crashes months later, and at that point you cannot tell whether the corruption happened in transfer or in the original sequencing run. **Uploading before securing access.** Teams frequently create a bucket, upload terabytes, and then think about permissions. Published guidance on genomic cloud computing treats data security, confidentiality, and transfer as preconditions to negotiate with the provider, not afterthoughts [3]. Create the bucket policy, encryption defaults, and least-privilege role first. **Using browser uploads for large files.** Browser-based upload forms choke on multi-gigabyte files, lose progress on network blips, and provide no checksum verification. Dedicated repositories solved this by shipping desktop clients for large NGS uploads [5], which is an admission that the browser path fails. Use command-line tools with retry logic for anything above a few gigabytes. **Ignoring parallelism and default tuning.** Default CLI settings are tuned for small files. A 100 GB FASTQ uploaded as a single stream on defaults can take an order of magnitude longer than the same file uploaded with multipart and 20 concurrent requests. The tuning step in this walkthrough is not optional decoration. **Confusing sync with copy.** Re-running `cp` after an interruption re-uploads completed files. `sync` with `-size-only` resumes cleanly. Using the wrong one doubles transfer time and bills. **Neglecting metadata at transfer time.** Metadata captured at upload is authoritative. Metadata reconstructed six months later from file names is guesswork. Automated metadata capture and validation, as implemented in cloud platforms built for GWAS meta-analysis [9], exists because manual metadata handling is where reproducibility dies. **Transferring data twice.** Every hop adds time, egress cost, and corruption risk. Stage data directly to the region where analysis will run. Within one cloud Region, ingest, storage, and compute compose into pipelines that have processed petabytes without the data leaving [2]. **Underestimating legal review time.** Institutional and provider agreements take weeks. Teams that discover consent restrictions or contractual limits after upload have only bad options left. The accountability point in published cloud-genomics guidance exists for exactly this reason. **A limitation of scale estimates.** Every throughput and time figure in this article is a planning estimate based on sustained link capacity. Your real numbers depend on network hardware, concurrent load, and provider-side throttling. Run the pilot transfer, measure it, and extrapolate from your own measurements rather than from any published figure. ## Frequently Asked Questions ### How long should a 10 TB genomic transfer take? ![Calculation of 10 TB at 100 MB/s giving about 28 hours, and the same dataset at 10 MB/s giving roughly 12 days.](/article-images/illustrations/large-genomic-data-transfer-to-cloud-best-practices-transfer-time-estimate.webp) At a sustained 100 MB/s on a 1 Gbps link, about 28 hours of continuous transfer. Measure your actual sustained throughput with the pilot batch before committing to a schedule. ### Should I compress FASTQ files before uploading? FASTQ files are normally already gzip-compressed by the sequencer or bcl-conversion software. Do not recompress, and do not upload uncompressed FASTQ if a gzipped equivalent exists, since storage bills by the byte. ### What is the most secure way to transfer human genomic data? Use encrypted transport (TLS-based tools such as the AWS CLI or rclone), server-side encryption at rest, least-privilege IAM roles, federated identity instead of shared permanent credentials, and a signed agreement that covers data control, confidentiality, transfer, and accountability with the provider [3][8]. ### How do I prove files arrived intact? Generate an MD5 or SHA-256 manifest before upload, then compare object counts, total sizes, and sampled checksums against it after upload, and run a functional smoke test such as `samtools quickcheck` on a sample of BAM files. ### Should I use aws s3 cp or aws s3 sync? Use `cp` for the initial bulk load and `sync, size-only` for resuming interrupted transfers or topping up immutable datasets, because sync copies only missing or changed objects. ### How much does cloud genomic transfer cost? Data ingress to major clouds is generally free, while storage, cross-region traffic, and egress accrue charges that vary by provider and tier. The cost that dominates most genomics projects is compute, and published cloud pipelines have achieved population-scale variant calling at low per-sample compute cost [12]. Price your own plan with your provider's current calculator rather than relying on any single published figure. ### Can I start analyzing data before the upload finishes? Yes, file by file, if your upload order matches your analysis order and your pipeline handles partial directories gracefully. For safety, gate automated analysis on the verification step completing for each batch. ### What metadata must travel with the files? At minimum: sample identifier, library and assay type, sequencing platform, reference genome version, consent and access restrictions, producing lab, and transfer date. Upload it as a machine-readable file and, where supported, as object tags, following FAIR principles [11]. ## Sources 1. [GeniePool: genomic database with corresponding annotated samples based on a cloud data lake architecture.](https://pubmed.ncbi.nlm.nih.gov/37311148/) 2. [Celeste: A cloud-based genomics infrastructure with variant-calling pipeline suited for population-scale sequencing projects.](https://pubmed.ncbi.nlm.nih.gov/40343041/) 3. [Genomic cloud computing: legal and ethical points to consider.](https://pubmed.ncbi.nlm.nih.gov/25248396/) 4. [Cloud-based bioinformatics workflow platform for large-scale next-generation sequencing analyses.](https://pubmed.ncbi.nlm.nih.gov/24462600/) 5. [GenoVault: a cloud based genomics repository.](https://pubmed.ncbi.nlm.nih.gov/34325724/) 6. [VDJServer: A Cloud-Based Analysis Portal and Data Commons for Immune Repertoire Sequences and Rearrangements.](https://pubmed.ncbi.nlm.nih.gov/29867956/) 7. [A cloud-based solution for managing next-generation sequencing data to comprehend forensic population statistics.](https://pubmed.ncbi.nlm.nih.gov/41805780/) 8. [Cloud bursting galaxy: federated identity and access management.](https://pubmed.ncbi.nlm.nih.gov/31197310/) 9. [GWASHub: An Automated Cloud-Based Platform for Genome-Wide Association Study Meta-Analysis.](https://pubmed.ncbi.nlm.nih.gov/41282854/) 10. [The SARS-CoV-2 Integrated Genomic Epidemiology Database (IGED): Linking viral genomes with patient-level metadata to advance statewide genomic surveillance in California.](https://pubmed.ncbi.nlm.nih.gov/42672077/) 11. [Biofilm marker discovery with cloud-based dockerized metagenomics analysis of microbial communities.](https://pubmed.ncbi.nlm.nih.gov/39266450/) 12. [Inexpensive and Highly Reproducible Cloud-Based Variant Calling of 2,535 Human Genomes.](https://pubmed.ncbi.nlm.nih.gov/26110529/) 13. [CloudASM: an ultra-efficient cloud-based pipeline for mapping allele-specific DNA methylation.](https://pubmed.ncbi.nlm.nih.gov/32119067/) 14. [PyGMQL: scalable data extraction and analysis for heterogeneous genomic datasets.](https://pubmed.ncbi.nlm.nih.gov/31703553/)