# Federated Data Access for Population Genomics By the end of this guide you will be able to (1) explain how federated data access for population genomics differs from bulk download and central rehosting, (2) choose among the major access models used in cloud bioinformatics, (3) set up a working AWS genomics environment with credentials, storage, and a workflow engine, (4) verify that a run completed correctly, and (5) avoid the mistakes that most often burn budgets or break reproducibility. To work through the hands-on portion you need a computer with the AWS command line interface (CLI) installed, an AWS account with an access key, about 10 GB of free disk space, and either Docker or a container runtime such as Apptainer. If you plan to work on a controlled-access cohort, you also need an approved data access request, which your institution's signing official usually files on your behalf. ## Federated Data Access for Population Genomics: What It Actually Means Federated data access is a pattern in which data stays under the control of its original custodian while the analysis moves to the data. Instead of copying thousands of genomes to your laptop or your institutional cluster, you run your computation on the platform where the data already lives, and only derived results leave. Three forces push population genomics toward this pattern. **Volume.** National-scale sequencing programs now generate genomic data for cohorts approaching or exceeding one million individuals, integrated with longitudinal clinical records [1], [2]. At that scale, a "copy everything to one place" strategy breaks down on transfer time, storage duplication, and governance review. **Governance.** Clinical and cohort genomic data is usually bound by privacy policy and jurisdictional rules. Much of it sits siloed inside hospitals and cohort custodians, and one frequently cited estimate holds that more than 97 percent of hospital data goes unused [3]. Federated platforms let a custodian grant permissioned access without the data physically crossing organizational or national boundaries. **Reproducibility.** When the analysis code travels to the data as a versioned workflow, the same container, the same reference files, and the same workflow definition can be re-executed by any collaborator. Early cloud workflow platforms were built explicitly around this idea, combining Galaxy-style interfaces with reliable bulk data transfer and automatic cloud provisioning so that a workflow, not a pile of scripts, is the unit of sharing [4]. A useful mental model is three layers: 1. **Data layer.** Genomic objects (FASTQ, BAM or CRAM, gVCF, multi-sample VCF) plus metadata, held by a custodian or a program such as a national biobank. 2. **Execution layer.** The compute substrate where your workflow runs, typically serverless orchestration, container schedulers, or batch systems on a cloud such as AWS [1], [5]. 3. **Governance layer.** The access request, audit logging, and data use limitations that determine which queries and which users are permitted. Federation is a spectrum. At one end, a custodian hosts your container next to the data and returns only your result tables. At the other end, you pull data into your own cloud account under a data use agreement. Both are common, and knowing which one you are in changes every design decision that follows. ## Why Cloud Bioinformatics Changed the Equation Before reliable cloud platforms, a population-scale project had to own hardware sized for its peak load, which sat idle most of the time. Cloud bioinformatics inverted that. Three published results show the shape of the change. First, elasticity at the top end is real. The Celeste infrastructure, built to process whole-genome sequencing for the All of Us Research Program, targets a cohort of more than one million participants and routinely handles fluctuating workloads of up to 9,000 whole-genome samples per month within a single AWS Region. In 12 months of production it coordinated roughly 200 million serverless function invocations and more than 20 million messages to complete 1.8 million analysis, quality control, and reporting jobs [1]. Second, elastic parallelism collapses wall-clock time. A cloud curation study on more than 126,000 CT volumes from over 26,000 patients used more than 21,000 virtual machines to finish in under 9 hours what a single workstation would have needed an estimated 522 days to complete, at a total compute cost of $1,011.05 [6]. The task was imaging rather than sequencing, but the price and performance logic transfers directly to genomics pipelines. Third, aggressive cost engineering works. GT-WGS, a distributed whole-[genome analysis](/blog/guides/genome-analysis) system built on AWS, exploited the cloud's dynamic pricing mechanisms to complete a 55x WGS run on a 400 GB FASTQ in as little as 18.4 minutes for $16.50, with variant calls 99.9 percent consistent with the GATK best practices workflow it was benchmarked against [5]. For context, a conventional 50x WGS analysis commonly took 30 to 40 hours on fixed infrastructure. The consequence for a researcher is that population-scale computing is now a budgeting and design problem rather than a procurement problem. The harder problems are choosing the right access model and avoiding the operational traps described later in this guide. ## Access Models Compared: Where the Data Lives and Where Your Code Runs Not every "cloud" project is federated in the same way. The table below summarizes the four models you will actually encounter. | Model | Where data lives | Where your code runs | Typical examples | Best for | Main risk | |--|--|--|--|--|--| | Co-located analysis platform | Program-hosted bucket or workspace | Same platform, on-demand VMs | Terra, Seven Bridges Cancer Genomics Cloud with more than 200 curated tools over The Cancer Genome Atlas [7] | Analyzing large public or controlled-access cohorts with no transfer | Vendor lock-in of workspace logic | | Precomputed query layer | Cloud data warehouse or relational store | Query engine, not your pipeline | Mitelman Database coordinate mappings in Google BigQuery with ISB-CGC integration [8], [Functional Genomics](/blog/guides/functional-genomics) Platform with 200,000 curated genomes and 68 million genes [9] | Exploration, feasibility checks, integrative joins | Query results can mislead if you skip upstream QC | | Federated execution | Custodian's environment | Custodian's environment, your workflow | Federated data platforms that avoid moving data outside organizational or jurisdictional boundaries [3] | Controlled-access clinical data, multi-country consortia | Limited tool control, dependency on custodian's stack | | Bring your own cloud | Your own bucket | Your own cluster or batch system | AWS-based pipelines such as GT-WGS [5] and Celeste-style serverless architectures [1], hybrid on-premise plus cloud systems [10] | Novel pipelines, maximum flexibility | You own the security, cost, and reproducibility burden | Two practical notes on choosing among these. If the cohort you need is already hosted on a platform, working co-located is almost always cheaper and faster than replicating the data, because you pay for compute only and the platform absorbs the storage engineering. If you need a method the platform cannot express, the hybrid pattern described by a Japanese group, which combined an on-premise supercomputer, university science clouds, and AWS with distributed, reproducible pipelines, is a proven middle path and scales to analyses of more than 11,000 human genomes [10]. ## Step-by-Step: Building a Working Federated-Style Analysis on AWS Genomics Infrastructure ![Six-step flow: configure credentials, create a bucket, stage data, run a workflow engine, query the VCF, verify the result.](/article-images/illustrations/federated-data-access-population-genomics-cloud-federated-setup-steps.webp) The walkthrough below gets you from a bare AWS account to a verified variant query against cohort data in your own account. It assumes the bring-your-own-cloud model, which exercises the most skills and translates directly to the other models. ### Step 1: Configure credentials and confirm your identity Install the AWS CLI, then run: ``` aws configure ``` You will be prompted for four values: ``` AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY Default region name [None]: us-east-1 Default output format [None]: json ``` Choose the region your cohort's data lives in if you know it. Keeping compute and data in the same region avoids cross-region transfer charges and latency, a lesson baked into the design of population-scale systems that deliberately ran within a single AWS Region [1]. Verify the configuration: ``` aws sts get-caller-identity ``` Expected output: ``` { "UserId": "AIDACKCEVSQ6C2EXAMPLE", "Account": "123456789012", "Arn": "arn:aws:iam::123456789012:user/popgen-analyst" } ``` If you see that JSON, your credentials work. Do not proceed with root account keys. Create an IAM user or role with programmatic access and scope its permissions to the buckets and services this project needs. ### Step 2: Create a working bucket ``` aws s3 mb s3://popgen-work-yourinitials-us-east-1 ``` Expected output: ``` make_bucket: popgen-work-yourinitials-us-east-1 ``` Bucket names are globally unique across all of AWS, so append your initials or a project code. Use one bucket per project and prefix object keys by stage, for example `raw/`, `ref/`, `results/`. This prefix discipline is what makes later lifecycle rules and access audits tractable. ### Step 3: Stage reference and cohort data List what is available in a public bucket before copying anything: ``` aws s3 ls s3://1000genomes/phase3/ ``` Expected output (abbreviated): ``` PRE analysis/ PRE data/ PRE integrated_call_sets/ 2025-01-14 09:12:03 ... ``` The 1000 Genomes Project data is publicly readable, so no special access flags are required. If you instead need controlled-access cohort data, your data access agreement will specify the staging path and who may copy it. Sync a focused subset rather than the full corpus: ``` aws s3 sync s3://1000genomes/phase3/integrated_call_sets/ s3://popgen-work-yourinitials-us-east-1/phase3/ ``` Monitor progress with `aws s3 ls s3://popgen-work-yourinitials-us-east-1/phase3/, recursive` if you need a full size inventory, or simply spot-check with: ``` aws s3 ls s3://popgen-work-yourinitials-us-east-1/phase3/ ``` Expected output: ``` 2026-11-11 14:02:11 12593412 ALL.wgs.phase3_integrated_v1.20130502.genotypes.vcf.gz.tbi ``` The companion `.tbi` or `.csi` index file must travel with every compressed VCF. Losing the index is the single most common cause of "mystery" query failures later. ### Step 4: Run a workflow engine rather than loose scripts For population genomics you want a workflow engine with containerized tools, resumable execution, and provenance logs. Two published platforms illustrate the pattern: the Galaxy-based cloud workflow system added Globus Transfer for reliable data movement, HTCondor-based auto-scaling, and semantic verification of workflow correctness [4], and modern ports of established tools such as rMATS publish the same workflow in CWL, WDL, and Nextflow so it runs on Cavatica, Terra, or Seqera alike [11]. Pick the ecosystem your collaborators use, because the workflow language matters less than the shared definition. With Nextflow installed and Docker running, smoke-test a community pipeline in test mode: ``` nextflow run nf-core/sarek -profile test -with-report run_report.html ``` Expected output at the end of a successful test run: ``` Pipeline Status : SUCCEEDED Duration : 12m 34s CPU hours : 1.2 Succeeded processes : 47 ``` Then point the pipeline at your own data using a parameters file, which keeps every run definition reviewable in version control: ``` nextflow run nf-core/sarek -params-file params.json -profile docker -resume ``` with `params.json`: ``` { "input": "s3://popgen-work-yourinitials-us-east-1/samplesheet.csv", "outdir": "s3://popgen-work-yourinitials-us-east-1/results" } ``` The `-resume` flag reuses successfully completed tasks if a run fails partway, which converts a 3,000-sample crash at sample 2,871 from a disaster into an 11-minute restart. The `-profile docker` setting tells Nextflow to execute each tool in its container image, which is what makes the run portable to another cloud or to a hybrid on-premise plus cloud system later [10]. ### Step 5: Query a large VCF without copying it htslib, the library underneath samtools and bcftools, can stream directly from S3. For a quick per-chromosome check on a remote VCF: ``` bcftools view -r chr20 -Oz -o chr20_cohort.vcf.gz s3://popgen-work-yourinitials-us-east-1/results/cohort.vcf.gz ``` Then summarize: ``` bcftools stats chr20_cohort.vcf.gz | grep SNPs ``` Expected output (values will differ by cohort): ``` SN 0 number of SNPs: 1187654 ``` Keep queries region-scoped with `-r` whenever possible. Whole-file streaming from object storage is far slower than an indexed random access read, and at cohort scale the difference is hours versus days. ### Step 6: Verify the result before you believe it Build verification into every run: 1. **Checksum the inputs.** Run `md5sum` on every FASTQ and reference file at intake and record the values. A mismatch later explains most unexplained result drift. 2. **Check counts against expectations.** Compare per-sample variant counts against the cohort median. A sample with half the expected heterozygous SNPs usually means a failed library, not biology. 3. **Validate a subsample against a trusted reference.** GT-WGS validated its pipeline by benchmarking variant calls for consistency against the GATK best practices result [5], and you should do the same at whatever scale you can afford, even if it is 20 samples. 4. **Read the workflow report.** The `-with-report run_report.html` output from Step 4 lists per-task CPU, memory, and wall time, which you need for cost estimation before the full cohort run. ## Key Parameters That Control Cost, Speed, and Correctness ![Worked calculation: a 10-sample pilot at $30 and 4 hours gives about $3,000 for 1,000 samples, with wall clock set by concurrency.](/article-images/illustrations/federated-data-access-population-genomics-cloud-pilot-cost-extrapolation.webp) | Parameter | Where it appears | What it controls | Sensible default for population genomics | |--|--|--|--| | Region | `aws configure`, bucket location | Data locality, latency, transfer charges | Same region as the cohort data [1] | | Spot versus on-demand pricing | EC2 purchase option | Cost versus interruption risk | Spot for embarrassingly parallel per-sample work [5] | | `-profile` | Nextflow | Execution backend, containers, test data | `docker` or `singularity` for real runs, `test` for smoke tests | | `-resume` | Nextflow | Caching of completed tasks | Always on for long pipelines | | `-r` versus `-R` | bcftools | Indexed region query versus whole-file region filter | `-r` with a present index for speed | | Memory and disk directives | Workflow task definitions | Task success and cost | Set from the report of a pilot run, not guessed | | Storage class | S3 object lifecycle | Cost of archived versus active data | Keep active intermediates standard, archive raw reads after pipeline completion | One worked calculation ties these together. Suppose a pilot of 10 samples costs $30 and takes 4 hours. A linear extrapolation to 1,000 samples suggests $3,000 and a single long run, but per-sample tasks embarrassingly parallelize, so the real wall clock depends on your concurrency limit while the cost stays near $3,000. The imaging curation study cited earlier is the extreme case of that arithmetic, trading 21,000 virtual machines for a 522-day serial job compressed into 9 hours at roughly $1,000 [6]. Estimate cost from per-sample cost times sample count, and estimate wall clock from per-sample time divided by your achievable concurrency. ## Cost and Runtime Anchors From Published Runs Budget conversations go better with real numbers. Three published anchors, each stated once here: - A 55x WGS sample (400 GB FASTQ) was processed in 18.4 minutes for $16.50 on AWS with 99.9 percent agreement to GATK best practices calls [5]. - A cloud AI curation job on more than 126,000 medical images used more than 21,000 virtual machines and cost $1,011.05 total, finishing in under 9 hours [6]. - New cloud users can prototype for free, since DeepVariant-on-Spark was deployed on Google Cloud Platform so that users could start analyzing at least ten WGS datasets within 20 minutes of setup using free platform credits [12]. Treat these as order-of-magnitude anchors, not quotes. Your per-sample cost depends on depth, pipeline, pricing changes, and how much parallelism you actually achieve. ## Common Mistakes and Limitations **Copying data you could query.** The most expensive mistake in federated work is transferring data that a query layer could have answered. Several programs now publish derived tables directly to cloud warehouses for exactly this reason, as with the Mitelman Database coordinate mappings exposed through Google BigQuery and the ISB-CGC resource [8], and precomputed knowledge bases covering hundreds of millions of genes, proteins, and domains that answer questions in a fraction of the time raw-tool workflows need [9]. Before staging a petabyte, ask whether a join or a precomputed resource answers the question. **Region mismatch.** Launching compute in one region against data in another silently adds transfer fees and can triple run time. Population-scale production systems stay inside one region on purpose [1]. Check your bucket region with `aws s3 ls` on the bucket and your CLI default with `aws configure get region`. **Treating federation as a compliance shortcut.** Federated access reduces exposure, but it does not remove governance obligations. Data use limitations, audit trails, and participant consent still bind every query, and the platforms that succeed are the ones managed to maintain patient and public trust [3]. A federated query that leaks an individual's genotype through a small cell count is a disclosure regardless of where the compute ran. **Ignoring spot interruptions.** Discounted spot capacity can vanish mid-task. Mitigate by checkpointing per-sample work, using `-resume`, and keeping batch sizes small enough that an interruption costs minutes, not days. Systems that won benchmark prizes on AWS did so by exploiting dynamic pricing deliberately [5]. **Wrong contig naming.** One pipeline emits `chr1`, your reference says `1`, and bcftools fails with a contig not defined in the header error. Normalize naming once, at intake, with a documented renaming step. Do not fix it ad hoc inside individual commands. **Missing or stale index files.** htslib fails with a "Could not retrieve index file" error, or worse, silently uses an index older than the data. Regenerate with `bcftools index -t file.vcf.gz` after every rewrite and confirm the index timestamp is newer than the VCF. **Out-of-memory kills at scale.** Tasks that pass on 10 samples die at 500 because merged VCFs and joint genotyping scale super-linearly. The signature is a Nextflow error block showing `Command exit status: 137`, the Linux out-of-memory kill signal. Raise the task's memory directive based on the report from your largest successful pilot and rerun with `-resume`. **Skipping the pilot.** DeepVariant-on-Spark exists in part because small-scale cloud pilots are cheap and fast, letting researchers validate an analysis before committing a cohort [12]. Run a 5 to 20 sample pilot through the full pipeline, including output verification, before any large launch. **Underestimating the reproducibility debt.** A run you cannot re-execute is a result you cannot defend. Pin container image digests, version your reference files with checksums, and keep the workflow definition in version control. This is the core design principle that early cloud workflow platforms built in with semantic workflow verification [4] and that national-scale hybrid systems enforce across multiple sites [10]. **Assuming statistical conclusions transfer.** Large cohorts with small effect sizes demand methods that control model misspecification bias and weak population structure, which is why semiparametric, doubly robust estimation frameworks such as TarGene target exactly these cohort-scale questions [13]. A pipeline that produces VCFs has not yet produced a genetic association result you can publish. ## Governance, Trust, and Who Gets Access The federated movement is as much a policy story as a technical one. Clinical-genomic data accumulates faster than it is used, benefits concentrate where sequencing happens, and populations outside the traditional sequencing centers remain underrepresented [3]. Federated platforms address this by letting a researcher anywhere run an approved analysis against a custodian's data, with no-code interfaces opening participation to end users who are not programmers. For national cohorts, the pattern is already operating at scale. Reviews of national WGS biobank programs, including the UK Biobank, the All of Us Research Program, Singapore's PRECISE, Biobank Japan, and Korea's National Project of Bio-Big Data, describe cloud-based computing, multi-sample VCF construction, and optimized storage formats as the standard enabling stack for these resources [2]. If your research question depends on population diversity, the practical route is often to join one of these program's researcher workspaces rather than to request raw transfers. When you design your own project's access policy, write down three things before the first external user is granted access: which results may leave the platform, what audit records are kept, and how a data use limitation is technically enforced rather than merely promised. ## Next Steps Once Your First Run Succeeds 1. **Containerize anything still running as loose scripts.** Every tool in a versioned container is the difference between a rerun and a forensic investigation. 2. **Automate the verification step.** Wrap the checksum and count checks from Step 6 into a workflow task so verification cannot be skipped. 3. **Consider hybrid capacity.** If deep learning or long queues enter your future, the hybrid pattern of on-premise plus science cloud plus public cloud, proven at 11,000-genome scale with GPU nodes added on demand, is a tested blueprint [10]. 4. **Publish the workflow, not the notebook.** Porting your pipeline to a registry-hosted definition in CWL, WDL, or Nextflow, as the rMATS team did across Cavatica, Terra, and Seqera, makes it executable by others on their platform of choice [11]. 5. **Scale the statistics with the data.** When you move from variant calling to association testing across hundreds of thousands of samples, use frameworks built for cohort-scale effect estimation, including distributed machine learning approaches such as VariantSpark, which scales to 100 million variants by 100,000 samples for epistasis-aware association testing [14]. ## Frequently Asked Questions ### What does federated data access for population genomics actually mean? It means analysis travels to the data instead of data traveling to the analyst. The custodian keeps control of the genomic files, your approved workflow runs on the platform where they reside, and only aggregate or permitted results come back. ### Do I need my own AWS account to do cloud bioinformatics? Not always. Platforms such as Terra and the Seven Bridges Cancer Genomics Cloud provide hosted workspaces with curated tools and on-demand compute, so you can analyze hosted cohorts without administering infrastructure [7]. You need your own account mainly for the bring-your-own-cloud model or for novel pipelines a hosted platform cannot express. ### How much does cloud genomics cost? It varies enormously with scale and engineering, but published anchors help. One AWS whole-genome run processed a 400 GB FASTQ for $16.50 [5], and a 126,000-image cloud curation job cost just over $1,000 total [6]. ### Can I do association testing without moving raw VCFs? Yes. VariantSpark was designed for exactly this, using distributed machine learning to search polygenic and epistatic associations across up to 100 million variants and 100,000 samples in a cloud framework [14]. Precomputed database resources and cloud query layers similarly let you filter and join before any bulk transfer [8], [9]. ### What is the quickest way to test a pipeline before scaling up? Use a test profile on a community workflow and a small pilot cohort. DeepVariant-on-Spark was set up on Google Cloud Platform so users could go from zero to analyzing at least ten WGS datasets in about 20 minutes with free credits [12], and the same smoke-test logic applies to any Nextflow or WDL pipeline. ### How do I keep a federated cloud analysis reproducible? Pin versions everywhere. Use container digests, checksummed reference files, a version-controlled workflow definition, and a workflow engine that logs provenance, the combination that early cloud workflow platforms [4] and later national hybrid systems [10] both treat as non-negotiable. ### How is federated access different from a bulk download? A bulk download gives you a copy and every burden that follows, storage, security, and re-approval at each use. Federated access keeps the single authoritative copy with the custodian and grants permissioned execution, which is how platforms avoid moving data across organizational and jurisdictional boundaries [3]. ### Which large national cohorts can I access through these models? Major WGS-based national programs include the UK Biobank, the All of Us Research Program, Singapore's PRECISE, Biobank Japan, and Korea's National Project of Bio-Big Data, each with defined researcher access pathways and cloud-based analysis capabilities [2]. ## Sources 1. [Celeste: A cloud-based genomics infrastructure with variant-calling pipeline suited for population-scale sequencing projects](https://pubmed.ncbi.nlm.nih.gov/40343041/) 2. [Lessons from national biobank projects utilizing whole-genome sequencing for population-scale genomics](https://pubmed.ncbi.nlm.nih.gov/40050991/) 3. [Democratizing clinical-genomic data: How federated platforms can promote benefits sharing in genomics.](https://pubmed.ncbi.nlm.nih.gov/36704354/) 4. [Cloud-based bioinformatics workflow platform for large-scale next-generation sequencing analyses](https://pubmed.ncbi.nlm.nih.gov/24462600/) 5. [GT-WGS: an efficient and economic tool for large-scale WGS analyses based on the AWS cloud service](https://pubmed.ncbi.nlm.nih.gov/29363427/) 6. [Cloud-based large-scale curation of medical imaging data using AI segmentation](https://pubmed.ncbi.nlm.nih.gov/38746269/) 7. [The Cancer Genomics Cloud: Collaborative, reproducible, and democratized-a new paradigm in large-scale computational research](https://pubmed.ncbi.nlm.nih.gov/29092927/) 8. [A cloud-based resource for genome coordinate-based exploration and large-scale analysis of chromosome aberrations and gene fusions in cancer](https://pubmed.ncbi.nlm.nih.gov/36695636/) 9. [Functional Genomics Platform, A Cloud-Based Platform for Studying Microbial Life at Scale](https://pubmed.ncbi.nlm.nih.gov/32877338/) 10. [Design and implementation of a hybrid cloud system for large-scale human genomic research](https://pubmed.ncbi.nlm.nih.gov/36755016/) 11. [rMATS-cloud: Large-scale Alternative Splicing Analysis in the Cloud](https://pubmed.ncbi.nlm.nih.gov/40341961/) 12. [DeepVariant-on-Spark: Small-Scale Genome Analysis Using a Cloud-Based Computing Framework](https://pubmed.ncbi.nlm.nih.gov/32952600/) 13. [Semiparametric efficient estimation of small genetic effects in large-scale population cohorts](https://pubmed.ncbi.nlm.nih.gov/41024427/) 14. [VariantSpark: Cloud-based machine learning for association study of complex phenotype and large-scale genomic data](https://pubmed.ncbi.nlm.nih.gov/32761098/)