How to Build a Federated Genomic Data Network in the Cloud

By Dr. Zubair Khalid, DVM, MS, PhD · 2026-11-11

By the end of this guide you will be able to stand up a working two-site federated genomic data network in the cloud. Each participating site keeps its own sequencing data under its own encryption keys, runs its own pipelines on its own compute, and answers cross-site queries by returning summary statistics instead of moving genomes. You will provision the storage and compute at each site, deploy a workflow engine, wire up a federation layer that enforces machine-readable data use policies, run a federated allele frequency and summary-statistics job end to end, and leave behind an audit trail an access body can actually review.

This is a practical build guide. Where design choices depend on regulation or scientific goals, I say so directly and point you to the underlying literature rather than inventing rules.

What You Need Before You Start

Have these ready before typing any commands:

• An AWS account with an IAM user or role that can create S3 buckets, KMS keys, EKS clusters, and CloudTrail trails. Verify identity with aws sts get-caller-identity and confirm the returned account ID matches the site you are provisioning.
• The AWS CLI (version 2), eksctl, and kubectl installed and on your PATH.
• Nextflow (Java 17 or later installed), bcftools and samtools from HTSlib, and Python 3.10 or later on the machine you use to drive the federation.
• A signed data use agreement or material transfer framework with each partner site before any real participant data moves or is queried. Technical federation does not substitute for the legal layer.
• A reference genome build decision. GRCh38 is the practical default for new networks. Sites on GRCh37 need a documented liftOver policy before queries will agree.
• Test data that is safe to share, such as a public reference sample or synthetic VCFs, for the first end-to-end run.

Two sites are enough for a pilot. Three sites teach you most of the operational lessons, but two will surface nearly every design mistake at a fraction of the coordination cost.

Why Federation Instead of a Central Warehouse

A federated genomic data network is a set of sites, each holding its own genomic data under its own control, joined by a coordination layer that lets them discover each other, negotiate access, and answer joint questions without copying the underlying genomes to a single location.

The alternative models each carry known trade-offs. Centralized archives such as the European Genome-phenome Archive (EGA) and dbGaP require data to be deposited and controlled centrally. Cloud-native platforms place analysis tools next to a large central data store, which scales well but concentrates both the data and the trust. Federated networks, exemplified in Europe by the Genomic Data Infrastructure (GDI), keep data at the source and move questions and results instead [1]. Recent comparative analysis of these three architectural families frames the trade-offs along three axes: scalability, data sovereignty, and analytical flexibility. Centralized models win on scalability and analytical flexibility. Federated models win on sovereignty. Cloud-native models sit in between [1].

Federation buys you two things a central bucket cannot:

1. Sovereignty. The hospital or sequencing center that generated the genomes remains the controller. Local consent scopes stay attached to local data.
2. Legal fit. Large-scale secondary use under the GDPR sits in tension with broad central deposition. The European Health Data Space and national Health Data Access Bodies push toward models where data stays local and permission is enforced at query time [1].

The honest cost is that federation is harder operationally. Semantic harmonization, schema alignment across sites, and machine-readable authorization remain active engineering problems rather than solved ones [1]. Build your pilot with that expectation.

The Core Architecture Components

Every federated genomic data network cloud architecture, regardless of provider, decomposes into five planes. Naming them separately keeps design conversations honest, because most failed pilots failed in exactly one of them.

1. Data plane. Per-site encrypted object storage holding CRAM, gVCF, and phenotype tables. Each site holds its own keys.
2. Compute plane. A workflow engine per site (Nextflow, Cromwell for WDL, or Snakemake) running on Kubernetes or on managed batch infrastructure. Container orchestration plus serverless triggering is the pattern proven at population scale by the Celeste infrastructure, which coordinates All of Us whole-genome analysis on AWS [2].
3. Federation control plane. A service registry listing what each site offers, a query API, and a mechanism for sites to authenticate each other and to users.
4. Policy layer. Machine-readable data use conditions attached to each dataset. The GA4GH framework of Passports (signed claims about who a user is and what they are cleared for) plus the Data Use Ontology (standardized terms such as disease-specific research use or no commercial use) is the emerging interoperable answer, and European infrastructures are converging on it [1].
5. Audit layer. Immutable, timestamped records of every query, who issued it, under which dataset permissions, and what was returned. Governance-oriented federated frameworks increasingly treat audit logging as a first-class architectural module rather than an afterthought [3].

A Reference Architecture Template

Copy this template into your design document and fill in the values per site. The format below is deliberately minimal so it fits in a pull request review or a grant appendix, and it doubles as a machine-readable source of truth for your deployment tooling.

# federated-network-template.yaml
network:
  name: my-genomics-federation
  reference_build: GRCh38
  sites:
    - id: site-a
      org: Partner Institution A
      cloud_region: us-east-1
      storage: s3://fedgen-site-a-genomes
      compute: eks/fedgen-site-a
      query_endpoint: https://site-a.genomics.example.org
      data_types: [wgs-cram, gvcf, phenotype-parquet]
      local_key_policy: one-kms-key-per-site
    - id: site-b
      org: Partner Institution B
      cloud_region: us-east-1
      storage: s3://fedgen-site-b-genomes
      compute: eks/fedgen-site-b
      query_endpoint: https://site-b.genomics.example.org
      data_types: [wgs-cram, gvcf, phenotype-parquet]
      local_key_policy: one-kms-key-per-site
  federation:
    discovery: service-registry-with-signed-descriptors
    authorization: passport-based-tokens-with-duo-terms
    query_model: summary-statistics-only-for-pilot
    aggregation: coordinator-collects-counts
  audit:
    log_store: cloudtrail-plus-s3-access-logs
    retention_years: 7

Every field in that file maps to a concrete build step in the walkthrough that follows. Treat a missing field as an unfinished decision, not a default.

The key design decisions and their standard options:

Step-by-Step Build on AWS

This walkthrough builds one site completely, then replicates it. The second site differs only in names and bucket identifiers. All commands assume us-east-1 and the placeholder bucket names shown. Run aws configure first if your CLI has no default profile.

Step 1: Create Per-Site Encrypted Storage

First confirm you are operating as the right identity, because every subsequent resource lands in that account.

aws sts get-caller-identity

Expected output:

{
    "UserId": "AIDAEXAMPLE123456789",
    "Account": "123456789012",
    "Arn": "arn:aws:iam::123456789012:user/fedgen-admin"
}

Create a dedicated KMS key for the site. One key per site, never shared, is what makes per-site sovereignty real. If the coordinating server cannot decrypt the genomes, it cannot leak them.

aws kms create-key \
  --description "fedgen site A CRAM key" \
  --key-usage ENCRYPT_DECRYPT \
  --origin AWS_KMS

Note the "KeyId" value in the JSON response, for example 1234abcd-12ab-34cd-56ef-1234567890ab. You will use it immediately. Then create the bucket and bind encryption to that key:

aws s3 mb s3://fedgen-site-a-genomes --region us-east-1

aws s3api put-bucket-encryption \
  --bucket fedgen-site-a-genomes \
  --server-side-encryption-configuration \
  '{"Rules":[{"ApplyServerSideEncryptionByDefault":{"SSEAlgorithm":"aws:kms","KMSMasterKeyID":"1234abcd-12ab-34cd-56ef-1234567890ab"},"BucketKeyEnabled":true}]}'

aws s3api put-public-access-block \
  --bucket fedgen-site-a-genomes \
  --public-access-block-configuration \
  BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true

Parameter notes:

• BucketKeyEnabled: true reduces per-object KMS API calls and lowers cost on the very high object counts typical of gVCF shards.
• The four true values in put-public-access-block are non-negotiable. A publicly readable CRAM is a reportable privacy incident.
• Set a bucket policy later that denies any request without TLS, so no client can accidentally upload or download over plaintext.

Common error at this step: AccessDenied on put-bucket-encryption usually means your identity lacks kms:DescribeKey on the new key. Fix by attaching a key policy that permits the site administrator role, then retry.

Verification: run aws s3api get-bucket-encryption, bucket fedgen-site-a-genomes. The returned JSON should show your KMS key ID and BucketKeyEnabled: true.

Step 2: Provision the Compute Cluster at Each Site

eksctl is the least error-prone path to a managed Kubernetes cluster suitable for genomics workloads. The node sizing here supports four concurrent whole-genome analysis tasks comfortably.

eksctl create cluster \
  --name fedgen-site-a \
  --region us-east-1 \
  --nodegroup-name compute \
  --nodes 4 \
  --node-type m5.2xlarge \
  --managed

Expect this to take 15 to 20 minutes. The final line of successful output reads:

[✔]  EKS cluster "fedgen-site-a" in "us-east-1" region is ready

Parameter notes:

• m5.2xlarge gives 8 vCPU and 32 GB per node. Alignment-heavy tasks want memory, and four such nodes let a small Nextflow workflow saturate without thrashing.
• Start with a single nodegroup. Split CPU and memory-optimized nodegroups later only if your pipeline profile shows a real imbalance.

Point your local client at the cluster and confirm:

aws eks update-kubeconfig --name fedgen-site-a --region us-east-1
kubectl get nodes

Expected output:

NAME                                          STATUS   ROLES    AGE   VERSION
ip-192-168-12-101.ec2.internal                Ready    <none>   3m    v1.3x.x
ip-192-168-12-44.ec2.internal                 Ready    <none>   3m    v1.3x.x
ip-192-168-12-77.ec2.internal                 Ready    <none>   3m    v1.3x.x
ip-192-168-12-9.ec2.internal                  Ready    <none>   3m    v1.3x.x

Common error: error: You must be logged in to the server (Unauthorized) from kubectl get nodes. This means the kubeconfig is stale or points at the wrong cluster. Re-run aws eks update-kubeconfig with the correct -name, and confirm with kubectl config current-context.

Repeat Step 1 and Step 2 for site-b with a second KMS key, bucket fedgen-site-b-genomes, and cluster fedgen-site-b.

Step 3: Deploy the Workflow Engine

For the pilot, run Nextflow on the EKS cluster and keep intermediate work files in the site bucket. The nf-core/sarek pipeline performs germline variant calling from fastq or CRAM through gVCF and VCF, which is exactly the artifact your federated queries will consume.

nextflow run nf-core/sarek \
  --input samplesheet.csv \
  --outdir s3://fedgen-site-a-genomes/results \
  -w s3://fedgen-site-a-genomes/work \
  --genome GRCh38 \
  -profile test

The -profile test run uses bundled test data and finishes in minutes, which makes it the right first integration check for the cluster, the S3 credentials, and the network path all at once.

Parameter notes:

• -input expects a samplesheet CSV with sample, fastq, and status columns. nf-core publishes the exact schema, and the pipeline validates it before launching anything expensive.
• -genome GRCh38 pulls the nf-core iGenomes reference configuration. Every site in your network must resolve this to the same reference build, or per-site counts will describe different coordinate systems.
• -w places work directories in S3 so failed tasks can be resumed after spot interruptions.

Common errors and fixes:

• Process 'NFCORE_SAREK:SAREK:... ' terminated for an unknown reason with exit code 137, meaning the task was killed for exceeding memory. Raise that process's memory directive in a local Nextflow config rather than shrinking node types.
• NoSuchBucket from an S3 stage-in step almost always means the bucket name or region mismatched between the samplesheet and the -w path. Run aws s3 ls | grep fedgen and compare.

Verification: the run summary table Nextflow prints at completion should show every process as ✔ (cached or executed successfully), and s3://fedgen-site-a-genomes/results/multiqc/multiqc_report.html should exist. Download and open that report. It is your fastest per-site quality gate.

At production scale this same pattern becomes serverless. Celeste, the infrastructure processing All of Us sequencing, combines serverless functions with container orchestration and in 12 months of production within a single AWS region coordinated roughly 200 million serverless function invocations and over 20 million messages to execute 1.8 million bioinformatics, quality control, and clinical reporting jobs, routinely absorbing monthly workloads of up to 9,000 whole-genome samples [2]. You do not need that shape on day one, but designing your pipeline steps as discrete, independently resumable container tasks is what makes that scale reachable later [2].

Step 4: Stand Up the Federation Query API at Each Site

Each site exposes one small HTTPS service that answers summary-statistics queries against its local VCFs. It never returns genotypes. Before exposing it, install an ingress with TLS termination and lock the service to the coordinator's network path.

At each site, produce counts for a query interval locally. For a single variant:

bcftools view -H -r chr17:43044295-43044295 siteA.vcf.gz \
  | cut -f8 \
  | tr ';' '\n' \
  | grep -E '^(AC|AN)='

Expected output:

AC=250
AN=10000

AC is the alternate allele count and AN is the total allele count in the local cohort. Wrap that logic in a service. A minimal FastAPI example shows the shape of the site-side endpoint:

from fastapi import FastAPI, Header, HTTPException
import subprocess

app = FastAPI()

@app.get("/query/af")
def query_af(chrom: str, pos: int, authorization: str = Header(default="")):
    if not authorization.startswith("Bearer "):
        raise HTTPException(status_code=401, detail="missing access token")
    cmd = [
        "bcftools", "view", "-H",
        "-r", f"{chrom}:{pos}-{pos}",
        "/data/cohort.vcf.gz",
    ]
    out = subprocess.run(cmd, capture_output=True, text=True)
    if out.returncode != 0:
        raise HTTPException(status_code=500, detail="index lookup failed")
    ac = an = 0
    for line in out.stdout.strip().splitlines():
        for tag in line.split("\t")[4].split(";"):
            if tag.startswith("AC="):
                ac = int(tag[3:])
            if tag.startswith("AN="):
                an = int(tag[3:])
    if an == 0:
        raise HTTPException(status_code=404, detail="variant not found in cohort")
    return {
        "site": "site-a",
        "alt_allele_count": ac,
        "total_allele_count": an,
    }

This toy example checks only for a token. Production systems should evaluate GA4GH Passport claims, signed assertions about a researcher's identity and clearances, against the Data Use Ontology terms attached to each dataset, checking that the requested use matches the permitted use before answering [1]. Build the plumbing now, swap the token check for Passport verification in the same code path.

Step 5: Run the First Federated Query End to End

The coordinator calls each site and pools the counts. The essential property is visible in the arithmetic: only integers travel between sites.

import httpx

SITES = {
    "site-a": "https://site-a.genomics.example.org",
    "site-b": "https://site-b.genomics.example.org",
}

def pooled_af(chrom, pos, token):
    ac = an = 0
    per_site = {}
    for site, base in SITES.items():
        r = httpx.get(
            f"{base}/query/af",
            params={"chrom": chrom, "pos": pos},
            headers={"Authorization": f"Bearer {token}"},
            timeout=60,
        )
        r.raise_for_status()
        body = r.json()
        per_site[site] = body
        ac += body["alt_allele_count"]
        an += body["total_allele_count"]
    af = ac / an if an else None
    return {"pooled_af": af, "alt_allele_count": ac,
            "total_allele_count": an, "sites": per_site}

print(pooled_af("chr17", 43044295, "ey..."))

Worked calculation for the two test cohorts:

site A: AC=250, AN=10000
site B: AC=150, AN=10000

pooled_af = (250 + 150) / (10000 + 10000)
          = 400 / 20000
          = 0.04

That single number, an alternate allele frequency of 0.04 across a pooled 20,000 alleles, is all any site learned about the other. No genotype, no haplotype, and no individual-level record crossed a site boundary. This is the canonical minimal federated genomic data network cloud architecture example, and it is worth running because it forces every operational decision, TLS, tokens, timeouts, logging, to work before you attempt anything harder.

Verification: recompute the same variant by pooling the two VCFs locally on synthetic data where that is permitted. pooled_af from the network must match the centralized recomputation to full floating-point precision. If it differs, one site's AN is wrong, which usually traces back to a reference build mismatch or to counting sex chromosomes with a different ploidy convention.

Common error: httpx.ReadTimeout on one site. The coordinator's 60 second timeout is a design parameter, not a detail. Sites with hundreds of shards need precomputed per-variant indexes, such as site-local queryable count caches, so a query is a lookup rather than a scan. Measure p95 latency per site before scaling the network beyond two sites.

Step 6: Audit Logging and Monitoring

Create an audit trail at each site before the first real query, not after the first review request.

aws s3 mb s3://fedgen-site-a-auditlogs --region us-east-1

aws cloudtrail create-trail \
  --name fedgen-audit \
  --s3-bucket-name fedgen-site-a-auditlogs \
  --is-multi-region-trail

aws cloudtrail start-logging --trail-name fedgen-audit

Verify logging is active with aws cloudtrail get-trail-status, name fedgen-audit, which should return "IsLogging": true. In addition to control-plane events, enable S3 server access logging or CloudTrail data events on the genome buckets so that object reads are recorded.

Governance-focused federated architectures treat this audit stream as a core module, combining policy enforcement, versioned federated state, and immutable audit logging, with differential privacy applied to anything derived features leave the site boundary [3]. Mirror that posture early. Retrofitting audit trails onto a running network is far more painful than enabling them at creation.

Scaling Analysis Beyond Counts: Federated Statistics and Learning

Allele frequencies are the entry point. The same pattern extends to real analytical workloads in three tiers.

Tier 1, federated summary statistics. Each site computes local GWAS summary statistics, effect sizes, standard errors, and p-values per variant, and the coordinator performs fixed-effect or random-effect meta-analysis. This tier is where most networks should start, because the exchanged artifacts are small, statistically interpretable, and straightforward to review.

Tier 2, federated machine learning on per-site data. Complex phenotypes with polygenic and epistatic architecture strain single-site samples. VariantSpark, a distributed cloud-based framework for association analysis, scales to roughly 100 million genomic variants across 100,000 samples and was measured at 3.6 times the speed of ReForeSt on that problem class [5]. A practical federated deployment runs a tool like this per site and federates the resulting local statistics or model components, keeping genotypes local. Multi-institution epistasis detection faces the same underlying constraint that motivates the whole field, that a single institution's cohort is usually too small for statistically meaningful detection of complex disease-associated variants, and that privacy and trust concerns impede central pooling, which is why federated frameworks with sequence perturbation and privacy-preserving aggregation have been proposed for exactly this task [6].

Tier 3, federated model training. In this tier sites train local models and share updates rather than data. Reviews of federated learning architectures describe the practical constraints you will meet here, including data and device heterogeneity, communication cost, and the robustness, fairness, and explainability pillars that trustworthy federated systems must address [7]. Prototype implementations combining Kubernetes orchestration with federated TensorFlow have demonstrated the orchestration, communication optimization, and fault-tolerance patterns such deployments need. Expect convergence issues from site heterogeneity. If your sites differ in ancestry composition, sequencing platform, or phenotype definitions, a naive averaging scheme will drift toward the largest sites.

Cloud bioinformatics makes each tier cheaper to run. Kubernetes-based distributed cloud platforms for GWAS-scale analysis, integrating resources spanning tens of thousands of phenotypes, have demonstrated second-level data extraction and reductions in local storage requirements above 99 percent compared with terabyte-scale local downloads [8]. The operational lesson generalizes: in a federation, keep the heavy compute near the data and move small, well-defined artifacts.

Security Choices and Where Homomorphic Encryption Fits

Security in a federated genomic data network has four layers, and you should implement them in this order:

1. Encryption at rest with per-site KMS keys, as built in Step 1. This is table stakes and has no performance excuse.
2. Encryption in transit with TLS on every hop, including site-to-site calls, not just user-facing endpoints.
3. Fine-grained authorization with Passport-style claims evaluated against machine-readable data use terms [1].
4. Computation on protected data where the threat model demands it.

That fourth layer is where homomorphic encryption (HE) enters. HE allows computation directly on ciphertext, so a site can contribute to a computation without revealing its inputs even to the coordinator. A review of cryptographic approaches for genomic data on federated clouds analyzes existing attacks on genomic data and the case for HE schemes as a safeguard for genomics data held in federated cloud environments [4]. The practical position for a first network: HE is a valid research direction and a sensible choice for specific high-sensitivity aggregates, but its computational overhead makes it a poor default for whole-cohort workloads today. Design your architecture so the aggregation interface could be swapped for an HE-based implementation later, which is exactly the benefit of keeping the query contract small, as in Step 5.

Common Mistakes and Limitations

• Treating federation as a purely technical project. The binding constraints in real genomic infrastructures are legal and semantic as often as technical. Consent scope, access body processes, and schema harmonization across sites remain open problems even in mature European networks [1]. Budget coordination time, not just engineering time.
• Skipping the machine-readable policy layer. A PDF data use agreement nobody can query is not a policy layer. Attach Data Use Ontology terms to each dataset at registration time and enforce them in the query path.
• Leaking genotypes through side channels. Error messages, logs, and profile summaries can carry individual-level fragments. A stack trace that echoes a variant line, or a slow query enabling a membership inference attack against small cohorts, defeats the whole design. Filter error text at the site boundary and suppress or perturb results for variants present in very few individuals.
• Reference build drift. One site on GRCh37 answering chr17:43044295 from the wrong coordinate system silently poisons every pooled statistic. Pin the build in the network template, verify at onboarding, and reject queries that arrive without a build declaration.
• Assuming the coordinator is trusted. A central aggregator is a reduction in trust, not an elimination of it. If sites cannot agree to trust one coordinator, design for peer-to-peer or committee-based aggregation from the start rather than bolting it on.
• Confusing federated queries with federated learning. They share a name and little else. Federated queries move counts and statistics with deterministic correctness checks. Federated learning moves model updates, inherits convergence and heterogeneity problems, and needs the robustness and fairness machinery described in the federated learning literature [7]. Start with queries.
• Pilot results as production predictions. A two-site run on synthetic or reference data proves the plumbing works. It does not predict latency, cost, or statistical behavior at population scale, where burst workloads like the 9,000 WGS samples per month seen in production systems [2] stress queues, indexes, and network paths in ways small pilots never will.

Frequently Asked Questions

What is a federated genomic data network in the cloud?

It is a set of institutions whose genomic data stays within each institution's own cloud storage and compute, joined by a shared control plane that handles discovery, authorization, and joint queries. Only questions and small results cross site boundaries, never the raw genomes.

Does federation mean no data ever leaves the local site?

Correctly built, raw genotypes never leave the site. Summaries such as allele counts, aggregate statistics, or model updates do cross boundaries, and your access agreements and privacy protections must cover those outputs explicitly.

Which AWS services should a genomics federation use?

The core pattern is S3 for per-site encrypted storage, KMS for per-site keys, EKS or batch services for containerized workflow execution, and CloudTrail for audit. Population-scale systems on AWS have combined serverless functions with container orchestration to coordinate millions of analysis jobs [2].

What are GA4GH Passports and where do they fit?

Passports are signed claims about a researcher's identity and clearances, evaluated together with Data Use Ontology terms that state what a dataset may be used for. European genomic data infrastructures are adopting this pairing as the interoperable machine-readable authorization layer [1].

Can I run a federated genome-wide association study?

Yes, most commonly by having each site compute local summary statistics and the coordinator perform meta-analysis. For polygenic and epistatic questions, distributed cloud frameworks can handle roughly 100 million variants across 100,000 samples per site, and federated designs extend multi-institution analysis without pooling genotypes [5].

Is homomorphic encryption a practical requirement?

Not for a first network. Homomorphic encryption can safeguard genomic computations on federated clouds by operating on ciphertext, but its computational cost makes it a targeted tool for specific high-sensitivity aggregates rather than a default for every query [4].

How large should a first federated pilot be?

Two sites, one reference build, one query type such as pooled allele frequency, and synthetic or reference test data. Two sites expose nearly every design flaw, and you can fix them before real cohorts and multiple access bodies raise the stakes.

How do I prepare for access body review and audits?

Log every query with requester identity, the datasets and data use terms invoked, the query content, and the response, in an immutable store, from the first day of operation. Governance-oriented federated frameworks treat policy enforcement and audit logging as core architectural modules, which is the posture access bodies expect [3].

Sources

1. Federated, governed, and interoperable? The emerging architecture of public human genomic data infrastructures: a European perspective.
2. Celeste: A cloud-based genomics infrastructure with variant-calling pipeline suited for population-scale sequencing projects.
3. Federated Learning Framework for Secure Data Sharing and Governance in Oracle Cloud Environments
4. Homomorphic Encryption for Genomics Data Storage on a Federated Cloud: A Mini Review
5. VariantSpark: Cloud-based machine learning for association study of complex phenotype and large-scale genomic data.
6. A Novel Centralized Federated Deep Fuzzy Neural Network with Multi-objectives Neural Architecture Search for Epistatic Detection
7. A Review on Federated Learning Architectures for Privacy-Preserving AI: Lightweight and Secure Cloud-Edge-End Collaboration
8. Cloud-Based GWAS Platform: An Innovative Solution for Efficient Acquisition and Analysis of Genomic Data