Data Federation vs. Data Centralization for Population Genomics

By Dr. Zubair Khalid, DVM, MS, PhD · 2026-11-11

Data centralization copies genomic data into one controlled store or platform where compute runs next to the data. Data federation leaves the data where each site produced it and moves queries, indexes, or workflow code to the data instead. The rule of thumb is simple: centralize when one organization owns the cohort and the analyses are compute heavy and iterative, federate when the data cannot or should not move because of governance, ownership, or sheer size and the analyses are largely performed per site.

This article explains each architecture on its own terms, compares them on the criteria that decide real infrastructure choices, walks through worked scenarios at different scales, and lists the mistakes that cost teams the most time and money. The examples draw on published population-scale systems, including the All of Us genomics platform, the Seven Bridges Cancer Genomics Cloud, and the hybrid cloud built for Japanese national genomics projects.

Background: Why This Choice Exists at All

A single 30x whole-genome sequence produces on the order of 100 gigabytes of raw data, and population projects now sequence tens of thousands to millions of people. National biobank programs such as the UK Biobank, the All of Us Research Program, Singapore's PRECISE, Biobank Japan, and Korea's National Project of Bio-Big Data are generating unprecedented volumes of genomic data integrated with phenotypic, environmental, and clinical information [1]. At that scale, the physics and economics of moving bytes dominate architectural decisions. This is the core tension in cloud bioinformatics: analysis tools want data in one place, while the institutions holding the data often have legal, ethical, or practical reasons to keep it distributed.

The phrase "AWS genomics" and its equivalents on Google Cloud and Azure refer to the managed building blocks, object storage, elastic virtual machines, batch schedulers, serverless functions, and query warehouses, that both architectures sit on top of. The choice between federation and centralization is not a choice between vendors. It is a choice about where data lives and where computation happens relative to the data.

What Data Centralization Means in Cloud Bioinformatics

Data centralization means the cohort's sequence files, variant calls, and metadata are ingested into storage controlled by one operator, usually object storage in a single cloud region, and all downstream analysis runs against that copy. The defining property is colocalization: data, tools, and compute live close together, so analyses never pay network transfer costs after ingestion.

The Canonical Pattern: Colocalized Data, Tools, and Compute

The Seven Bridges Cancer Genomics Cloud (CGC) is a clean illustration. The CGC provides secure on-demand access to massive public cancer genomic datasets, including The Cancer Genome Atlas, alongside more than 200 pre-installed and curated bioinformatics tools and workflows. By colocalizing data, analysis tools, and computing resources in the cloud, the platform enables scalable and reproducible analyses for researchers worldwide [2]. A researcher can open the platform, select a tumor cohort, and run a variant-calling workflow without downloading a single file. The data was moved once, at ingestion, and every subsequent analysis benefits.

The All of Us Research Program shows the same pattern at extreme scale. Celeste, the open-source cloud architecture behind All of Us genomics, runs within a single Amazon Web Services Region and processes fluctuating workloads of up to 9,000 whole-genome sequencing samples per month. In 12 months of production it executed around 200 million serverless functions and coordinated over 20 million messages to run 1.8 million bioinformatics, quality control, and clinical reporting jobs [3]. Two design elements matter for our comparison. First, the entire pipeline estate lives in one region, which keeps the petabyte-scale data movement internal to the cloud provider's network. Second, serverless technology and container orchestration let the system absorb workload spikes without a standing fleet of machines [3]. That combination is only economical because the data is centralized and the compute is elastic.

Centralization also enables a different class of resource: precomputed knowledge bases. The Functional Genomics Platform captures relationships among 200,000 curated bacterial genomes, over 68 million genes, 52 million proteins, and 239 million protein domains in a relational database that returns cross-entity queries in a fraction of the time traditional per-analysis methods require [4]. You cannot build that artifact without first centralizing the underlying data, normalizing it, and keeping it under continuous curation.

Why Centralization Wins on Analytical Depth

Population genomics methods that need to see all samples simultaneously favor centralization. VariantSpark, a distributed machine-learning framework for association studies, scales to 100 million genomic variants and 100,000 samples and identifies variants tied to complex polygenic phenotypes, including epistatic interactions that single-variant genome-wide association studies miss [5]. Multi-sample joint genotyping, principal component analysis for population structure, and genome-wide association scans all assume the analyst can shuffle records across the full cohort. Moving the whole cohort into one system makes those operations straightforward. Large-scale comparative studies follow the same logic. The Salmonella enterica population-structure study that redefined serovar classification drew on more than 180,000 genomes aggregated from the GenomeTrakr and PubMLST databases [6]. Ingest, harmonize, then compute against everything at once is the centralization playbook.

What Data Federation Means

Data federation is the counter-architecture. Data stays at its source institution, under that institution's control, and analysis reaches it through a virtual layer: federated query engines, portable workflows shipped to each site, or indexing schemes that summarize data locally and share only the summaries. No single operator ever holds the full cohort. The unifying idea is that the query or the code travels, not the genome.

Federated Querying with Data Warehouses

Cloud data warehouses blur the line between the two architectures in an instructive way. The Mitelman Database of Chromosome Aberrations and Gene Fusions in Cancer, covering more than 70,000 cases, was mapped to genomic coordinates and made publicly available in Google BigQuery, a cloud-based data warehouse, which allows integrative analysis with other cancer datasets hosted by the Institute for Systems Biology Cancer Gateway in the Cloud resource [7]. The coordinate mappings were centralized into BigQuery, but the analytical integration happens through queries that join BigQuery tables to other hosted datasets without the analyst ever exporting full copies. This is federation at the query layer: the analyst expresses intent, the warehouse resolves where the data physically resides, and only results move.

The same principle applies when each institution maintains its own warehouse and a federation layer dispatches sub-queries to each one. Each site returns aggregate statistics rather than individual-level records. For analyses expressible as sums, counts, and model summaries, this preserves the analytical question while keeping records local.

Workflow Portability as Lightweight Federation

A second federation pattern ships the workflow to the data. The rMATS-cloud project packaged the rMATS alternative-splicing workflow in CWL, WDL, and Nextflow formats on Dockstore and demonstrated it on three platforms, Cavatica, Terra, and Seqera, over datasets with thousands of samples [8]. When the same portable workflow can run at every holding site, the consortium as a whole behaves like one federated supercomputer even though no data moved. Each site runs the workflow on its own samples, and only per-sample outputs or summary statistics are pooled.

The Galaxy-based cloud platform described by Liu and colleagues adds the complementary piece: reliable bulk data transfer when some movement is unavoidable. That platform extends Galaxy with data management via Globus Transfer, preconfigured domain-specific tools, automatic cloud deployment through Globus Provision, and auto-scaling through an HTCondor scheduler [9]. In practice, many consortia use exactly this hybrid of patterns: harmonized portable workflows plus controlled transfer pipelines for the subset of data that must be colocated for a given analysis.

Hybrid Architectures

Federation and centralization are endpoints of a spectrum, and national programs frequently land in the middle. The Japanese hybrid cloud system for human genomic research combines an on-premise supercomputer, university science clouds, and Amazon Web Services. The team demonstrated the design by analyzing more than 11,000 human genome sequences, distributing pipeline stages across sites while enforcing reproducibility, and adding resources on demand, for example GPU nodes for deep-learning-dependent workflows such as pathology image analysis [10]. Data that Japanese institutions must keep on domestic infrastructure stays there. Workloads that benefit from elastic cloud capacity burst to AWS. The architecture is federated in ownership and centralized where computation needs it.

Data Federation vs. Data Centralization for Population Genomics: Side by Side

The table below is the core of the decision. Read it as a set of tradeoffs, not a scorecard.

Criterion	Data Centralization	Data Federation
Data location	Copied into one operator's storage, typically one cloud region	Remains at each source institution or repository
How analysis runs	Compute moves next to the data once, then runs repeatedly	Queries or portable workflows move to each data site
Example systems	All of Us Celeste on AWS [3], Seven Bridges CGC with TCGA [2], Functional Genomics Platform [4]	BigQuery-based integrative analysis with ISB-CGC datasets [7], portable rMATS workflows on Dockstore across Terra, Cavatica, and Seqera [8], hybrid Japanese national system [10]
Analytical depth	Full-cohort joint methods, machine learning across all samples [5]	Analyses decomposable into per-site computations plus aggregation
Governance	One consent framework, one security boundary	Per-site consent rules, per-site security review
Cost profile	High ingestion and storage cost, then cheap repeated compute; elastic compute can be very cheap when scheduled well [11]	Avoids bulk transfer and long-term duplication, pays in coordination and cross-site latency
Reproducibility	Straightforward, one pinned environment	Requires disciplined workflow portability (CWL, WDL, Nextflow) and version pinning at every site
Time to first result	Fast after ingestion for any analysis on that cohort	Fast for per-site questions, slower for cross-cohort harmonization
Typical failure mode	Consent or export restrictions block data sharing, storage bills grow with duplication	Inconsistent tool versions and reference data across sites produce incomparable results

Two clarifications keep this table honest. First, the difference between data federation and centralization in genomics cloud settings is not centralization versus no infrastructure. Both need object storage, schedulers, workflow engines, and identity management. The difference is who holds the data and in which direction the heavy payloads move. Second, the choice is rarely permanent. Projects commonly centralize public data (where consent permits) and federate controlled-access data, adjusting the boundary as agreements evolve.

The Criteria That Decide Real Choices

Side-by-side comparison of centralization and federation on transfer cost, coordination, governance and team burden.

Data Gravity and Transfer Economics

Data gravity is the observation that large datasets attract computation, because moving petabytes repeatedly is slower and more expensive than moving code. Centralization pays the transfer cost once, at ingestion, and then amortizes it across every analysis. This is why the Cancer Genomics Cloud colocates TCGA with 200-plus tools so researchers can analyze data immediately in the cloud [2], and why Celeste keeps its entire pipeline in a single AWS Region despite workloads spanning thousands of genomes per month [3].

The counter-example shows when gravity does not bind. The National Lung Screening Trial curation study used the NCI Cancer Research Data Commons cloud resources, Terra (FireCloud) and Seven Bridges CGC, to run AI segmentation over more than 126,000 CT volumes from more than 26,000 patients, using over 21,000 virtual machines to finish in under 9 hours what a single workstation would need an estimated 522 days to complete, at a total compute cost of $1,011.05 [12]. The data was already cloud-hosted by the repository, so the "centralization" had effectively been paid for by the data provider, and the research team paid only for elastic compute. The lesson: if a trusted repository has already centralized the data and granted you access, re-centralizing it yourself buys nothing.

Federation pays ongoing coordination costs instead of transfer costs. Every cross-site analysis needs agreement on tool versions, reference genomes, and output schemas. Portable packaging through Dockstore and format standards such as CWL and WDL exists precisely to reduce that cost [8].

Governance, Consent, and Data Sovereignty

Consent terms decide more architectures than engineering does. Controlled-access human genomic data often carries legal or consent-based restrictions on where it may be stored and who may touch it. When multiple countries or sovereign institutions are involved, copying raw genotypes to a foreign cloud region may simply be prohibited. Federation is the only workable answer for those datasets. The Japanese hybrid system exists for exactly this reason, keeping data under national institutional control while still enabling 11,000-genome-scale analysis through distributed, reproducible pipelines spanning on-premise and cloud resources [10].

Centralization is appropriate when a single program holds consent for the whole cohort. All of Us sequenced over one million participants from diverse backgrounds with an explicit design goal of integrating genetic data with longitudinal health information, which justifies a single governed estate [3]. National biobank reviews emphasize that each program makes distinct choices about data collection, access, and processing infrastructure, including multi-sample VCF construction, optimized storage formats, and cloud-based computing, precisely because governance models differ [1].

Cost Models: Elastic Versus Fixed

Centralization concentrates spend on storage and egress. Federation concentrates spend on engineering time and duplicate storage at every site. Elastic cloud compute changes the calculus for both. GT-WGS, a distributed WGS analysis system on AWS, exploited AWS's dynamic pricing mechanism so that a 55x WGS analysis of a 400 GB FASTQ dataset finished in 18.4 minutes at a total AWS cost of 16.5 US dollars, with accuracy 99.9 percent consistent with the GATK best practices, and it won first prize at the GCTA 2017 Wind and Cloud challenge [11]. Numbers like that show why centralized, elastic compute is attractive for bursty per-sample workloads: you rent thousands of core-hours for minutes, then pay nothing.

But those numbers apply to per-sample embarrassingly parallel work. Cross-cohort statistical methods with small effect sizes and demanding model assumptions, such as semiparametric estimation of genetic effects in large cohorts, need the full cohort in one analytical environment [18 inapplicable here, so stated qualitatively]. When the algorithm itself requires joint access to all samples, federation must either aggregate to summaries that preserve the estimand or accept a centralized component, which is exactly the design of modern national pipelines that construct multi-sample VCFs at a central tier [1].

Reproducibility and Provenance

Centralized platforms make reproducibility nearly free because there is one environment to pin. Federated systems must enforce reproducibility across many environments simultaneously. The Japanese hybrid system explicitly designed for distributed execution while ensuring reproducibility [10], and rMATS-cloud's packaging in three workflow languages on a public registry [8], represent the current standard answer: containerized, version-pinned, registry-hosted workflows that any site can execute identically. If your consortium cannot commit to that discipline, federation will produce results that cannot be compared across sites.

Team Skills and Operational Burden

Running a centralized estate at population scale is an engineering discipline. Celeste's throughput of roughly 200 million serverless functions per year implies dedicated platform engineering, monitoring, and cost management [3]. Federation spreads operational burden across sites but adds a coordination role that many consortia underestimate. A small lab without platform engineers should start with managed, centralized platforms. DeepVariant-on-Spark, deployable on Google Cloud Platform within about 20 minutes using free credits, with capacity for at least ten whole-genome datasets, targets exactly this situation: pilot-scale analysis without an infrastructure team [13].

When to Use Data Federation

A short triage path ending in a federated, hybrid or centralized design for a genomics cohort.

Use federation when one or more of the following holds.

Legal, consent, or sovereignty rules prevent copying raw genomic data into a shared location. This is the strongest and most common trigger, and it drives designs like the Japanese hybrid national system [10].
Multiple institutions already hold harmonized sub-cohorts and each can run identical, containerized workflows locally, with only summary outputs pooled, as the portable-workflow model enables [8].
The analysis decomposes cleanly into per-site computations. Allele counting, quality metrics, per-sample variant calling, and most summary statistics federate naturally. Joint genotyping and cross-cohort machine learning do not, unless you adopt specialized distributed methods such as VariantSpark's parallelized association testing over 100 million variants and 100,000 samples [5].
Query-level access suffices. If the research question can be answered by joining aggregate or coordinate-level resources, a warehouse-based federation pattern, as demonstrated for the Mitelman Database with BigQuery and ISB-CGC [7], avoids any bulk movement.

Use centralization when the inverse holds: one governance umbrella, analyses that need all samples together, and a plan to run many analyses against the same cohort over years. The economies are real. A centralized cohort ingested once supports unlimited elastic compute at spot-like prices, and the cloud platforms built for this purpose come with curated tool ecosystems that remove most setup work [2].

Worked Scenarios

Scenario 1: A National Program Sequencing One Million Genomes

You are designing the genomics arm of a national cohort. Consent permits centralized processing. The pipeline must handle thousands of new whole genomes per month with fluctuating arrival rates. This is the centralization case, and Celeste is the reference design: one cloud region, serverless orchestration, containerized workflow steps, millions of jobs per year, and scaling to 9,000 WGS samples monthly [3]. Federation has nothing to offer here, because the program itself is the single data holder and the analytical roadmap includes joint genotyping and population-scale association testing that require the full cohort.

Scenario 2: A Multi-Country Consortium with Restricted Data

Six universities in different jurisdictions each hold 2,000 genomes under national data-protection rules. Raw genotypes cannot leave each country. The consortium wants to identify rare variants and estimate variant frequencies across the full 12,000-genome cohort. The workable design is federated: agree on a single versioned variant-calling workflow distributed through a public workflow registry in CWL or WDL [8], run it independently at each site, and pool only variant-level aggregate counts with a defined minimum cell size. For any downstream method requiring individual-level joint analysis, the consortium must either negotiate a trusted central enclave for derived, de-identified matrices (a partial centralization) or choose methods designed for summary-level combination. The Japanese system demonstrates that even within one country, hybrid designs mixing on-premise and cloud resources can reach 11,000-genome scale while respecting institutional control [10].

Scenario 3: A Small Lab Running a Pilot

You have 50 transcriptomes and no infrastructure team. The question is whether your splicing event of interest changes across conditions. Do not build either architecture. Use a managed platform where the data and tools are colocated, run a portable workflow such as rMATS-cloud on one of its supported platforms [8], or deploy a prebuilt cloud pipeline for variant discovery if sequencing analysis is the goal [13]. At this scale the federation-versus-centralization question is premature. Your real constraints are budget, workflow portability for when you scale up, and picking a platform whose governance terms will still suit you when the pilot becomes a grant.

Scenario 4: Adding a New Data Type to an Existing Cloud Resource

You maintain a curated coordinate-level resource of 70,000 cancer cases and want researchers to combine it with genomic datasets they already use. Copying other people's datasets into your system would be duplication without consent. Publishing your resource as queryable tables in a cloud data warehouse, as the CytoConverter team did with the Mitelman Database in BigQuery [7], lets others join it against their datasets of choice. This is federation as a publication strategy, and it is increasingly the norm for reference resources.

Common Mistakes and Limitations

Treating the choice as all-or-nothing. Nearly every production system at scale is hybrid. All of Us centralizes within one cloud region [3]. The Japanese national system federates across on-premise, university, and cloud resources [10]. Deciding "federation, period" or "centralization, period" usually signals that the governance analysis has not been done per dataset.

Ignoring data gravity until the first big transfer. Teams that store data in one region and compute in another discover egress costs and latency at exactly the wrong moment. The entire value proposition of platforms like the CGC is colocalization of data, tools, and compute [2]. If you centralize, centralize compute too.

Assuming portable workflows make federation free. A workflow packaged in CWL, WDL, and Nextflow on Dockstore [8] solves the software side. It does not solve reference data harmonization, metadata schema agreement, or per-site security review. Budget real calendar time for those.

Underestimating centralization's governance bottleneck. A single data holder is a single point of policy failure. If consent scope changes or a partner exits, the centralized design must be renegotiated. National biobank experience shows governance and data-access models vary substantially between programs and drive technical choices [1].

Confusing warehouse centralization with federation. Putting derived tables in BigQuery and joining them to other hosted datasets [7] feels federated to the analyst, but the published tables are physically centralized. Be precise in project documentation about what moved and what did not, because audit and consent language cares.

Picking the architecture before picking the methods. If your headline analysis requires joint modeling of all samples, plan for a centralized tier or a method specifically built for distributed execution [5]. If your analyses are per-sample, federation is cheap. Sequencing the architecture decision before the analysis plan leads to expensive rework.

A limitation of the evidence itself. Published comparisons of federation versus centralization at population scale come from specific programs with specific consent regimes, cloud vendors, and budgets. Reported figures, such as Celeste's job volumes [3] or GT-WGS's per-sample cost [11], reflect those implementations and pricing at the time of publication. Treat them as demonstrated feasibility, not guaranteed quotes for your project. And as with any research infrastructure decision, individual studies and clinical interpretations still require appropriately qualified oversight within your own institution.

Frequently Asked Questions

What is the difference between data federation and data centralization in genomics cloud systems?

Centralization copies genomic data into one operator's storage and runs compute next to it, while federation leaves data at each source site and moves queries or portable workflows to the data. The difference is which asset travels: centralized systems move data once and then run code locally, federated systems move code and queries to each data holder.

When should I use data federation for a genomics project?

Use federation when consent, law, or sovereignty prevents copying raw genomic data to a shared location, when multiple institutions can run identical containerized workflows locally, and when your analyses decompose into per-site computations plus pooled summaries. If your key analysis requires joint access to all individual samples, federation needs a centralized component or specialized distributed methods.

Is federation cheaper than centralization for population genomics?

Neither is categorically cheaper. Centralization front-loads ingestion and storage costs but allows very inexpensive elastic compute afterward, with published examples of a 55x whole-genome analysis finishing in under 19 minutes for about 17 US dollars on AWS using dynamic pricing [11]. Federation avoids bulk transfer and duplication but pays in coordination labor, duplicated infrastructure at each site, and cross-site harmonization time.

Can a project mix federated and centralized components?

Yes, and most production systems do. A common design centralizes public or program-consented data for compute-heavy joint analyses, and federates controlled-access data through portable workflows or aggregate-only queries. The Japanese national genomic analysis system is a published example, combining on-premise supercomputing, university resources, and AWS in one reproducible architecture [10].

How do national biobank programs handle this decision?

Each program chooses independently based on its governance model, consent scope, and analysis roadmap, with published reviews covering the UK Biobank, All of Us, Singapore's PRECISE, Biobank Japan, and Korea's national bio-big-data project [1]. Shared technical patterns include multi-sample VCF construction, optimized storage formats, and cloud-based processing, but the federation-versus-centralization boundary differs by program.

Do I need a large engineering team to run centralized cloud genomics?

For population scale, effectively yes, since systems like Celeste coordinate millions of jobs and hundreds of millions of serverless invocations per year [3]. For pilot-scale work, no: managed platforms and prebuilt deployable pipelines let small teams start analysis within hours, as demonstrated by cloud-deployable variant-calling frameworks usable with free cloud credits [13]. Match the architecture to your team size before committing.