Benchmarking Atlas-Level Data Integration in Single-Cell Genomics: Methods and Best Practices

Single-cell atlases frequently combine samples collected across multiple locations, laboratories, conditions, and sequencing protocols, which introduces complex and nested batch effects into the data. Reliable data integration is therefore a prerequisite for joint analysis of atlas-scale datasets, and the choice of integration method can materially change biological conclusions. This article provides a comparison framework for evaluating integration methods such as Harmony and scVI, with explicit scoring criteria for batch correction and biological preservation, practical workflow steps, record-keeping guidance, and escalation criteria for when results warrant professional review.

Scope and Reader Context

This guidance is written for students, researchers, analysts, and life-science professionals who need to select and evaluate data integration methods for large-scale single-cell datasets. The focus is on atlas-level integration tasks, where datasets may include gene expression, chromatin accessibility, protein abundance, and spatial transcriptomics data. The framework presented here draws on published benchmarking studies that compared dozens of methods across hundreds of batches and millions of cells. The practical outcome is a reproducible evaluation workflow that lets you score candidate methods on your own data before committing to a production analysis pipeline.

The Integration Problem in Atlas-Scale Single-Cell Genomics

Atlas-level datasets are distinct from typical single-experiment datasets in several ways that affect integration method choice. First, the number of cells is large, often exceeding one million cells across dozens of batches. Second, the batches themselves are nested, meaning that cells from one laboratory may have been processed in multiple sequencing runs, and cells from one sequencing run may come from multiple donors or conditions. Third, the biological variation of interest is often rare cell types or subtle cell states that can be obscured by technical variation.

The benchmark by Luecken et al. evaluated 68 method and preprocessing combinations on 85 batches of gene expression, chromatin accessibility, and simulation data from 23 publications, representing more than 1.2 million cells distributed across 13 atlas-level integration tasks. The authors assessed methods on scalability, usability, and the ability to remove batch effects while retaining biological variation, using 14 evaluation metrics. This study remains the most extensive public comparison of atlas-level integration methods and provides a useful reference point for method selection.

A separate benchmark of single-cell multi-modal integration methods evaluated 40 algorithms across DNA, RNA, protein, and spatial omics modalities for paired, unpaired, and mosaic datasets. Mosaic datasets contain a mixture of paired and unpaired measurements, which is common in real-world multi-omics studies. The authors assessed usability, accuracy, and robustness to guide method selection for different dataset configurations.

Core Principles of Integration Method Evaluation

Batch Correction and Biological Preservation Are Competing Objectives

Every integration method makes a tradeoff between removing technical variation and preserving biological variation. Methods that aggressively remove batch effects can also remove genuine biological differences between conditions or individuals. Methods that preserve biological variation may leave residual batch signal that confounds downstream analysis. The evaluation framework must therefore score methods on both objectives simultaneously.

The kBET method was introduced to quantify batch effects in single-cell transcriptomics using a k-nearest-neighbor approach. It measures whether the local neighborhood composition of each cell reflects the expected batch composition under the null hypothesis of perfect mixing. kBET was used to assess commonly used batch-regression and normalization approaches and to quantify the extent to which they remove batch effects while preserving biological variability. The Batch Probing Score (BPS) is a supervised metric that directly measures residual batch signal after correction, and it has been reported to have the highest sensitivity among existing metrics used to evaluate batch correctors.

Preprocessing Choices Affect Integration Outcomes

The choice of feature space and preprocessing steps can have a larger effect on integration quality than the choice of integration algorithm itself. The atlas-level benchmark found that highly variable gene selection improves the performance of data integration methods, whereas scaling pushes methods to prioritize batch removal over conservation of biological variation. This finding has direct practical implications: the same integration method can produce different results depending on whether you use all genes, highly variable genes, or scaled data.

For single-cell ATAC-sequencing data, integration performance is strongly affected by the choice of feature space. Methods that work well for gene expression data may perform poorly on chromatin accessibility data if the feature representation is not appropriate. The benchmark recommended evaluating multiple feature spaces when integrating ATAC-seq data.

Method Performance Depends on Dataset Configuration

The optimal integration method depends on the structure of your data. For atlas-level gene expression integration, scANVI, Scanorama, scVI, and scGen performed well in the benchmark, particularly on complex integration tasks. For multi-modal integration, the choice of method depends on whether your data are paired, unpaired, or mosaic. A benchmark of 14 protein abundance and chromatin accessibility prediction algorithms and 18 multi-omics integration algorithms found that totalVI and scArches outperformed other algorithms for predicting protein abundance, while LS_Lab was the top performer for chromatin accessibility prediction. For vertical integration, Seurat, MOJITOO, and scAI emerged as leading algorithms, whereas totalVI and UINMF excelled in horizontal and mosaic integration scenarios.

For spatial transcriptomics integration, the task matters. A benchmark of 16 integration methods using 45 paired datasets and 32 simulated datasets found that Tangram, gimVI, and SpaGE outperformed other methods for predicting the spatial distribution of RNA transcripts, whereas Cell2location, SpatialDWLS, and RCTD were the top performers for cell type deconvolution of spots.

At a Glance: Integration Method Selection by Dataset Type

Building an Integration Evaluation Workflow

Step 1: Define the Integration Task and Success Criteria

Before running any integration method, write down the specific task you are trying to solve. Are you integrating multiple batches of the same tissue type to increase cell number for rare cell type discovery? Are you integrating different modalities to link gene expression with chromatin accessibility? Are you integrating across species to identify conserved cell types? Each task has different success criteria.

Define success criteria in measurable terms. For batch correction, this could be the proportion of batches represented in each cell's local neighborhood. For biological preservation, this could be the ability to recover known cell type markers or the separation of known biological conditions. Write these criteria down before running any method to avoid post hoc rationalization of results.

Step 2: Prepare and Document Input Data

Document the source of each dataset, including the sequencing protocol, the number of cells, the number of genes or features, the donor or condition labels, and any known technical artifacts. The NCBI Data Resources provide access to many public single-cell datasets, and the EMBL-EBI Training portal offers guidance on data handling and analysis workflows. Record the exact preprocessing steps applied to each dataset, including quality control thresholds, normalization method, and feature selection.

The FAIR Guiding Principles provide a framework for making data findable, accessible, interoperable, and reusable. Applying these principles to your integration inputs means using standard file formats, recording metadata in machine-readable form, and documenting provenance. This documentation is essential for reproducibility and for troubleshooting when integration results are unexpected.

Step 3: Select Candidate Methods

Choose a small set of candidate methods based on your dataset configuration. For atlas-level gene expression data, include at least one variational autoencoder method such as scVI or scANVI, one mutual nearest neighbors method such as Scanorama, and one reference-based method such as scGen. For multi-modal data, include methods that support your specific modality combination. For spatial data, include methods appropriate for your task, whether transcript prediction or deconvolution.

Do not include every available method in your initial comparison. The benchmark by Luecken et al. evaluated 68 method and preprocessing combinations, but this scale is not necessary for most projects. A focused comparison of three to five methods with two preprocessing choices each is sufficient to identify the best approach for your data.

Step 4: Run Integration with Multiple Preprocessing Configurations

For each candidate method, run the integration with at least two preprocessing configurations. The atlas-level benchmark found that highly variable gene selection improves integration performance, so include a configuration with highly variable genes and a configuration with all genes. Include a configuration with scaling and a configuration without scaling, because scaling pushes methods to prioritize batch removal over biological conservation.

Record the runtime and memory usage for each method and configuration. Scalability is an important criterion for atlas-level data, where some methods may be computationally infeasible for datasets with millions of cells.

Step 5: Score Methods on Batch Correction and Biological Preservation

Apply multiple evaluation metrics to each integration result. For batch correction, use metrics that measure local mixing, such as kBET, and metrics that measure residual batch signal, such as the Batch Probing Score. For biological preservation, use metrics that measure the recovery of known cell types, the separation of known biological conditions, and the conservation of cell type proportions.

The atlas-level benchmark used 14 evaluation metrics to assess methods. You do not need to use all 14, but you should use at least two metrics for batch correction and two for biological preservation. Using multiple metrics is important because single metrics can be gamed by specific methods.

Step 6: Inspect Results Visually and Biologically

Quantitative metrics are necessary but not sufficient for method evaluation. Visualize the integrated data using dimensionality reduction and examine whether known cell types form coherent clusters, whether batches are mixed within cell types, and whether rare cell types are preserved. The deep visualization method (DV) was developed to preserve local and global structure while handling batch effects, and it can be useful for this inspection step.

Check whether known biological relationships are preserved. For example, if you are integrating data from healthy and diseased tissue, verify that disease-associated cell states are still distinguishable after integration. If you are integrating across species, verify that orthologous cell types cluster together.

Step 7: Document Results and Decisions

Record the scores for each method and preprocessing configuration, the runtime and memory usage, and the visual inspection results. Document which method you selected and why. Record any limitations of the selected method, such as known sensitivity to certain preprocessing choices or poor performance on specific cell types.

This documentation is important for reproducibility and for future analyses. If you or a collaborator later need to integrate additional data, the documentation will tell you which method and preprocessing configuration to use.

Records and Measurements for Integration Evaluation

Essential Records

Maintain a laboratory notebook or electronic record for each integration evaluation. Include the following information for each run:

• Dataset identifiers and source repositories
• Sequencing protocols and versions
• Quality control thresholds and numbers of cells removed
• Normalization method and parameters
• Feature selection method and number of features
• Integration method and version
• Integration parameters and hyperparameters
• Runtime and peak memory usage
• Evaluation metric scores
• Visual inspection notes
• Software environment and package versions

Measurements to Track

Track the following measurements across all integration runs:

• Number of cells and features before and after quality control
• Number of batches and the distribution of cells per batch
• Proportion of cells assigned to each cell type before and after integration
• Batch mixing metrics such as kBET scores
• Residual batch signal metrics such as BPS
• Biological preservation metrics such as cell type recovery scores
• Runtime and memory usage

Reproducibility Controls

Use version control for all analysis scripts and record the exact software environment. The EMBL-EBI Training portal provides guidance on reproducible bioinformatics workflows. Containerization tools can help ensure that the same software versions are used across analyses.

Common Failure Patterns in Integration Evaluation

Over-Correction Removes Biological Variation

The most common failure pattern is over-correction, where the integration method removes genuine biological differences along with technical variation. This is particularly problematic for rare cell types and subtle cell states. The atlas-level benchmark found that scaling pushes methods to prioritize batch removal over conservation of biological variation, so over-correction is more likely when scaling is applied.

Detection: Compare cell type proportions before and after integration. If rare cell types disappear or known biological conditions become indistinguishable, over-correction is likely. Check whether known marker genes for rare cell types are still expressed in the integrated data.

Under-Correction Leaves Residual Batch Signal

The opposite failure pattern is under-correction, where batch effects remain after integration. This is more common when batch effects are strong or when the integration method is not appropriate for the dataset configuration.

Detection: Use the Batch Probing Score to measure residual batch signal after correction. Visualize the integrated data and check whether cells from the same batch cluster together within cell types.

Preprocessing Artifacts Propagate into Integration Results

Integration methods cannot correct for poor preprocessing. If quality control thresholds are too lenient, doublets and low-quality cells will be integrated into the data and can form spurious clusters. If normalization is inappropriate for the sequencing protocol, the integration method will attempt to correct for normalization artifacts instead of true batch effects.

Detection: Examine the quality control metrics for each batch before integration. Check whether the number of genes detected per cell and the mitochondrial read fraction are comparable across batches. If not, adjust quality control thresholds before integration.

Method Selection Based on Default Parameters

Many integration methods have default parameters that are not optimal for atlas-level data. The benchmark by Luecken et al. found that method performance varies substantially with preprocessing choices, and the same is likely true for parameter choices. Using default parameters without evaluation can lead to suboptimal integration.

Detection: Run each method with at least two parameter configurations and compare the evaluation metric scores. If the scores vary substantially between configurations, the method is sensitive to parameter choice and requires careful tuning.

Ignoring Computational Constraints

Some integration methods do not scale to atlas-level datasets. The benchmark evaluated scalability as one of the criteria, and methods that perform well on small datasets may be computationally infeasible on datasets with millions of cells. Ignoring computational constraints can lead to excessive runtime and memory usage.

Detection: Record runtime and memory usage for each method. If a method takes more than a few days to run on your dataset or requires more memory than is available, consider alternative methods or downsampling strategies.

Limitations of Integration Evaluation

Benchmark Results Do Not Guarantee Performance on Your Data

Published benchmarks provide guidance on method selection, but they cannot predict performance on your specific dataset. The benchmark by Luecken et al. evaluated methods on 13 atlas-level integration tasks, but your data may have different batch structures, different cell type compositions, or different technical artifacts. Always evaluate methods on your own data before committing to a production analysis.

Evaluation Metrics Measure Different Aspects of Integration Quality

No single metric captures all aspects of integration quality. The atlas-level benchmark used 14 evaluation metrics, and the authors noted that methods that perform well on one metric may perform poorly on another. The Batch Probing Score was reported to have the highest sensitivity among existing metrics, but sensitivity alone does not guarantee that the metric measures what you care about.

Integration Methods Are Evolving Rapidly

New integration methods are published regularly, and existing methods are updated with new versions. The benchmark results from 2022 may not reflect the current performance of methods that have been updated since then. The multi-modal integration benchmark from 2025 evaluated 40 algorithms, reflecting the rapid growth of the field. Stay current with the literature and re-evaluate methods periodically.

Cross-Species Integration Has Unique Challenges

Cross-species integration requires orthology mapping, which introduces additional sources of error. A benchmark of cross-species integration strategies evaluated methods on their ability to identify conserved cell types across species. Species-specific genes and regulatory elements can obscure shared biology, and the choice of orthology mapping method can affect integration results.

Safety and Regulatory Context for Data Integration

Data Sharing and Privacy Considerations

Single-cell datasets may contain sensitive information about human subjects. The NIH Genomic Data Sharing Policy provides requirements for the sharing of genomic data generated with NIH funding, including data security, informed consent, and privacy protections. When integrating public datasets, verify that the data were collected and shared in compliance with applicable policies and that any restrictions on data use are respected.

Data Provenance and Attribution

The FAIR Guiding Principles emphasize the importance of data provenance for findability and reusability. When integrating data from multiple sources, document the origin of each dataset and provide appropriate attribution. The NCBI Data Resources and other repositories provide citation guidance for datasets deposited in their systems.

Reproducibility Requirements

Many journals and funding agencies require that analyses be reproducible. This includes providing the code, data, and software environment used for integration. The EMBL-EBI Training portal provides guidance on reproducible bioinformatics workflows, and the FAIR Guiding Principles provide a framework for making data and analysis outputs findable, accessible, interoperable, and reusable.

Professional Escalation Criteria

When to Consult a Bioinformatics Specialist

Escalate to a bioinformatics specialist or computational biology collaborator when any of the following conditions apply:

• Integration results are highly sensitive to preprocessing choices, and you cannot determine which configuration is correct
• The Batch Probing Score or other residual batch metrics indicate substantial remaining batch signal after correction with multiple methods
• Rare cell types or known biological conditions disappear after integration with all candidate methods
• You are integrating data from modalities or species that are not well covered by published benchmarks
• You need to integrate data with complex nested batch structures that are not handled by standard methods

When to Consult a Statistician

Escalate to a statistician when:

• You need to design a formal comparison of integration methods with appropriate statistical controls
• You are concerned about multiple testing issues in downstream analyses after integration
• You need to quantify uncertainty in integration results
• You are developing new evaluation metrics or benchmarking pipelines

When to Consult an Ethics or Compliance Officer

Escalate to an ethics or compliance officer when:

• You are integrating data that may contain sensitive information about human subjects
• You are uncertain whether data use restrictions apply to your analysis
• You plan to share integrated data that includes data from multiple sources with different consent agreements

Frequently Asked Questions

What is the difference between batch correction and data integration?

Batch correction refers to the removal of technical variation that arises from processing samples in different batches, such as different sequencing runs, laboratories, or protocols. Data integration is a broader concept that includes batch correction but also encompasses the alignment of different modalities, the mapping of cells across datasets, and the creation of a unified representation for downstream analysis. Integration methods typically perform batch correction as part of their algorithm, but the goals are distinct. Batch correction focuses on removing technical noise, while integration focuses on creating a coherent dataset that supports biological discovery.

How do I choose between Harmony and scVI for my dataset?

The choice between Harmony and scVI depends on your dataset size, the complexity of batch effects, and your downstream analysis goals. Harmony is a fast method that performs well on large datasets with relatively simple batch structures. scVI is a variational autoencoder method that can model complex batch effects and is particularly effective for atlas-level integration tasks. The atlas-level benchmark found that scVI and scANVI performed well on complex integration tasks, while Harmony was not among the top performers in that benchmark. However, Harmony may be preferable when computational resources are limited or when you need a quick initial integration for exploratory analysis. Evaluate both methods on your data using the framework described in this article.

What preprocessing steps are most important for integration success?

Highly variable gene selection is the most important preprocessing step for integration success. The atlas-level benchmark found that highly variable gene selection improves the performance of data integration methods. Scaling is also important, but it pushes methods to prioritize batch removal over conservation of biological variation, so use scaling with caution. Quality control to remove doublets and low-quality cells is essential before integration. Normalization should be appropriate for the sequencing protocol, and the choice of feature space is critical for chromatin accessibility data.

How do I evaluate whether integration preserved biological variation?

Use multiple evaluation metrics that measure different aspects of biological preservation. Check whether known cell type markers are still expressed in the integrated data, whether known biological conditions are distinguishable, and whether cell type proportions are conserved. Visualize the integrated data and examine whether known cell types form coherent clusters. The atlas-level benchmark used 14 evaluation metrics, and you should use at least two metrics for biological preservation in your evaluation.

What should I do if integration removes a rare cell type?

If integration removes a rare cell type, first check whether the cell type was present in the input data before integration. If it was present, try a different integration method or a different preprocessing configuration. The atlas-level benchmark found that scaling pushes methods to prioritize batch removal over biological conservation, so try integration without scaling. You may also need to adjust quality control thresholds to ensure that rare cell types are not removed during preprocessing.

How do I integrate single-cell RNA-seq data with spatial transcriptomics data?

The choice of integration method depends on your task. For predicting the spatial distribution of RNA transcripts, Tangram, gimVI, and SpaGE were the top performers in a benchmark of 16 methods. For cell type deconvolution of spatial spots, Cell2location, SpatialDWLS, and RCTD were the top performers. These methods require a single-cell RNA-seq reference dataset and spatial transcriptomics data from the same tissue type. The integration approach differs from standard batch correction because the goal is to map single-cell data onto spatial coordinates instead of to mix cells from different batches.

What are the limitations of using published benchmarks to choose an integration method?

Published benchmarks provide guidance on method selection, but they cannot predict performance on your specific dataset. The benchmark by Luecken et al. evaluated methods on 13 atlas-level integration tasks, but your data may have different batch structures, cell type compositions, or technical artifacts. Benchmarks also become outdated as methods are updated and new methods are published. Always evaluate methods on your own data before committing to a production analysis pipeline.

How do I document integration results for publication?

Document the exact software versions, parameters, and preprocessing steps used for integration. Provide the evaluation metric scores for all candidate methods and explain why you selected the final method. The FAIR Guiding Principles provide a framework for making data and analysis outputs findable, accessible, interoperable, and reusable. Many journals require that analysis code and data be deposited in public repositories, and the NCBI Data Resources and EMBL-EBI provide infrastructure for this purpose.

Related Bioinformatics Guides

• Single-Cell RNA-Seq Normalization: Batch Effect Correction and Dimension Reduction (PCA, t-SNE, UMAP)
• Variant Calling Pipelines: GATK Best Practices, FreeBayes, and DeepVariant Comparison
• The Human Cell Atlas: A Computational Perspective
• Foundation Models for Single-Cell Biology
• Single-Cell ATAC-Seq Bioinformatics

References and Further Reading

• EMBL-EBI Training. European Bioinformatics Institute.
• NCBI Data Resources. National Center for Biotechnology Information.
• Genomic Data Sharing Policy. National Institutes of Health.
• The FAIR Guiding Principles. Scientific Data.
• Benchmarking atlas-level data integration in single-cell genomics.. Nature methods, 2022.
• Comparative Analysis of Single-Cell RNA Sequencing Methods.. Molecular cell, 2017.
• Benchmarking single-cell multi-modal data integrations.. Nature methods, 2025.
• Benchmarking algorithms for single-cell multi-omics prediction and integration.. Nature methods, 2024.
• Systematic comparison of single-cell and single-nucleus RNA-sequencing methods.. Nature biotechnology, 2020.
• Benchmarking spatial and single-cell transcriptomics integration methods for transcript distribution prediction and cell type deconvolution.. Nature methods, 2022.
• An atlas of healthy and injured cell states and niches in the human kidney.. Nature, 2023.
• Benchmarking integration of single-cell differential expression.. Nature communications, 2023.
• Artificial Intelligence in Transcriptomics: From Human-in-the-Loop to Agentic AI.. 2026.
• Spatially multimodal and multiscale network for representation learning from spatial multi-omics.. 2026.
• A test metric for assessing single-cell RNA-seq batch correction. Nature Methods, 2018.
• Flexible comparison of batch correction methods for single-cell RNA-seq using BatchBench. bioRxiv, 2020.
• Probing as a new technique to assess single-cell RNA-seq batch correction. bioRxiv, 2025.
• Structure-preserving visualization for single-cell RNA-Seq profiles using deep manifold transformation with batch-correction. bioRxiv, 2022.
• scStarCorrect: A StarGAN-Based Adversarial Model for Reference-Guided Batch Correction in Single-Cell RNA-Seq Data. IEEE International Conference on Bioinformatics and Biomedicine, 2025.
• Integrating single-cell RNA-seq datasets with substantial batch effects. BMC Genomics, 2025.
• cKBET: assessing goodness of batch effect correction for single-cell RNA-seq. Frontiers of Computer Science, 2023.
• Benchmarking deep learning methods for biologically conserved single-cell integration. Genome Biology, 2025.
• Multitask benchmarking of single-cell multimodal omics integration methods. Nature Methods, 2025.
• Benchmarking strategies for cross-species integration of single-cell RNA sequencing data. Nature Communications, 2023.
• Benchmarking Preprocessing and Integration Methods in Single-Cell Genomics. Communications in Computer and Information Science, 2026.

This article is educational and does not replace validated analysis plans, institutional policy, clinical interpretation, or specialist review.