Foundation Models for Single-Cell Biology: Architectures, Benchmarks, and Applications in Transcriptomics

Introduction

The advent of high-throughput single-cell RNA sequencing (scRNA-seq) has enabled the profiling of transcriptomes at unprecedented resolution, revealing cellular heterogeneity within tissues and across physiological states [1]. The complexity and volume of these data have motivated the development of foundation models, large-scale neural networks pretrained on vast corpora of single-cell transcriptomic profiles, which can be fine-tuned for a wide array of downstream tasks [2, 3]. These models, often built upon transformer architectures originally developed for natural language processing, treat gene expression patterns as a language of cellular states [1, 4]. In veterinary medicine and comparative biology, such models offer the potential to decode host-pathogen interactions, characterize immune cell populations across species, and predict tissue-level responses to infection or therapeutic intervention [5, 3]. This article provides a technical review of the architectural principles, pretraining strategies, evaluation benchmarks, and emerging applications of foundation models for single-cell biology, with an emphasis on transcriptomic analysis.

Transformer Architectures for Single-Cell Data

Foundation models for single-cell biology predominantly employ transformer architectures that leverage self-attention mechanisms to capture dependencies between genes within a cell [6, 7]. In these models, each gene is represented as a token, and its expression level (often normalized and log-transformed) serves as an input feature [1, 4]. The model learns contextualized embeddings for each gene, enabling it to capture gene-gene interactions and regulatory relationships that are not explicitly encoded in the raw count matrix [7, 4].

A representative example is scGPT, a generative pretrained transformer that models single-cell transcriptomes by predicting masked gene expression values [7]. The model is trained on a large corpus of scRNA-seq data using a denoising objective, where a fraction of gene expression values are masked and the model must reconstruct them from the remaining context [7]. This pretraining phase allows the model to learn a rich representation of cellular states, which can then be transferred to downstream tasks such as cell type annotation, gene regulatory network inference, and perturbation prediction [6, 7].

Another notable architecture is CELLama, which transforms single-cell and spatial transcriptomics data into textual "sentences" by concatenating gene symbols with their expression levels and associated metadata [1]. These sentences are then processed by a language model to generate universal cell embeddings that can be used for clustering, cell typing, and spatial context analysis without requiring dataset-specific normalization or reference selection [1]. The CELLama framework exemplifies how language model capabilities can be leveraged to create flexible, reference-free analytical pipelines for single-cell data [1].

The scGenePT model extends this paradigm by integrating textual representations of genes derived from scientific literature with expression-based embeddings learned from scRNA-seq data [4]. This multimodal approach combines the complementary strengths of biological sequence data and natural language descriptions of gene function, subcellular localization, and protein interactions [4]. The study demonstrated that textual priors, particularly those describing subcellular location, significantly improve the prediction of single-gene perturbation outcomes, while protein interaction information enhances the modeling of combinatorial gene perturbations [4].

Pretraining Data and Scaling Properties

The performance of single-cell foundation models is critically dependent on the size, diversity, and quality of the pretraining dataset [2]. Systematic evaluations have shown that increasing the number of cells and the diversity of tissue types and conditions in the pretraining corpus generally improves model performance on downstream tasks, although the relationship is not strictly monotonic [2]. Models pretrained on highly heterogeneous datasets, encompassing multiple species, tissues, and disease states, tend to generalize better to unseen biological contexts [2, 3].

However, the benefits of scaling are modulated by the specific task and the degree of domain shift between pretraining and fine-tuning data [2, 8]. For tasks such as cell type annotation, where the target cell types are well represented in the pretraining data, larger pretraining corpora yield substantial improvements [2]. In contrast, for tasks involving rare cell populations or novel perturbations, the marginal benefit of additional pretraining data may diminish, and fine-tuning strategies become more critical [2, 8].

The diversity of the pretraining data also influences the model's ability to capture biological variability. Models trained exclusively on healthy tissue samples may struggle to represent disease-associated cell states, whereas those exposed to a wide range of pathological conditions exhibit greater robustness [3, 9]. This observation has direct implications for veterinary applications, where the availability of species-specific single-cell atlases is often limited, and models must generalize across host species and tissue types [5].

Fine-Tuning and Hyperparameter Optimization

Fine-tuning is the process of adapting a pretrained foundation model to a specific downstream task by updating its weights on a smaller, task-specific dataset [6, 7]. The effectiveness of fine-tuning depends on several factors, including the choice of learning rate, the number of layers updated, and the regularization strategy [6]. Bayesian hyperparameter optimization has been shown to significantly improve the performance of scGPT fine-tuning for multi-omics integration tasks, outperforming manual or grid-based search methods [6]. This approach models the objective function as a probabilistic surrogate and selects hyperparameters that maximize expected improvement, thereby reducing the computational cost of fine-tuning while achieving superior results [6].

Fine-tuning can be applied to a variety of downstream tasks, including cell type annotation, gene expression prediction, treatment perturbation modeling, and gene regulatory network inference [7]. In the context of cancer research, fine-tuned scGPT models have demonstrated the ability to distinguish between individuals and treatment groups at the single-cell level, capturing information that reflects both patient-specific and treatment-specific transcriptional responses [7]. The analysis of attention matrices from fine-tuned models has revealed that different attention heads and layers capture distinct biological signals, such as gene co-expression modules and pathway-level interactions [7].

Evaluation and Benchmarking

Rigorous benchmarking is essential for assessing the capabilities and limitations of single-cell foundation models [3]. A comprehensive benchmark study evaluated six scFMs against well-established baseline methods across two gene-level tasks and four cell-level tasks, using 12 metrics spanning unsupervised, supervised, and knowledge-based approaches [3]. The study introduced scGraph-OntoRWR, a novel metric designed to quantify the intrinsic biological knowledge encoded by scFMs by measuring the consistency between model embeddings and known gene ontology relationships [3].

The benchmark revealed that scFMs are robust and versatile tools for diverse applications, but that no single model consistently outperforms all others across every task [3]. Simpler machine learning models, such as regularized logistic regression or gradient-boosted trees, can be more adept at efficiently adapting to specific datasets, particularly under resource constraints [3]. The choice of model should therefore be guided by factors such as dataset size, task complexity, biological interpretability, and available computational resources [3].

Partial-label metric ceilings have been proposed as a framework for evaluating gene regulatory networks inferred from single-cell foundation models [10]. This approach acknowledges that ground-truth regulatory relationships are often incomplete or uncertain, and defines an upper bound on achievable performance given the quality of the reference annotations [10]. By comparing model predictions against this ceiling, researchers can distinguish between limitations imposed by the model architecture and those inherent to the reference data [10].

Large language model consensus methods have been shown to substantially improve the accuracy of cell type annotation for scRNA-seq data [11]. By aggregating predictions from multiple foundation models or multiple runs of the same model with different random seeds, consensus approaches reduce the variance and bias associated with individual model predictions [11]. This ensemble strategy is particularly valuable for annotating cell types in datasets from non-model organisms, where reference atlases may be sparse or absent [11].

Applications in Transcriptomics and Spatial Biology

Foundation models for single-cell biology have been applied to a wide range of transcriptomic analyses, including cell type annotation, trajectory inference, gene regulatory network reconstruction, and perturbation prediction [8, 3, 7]. The learnability of these models across multiple tasks has been systematically evaluated, revealing that performance varies considerably depending on the task and the degree of similarity between pretraining and fine-tuning data [8].

In spatial transcriptomics, foundation models such as CELLama have been adapted to analyze the spatial organization of cells within tissues [1]. By embedding both gene expression and spatial coordinates into a unified representation, these models can identify cellular neighborhoods, characterize ligand-receptor interactions, and predict tissue-level phenotypes [5, 1]. The Tissueformer model extends single-cell foundation models to predict population-level phenotypes from spatial transcriptomics data, enabling the identification of tissue regions associated with disease states or treatment responses [5].

The application of foundation models to veterinary transcriptomics presents unique opportunities and challenges. While many models are pretrained primarily on human data, transfer learning approaches can adapt these models to analyze scRNA-seq data from livestock, companion animals, and wildlife species [5, 3]. For example, a foundation model pretrained on a diverse mammalian cell atlas could be fine-tuned to identify immune cell subtypes in porcine lung tissue during influenza infection, or to characterize the transcriptional response of bovine alveolar macrophages to Mycobacterium bovis exposure [5]. The ability to leverage cross-species transcriptional similarities is particularly valuable for veterinary research, where the generation of large-scale, species-specific reference atlases may be cost-prohibitive [5].

Gene Regulatory Network Inference and Perturbation Modeling

One of the most promising applications of single-cell foundation models is the inference of gene regulatory networks (GRNs) from transcriptomic data [10, 7]. GRNs describe the interactions between transcription factors and their target genes, and are essential for understanding the molecular mechanisms underlying cell fate decisions, disease progression, and responses to perturbations [10]. Foundation models can infer GRNs by analyzing the attention weights learned during pretraining or fine-tuning, which reflect the degree to which the model attends to one gene when predicting the expression of another [7].

The evaluation of GRNs inferred from foundation models requires careful consideration of the partial and noisy nature of reference annotations [10]. Partial-label metric ceilings provide a principled framework for assessing model performance in this context, accounting for the fact that many true regulatory relationships may be unknown or unverified [10]. This approach has been used to benchmark GRN inference methods across multiple datasets and model architectures, revealing that foundation models can capture both known and novel regulatory interactions [10].

Perturbation prediction, the task of forecasting how gene expression profiles change in response to genetic or chemical perturbations, is another key application of foundation models [4]. The scGenePT model demonstrated that integrating textual gene representations with expression-based embeddings improves the accuracy of perturbation predictions, particularly for combinatorial perturbations involving multiple genes [4]. This capability has direct relevance for veterinary pharmacology and toxicology, where predicting the transcriptional effects of drug treatments or environmental exposures in target species can inform safety assessments and therapeutic strategies [4].

Limitations and Challenges

Despite their promise, single-cell foundation models face several limitations that must be addressed for reliable application in veterinary and comparative biology. First, the pretraining data are overwhelmingly derived from human samples, which may limit the models' ability to capture species-specific transcriptional programs and regulatory mechanisms [2, 3]. Transfer learning from human to non-human species is feasible but requires careful validation, as orthologous genes may have divergent functions or expression patterns across species [5].

Second, the computational cost of training and fine-tuning large foundation models can be prohibitive for many veterinary research groups [6, 3]. Bayesian hyperparameter optimization and other efficient fine-tuning strategies can mitigate this burden, but access to high-performance computing infrastructure remains a barrier [6]. Third, the interpretability of foundation models is limited, as the high-dimensional embeddings and attention weights do not directly correspond to biologically meaningful units [3, 7]. Efforts to develop interpretability metrics, such as scGraph-OntoRWR, represent important steps toward bridging this gap [3].

Fourth, the evaluation of foundation models on clinically relevant tasks, such as cancer cell identification and drug sensitivity prediction, has revealed that performance can be highly variable across datasets and conditions [3, 7]. Models that perform well on benchmark datasets may not generalize to real-world clinical or field samples, which often exhibit greater technical and biological variability [7]. Rigorous validation on independent, species-specific datasets is therefore essential before deploying these models in veterinary diagnostic or research settings [3].

Future Directions

The continued development of single-cell foundation models for veterinary and comparative biology will require several key advances. First, the creation of large-scale, multi-species single-cell atlases that encompass a broad range of tissues, developmental stages, and disease states will provide a more representative pretraining corpus for cross-species applications [2, 5]. Second, the integration of additional data modalities, such as chromatin accessibility (scATAC-seq), protein abundance (CITE-seq), and spatial context, will enable more comprehensive models of cellular regulation [6, 1].

Third, the development of explainable AI techniques tailored to single-cell data will enhance the interpretability of foundation models and facilitate the discovery of novel biological mechanisms [3, 12]. The eSPred model, which incorporates pathway-aware fine-tuning and cell-type grouping strategies, represents a step toward more interpretable and biologically grounded predictions [12]. Fourth, the application of foundation models to predict host-pathogen interactions at the single-cell level, as discussed in the article on Single-Cell Transcriptomics of Host-Pathogen Interactions During Viral Infection, will open new avenues for understanding viral pathogenesis and identifying therapeutic targets in veterinary species.

Frequently Asked Questions

What is a foundation model for single-cell biology?

A foundation model for single-cell biology is a large-scale neural network, typically based on a transformer architecture, that is pretrained on a vast corpus of single-cell transcriptomic data to learn general representations of cellular states [2, 3]. These representations can be fine-tuned for a wide range of downstream tasks, including cell type annotation, gene regulatory network inference, and perturbation prediction [6, 7].

How do foundation models differ from traditional single-cell analysis methods?

Traditional single-cell analysis methods, such as clustering and differential expression, are typically applied to individual datasets and require extensive preprocessing, normalization, and manual annotation [1]. Foundation models, in contrast, learn universal representations from large, heterogeneous datasets and can be applied to new data with minimal preprocessing, often in a zero-shot or few-shot manner [2, 1]. They also capture complex, non-linear gene-gene interactions that are not accessible to linear or distance-based methods [3, 7].

Can foundation models trained on human data be applied to veterinary species?

Yes, but with caveats. Transfer learning from human to non-human species is feasible because many transcriptional programs and regulatory mechanisms are conserved across mammals [5]. However, species-specific differences in gene expression, isoform usage, and regulatory networks can reduce model accuracy [2, 3]. Fine-tuning on species-specific data is strongly recommended to adapt the model to the target species [6, 5].

What are the main limitations of current single-cell foundation models?

The main limitations include a heavy reliance on human pretraining data, high computational costs for training and fine-tuning, limited interpretability of model embeddings and attention weights, and variable performance across different downstream tasks and datasets [6, 2, 3, 7]. Additionally, the evaluation of these models on clinically relevant tasks remains an active area of research [3, 7].

How are foundation models evaluated for gene regulatory network inference?

Gene regulatory networks inferred from foundation models are evaluated using metrics that compare predicted regulatory relationships against reference annotations [10]. Partial-label metric ceilings provide a framework for accounting for the incomplete and uncertain nature of these annotations, defining an upper bound on achievable performance [10]. This approach allows researchers to distinguish between model limitations and reference data quality issues [10].

Conclusion

Foundation models for single-cell biology represent a paradigm shift in transcriptomic data analysis, enabling the integration of heterogeneous datasets, the transfer of knowledge across biological contexts, and the prediction of cellular responses to perturbations [2, 1, 3]. While the majority of current models are trained on human data, their application to veterinary and comparative biology holds significant promise for advancing our understanding of host-pathogen interactions, immune cell dynamics, and tissue-level responses to disease and therapy [5, 3]. Continued efforts to expand pretraining data diversity, improve model interpretability, and develop species-specific benchmarks will be essential for realizing the full potential of these models in veterinary medicine and diagnostics [10, 2, 3].

References

[1] Choi H, Park J, Kim S, et al. CELLama: Foundation Model for Single Cell and Spatial Transcriptomics by Cell Embedding Leveraging Language Model Abilities. bioRxiv. 2024. URL: https://www.semanticscholar.org/paper/27c4d45afa0f09312ed61002da36b33744fbf684

[2] DenAdel A, Hughes M, Thoutam A, et al. Evaluating the role of pretraining dataset size and diversity on single-cell foundation model performance. Nat Methods. 2026. URL: https://pubmed.ncbi.nlm.nih.gov/42265208/

[3] Wu J, Ye Q, Wang Y, et al. Biology-driven insights into the power of single-cell foundation models. Genome Biology. 2025. URL: https://www.semanticscholar.org/paper/edb8dc4318c715c34ff0a5db7659e633e0315cd9

[4] Istrate AM, Li D, Karaletsos T. scGenePT: Is language all you need for modeling single-cell perturbations? bioRxiv. 2024. URL: https://www.semanticscholar.org/paper/8ae96f34d1459b39c13524b4fd0c382092df7405 *** Disclaimer: This article is for educational and informational purposes only. It is not intended to substitute for professional veterinary advice, diagnosis, treatment, or regulatory guidance. Always consult a licensed veterinarian or qualified specialist regarding animal health, disease diagnosis, and therapeutic decisions.

[5] Benjamin AS, Zador A. Tissueformer: extending single-cell foundation models to predict population-level phenotypes. BMC Bioinformatics. 2026. URL: https://pubmed.ncbi.nlm.nih.gov/42243667/

[6] Jun Tay DY, Khanh Le NQ, Heng Chua MC. Bayesian Hyperparameter Optimization Improves scGPT Fine-Tuning for Single-Cell Multi-Omics Integration. Bioinformatics. 2026. URL: https://pubmed.ncbi.nlm.nih.gov/42286785/

[7] Tan R, Cui H, Wang B, et al. Abstract A029: Evaluating and interpreting scGPT: A foundation model for single-cell biology in real-world cancer clinical trial data. Cancer Research. 2024. URL: https://www.semanticscholar.org/paper/057a81022d62d0646ab9e6f0c4fd53f24b2847d0

[8] Yan Y, Wang X, Song D. Evaluating the learnability of single-cell large language models on multiple tasks. BMC Genomics. 2026. URL: https://pubmed.ncbi.nlm.nih.gov/42249309/

[9] Xu J, Zhou Y, Lorincz-Comi NJ, et al. The Alzheimer's Cell Atlas: A comprehensive brain single-cell transcriptomic atlas using a generative AI foundation model. Alzheimer's & Dementia. 2025. URL: https://www.semanticscholar.org/paper/f52984c71d006a48286a2062fe7e74ddba53f5e4

[10] Kendiukhov I. Partial-label metric ceilings for evaluating gene regulatory networks inferred from single-cell foundation models. Biosystems. 2026. URL: https://pubmed.ncbi.nlm.nih.gov/42379339/

[11] Yang C, Zhang X, Chen J. Large language model consensus substantially improves the cell type annotation accuracy for scRNA-seq data. Commun Biol. 2026. URL: https://pubmed.ncbi.nlm.nih.gov/42260142/

[12] Sun L, Yang Q, Zhang J, et al. eSPred: Explainable scRNA-seq Prediction via Customized Foundation Models and Pathway-Aware Fine-tuning. bioRxiv. 2025. URL: https://www.semanticscholar.org/paper/92f464d15e39cce85b5a116e610477c7b209dcd1