Spatial Transcriptomics Cell Segmentation

This guide explains how to segment cells from spatial transcriptomics data a critical step that assigns gene expression measurements to individual cells rather than to tissue pixels. You should read this if you are a computational biologist, bioinformatics analyst, or advanced graduate student working with platforms like Visium, Slide‑seq, MERFISH, or Xenium and need a practical framework for choosing, implementing, and validating a segmentation method. The core challenge is that spatial transcriptomics data often lacks a direct cell boundary image, so you must combine histological stains, nuclear stains, or computational heuristics to recover cellular resolution.

Core Concepts and Decision Points

Spatial transcriptomics cell segmentation is not a single algorithm. The two broad families are image‑based segmentation, which uses a co‑registered histological image to delineate cell membranes or nuclei, and spot‑based segmentation, which aggregates transcript spots into a grid or clustering without an image. Image‑based segmentation generally yields higher biological accuracy when a high‑quality stain is available, but it requires careful alignment and stain specificity. Spot‑based methods are simpler and avoid alignment errors but lose cell morphology and can merge adjacent cells.

The official Bioconductor documentation provides workflows for spatial transcriptomics analysis, including cell segmentation using the SpatialExperiment framework Bioconductor. The Galaxy Training Network also offers step‑by‑step tutorials for processing spatial transcriptomics data with common tools Galaxy Training Network. These resources are excellent starting points for implementation, but they often assume you already have a segmentation strategy in mind.

Your decision depends on three factors:

1. Image quality and resolution. If your tissue section has a clear nuclear stain (e.g., DAPI) or membrane marker, an image‑based approach like Cellpose or watershed can work. With poor image quality or no image at all, you must rely on spot density clustering.
2. Platform type. Visium spots are 55 µm apart and each captures multiple cells. Image‑based segmentation can refine assignment but you must accept that each spot may contain partial cells. Slide‑seq and MERFISH often have sub‑cellular transcript resolution, there, spot‑based methods with distance thresholds are common.
3. Tissue density and cell type. In regions with tightly packed cells, image‑based segmentation may under‑segment, merging two nuclei into one. The NCBI Bookshelf contains general histology references that describe cell packing in different tissues, understanding your tissue’s native architecture helps set segmentation parameters NCBI Bookshelf.

Practical Workflow or Implementation Sequence

The following sequence applies to most image‑based segmentation pipelines. Adapt it if you are using only spot data.

Step 1: Align the Histological Image to Transcript Coordinates

Most platforms provide a registration between the tissue image and the spatial barcode array. Visually inspect the alignment by overlaying transcript spots on the image. A misalignment of even a few micrometers will cause transcripts to be assigned to the wrong cell. The DISSECT framework integrates cytological images with spatial transcriptomics to improve alignment and segmentation DISSECT. Use fiducial markers or manual landmark points if automatic alignment fails.

Step 2: Preprocess the Image

Apply flat‑field correction to remove uneven illumination, and normalize intensities across the whole image. For DAPI images, a median filter reduces noise without blurring nuclear edges. Convert the image to 8‑bit if needed. Avoid excessive contrast stretching that might create false boundaries.

Step 3: Execute Segmentation on the Image

Choose a segmentation algorithm based on your image quality and available computing power.

• Watershed: Works well on clean DAPI images with separated nuclei. You need to estimate a minimum nuclear area (e.g., 50 pixels for a 10x image) and a distance transform threshold.
• Cellpose: A deep learning model trained on diverse cell morphologies. It requires a GPU but handles touching cells better than watershed. Cellpose can run on H&E if you have membrane signal.
• Stardist: Often faster than Cellpose for star‑convex nuclei. Works well for round nuclei but struggles with elongated or irregular shapes.

Parameter tuning: Test segmentation on a small region and compare with manual outlines. The Odon viewer can be used to visually inspect segmentation results and overlay transcript spots, providing rapid feedback Odon.

Step 4: Assign Transcripts to Cells

For each transcript spot (or barcoded bead), check whether its centroid falls inside a segmented cell polygon. If using Visium spots, you may need to assign a spot to the cell whose polygon overlaps the spot centroid by the largest fraction. Record ambiguous spots (within 5 µm of a boundary) for later filtering.

Some pipelines use a Gaussian mixture model on the spatial coordinates of transcripts instead of polygons. This is faster but less accurate. The SPICE framework handles segmentation in the context of copy number variation detection, where precise cell boundaries are essential for accurate inference SPICE.

Step 5: Filter and Normalize

Remove cells with fewer than 100 transcripts likely empty droplets or debris. Normalize counts per cell using SCTransform or scran. Create a quality metric: “cell area versus transcript count”. A correlation below 0.5 may indicate poor segmentation.

Quality Checks

Before proceeding to downstream analysis, perform these checks.

1. Visual overlay of segment boundaries on the image. Do the polygons align with visible nuclear or membrane edges? Run this on at least five fields of view.
2. Transcript density inside vs. outside cells. Bulk RNA transcripts should be inside, not in the stroma. If more than 20% of transcripts fall outside any cell boundary, your segmentation is too conservative or the image is misaligned.
3. Cell size distribution. Does the median cell area match expected values for your tissue (e.g., 100 200 µm² for a mammalian cell)? An unexpected mode of very small “cells” usually indicates noise.
4. Reproducibility. Run the segmentation twice with slightly different parameters. The Jaccard index between the two sets of cell polygons should exceed 0.7 for the same region.

Common Mistakes

• Using the wrong nuclear stain. DAPI works well for intact nuclei. In tissue with high RNA fragmentation, DAPI signal may be dim, leading to under‑segmentation. Post‑mortem RNA fragmentation can also affect marker gene detection and should be considered when interpreting results RNA fragmentation.
• Over‑relying on default parameters. Every dataset is different. The defaults in Cellpose or watershed were trained on specific organisms or magnification levels. Test and adjust.
• Ignoring batch effects. Different sections from the same experiment may have different staining intensities. Normalize images separately to avoid systematic bias in cell detection.
• Forgetting that spots are not perfect. Even sub‑cellular resolution data has localization error. A transcript assigned to the wrong cell can propagate errors in all downstream analyses.
• Not validating with known markers. If you have a cell type marker that should be nuclear or membrane bound, check whether its expression pattern matches your segmentation.

Limits of Interpretation

Segmentation cannot resolve true single‑cell resolution in every context. In dense neural tissue or tumor regions with high cellularity, two cells may share a common boundary that is blurred beyond the imaging resolution. The resulting “cell” is a mixture of two transcriptomes. Downstream inference of differential expression or copy number alterations will be confounded.

The biological reality is even more granular: transcripts can diffuse up to several micrometers during tissue permeabilization. This smears the spatial signal, and no segmentation algorithm can perfectly correct for it. The SPICE framework addresses some of these artifacts by modeling copy number variations under spatial uncertainty SPICE.

Tissue‑stripping artifacts from VisiumHD slide chemistry can create systematic bands of empty spots. A recent method models nuclear counts to remove these stripes, but the underlying segmentation remains imperfect Striping artifact removal.

Finally, cell segmentation is only as good as the reference image. If the histological stain does not label every cell type (e.g., microglia may not take up DAPI well), some cells will be missed. An integrated multi‑omics cell atlas that combines morphology with molecular profiles can help define which cell populations are systematically under‑segmented Integrated multi‑omics cell atlas.

Frequently Asked Questions

1. How do I choose between Cellpose and watershed? Cellpose is generally superior for images where cells are touching or have irregular shapes. Watershed is faster and works well on clean DAPI images with well‑separated nuclei. If you have a GPU, start with Cellpose. If you need speed on a CPU, use watershed and then manually relabel a few mis‑segmented regions.

2. Can I segment cells without any image (only spot coordinates)? Yes. You can cluster transcript spots into “pseudo‑cells” using a nearest‑neighbor distance threshold or a Gaussian mixture model. This approach is simpler but cannot recover true cell boundaries. You will likely over‑merge adjacent cells, especially in dense tissue. For many exploratory analyses it is acceptable, but confirmatory studies should use image‑guided segmentation.

3. What is the minimum number of transcripts needed per cell for reliable downstream analysis? Most spatial transcriptomics protocols yield 100 1000 transcripts per cell. Below 100 transcripts, the expression profile is dominated by noise. Below 50, dropouts are severe. Filter out any “cell” with fewer than 100 transcripts and consider whether your segmentation is capturing real cells or artifacts.

4. How do I handle Visium spots that cover multiple cells? You cannot split a Visium spot into single cells because each spot is a single barcode. Instead, perform “cell deconvolution” using known cell type signatures to estimate the proportions of each cell type within the spot. This is distinct from segmentation. For true cellular resolution, move to Slide‑seq, MERFISH, or Xenium platforms.

References and Further Reading

• Bioconductor Open source software and workflows for genomic data, including SpatialExperiment and segmentation vignettes.
• NCBI Bookshelf Free authoritative textbooks on histology, tissue architecture, and cell biology.
• EMBL-EBI Training Online courses that cover spatial transcriptomics data analysis and image processing.
• Galaxy Training Network Interactive tutorials for spatial transcriptomics processing including segmentation steps.
• NCBI Sequence Read Archive Repository where raw sequencing data and spatial barcode files are deposited.
• DISSECT paper A method that integrates cytological images with spatial transcriptomics to improve cell segmentation.
• SPICE paper Framework for identifying copy number variations and handling segmentation uncertainty.
• Odon viewer Ultra‑fast viewer for spatial proteomics that can also aid visual validation of segmentation.

Related Articles

• Structural Variant Validation
• Repeat Expansion Sequencing
• Copy Number Variant Analysis
• Somatic Mosaic Variant Calling
• Germline Variant Filtering