Medical Image Annotation Tools: A Practical Guide for Building Segmentation Datasets

Building a medical image segmentation dataset requires selecting annotation tools that match your imaging modality, dataset scale, and team expertise. This guide compares open-source annotation platforms available on GitHub, focusing on practical features such as format support, automation capabilities, and deep learning framework integration. Researchers, students, and analysts will find concrete criteria for tool selection, workflow design, and quality control.

At a Glance

The table below summarizes key open-source annotation tools and platforms discussed in this guide, with their primary use cases and integration strengths.

Understanding the Annotation Bottleneck

Medical image segmentation models depend on pixel-level labels that capture the exact boundaries of anatomic structures or pathologic regions. Manual annotation by medical experts is time-intensive and costly, which limits the scale of training datasets. The development of deep learning algorithms for medical imaging requires software that supports training data annotation and graphical user interface construction so that developers can focus on core algorithm implementation instead of infrastructure [5].

Several studies demonstrate the scale of this challenge. A large-scale cone beam computed tomography dataset for impacted mandibular third molar segmentation required 186 expert manual annotations supplemented by 1,076 deep learning-assisted annotations to reach 1,262 total image volumes [21]. A pediatric supracondylar humerus fracture dataset used a double-blind cross-review protocol with two experienced orthopedic surgeons to annotate 2,015 images from 10,325 total elbow X-rays [18]. These examples show that annotation strategies vary by clinical question and available expertise.

The annotation process itself affects model performance. A quality control pipeline for COCO-format polygon annotations in endoscopic airway imaging found that fragmented regions and geometrically inconsistent polygons reduced reproducibility and spatial stability in segmentation models. After applying a systematic quality control workflow that validated polygon structure, generated masks at native resolution, and removed small noise-like instances, box [email protected]:0.95 increased from 0.602 to 0.628 and mean Dice similarity coefficient improved from 0.823 to 0.830 [17]. This result shows that annotation quality directly influences downstream model performance without requiring network architecture changes.

Core Principles for Tool Selection

Match the Tool to the Imaging Modality

Different imaging modalities produce different data structures that affect annotation tool requirements. Whole slide images in pathology are extremely large and require specialized viewing and annotation approaches. QuPath is one of the most widely used software applications for histopathology image analysis in research and provides a comprehensive set of computational tools to evaluate histology slides [6]. The CRS4 Digital Pathology Platform manages whole slide image collections with structured multi-label morphological and clinical image annotation [7].

For radiological images such as MRI and CT, tools must handle volumetric data and support slice-by-slice or cross-sectional annotation. AnatomySketch supports image annotation using stylus and multi-touch screens and provides efficient tools for collaboration between human experts and artificial intelligence algorithms [5]. The MRSegmentator model used a human-in-the-loop annotation workflow with cross-modality transfer learning to segment 40 anatomic structures in whole-body MRI and CT [11].

Ultrasound images present unique challenges including lower signal-to-noise ratios and operator-dependent acquisition. Annotation Web is an open-source web-based annotation tool designed specifically for ultrasound images [25]. Web-based tools enable distributed annotation teams and reduce installation requirements.

Evaluate Automation and AI Assistance

Modern annotation tools increasingly integrate artificial intelligence to accelerate the labeling process. The Segment Anything Model has been adapted for medical imaging through fine-tuning strategies. MedicoSAM, a fine-tuned version of SAM for medical images, shows performance improvements especially for interactive segmentation tasks and is compatible with existing data annotation tools [15].

Deep learning-assisted annotation can substantially reduce the time required per case. In the impacted third molar segmentation study, the AttentionUnet3D model achieved annotation speeds of 16.51 seconds per case with quality assessment scores of 3.07 and 3.29 from two physicians [21]. This speed improvement enables dataset construction at scales that would be impractical with manual annotation alone.

Human-AI collaboration tools allow experts to correct and refine AI-generated annotations instead of creating labels from scratch. AnatomySketch demonstrates this approach with human-AI collaborated spine disc segmentation and Annotation-by-iterative-Deep-Learning for model training [5]. The framework described in the citizen science study combines MedSAM segmentation AI with crowd annotators and uses an algorithm to merge crowd-labeled images, maintaining expert-level quality while accelerating the annotation process [19].

Consider Structured Annotation and Provenance

Standard annotation tools often employ single-label approaches that provide a flat representation of whole slide images, limiting their ability to capture the complexity of diagnosis-significant elements. The CRS4 Digital Pathology Platform addresses this limitation with structured multi-label morphological and clinical image annotation, support for controlled but customizable annotation protocols, and workflow-based computational analysis with integrated provenance tracking [7].

Provenance tracking is critical for research reproducibility. Without it, high variability in annotations and lack of process documentation limit reproducibility and reusability of collected data [7]. When selecting an annotation tool, verify that it records who annotated each image, when the annotation was created, and what protocol or guidelines were followed.

Practical Workflow for Building Segmentation Datasets

Step 1: Define the Annotation Protocol

Before selecting tools, document the annotation protocol. This protocol should specify the anatomic structures or pathologic regions to label, the labeling categories, and the boundary definitions. The CRS4 Digital Pathology Platform demonstrates the value of well-defined protocols for gathering high-quality structured annotations [7].

The protocol should also define the review process. The pediatric fracture dataset used a double-blind cross-review protocol where two experienced surgeons independently annotated images and then cross-reviewed each other's work [18]. This approach reduces individual bias and improves labeling accuracy.

Step 2: Select the Annotation Tool

Choose the tool based on your imaging modality, dataset size, and available computational resources. For histopathology, QuPath provides comprehensive computational tools for evaluating histology slides [6]. For general medical imaging, AnatomySketch offers a flexible plugin interface for integrating user-developed algorithm modules [5]. For ultrasound, Annotation Web provides a web-based interface suitable for collaborative annotation [25].

Consider whether the tool supports the annotation format required by your downstream deep learning framework. Many segmentation models expect COCO-format polygon annotations or mask files. The quality control pipeline for endoscopic airway imaging worked with COCO-format datasets and validated polygon structure before model training [17].

Step 3: Implement Quality Control Checks

Annotation quality control should occur before model training, not after. The systematic quality control workflow for polygon annotations validated polygon structure, generated masks at native image resolution, and removed small noise-like instances using an area threshold [17]. These checks improved boundary consistency and reduced mask fragmentation.

Quality control should also include inter-annotator agreement assessment. When multiple annotators label the same images, measure agreement using metrics such as Dice similarity coefficient or Cohen's kappa. The HOTSPoT model validation showed strong concordance between automated portal tract quantification and manual assessments with kappa values up to 0.90 [8].

Step 4: Document Dataset Characteristics

Record the demographic and clinical characteristics of the imaging data. The MRSegmentator study reported demographic information for 70 of 139 evaluated participants with mean age 52.7 years and 36 female participants [11]. This documentation supports external validation and helps identify potential biases in model performance.

Dataset documentation should also include acquisition parameters such as imaging protocol, scanner type, and resolution. The framework for a national cancer imaging repository in Nigeria emphasizes standardized protocols for data ingestion and preprocessing to ensure that repositories maintain only high-quality images and support interoperability with existing databases [14].

Options and Tradeoffs in Annotation Approaches

Manual Annotation

Manual annotation by expert clinicians remains the reference standard for medical image segmentation datasets. Expert hepatopathologists annotated the multi-institutional dataset of 223 cases used to train and validate the HOTSPoT model [8]. Manual annotation provides the highest quality labels but is time-intensive and expensive.

The time cost of manual annotation varies by modality and structure complexity. Segmenting impacted third molars in cone beam computed tomography images requires significant time and effort from clinicians and may lead to treatment delays [21]. For large datasets, manual annotation of every image may be impractical.

Deep Learning-Assisted Annotation

Deep learning models can generate initial annotations that experts then review and correct. This approach reduces the time required per case while maintaining quality through expert oversight. The impacted third molar dataset used AttentionUnet3D with a merged annotation strategy to generate annotations that physicians then quality-assessed [21].

The MRSegmentator model used a human-in-the-loop annotation workflow with cross-modality transfer learning from an existing CT segmentation model to segment 40 anatomic structures in MRI [11]. This approach leverages existing models to accelerate annotation of new modalities.

Crowdsourcing and Citizen Science

Crowdsourcing platforms enable distributed annotation by non-experts with expert oversight. The framework combining AI and citizen science used a user-friendly online platform enabling diverse crowd annotators to label medical images efficiently, with MedSAM segmentation AI accelerating the process and an algorithm merging crowd-labeled images to maintain expert-level quality [19].

This approach can substantially increase dataset size but requires careful quality control. The framework also used pix2pixGAN to expand the training dataset with synthetic images capturing realistic morphological features, which improved model performance especially when training data was limited [19].

Synthetic Data Generation

Generative models can create synthetic images with corresponding labels, expanding training datasets without additional manual annotation. HistoSmith uses a latent diffusion model to learn the joint distribution of cellular layouts, classification masks, and histology images, enabling tailored data generation conditioned on cell types, quantities, and tissue types [23]. This approach improved cell instance segmentation and classification, particularly for underrepresented cell types like neutrophils.

Synthetic data generation raises validation questions. Models trained on synthetic data must be evaluated on real clinical images to confirm generalization. The zero-shot domain adaptation framework using CycleGAN generated X-ray-like images from breast ultrasound images while preserving structural information, enabling existing segmentation masks to be reused as training labels for a U-Net model tested on real X-ray images [20].

Records and Measurements for Annotation Projects

Annotation Logs

Maintain detailed logs of the annotation process including annotator identifiers, dates, image identifiers, and protocol versions. The CRS4 Digital Pathology Platform includes provenance tracking as a core feature, enabling researchers to trace how each annotation was created [7]. This documentation supports reproducibility and enables analysis of annotator variability.

Quality Metrics

Track quality metrics throughout the annotation process. Common metrics include:

• Dice similarity coefficient for comparing annotations against reference standards
• Intersection over Union for measuring boundary agreement
• Inter-annotator agreement for assessing consistency across annotators
• Annotation time per case for estimating project timelines

The HOTSPoT model achieved mean Dice scores of 0.92 on training and validation data and 0.91 on test data, with mean Intersection over Union values of 0.86, 0.85, and 0.84 respectively [8]. These metrics provide benchmarks for annotation quality in liver pathology.

Dataset Versioning

Treat annotation datasets as versioned assets. When annotations are corrected or protocols change, create new dataset versions instead of overwriting existing files. This practice supports reproducibility and enables analysis of how annotation changes affect model performance.

Common Failure Patterns in Annotation Projects

Inconsistent Boundary Definitions

When multiple annotators label the same structures, boundary definitions often vary. This inconsistency reduces model performance and complicates validation. The quality control pipeline for endoscopic airway imaging found that fragmented regions and geometrically inconsistent polygon annotations reduced reproducibility and spatial stability in segmentation models [17].

Mitigation: Develop detailed annotation guidelines with visual examples of correct and incorrect boundaries. Conduct training sessions for all annotators and measure inter-annotator agreement before scaling up annotation efforts.

Insufficient Quality Control

Skipping quality control checks before model training leads to downstream performance issues. The annotation quality control workflow improved segmentation robustness without modifying the network architecture, demonstrating that annotation-level fixes are effective [17].

Mitigation: Implement automated quality checks for polygon structure, mask consistency, and area thresholds. Include manual review of a random sample of annotations by an expert.

Inadequate Protocol Documentation

Without documented protocols, annotation variability increases and reproducibility suffers. The CRS4 Digital Pathology Platform emphasizes the difficulty of strictly following precise review protocols and the lack of provenance tracking during annotation processes, which can result in high variability and limit reproducibility and reusability of collected data [7].

Mitigation: Write protocols before annotation begins. Include structure definitions, boundary rules, labeling categories, and review procedures. Update protocols through a versioned process.

Ignoring Domain Shift

Models trained on one imaging domain may not generalize to new clinical domains because of variations in image properties and lack of annotated data [20]. Annotation datasets built from a single institution or scanner may not represent the diversity of clinical practice.

Mitigation: Include data from multiple institutions and imaging systems when possible. The HOTSPoT model used a multi-institutional dataset of 223 cases and showed minimal domain shift with mean Dice scores of 0.92 on training and validation data and 0.91 on test data [8].

Limitations of Current Annotation Tools

Single-Label Limitations

Many open-source annotation tools employ single-label approaches that provide a flat representation of whole slide images, limiting their ability to capture the complexity of diagnosis-significant elements in a detailed and structured way [7]. For research questions requiring multi-label annotations, such as distinguishing multiple tissue types or grading pathologic features, these tools may be insufficient.

Scalability Constraints

Some annotation tools struggle with very large datasets or whole slide images. The rapid evolution of digital pathology has enabled large-scale data acquisition, but annotation tools must keep pace with the computational demands of processing and displaying these images [7].

Integration Challenges

Annotation tools vary in their ability to export annotations in formats compatible with deep learning frameworks. The HOTSPoT model is available as a TorchScript file with a modified WSInfer library, enabling efficient whole slide image inference and integration with QuPath for advanced pathology analysis [8]. When selecting tools, verify export format compatibility with your intended training pipeline.

Annotation Bias

Annotator and clinician biases can affect segmentation quality. Research on assessing annotator and clinician biases in an open-source-based tool used to generate head CT segmentations for deep learning training examines these effects [27]. Bias can arise from prior experience, interpretation of guidelines, or systematic differences in boundary placement.

Safety and Regulatory Context

Data Privacy and Sharing

Medical image datasets contain protected health information. Researchers must follow applicable data sharing policies and institutional review board requirements. The National Institutes of Health Genomic Data Sharing Policy provides a framework for responsible data sharing that can inform medical imaging data practices [3]. The National Center for Biotechnology Information provides data resources that support responsible sharing and reuse of biomedical data [2].

The framework for a national cancer imaging repository in Nigeria emphasizes adherence to ethical standards and support for interoperability with existing databases [14]. When building annotation datasets, document the data use agreements and privacy protections applicable to your imaging data.

FAIR Data Principles

The FAIR Guiding Principles provide a framework for making data findable, accessible, interoperable, and reusable [4]. Applying these principles to annotation datasets improves their utility for the research community. The European Bioinformatics Institute provides training resources that cover data management practices applicable to biomedical research [1].

Clinical Validation Requirements

Models trained on annotated datasets require clinical validation before deployment. The MRSegmentator study evaluated model performance on external datasets including the German National Cohort and Multi-Modality Abdominal Multi-Organ Segmentation Challenge 2022 dataset [11]. External validation is essential for establishing generalizability.

Professional Escalation Criteria

When to Consult a Domain Expert

Annotation projects should include domain expert involvement at key decision points. If annotators disagree on structure boundaries, escalate to a domain expert for adjudication. If the annotation protocol does not capture clinically relevant features, consult with clinicians to revise the protocol.

When to Seek Institutional Review

If the annotation project involves protected health information or human subjects data, consult with your institutional review board before beginning annotation. The National Institutes of Health Genomic Data Sharing Policy provides guidance on responsible data sharing that may apply to your project [3].

When to Reconsider Tool Selection

If the selected annotation tool cannot handle your data format, does not support required annotation types, or cannot export annotations in the format needed for model training, reconsider the tool selection. The CRS4 Digital Pathology Platform was developed specifically because currently available open-source annotation tools could not provide the structured multi-label annotations and provenance tracking needed for research studies [7].

Frequently Asked Questions

What is the difference between semantic segmentation and instance segmentation annotation?

Semantic segmentation assigns a class label to every pixel in an image, so all pixels belonging to the same class share the same label. Instance segmentation distinguishes individual objects within the same class, so each cell or organ instance receives a separate label. The choice affects annotation tool requirements and dataset format. Instance segmentation datasets typically use COCO-format polygon annotations, while semantic segmentation datasets often use mask files. The quality control pipeline for endoscopic airway imaging worked with COCO-format datasets containing polygon instances [17].

How many annotated images do I need for a medical image segmentation project?

The required number of annotated images depends on the task complexity, imaging modality, and model architecture. The impacted third molar dataset used 186 expert manual annotations supplemented by 1,076 deep learning-assisted annotations [21]. The pediatric fracture dataset used 2,015 carefully annotated images from 10,325 total images [18]. Frugal learning methods can reduce annotation requirements, with semi-supervised and transfer learning approaches achieving a Dice similarity coefficient of 0.89 compared to 0.92 for fully supervised models in kidney segmentation [13].

What annotation format should I use for deep learning segmentation models?

The annotation format should match your training pipeline requirements. COCO-format polygon annotations are common for instance segmentation and support quality control validation [17]. Mask files at native image resolution are used for semantic segmentation. Verify that your annotation tool can export the format required by your deep learning framework before committing to a tool.

Can I use the Segment Anything Model for medical image annotation?

The Segment Anything Model can be adapted for medical imaging through fine-tuning. MedicoSAM, a fine-tuned version of SAM for medical images, shows performance improvements especially for interactive segmentation tasks and is compatible with existing data annotation tools [15]. The citizen science framework integrated MedSAM segmentation AI with crowd annotators to accelerate the annotation process [19].

How do I ensure annotation quality across multiple annotators?

Implement a structured annotation protocol with clear boundary definitions and labeling categories. Use a double-blind cross-review protocol where annotators independently label images and then review each other's work [18]. Measure inter-annotator agreement using metrics such as Dice similarity coefficient or Cohen's kappa. Apply automated quality control checks for polygon structure and mask consistency before model training [17].

What is human-in-the-loop annotation?

Human-in-the-loop annotation combines automated segmentation with expert review and correction. The MRSegmentator model used a human-in-the-loop annotation workflow with cross-modality transfer learning from an existing CT segmentation model to segment 40 anatomic structures in MRI [11]. This approach reduces annotation time while maintaining quality through expert oversight.

How can I reduce the annotation burden for my project?

Several strategies can reduce annotation requirements. Frugal learning methods including data augmentation, semi-supervised learning, and weak supervision achieved a Dice similarity coefficient of 0.89 in kidney segmentation, only moderately lower than the fully supervised model at 0.92 [13]. Deep learning-assisted annotation can reduce time per case to seconds [21]. Synthetic data generation with generative models can expand training datasets without additional manual annotation [23].

Where can I find existing medical image segmentation datasets?

Public datasets are available through repositories such as The Cancer Imaging Archive and Kaggle. The leukocyte segmentation study used a publicly available dataset from Kaggle consisting of 12,444 images of four types of leukocytes [22]. The MRSegmentator study used the TotalSegmentator CT dataset and Multi-Modality Abdominal Multi-Organ Segmentation Challenge 2022 dataset [11]. The National Center for Biotechnology Information provides data resources that support biomedical research [2].

Related Bioinformatics Guides

• Deep Learning for Histopathology Image Classification in Veterinary Oncology
• Deep Learning for Functional Genomics
• Deep Learning for Gene Regulatory Network Reconstruction
• Deep Learning for Annotating Structural Variants in Viral Genomes
• Deep Learning for Predicting Viral Antigenic Drift from Sequence Surveillance

References and Further Reading

• EMBL-EBI Training. European Bioinformatics Institute.
• NCBI Data Resources. National Center for Biotechnology Information.
• Genomic Data Sharing Policy. National Institutes of Health.
• The FAIR Guiding Principles. Scientific Data.
• AnatomySketch: An Extensible Open-Source Software Platform for Medical Image Analysis Algorithm Development.. Journal of digital imaging, 2022.
• QuPath Edu and OpenMicroanatomy: Open-source virtual microscopy tools for medical education.. Journal of anatomy, 2025.
• An open-source platform for structured annotation and computational workflows in digital pathology research.. Scientific reports, 2025.
• Automating liver biopsy segmentation with a robust, open-source tool for pathology research: the HOTSPoT model.. NPJ digital medicine, 2025.
• GALBA: Genome Annotation with Miniprot and AUGUSTUS.. bioRxiv : the preprint server for biology, 2023.
• Galba: genome annotation with miniprot and AUGUSTUS.. BMC bioinformatics, 2023.
• Segmenting Whole-Body MRI and CT for Multiorgan Anatomic Structure Delineation.. Radiology. Artificial intelligence, 2025.
• A deep learning pipeline for accurate and automated restoration, segmentation, and quantification of dendritic spines.. Cell reports methods, 2025.
• Frugal Learning Methods for Kidney Segmentation in Non-Contrast MRI.. 2026.
• A framework for a national cancer imaging repository in Nigeria.. 2026.
• MedicoSAM: Robust Improvement of SAM for Medical Imaging.. 2026.
• OOD-SEG: Exploiting out-of-distribution detection techniques for learning image segmentation from sparse multi-class positive-only annotations.. 2026.
• A Reproducible COCO-Polygon Quality-Control Pipeline Improves Segmentation Stability in Endoscopic Airway Imaging.. 2026.
• A Benchmark X-ray Dataset for Pediatric Supracondylar Humerus Fractures with Improved YOLOv11-Based Detection.. 2026.
• Coupling AI and Citizen Science in Creation of Enhanced Training Dataset for Medical Image Segmentation. arXiv.org, 2024.
• Annotation-Free Zero-Shot Domain Adaptation for Robust Medical Image Segmentation Across Unseen Modalities and Diseases. 2026 6th International Conference on Expert Clouds and Applications (ICOECA), 2026.
• Deep learning-assisted creation of a large-scale cone beam computed tomography dataset for the segmentation of impacted mandibular third molar. Quantitative Imaging in Medicine and Surgery, 2023.
• Deep learning-based image annotation for leukocyte segmentation and classification of blood cell morphology. BMC Medical Imaging, 2024.
• HistoSmith: Single-Stage Histology Image-Label Generation via Conditional Latent Diffusion for Enhanced Cell Segmentation and Classification. arXiv.org, 2025.
• ExpACVO-Exponential Anti Corona Virus Optimization Enabled Hybrid Deep learning for Tongue Image Segmentation Towards Diabetes Mellitus Detection Via CNN-LSTM Network. 2023 International Conference on Artificial Intelligence for Innovations in Healthcare Industries (ICAIIHI), 2023.
• Annotation Web - An open-source web-based annotation tool for ultrasound images. IEEE International Ultrasonics Symposium Ius, 2021.
• An open source and free GUI based tool for annotating 3-D medical images. Proceedings of the IEEE EMBS Region 8 International Conference on Information Technology Applications in Biomedicine ITAB, 2010.
• Assessing Annotator and Clinician Biases in an Open-Source-Based Tool Used to Generate Head CT Segmentations for Deep Learning Training. Lecture Notes in Computer Science, 2026.
• HistoColAi: An open-source web platform for collaborative digital histology image annotation with AI-driven predictive integration. Computer Methods and Programs in Biomedicine, 2025.
• AI2Seg: A Method and Tool for AI-based Annotation Inspection of Biomedical Instance Segmentation Datasets. Proceedings of the Annual International Conference of the IEEE Engineering in Medicine and Biology Society EMBS, 2023.

This article is educational and does not replace validated analysis plans, institutional policy, clinical interpretation, or specialist review.