Multi-Class Segmentation and Classification of Intestinal Organoids: YOLO Stand-Alone vs. Hybrid Machine Learning Pipelines

Abstract

Background: The automated analysis of intestinal organoids in microscopy images are essential for high-throughput morphological studies, enabling precision and scalability. Traditional manual analysis is time-consuming and subject to observer bias, whereas Machine Learning (ML) approaches have recently demonstrated superior performance. Purpose: This study aims to evaluate YOLO (You Only Look Once) for organoid segmentation and classification, comparing its standalone performance with a hybrid pipeline that integrates DL-based feature extraction and ML classifiers. Methods: The dataset, consisting of 840 light microscopy images and over 23,000 annotated intestinal organoids, was divided into training (756 images) and validation (84 images) sets. Organoids were categorized into four morphological classes: cystic non-budding organoids (Org0), early organoids (Org1), late organoids (Org3), and Spheroids (Sph). YOLO version 10 (YOLOv10) was trained as a segmenter-classifier for the detection and classification of organoids. Performance metrics for YOLOv10 as a standalone model included Average Precision (AP), mean AP at 50% overlap (mAP50), and confusion matrix evaluated on the validation set. In the hybrid pipeline, trained YOLOv10 segmented bounding boxes, and features extracted from these regions using YOLOv10 and ResNet50 were classified with ML algorithms, including Logistic Regression, Naive Bayes, K-Nearest Neighbors (KNN), Random Forest, eXtreme Gradient Boosting (XGBoost), and Multi-Layer Perceptrons (MLP). The performance of these classifiers was assessed using the Receiver Operating Characteristic (ROC) curve and its corresponding Area Under the Curve (AUC), precision, F1 score, and confusion matrix metrics. Principal Component Analysis (PCA) was applied to reduce feature dimensionality while retaining 95% of cumulative variance. To optimize the classification results, an ensemble approach based on AUC-weighted probability fusion was implemented to combine predictions across classifiers. Results: YOLOv10 as a standalone model achieved an overall mAP50 of 0.845, with high AP across all four classes (range 0.797–0.901). In the hybrid pipeline, features extracted with ResNet50 outperformed those extracted with YOLO, with multiple classifiers achieving AUC scores ranging from 0.71 to 0.98 on the validation set. Among all classifiers, Logistic Regression emerged as the best-performing model, achieving the highest AUC scores across multiple classes (range 0.93–0.98). Feature selection using PCA did not improve classification performance. The AUC-weighted ensemble method further enhanced performance, leveraging the strengths of multiple classifiers to optimize prediction, as demonstrated by improved ROC-AUC scores across all organoid classes (range 0.92–0.98). Conclusions: This study demonstrates the effectiveness of YOLOv10 as a standalone model and the robustness of hybrid pipelines combining ResNet50 feature extraction and ML classifiers. Logistic Regression emerged as the best-performing classifier, achieving the highest ROC-AUC across multiple classes. This approach ensures reproducible, automated, and precise morphological analysis, with significant potential for high-throughput organoid studies and live imaging applications.

1. Introduction

The intestinal epithelium, a highly dynamic tissue, is extensively studied to elucidate stem cell biology and the pathogenesis of critical diseases, particularly colorectal cancer. Three-dimensional (3D) ex vivo organoid culture systems have revolutionized this field by providing physiologically relevant models that replicate key features of in vivo tissue architecture, such as crypt-villus domains and niche-derived growth factor dependence [1,2].

As organoids progress in culture, they exhibit distinct morphological transitions—from cystic structures enriched with stem and Paneth cells to budding organoids with crypt-like protrusions, and eventually, spheroid-like morphologies under specific conditions [3]. These changes serve as vital readouts for processes such as stemness, epithelial regeneration, and differentiation. For example, crypt budding quantifies stem cell activity, while spheroid morphology is associated with Wnt signaling, cellular plasticity, and tumorigenesis [3]. These spheroid-like morphologies represent a less differentiated epithelial state, occurring under particular growth conditions and often associated with specific signaling pathways, such as Wnt regulation [4]. These changes serve as vital readouts for processes such as stemness, epithelial regeneration, and differentiation. For instance, crypt budding quantifies stem cell activity, while spheroid morphology is associated with Wnt signaling, cellular plasticity, and tumorigenesis [3,5]. The ability to analyze these morphological transitions has been critical for studies investigating the effects of signaling pathways, drugs, and genetic perturbations on intestinal tissue homeostasis. For example, studies have demonstrated the role of Wnt and Notch signaling pathways in driving organoid growth and differentiation [6]. Furthermore, high-throughput organoid systems have been leveraged to investigate the impact of oncogenic mutations, such as APC loss or KRAS activation, on epithelial architecture and proliferative capacity [1,3]. Additionally, recent advancements have enabled the development of co-culture models where organoids are paired with stromal or immune cells to mimic more complex tissue microenvironments, facilitating translational studies on inflammation and cancer [7,8].

Despite these advances, current approaches to analyze organoid morphology remain largely manual, involving visual inspection and classification, which is time-consuming, subjective, and prone to variability between researchers [5,9,10,11,12,13,14,15,16,17]. This limitation becomes particularly evident when dealing with large datasets or longitudinal experiments that require continuous monitoring. Recent efforts have introduced computational tools to address medical challenge [18,19,20,21,22]. For instance, deep learning (DL)-based methods such as convolutional neural networks (CNNs) have been applied to automate the quantification of organoid size, shape, and viability [23,24]. In addition, new approaches leveraging machine learning (ML) algorithms, including object-detection frameworks like YOLO (You Only Look Once), have demonstrated significant promise for automating organoid analysis in real time [25]. These tools enable rapid detection of organoids across multiple morphological classes, overcoming the limitations of manual analysis while maintaining high accuracy. For instance, OrganoID provides a versatile platform for tracking and morphological classification of organoids in brightfield and phase-contrast microscopy, offering high accuracy and broad applicability for various organoid types [11]. Similarly, OrBITS facilitates label-free, real-time, and time-lapse monitoring of organoids, making it particularly valuable for drug screening and longitudinal studies [26].On the other hand, D-CryptO focuses specifically on colon organoid morphology, leveraging DL to evaluate structural features such as crypt formation and opacity from brightfield images, thus providing insights into organoid maturity and development.

Recent studies have underscored the transformative potential of computational approaches for organoid analysis, particularly in addressing the limitations of manual methods. Xiang et al. [27] reviewed the applications of intestinal organoids in disease modeling, drug screening, and regenerative medicine. They highlighted the use of cutting-edge technologies, including organoid-on-chip systems, genome editing, and microfluidics, to enhance high-throughput cultivation and better mimic patient-specific genetic characteristics.

Laudadio et al. [28] developed a fibrosis model using patient-derived intestinal organoids (PDOs) exposed to pro-inflammatory cytokines, such as TNF-α and TGF-β1. Their study revealed upregulation of epithelial–mesenchymal transition (EMT) genes in fibrotic PDOs, particularly in Crohn’s disease samples. This model provides an advanced tool for studying fibrogenesis in inflammatory bowel disease (IBD).

Matthews et al. [11] introduced OrganoID, a DL platform for real-time tracking and morphological classification of organoids. This versatile tool demonstrated high accuracy and broad applicability, facilitating automated monitoring of organoid dynamics across various experimental setups.

Deben et al. [26] presented OrBITS, a label-free platform for longitudinal and real-time analysis of organoids, particularly suited for drug screening applications. By enabling precise tracking of organoid responses, OrBITS bridges a critical gap in experimental reproducibility and scalability.

Abdul et al. [29] proposed D-CryptO, a DL-based tool tailored for analyzing colon organoid morphology. By evaluating structural features such as crypt formation and opacity in brightfield images, D-CryptO provides detailed insights into organoid development and maturity.

These studies collectively illustrate the rapid evolution of computational tools for organoid research, emphasizing their potential to enhance experimental throughput, reproducibility, and precision. However, challenges remain, particularly in generalizing across diverse experimental conditions and resolving morphologically similar structures.

In this study, we propose an innovative framework for automated segmentation and classification of intestinal organoids, combining the strengths of DL and ML approaches. Specifically:

• We evaluate YOLOv10 as a standalone model for organoid segmentation and classification, demonstrating its superior accuracy and efficiency over earlier YOLO architectures.
• We introduce a hybrid pipeline where features extracted from YOLO or ResNet50 are classified using state-of-the-art ML algorithms, including Logistic Regression, Random Forest, Naive Bayes, Multi-Layer Perceptrons (MLP), K-Nearest Neighbors (KNN), and eXtreme Gradient Boosting (XGBoost).
• To further enhance performance, we implement an AUC-weighted ensemble method that integrates predictions across multiple classifiers, achieving robust and scalable morphological classification.

This framework overcomes the limitations of manual annotation and current computational tools, enabling precise, high-throughput morphological analysis of organoids. With significant implications for oncology, stem cell biology, and therapeutic screening, our approach paves the way for advancements in cancer research, offering a scalable and reliable solution for organoid-based studies.

2. Methods

2.1. Dataset

The dataset used in this study was adopted from Domenech-Moreno et al. [25] and downloaded from the following link: https://datadryad.org/stash/dataset/doi:10.5061/dryad.jdfn2z3d5 (accessed on 20 April 2024). It consists of 840 transmitted-light microscopy images of intestinal organoids derived from crypts isolated from mouse small intestines. The dataset captures organoids at different morphological stages, including cystic non-budding organoids (Org0), early budding organoids with 1–2 crypt-like structures (Org1), advanced organoids with three or more budding crypts (Org3), and spheroids (Sph), which are thin-walled, non-budding structures with a larger diameter.

The dataset contains a total of 23,066 annotated organoids, unevenly distributed across the four morphological classes. Org0 represents 52% of the total annotations, followed by Org1 with 24%, Org3 with 14%, and Sph comprising 10%. Each annotation was saved in the YOLO format, where bounding boxes are represented by a class label and four normalized coordinates corresponding to the center, width, and height of the region of interest.

To facilitate robust model training and evaluation, the dataset was split into two subsets: 756 images (90%) were assigned to the training set, while the remaining 84 images (10%) were reserved for the validation set. The annotations were manually reviewed for accuracy to ensure high-quality morphological classification and precise bounding box placement.

The dataset was used in its original form without any additional preprocessing (e.g., denoising, contrast enhancement, or normalization), as released by Domenech-Moreno et al. via the Dryad repository, to maintain full comparability with the original study conditions.

2.2. YOLO Standalone Model for Segmentation and Classification

You Only Look Once (YOLO) is a state-of-the-art, real-time object detection algorithm. YOLO frames the object detection problem as a single regression problem, directly predicting bounding boxes and class probabilities from full images in one evaluation, making it extremely fast and accurate [30]. YOLO version 10 (YOLOv10) was employed as a standalone model for the simultaneous segmentation and classification of intestinal organoids into the four morphological classes. Two YOLOv10 configurations were tested: YOLOv10-nano and YOLOv10-medium, with the latter yielding the best performance. The training process used the pre-defined dataset split (90:10) provided by the original dataset authors [25].

The YOLOv10-medium model was trained for 150 epochs with a batch size of 28 and a learning rate of 0.01, using the default optimization strategies provided by the YOLO framework. No additional modifications or fine-tuning were introduced to the training process.

Model performance was evaluated on the validation set using standard object detection metrics. Specifically, the mean Average Precision at 50% overlap (mAP50) was calculated. The mAP50 metric is a widely used benchmark in object detection tasks that evaluates the ability of the model to correctly predict bounding boxes for objects within an image. It is computed by calculating the Average Precision (AP) for each class and averaging these values across all classes. The AP is determined by measuring the area under the Precision-Recall (PR) curve, which is generated by varying the detection confidence threshold. To qualify as a correct detection, the predicted bounding box must overlap with the ground truth bounding box by at least 50% as measured by the Intersection over Union (IoU) metric. The IoU is defined as the ratio between the area of overlap and the area of union of the predicted and ground truth bounding boxes. The mAP50, therefore, summarizes the Precision and Recall of the model across all classes and thresholds when using an IoU threshold of 50%, providing a comprehensive evaluation of detection accuracy. Precision and Recall were computed for each class to evaluate class-specific performance. Precision measures the proportion of correctly predicted positive instances out of all positive predictions, while Recall evaluates the proportion of true positives identified out of all actual positives. The workflow is described in Figure 1.

2.3. Hybrid Pipeline: YOLO Segmentation and Feature Extraction

In addition to evaluating YOLOv10 as a standalone model, a hybrid pipeline was implemented to combine object detection with feature extraction and classification. The workflow is described in Figure 2. The already trained YOLOv10-medium was used to segment the regions of interest (ROIs) by detecting bounding boxes in both the training and validation sets. The resulting bounding boxes, corresponding to the four morphological classes, were cropped from the images and saved as individual .jpeg files along with their corresponding class labels to ensure uniform input for feature extraction and subsequent analysis. Features were extracted from these cropped regions using two distinct approaches. The first approach utilized a YOLO model while the second approach employed ResNet50. ResNet50, introduced by Kaiming He et al. in 2015, is a deep convolutional neural network (CNN) known for its residual learning framework, which allows for the training of very deep networks by mitigating the vanishing gradient problem [31]. Neither model was fine-tuned on the specific organoid dataset, ensuring the extracted features remained generalized. Feature extraction was performed using ResNet50, selected for its proven robustness in medical imaging and efficient residual architecture. YOLOv10 was adopted for detection due to its improved accuracy and real-time capability. Hyperparameters were optimized following standard YOLO practices (input size, learning rate, batch size).

2.4. Machine Learning Classifiers

Following the extraction of features from the segmented bounding boxes using YOLO and ResNet50, a set of ML classifiers was employed to predict the morphological class of each organoid. The classifiers included Logistic Regression, Random Forest, Naive Bayes, MLP, KNN, and XGBoost. These classifiers were chosen because they natively support multi-class classification, making them well-suited for distinguishing among the four morphological classes of organoids without requiring additional modifications or wrapping strategies. Logistic Regression was trained with the lbfgs solver and a maximum of 500 iterations to guarantee convergence; the Random Forest classifier was configured with 100 estimators; the Naive Bayes classifier was implemented using the Gaussian Naive Bayes algorithm, which is based on probabilistic modeling assuming feature independence. The MLP classifier was trained with a maximum of 300 iterations, while the KNN classifier was configured with five neighbors. Finally, XGBoost was set up with the “mlogloss” evaluation metric for multi-class classification. All classifiers were trained on the features extracted from the training set and validated on the features extracted from the held-out validation set.

The classifiers were evaluated on the validation set using standard performance metrics, including the Receiver Operating Characteristic (ROC) curve and its corresponding Area Under the Curve (AUC), and confusion matrix analysis. The ROC curve illustrates the true positive rate (sensitivity) against the false positive rate (1-specificity) across different classification thresholds, while the AUC quantifies the overall performance of the classifier as a single value. A higher AUC reflects greater discriminative ability to distinguish between the morphological classes. Confusion matrices were used to identify class-specific performance, highlighting potential misclassifications and imbalances across classes. The ROC curve was utilized to assess the classifier performance and to determine an “optimal” prediction threshold that maximizes accuracy.

To investigate the impact of feature dimensionality on classification performance, Principal Component Analysis (PCA) was applied to reduce the number of features while retaining 95% of the cumulative variance. The performance of each classifier was compared both before and after feature reduction to evaluate the effect of dimensionality on model accuracy and robustness. However, since PCA did not yield consistent performance gains, it was not retained in the final pipeline; all classifiers were ultimately trained on the full feature sets.

2.5. AUC-Based Ensemble Method

To enhance the overall robustness and accuracy of the hybrid pipeline, an ensemble method was implemented by combining the predictions of all ML classifiers. The ensemble approach utilized an AUC-weighted probability fusion, where the contribution of each classifier to the final prediction was proportional to its AUC score.

For each bounding box, the predicted class probabilities from all classifiers were aggregated using the weighted fusion strategy. This method assigned higher weights to classifiers with superior performance, as reflected by their AUC scores, while reducing the influence of less effective models. The ensemble output represented a fused probability distribution across all classes, leading to more reliable and accurate predictions.

This ensemble strategy leverages the strengths of individual classifiers while mitigating their weaknesses, ultimately optimizing the classification performance of the hybrid pipeline. The final ensemble predictions were evaluated on the validation set using the same metrics as the individual classifiers, including ROC and AUC analysis.

2.6. Computing Environment and Runtime

Experiments were executed on the CINECA Leonardo supercomputing cluster (NVIDIA A100-class GPUs; Linux environment). YOLOv10 training completed in about one day on a single GPU. Training of downstream ML classifiers (using ResNet50 features) required ~5–8 h. During inference, YOLOv10 achieved millisecond-level latency per image on GPU (throughput depending on batch size and input resolution). To assess portability beyond HPC infrastructure, we also validated the pipeline on a workstation-class PC equipped with an NVIDIA GeForce RTX 3090 GPU, observing consistent behavior and comparable relative trends in training/inference performance. These figures are indicative and may vary across hardware configurations.

3. Results

3.1. YOLOv10 Standalone Model Performance

The YOLOv10-medium model demonstrated strong performance in both segmentation and classification tasks when evaluated on the validation set. The model achieved a mAP50 of 0.845, with a precision of 0.787, a recall of 0.763, reflecting robust overall performance.

As shown in Figure 3A, the PR curves highlight class-specific performance, with Org0 achieving a PR score of 0.885, followed by Org3 with 0.901, while Org1 and Sph exhibited slightly lower values of 0.797. The overall performance is summarized by the thick blue PR curve, corresponding to the model’s mAP50 of 0.845.

The confusion matrix (Figure 3B) further illustrates the classification performance. High accuracy was observed for Org0 (86%) and Organoid3 (82%), while Organoid1 and Spheroids were correctly classified in 74% and 68% of cases, respectively. Misclassifications were more frequent between Org0 and Sph, likely due to their morphological similarity. Background detections contributed minimally to false positives, indicating reliable localization and classification.

3.2. Hybrid Pipeline Performance

The hybrid pipeline, combining YOLOv10 segmentation with feature extraction and ML classifiers, demonstrated robust and scalable performance across the four morphological classes. Features were extracted from the bounding boxes using either YOLOv10 or ResNet50, and subsequently classified using Logistic Regression, Random Forest, Naive Bayes, MLP, KNN, and XGBoost.

The results, summarized in

Table 1. Performance comparison of ML classifiers in the hybrid pipeline. The table presents the AUC scores for each classifier across the four morphological classes (Org0, Org1, Org3, and Sph) using features extracted from YOLOv10 and ResNet50. The final column shows the AUC scores for the ensemble method based on ResNet50 features.

Classifier	Feature Extraction								Enseble ResNet50
	YOLO				ResNet50
	AUC				AUC				AUC
	Org0	Org1	Org3	Sph	Org0	Org1	Org3	Sph	Org0	Org1	Org3	Sph
LR	0.89	0.83	0.91	0.80	0.97	0.93	0.98	0.98	0.96	0.92	0.98	0.96
Random Forest	0.89	0.82	0.92	0.71	0.96	0.90	0.97	0.94
Naïve Bayes	0.85	0.77	0.89	0.74	0.91	0.84	0.96	0.87
KNN	0.80	0.75	0.69	0.71	0.92	0.83	0.93	0.87
XGBoost	0.92	0.87	0.94	0.82	0.96	0.92	0.98	0.97
MLP	0.92	0.87	0.94	0.84	0.96	0.91	0.97	0.97

LR = Logistic Regresssion; KNN = K-Nearest Neighbors; XGBoost = eXtreme Gradient Boosting; MLP = Multi-Layer Perceptron.

, reveal that classifiers trained on features extracted with ResNet50 consistently outperformed those using YOLOv10 features. Among the individual classifiers, Logistic Regression achieved the highest AUC scores with ResNet50 features, reaching AUC scores of 0.97 for Org0, 0.93 for Organoid1, 0.98 for Organoid3 and 0.98 for Spheroid (Figure 4), showcasing its superior performance across the four morphological classes. In contrast, classifiers trained on YOLOv10 features showed lower but competitive performance, with XGBoost and MLP achieving the highest AUC scores of 0.94 for Organoid3.

To further evaluate the classification performance, additional metrics such as accuracy, precision, and F1-score were computed for each classifier (

Table 2. Comparative performance of Machine Learning classifiers trained on features extracted using YOLOv10 and ResNet50. The table presents accuracy, precision, recall, and F1-score for each classifier. The final column shows the metrics scores for the ensemble method based on ResNet50 features.

Classifier	Feature Extraction								Ensemble ResNet50
	YOLO				ResNet50
	Acc	P	R	F1	Acc	P	R	F1	Acc	P	R	F1
LR	0.71	069	0.70	0.69	0.84	0.84	0.84	0.84	0.84	0.83	0.84	0.83
Random Forest	0.69	0.69	0.69	0.65	0.82	0.81	0.81	0.81
Naïve Bayes	0.57	0.66	0.57	0.60	0.72	0.75	0.73	0.73
KNN	0.60	0.60	0.60	0.54	0.79	0.77	0.78	0.77
XGBoost	0.74	0.74	0.74	0.72	0.83	0.83	0.83	0.83
MLP	0.74	0.74	0.74	0.73	0.82	0.81	0.82	0.82

LR = Logistic Regresssion; KNN = K-Nearest Neighbors; XGBoost = eXtreme Gradient Boosting; MLP = Multi-Layer Perceptron; Acc = Accuracy; P = Precision; R = Recall; F1 = F1 score.

). Logistic Regression trained on ResNet50 features achieved the highest overall accuracy, precision, and F1-score (0.84) across the four morphological classes, confirming its superior performance as indicated by the AUC scores in Table 1. XGBoost followed closely, attaining an accuracy of 0.83 with balanced precision and F1-score values.

In contrast, classifiers trained on features extracted using YOLOv10 displayed lower, yet competitive performance. XGBoost and MLP achieved the best results within this group, with accuracy scores of 0.74 and F1-scores around 0.72–0.73. However, other classifiers, such as Naïve Bayes and KNN, struggled to generalize effectively on YOLOv10 features, reflecting lower accuracies (0.57–0.60) and F1-scores (0.54–0.60).

Given the superior performance of ResNet50 features, the ensemble method was structured exclusively around classifiers trained on these features, leveraging an AUC-weighted fusion of their predictions to further improve the overall performance. The ensemble achieved AUC scores of 0.96, 0.92, 0.98, and 0.96 for Org0, Org1, Org3, and Sph, respectively. Interestingly, the results of the ensemble method were similar to those achieved by Logistic Regression with ResNet50 features. Figure 5 shows the ROC curves for the ensemble method.

4. Discussion

This study explored the application of YOLOv10 as a standalone model and within a hybrid pipeline combining segmentation and feature extraction for the morphological analysis of intestinal organoids across four distinct classes: cystic non-budding organoids (Org0), early budding organoids (Org1), late budding organoids (Org3), and spheroids (Sph). YOLOv10 was trained to perform simultaneous segmentation and classification, while the hybrid pipeline leveraged features from YOLOv10 or ResNet50 for downstream ML classifiers.

YOLOv10 proved effective as a standalone model for simultaneous segmentation and classification of intestinal organoids. It efficiently detected and labeled the four morphological classes, supporting real-time applications where both speed and accuracy are required. Class-wise analysis showed stronger performance for phenotypes with clearer features (e.g., Org3 and Org0), while confusion mainly arose between morphologically similar classes (Org0 vs. Sph). These errors likely reflect subtle differences that are challenging for object-detection models to capture.

Despite these limitations, YOLOv10 remains a powerful tool for organoid segmentation and classification, enabling real-time processing and streamlined workflows. Its performance supports use in high-throughput studies, where rapid analysis is required for large datasets [25,30,32,33,34].

A significant comparison can be drawn with Tellu [25], a recent YOLOv5-based tool trained on the same microscopy dataset (YOLOv5-small). Tellu achieved real-time performance, its reported results highlight certain limitations, particularly when distinguishing morphologically similar classes.

Compared to Domènech-Moreno et al., who employed YOLOv5 for organoid classification, our study introduces several methodological innovations. First, we adopted YOLOv10, which provides architectural improvements for more accurate detection and segmentation. Second, we designed a hybrid pipeline that combines YOLO-based segmentation with ResNet50 feature extraction and multiple machine learning classifiers, significantly improving classification performance (AUC up to 0.98). Finally, we implemented an AUC-weighted ensemble strategy to enhance robustness across classes. These enhancements offer a scalable and reproducible framework for high-throughput organoid analysis, extending the capabilities of previous approaches.

The hybrid pipeline, combining YOLOv10 segmentation with feature extraction and ML classifiers, demonstrated enhanced performance in the morphological classification of intestinal organoids. It addressed limitations of the standalone YOLOv10, particularly when distinguishing between morphologically similar classes. Using features from the detected bounding boxes (ResNet50 or YOLOv10) provided a more nuanced representation of organoid morphology, which proved particularly effective for classification.

A key finding was the superior performance of ResNet50 features compared to YOLOv10 features as input for ML classifiers. ResNet50, as a deep residual network pre-trained on large-scale image datasets, was able to extract richer and more discriminative features, allowing classifiers such as Logistic Regression and XGBoost to achieve higher accuracy and AUC scores across all organoid classes. Our choice of ResNet50 was motivated by its strong performance in medical imaging and computational efficiency. YOLOv10 was adopted for detection because of its architectural improvements over previous YOLO versions. Future work will explore alternative CNNs such as EfficientNet and DenseNet for feature extraction in organoid analysis. This underscores the importance of feature quality for subtle morphological distinctions, particularly between closely related classes like Org0 and Sph.

The integration of YOLOv10 for segmentation with ResNet50-based feature extraction and ML classifiers aligns with recent advances in medical image analysis. This hybrid approach has been effectively applied in various studies to enhance classification performance. For instance, a study on breast cancer detection combined convolutional neural networks (CNNs) with a pruned ensemble extreme learning machine to improve detection, segmentation, feature extraction, and classification, resulting in enhanced diagnostic accuracy [35]. Similarly, research on brain tumor detection utilized a hybrid feature extraction technique alongside Support Vector Machine (SVM) classifiers, achieving improved accuracy in distinguishing between healthy and abnormal MRI scans [36]. These examples underscore the efficacy of DL-based feature extraction paired with traditional ML classifiers in medical imaging.

The AUC-weighted ensemble, which fused predictions from multiple ResNet50-based classifiers, added robustness and improved consistency across all four classes. While individual classifiers achieved strong performance, the ensemble approach effectively mitigated variability and leveraged the strengths of each model to produce optimized predictions. In medical image analysis, ensemble approaches have been employed to improve diagnostic accuracy. For instance, a study on lung cancer detection using CT scan images demonstrated that ensemble methods outperformed individual models, leading to improved diagnostic accuracy [37].

However, ensemble methods do not always outperform single models. In our study, for example, the ensemble method did not surpass the performance of Logistic Regression, which emerged as the best-performing classifier when trained on ResNet50 features. A comparative study between SVM and Logistic Regression highlighted scenarios where Logistic Regression performed favorably [38]. In another study [39], Logistic Regression and Random Forest across 243 datasets found that Random Forest outperformed Logistic Regression in 69% of the cases. However, in the remaining 31% of datasets, Logistic Regression performed comparably or better, indicating that simpler models can be effective depending on the dataset characteristics.

The hybrid pipeline offers a versatile and scalable framework for organoid analysis, combining the speed and accuracy of YOLOv10 segmentation with the power of ML classifiers for refined morphological classification. This approach improves accuracy and better resolves closely related classes, yielding more precise and reliable outcomes in organoid morphology.

Recent studies have applied DL to organoid image analysis with notable success in both segmentation and classification. For instance, Mask R-CNN–based models like OrgaSegment, precisely segment patient-derived intestinal organoids [13], while U-Net architectures such as OrgaExtractor [12] achieve high-accuracy boundary segmentation across organoids of various sizes.

On the classification side, one strategy is to leverage one-stage object detectors: Deep-Orga [40] built on YOLO to increase detection speed and accuracy, and a recent lightweight YOLOv8-based detector (Deliod) achieved high precision (mAP ≈ 87%) in intestinal organoid recognition with a minimal model size [33]. Alternatively, several groups have explored deep CNN classifiers (ResNet, DenseNet, etc.) for organoid phenotyping: e.g., Cicceri et al. evaluated multiple CNN and vision transformer models for intestinal organoid morphology classification, reporting more than 85% accuracy [41]. Deep models have likewise been used to identify phenotypic heterogeneity in tumor organoids, for example, in distinguished “cystic” vs. “solid” subtypes in colorectal cancer organoids using a CNN-based approach [42].

Beyond deep networks, organoid image analysis can be achieved with classical, low-compute pipelines that combine global or adaptive thresholding (e.g., Otsu) and morphological operations with object splitting via watershed or active contours [43,44,45]. These segmentations can be summarized into hand-crafted descriptors (area, circularity, solidity; texture features such as GLCM, LBP, HOG) and then classified with traditional ML models like SVM or Random Forest [46,47,48]. In summary, while non-DL pipelines are computationally lighter (CPU-friendly, easy to deploy) and appropriate for standardized readout, DL detectors/segmenters generally provide higher robustness in heterogeneous, multi-site scenarios; our YOLOv10 setup is intended to retain much of this robustness at modest inference cost.

Despite the promising results, this study has some limitations. The dataset used for training and validation, while substantial, was limited to brightfield microscopy images of intestinal organoids. This may restrict the generalizability of the model to other organoid types or imaging modalities, such as fluorescence microscopy or multi-channel images. Expanding the dataset to include diverse imaging techniques and additional organoid types would help validate the robustness and versatility of the proposed pipeline.

In addition, the segmentation accuracy of YOLOv10, though high, is not perfect, particularly in dense cultures where organoids overlap. Bounding-box errors can propagate to feature extraction and downstream classification. Future work could incorporate advanced segmentation strategies, such as instance segmentation models to address this limitation.

Another limitation is class imbalance: Org0 comprises over half of the annotations, whereas Sph is underrepresented. Although some classifiers used internal class-weighting that partially mitigated this issue, residual effects on performance cannot be ruled out. Despite these limitations, our approach is particularly well-suited for high-throughput studies, where the ability to automate segmentation and classification while maintaining high precision is essential. From a biological standpoint, improving the automated classification of intestinal organoid morphologies has concrete implications for biomedical research. Accurate and reproducible identification of developmental stages, such as cystic, budding, or spheroid organoids, can serve as a quantitative readout of growth dynamics, differentiation status, and treatment response. In drug screening or disease modeling studies, automated image-based classification could substantially accelerate data processing and reduce observer bias, allowing consistent and objective evaluation across large organoid libraries. Therefore, the performance improvements achieved by YOLOv10 and the hybrid pipeline are not only of technical value but also enhance the reliability and scalability of organoid-based experimental and preclinical research. Future adaptations could explore additional DL architectures for feature extraction or integrate attention mechanisms to further enhance the pipeline’s ability to capture subtle morphological variations.

5. Conclusions

This study demonstrates the effectiveness of YOLOv10 as a standalone model for the simultaneous segmentation and classification of intestinal organoids, as well as the robustness of a hybrid pipeline that integrates DL-based feature extraction with ML classifiers. YOLOv10 achieved strong performance in detecting and classifying four distinct morphological classes of intestinal organoids, showcasing its suitability for real-time and high-throughput applications.

Beyond detection accuracy, the proposed framework provides a scalable and reproducible strategy for automated organoid image analysis, with potential applications in drug screening, disease modeling, and stem-cell research. Although large-scale performance depends on hardware and data-handling optimization, inference was achieved within milliseconds per image, supporting the feasibility of high-throughput use.

In conclusion, the combination of YOLOv10 segmentation and ML-based classification offers a robust, automated, and scalable solution for organoid morphological analysis, paving the way for more precise, reproducible, and high-throughput studies in the field of organoid biology.