Enhanced U-Net-Based Deep Learning Model for Automated Segmentation of Organoid Images

Abstract

Organoids have emerged as powerful in vitro models for studying human development, disease mechanisms, and drug responses. A critical aspect of organoid characterisation is monitoring changes in size and morphology during culture; however, extracting these metrics from high-throughput imaging datasets is time-consuming and often inconsistent. Automated deep-learning approaches can overcome this bottleneck by providing accurate and reproducible image analysis. Here, we present an enhanced U-net-based segmentation model that incorporates region-of-interest refinement to improve the delineation of organoid boundaries. The method was validated on bright-field organoid images and demonstrated robust performance, achieving an accuracy of 98.15%, a dice similarity coefficient of 97.19%, and a Jaccard index of 94.53%. Compared with conventional segmentation methods, our model provides superior boundary detection and morphological quantification. These results highlight the potential of this approach as a reliable tool for high-throughput organoid analysis, supporting applications in disease modelling, drug screening, and personalised medicine.

1. Introduction

Organoids, three-dimensional structures derived from stem cells, have emerged as powerful models for studying human development [1], disease mechanisms [2], and drug responses [3,4]. These miniature, organ-like systems recapitulate key aspects of human physiology and pathology [5], providing new opportunities to investigate complex biological processes in vitro. Despite their potential, the field faces significant challenges in reproducibility, scalability, and phenotypic characterisation [6,7]. High variability in organoid growth and morphology often leads to inconsistent results, making it difficult to derive reliable biological insights [8].

Artificial intelligence (AI) approaches, particularly deep learning, are increasingly being applied to overcome such limitations. Advanced segmentation models can precisely delineate organoid boundaries, enabling accurate quantification of size, shape, and structural features. Standardising image analysis through AI not only reduces variability but also improves reproducibility across laboratories [9]. When combined with high-throughput imaging, deep learning provides a scalable solution for analysing large organoid datasets, accelerating the identification of disease signatures and therapeutic targets.

Quantitative image analysis has long been central to medical research, including applications such as pathological grading and cancer diagnostics [10,11,12]. At the core of these efforts lies image segmentation, a process that remains technically challenging due to complex tissue structures and variable image quality. Traditional techniques, such as histogram-based thresholding, contour detection, and watershed algorithms [13], frequently struggle with noisy backgrounds and complex morphologies. While supervised learning approaches introduced in the 1990s improved performance [14], deep learning, particularly convolutional neural networks (CNNs) [15], has since transformed the field.

Semantic segmentation using CNNs has evolved rapidly, moving from patch-based approaches to architectures that exploit full-image context. U-net [16,17] and V-net [17] have become cornerstones in biomedical image segmentation, leveraging encoder–decoder structures to capture both local and global features. Enhancements such as ResNet feature extractors [18], multi-scale input layers [19], and scalable frameworks like Mask R-CNN [20] have further advanced segmentation accuracy and applicability across domains.

Compared with classical methods, CNN-based approaches consistently achieve superior performance and are now widely adopted in medical and biological imaging tasks, including histological analysis and vascular segmentation [19]. Building on these advances, we present an enhanced U-net-based algorithm for organoid segmentation. Our method incorporates a region-of-interest (ROI) refinement strategy to address challenges such as blurred boundaries and irregular morphologies, improving segmentation accuracy and consistency. We further demonstrate that this fully automated pipeline can reduce the workload associated with organoid image analysis and provide unbiased quantitative metrics.

2. Materials and Methods

2.1. Human Pluripotent Stem Cell Culture

Human pluripotent stem cell (hPSC) lines, including the GENEA22 and H9 human embryonic stem cell line as well as three human-induced pluripotent stem cell lines (1 control and 3 epilepsy lines) [21,22], were used for brain organoid generation. Cells were maintained on Geltrex™ LDEV-Free Reduced Growth Factor Basement Membrane Matrix (Thermo Fisher Scientific, Waltham, MA, USA, Cat. #A1413202) at a 1:100 dilution and cultured in mTeSR™ Plus medium (Stem Cell Technologies, Vancouver, BC, Canada, Cat. #100-0276), with daily medium changes. Colonies were passaged at 70–80% confluency using 0.5 mM EDTA (Thermo Fisher Scientific, MA, USA, Cat. #15575020) followed by Accutase Enzyme Cell Detachment Medium (Thermo Fisher Scientific, MA, USA, Cat. #00-4555-56). Cells were incubated under normoxic conditions (37 °C, 5% CO₂, humidified atmosphere).

2.2. Brain Organoid Generation

Cortical organoids were generated from hPSCs using a previously described protocol with slight modifications [23]. The hPSC lines [6,7] were included as sources for organoid derivation. Stem cells were expanded as colonies on a thin Geltrex layer and maintained in mTeSR Plus medium. Colonies were passaged every 4–7 days using 0.5 mM EDTA and singularised with Accutase at 65–75% confluency.

For embryoid body formation, 2.46 × 10⁶ hPSCs were seeded per well in an AggreWell800 plate (Stem Cell Technologies, Vancouver, BC, Canada, Cat. #34815) with 3 mL of mTeSR medium containing 10 µM of ROCK inhibitor (Sapphire Bioscience, Redfern, Australia, Cat. #10005583) for 24 h (Day 1). Embryoid bodies were collected, washed with DMEM:F12 medium (Thermo Fisher Scientific, MA, USA, Cat. #11320082), and transferred to Stage 1 medium [DMEM:F12 supplemented with 2% B27 without vitamin A (Thermo Fisher Scientific, MA, USA, Cat. #12587001), 1% N2 (Thermo Fisher Scientific, MA, USA, Cat. #17502048), 1% non-essential amino acids (Thermo Fisher Scientific, MA, USA, Cat. #11140050), 1% Penicillin–Streptomycin (Thermo Fisher Scientific, MA, USA, Cat. #15070063), and 55 µM of β-mercaptoethanol (Thermo Fisher Scientific, MA, USA, Cat. #21985023)], containing 2.5 µM of dorsomorphin (Sigma Aldrich, Burlington, MA, USA, Cat. #P5499) and 10 µM of SB-431542 (Miltenyi Biotec, Bergisch Gladbach, Germany, Cat. #130-106-543). One well of AggreWell800-derived embryoid bodies was plated into two 10 cm dishes and maintained until Day 6 with daily medium changes.

On Day 6, embryoid bodies were transferred to ultra-low-attachment 96-well plates to prevent fusion and cultured in Stage 2 medium [Neurobasal-A (Thermo Fisher Scientific, MA, USA, Cat. #10888022) with 2% B27 without vitamin A, 1% GlutaMAX (Thermo Fisher Scientific, MA, USA, Cat. #35050061), 1% Penicillin–Streptomycin], supplemented with 20 ng/mL of EGF (Stem Cell Technologies, BC, Canada, Cat. #78006.1) and 20 ng/mL of FGF-2 (Stem Cell Technologies, BC, Canada, Cat. #78003). Medium was changed daily from Day 6 to Day 15 and every other day until Day 22. From Day 22 onward, organoids were cultured in Stage 2 medium supplemented with 20 ng/mL of BDNF (Stem Cell Technologies, BC, Canada, Cat. #78005.1), 20 ng/mL of NT-3 (Stem Cell Technologies, BC, Canada, Cat. #78074.1), 200 µM of L-ascorbic acid (Sigma Aldrich, MA, USA, Cat. #A4544), 50 µM of cAMP (Stem Cell Technologies, BC, Canada, Cat. #73886), and 10 µM of DHA (Stem Cell Technologies, BC, Canada, Cat. #D2534), with feeding every second day.

At Day 46, organoids were transferred to ultra-low-attachment 24-well plates and maintained in Stage 4 medium [Neurobasal-A with 2% B27 Plus (Thermo Fisher Scientific, MA, USA, Cat. #A3582801), 1% GlutaMAX, 1% Penicillin–Streptomycin], refreshed every 4–5 days.

2.3. Data Acquisition

Individual organoids were labelled by well position and imaged longitudinally from Day 6 onwards. Bright-field images of control and epilepsy-derived organoids were acquired using an Olympus CKX53 inverted microscope with a 4× objective, 10× ocular, and a 5-megapixel Olympus SC50 colour camera. Images were captured with Olympus cellSens software (v2.3) and saved in TIFF format. The proposed system consists of two main steps: the first step is the determination of the boundaries of the cell, and the second step is ROI segmentation from the organoid images using the proposed segmentation method. The organoid images vary from one captured image to another. This variation includes the shape and size of the cell. In this study, the steps of the proposed method are as shown in Figure 1.

2.4. Deep-Learning Model Architecture

A convolutional neural network (CNN) segmentation model was implemented, comprising multiple convolutional layers, ReLU activation [24,25], pooling layers, and a pixel classification layer (Figure 2). Convolutional kernels were applied with fixed stride values to balance feature extraction and computational efficiency. Max-pooling layers reduced dimensionality, while a Softmax activation function assigned probabilities to each class. The pixel classification layer performed semantic segmentation by assigning categorical labels to each pixel, excluding undefined labels during training for improved accuracy.

The model was trained using the crossentropyex loss function to minimise the difference between predicted and true pixel values. Hyperparameters, including learning rate, batch size, and kernel size, were tuned to optimise convergence. Data augmentation strategies (rotation, flipping, scaling) were applied to improve robustness. Model performance was assessed on 128 × 128 pixel test images representing diverse organoid morphologies, with segmentation accuracy measured against ground-truth masks [26,27,28].

2.5. Image Preprocessing

To prepare a high-quality dataset, bright-field organoid images (2560 × 1920 pixels) were processed using ImageJ 1.54p macros (Figure 3). Files were organised into input/output folders and split into red, green, and blue channels, with the green channel selected for further analysis. Median filtering was applied to smooth edges, followed by FeatureJ-based edge detection. Images were thresholded, converted to binary, and processed with “Fill Holes” and “Erode” functions to remove background noise. Particle analysis was performed to extract area, circularity, and diameter measurements, and objects were labelled. Binary and original images were then cropped to generate paired ground-truth masks and training inputs.

2.6. Training of CNN Segmentation Model

The CNN was configured with four convolutional layers to reduce model complexity and avoid overfitting. A stride of 2 was applied uniformly across layers, equivalent to pooling stride, reducing computational burden while preserving feature resolution. The network was trained using the Adam optimiser with an initial learning rate of 0.001 for 300 epochs and categorical cross-entropy loss. Training and validation curves of accuracy and loss were used to determine optimal parameters (Figure 3).

3. Results and Discussion

The proposed segmentation framework was designed in two main steps: (i) identification of organoid boundaries and (ii) ROI segmentation. This strategy was necessary to accommodate the variability in organoid morphology across captured bright-field images. The workflow of the method is summarised in Figure 1. Training and testing were performed on an Intel i7-6700HQ CPU (2.6 GHz) with 8 GB RAM and a NVIDIA GTX 950 GPU, running MATLAB 2019b. Input images were resized to 240 × 240 pixels for training (Figure 2), which reduced computational complexity and accelerated processing while preserving structural information. Preprocessing steps are illustrated in Figure 3. Before training, the dataset of 1000 bright-field organoid images were divided into the following: 70% for training; 10% for validation, and 20% for testing, which is a final evaluation after training.

3.1. Evaluation Metrics

To assess the performance of the segmentation model, three widely used quantitative metrics were employed:

• Accuracy: the ratio of correctly classified pixels to the total number of pixels,

  $$Accuracy={\displaystyle \frac{TP+TN}{TP+FP+TN+FN}}$$
• Dice Similarity Coefficient: a measure of spatial overlap between prediction and ground truth,

  $$Dice={\displaystyle \frac{2TP}{2TP+FP+FN}}$$
• Jaccard Index: the proportion of correctly predicted pixels relative to the union of prediction and ground truth,

  $$Jaccard={\displaystyle \frac{TP}{TP+FP+FN}}$$

where TP, TN, FP, and FN represent true positive, true negative, false positive, and false negative pixels, respectively.

3.2. Model Testing

The trained model was evaluated on a test set of 128 × 128 pixel organoid images (Figure 4). Representative segmentation results are shown in Figure 5. Qualitative inspection demonstrated that the proposed method provided superior delineation of organoid structures compared with baseline approaches, including Hölder mean-based segmentation, K-means clustering, and standard semantic deep-learning segmentation (Figure 6). For computational efficiency, the proposed segmentation model achieved an average inference time of 0.0486 ± 0.0045 s per image (128 × 128 resolution) on an NVIDIA GTX 950 GPU, corresponding to a throughput of approximately 20 images per second. This highlights the scalability and potential applicability of the model in high-throughput organoid imaging workflows.

3.3. Comparative Analysis

Figure 6 illustrates side-by-side comparisons of the original test images, ground-truth masks, and segmentation outputs generated by the proposed model. The enhanced U-Net-based algorithm demonstrated robust boundary detection even in challenging cases, such as organoids with irregular edges and variable morphologies. Quantitative results further confirmed these observations where the proposed method achieved a mean accuracy of 98.15%, a Dice similarity coefficient of 0.9719, and a Jaccard index of 0.9453 (

Table 1. Performance comparison of segmentation methods compared with the proposed system.

Methods	Accuracy	Dice Similarity Coefficient	Jaccard Index
Hölder mean-based image [23]	0.9198 ± 0.001	0.9456 ± 0.001	0.8967 ± 0.002
K-means segmentation method [24]	0.776 ± 0.002	0.8739 ± 0.002	0.7761 ± 0.0011
Deep-learning semantic segmentation [25]	0.9198 ± 0.002	0.4776 ± 0.003	0.4137 ± 0.002
Proposal Model	0.9815 ± 0.007 *	0.9718 ± 0.007 *	0.94537 ± 0.012 *

Data represents Means ± Standard Deviations. * p < 0.05 via one-way ANOVA.

), outperforming the comparative methods. Although a small proportion of organoid regions with highly complex textures were less clearly distinguished, overall segmentation quality remained consistently high across diverse morphologies.

These results demonstrate that integrating ROI refinement into the U-net architecture markedly improves segmentation performance for organoid imaging. Accurate delineation of boundaries is particularly important for downstream analyses, such as growth kinetics, morphological classification, and drug-response studies. By providing a fully automated and reproducible workflow, this approach significantly reduces the manual workload typically associated with organoid image analysis and minimises observer bias.

Compared with traditional algorithms (e.g., K-means [29], contour-based methods [30,31]), our model achieved substantially higher Dice and Jaccard scores, highlighting its suitability for complex biological datasets. Furthermore, when comparing within the proposed method, at the 95% confidence level, the means of the three metric groups were not statistically different (p = 0.6164), indicating consistent and reliable performance across all metrics. Importantly, this method can be readily scaled for high-throughput applications, enabling the systematic analysis of large organoid collections.

In recent years, several advanced deep-learning frameworks have been developed for organoid image segmentation. For instance, TransUNet have improved long-range feature integration but require extensive computational resources and are less suited for rapid analysis of large bright-field datasets [32]. In contrast, the enhanced U-Net presented in this study was intentionally designed for accessibility and efficiency while maintaining high segmentation accuracy enough for use on standard laboratory GPUs. Other organoid-specific U-Net models such OrgaExtractor, RDAU-Net, and OrganoID addresses different practical needs. For example, OrgaExtractor emphasises multi-scale shape extraction [33], RDAU-Net focuses on feature-attention refinement for brain tumour segmentation [34], and OrganoID is optimised for time-lapse tracking [35], while the proposed model in this study instead integrates a ROI refinement step that minimises background noise and enhances boundary delineation in static bright-field images, thus improving throughput for high-content screening.

The current model was trained and validated on images containing spatially separated organoids. Since the segmentation of overlapping organoids remains a recognised challenge in the field, future work will focus on extending the present approach to address bright-field images with overlapping organoids to further improve its applicability in high-density culture conditions.

4. Conclusions

In this work, we developed and validated an enhanced U-net-based deep-learning model for the segmentation of organoid images. Our results highlight the robustness of the proposed approach in handling organoids with diverse shapes and irregular boundaries. This model can be employed to analyse organoid-based disease modelling and high-throughput drug screening, advancing personalised medicine. We acknowledge that the current study was limited to bright-field images and a relatively constrained dataset. Future work will focus on extending the model to multimodal imaging platforms, including fluorescence microscopy, live-cell tracking, and, potentially, medical imaging formats such as MRIs and X-rays, which will further strengthen scalability and clinical relevance.