AI-Driven Analysis for Real-Time Detection of Unstained Microscopic Cell Culture Images

Abstract

Staining-based assays are widely used for cell analysis but are invasive, alter physiology, and prevent longitudinal monitoring. Label-free, morphology-based approaches could enable real-time, non-invasive drug testing, yet detection of subtle and dynamic changes has remained difficult. We developed a deep learning framework for stain-free monitoring of leukemia cell cultures using automated bright-field microscopy in a semi-automated culture system (AICE3, LABMaiTE, Augsburg, Germany). YOLOv8 models were trained on images from K562, HL-60, and Kasumi-1 cells, using an NVIDIA DGX A100 GPU for training and tested on GPU and CPU environments for real-time performance. Comparative benchmarking with RT-DETR and interpretability analyses using Eigen-CAM and radiomics (RedTell) was performed. YOLOv8 achieved high accuracy (mAP@0.5 > 98%, precision/sensitivity > 97%), with reproducibility confirmed on an independent dataset from a second laboratory and an AICE3 setup. The model distinguished between morphologically similar leukemia lines and reliably classified untreated versus differentiated K562 cells (hemin-induced erythroid and PMA-induced megakaryocytic; >95% accuracy). Incorporation of decitabine-treated cells demonstrated applicability to drug testing, revealing treatment-specific and intermediate phenotypes. Longitudinal monitoring captured culture- and time-dependent drift, enabling separation of temporal from drug-induced changes. Radiomics highlighted interpretable features such as size, elongation, and texture, but with lower accuracy than the deep learning approach. To our knowledge, this is the first demonstration that deep learning resolves subtle, drug-induced, and time-dependent morphological changes in unstained leukemia cells in real time. This approach provides a robust, accessible framework for label-free longitudinal drug testing and establishes a foundation for future autonomous, feedback-driven platforms in precision oncology. Ultimately, this approach may also contribute to more precise and adaptive clinical decision-making, advancing the field of personalized medicine.

1. Introduction

In the search for new cancer therapeutics, in vitro drug testing and drug screening in cell lines and on primary samples and their derivatives have proven to be indispensable tools that accelerate the drug discovery process. This approach allows researchers to evaluate the efficacy and safety of potential drug candidates before advancing to more complex and ethically challenging in vivo studies [1,2,3,4]. One of the primary advantages of in vitro drug testing is the ability to conduct experiments in a highly controlled environment. Cell lines provide a simplified yet representative model of cellular behavior. This enables precise investigation of the impact of drug candidates on phenotypic changes and specific molecular pathways. Furthermore, it facilitates high-throughput screening, allowing to assess many compounds efficiently and accelerating the drug discovery timeline by quickly identifying promising candidates for further investigation. As such, in vitro drug testing may have utmost relevance in personalized medicine allowing researchers to tailor drug-screening assays to individual genetic profiles, predicting how a patient might respond to a specific treatment and optimizing therapeutic strategies [5]. Automated and advanced imaging technologies further enhance the speed and accuracy of compound screening, enabling researchers to sift through vast libraries of potential drugs and combinations [6]. However, the ultimate utility of cellular models for drug testing depends on accurate real-time longitudinal monitoring and recognizing relevant molecular and/or phenotypic changes such as differentiation, induction of cell death and others [7]. The ability to monitor such experiments in real time, without staining or fixation, remains a critical unmet need.

Conventional staining methods such as immunofluorescence provide valuable molecular insights, but they have fundamental drawbacks. Stains and labels are often invasive, may alter cellular physiology, and prevent true longitudinal observations. Moreover, marker expression is not always specific or universally available, and labeling can reduce viability or even activate signaling pathways [8]. These limitations make unbiased, real-time monitoring of cells in their native state essentially impossible with staining-based techniques.

Computer-aided approaches based on unstained, morphology-based analysis represents an attractive alternative. Yet, previous approaches to label-free phenotyping have been restricted to relatively coarse distinctions, such as separating cell groups with overt morphological differences (e.g., lymphocytes, granulocytes, and erythrocytes), or to viability assessments in simple contexts [9,10,11]. Classical radiomics-based pipelines, while interpretable, rely on handcrafted features such as size, shape, texturing of the cytoplasm and nucleus [9,12] and therefore lack the sensitivity to capture the subtle and dynamic phenotypes required for modern drug testing. Crucially, no method has yet demonstrated the ability to track fine-grained, time- and drug-dependent morphological changes in morphologically similar cells under fully stain-free conditions.

Beyond their methodological importance, such approaches are also relevant from a cellular pathology perspective. Reliable, morphology-based recognition could support not only preclinical drug testing but also key clinical tasks such as diagnosis, treatment monitoring, and prognosis (e.g., remission or relapse assessment, risk stratification, and detection of metastatic spread). Continuous, non-invasive observation has potential for disease surveillance and could provide insight into clonal evolution, a major contributor to relapse and therapy resistance. Equally important, minimizing false negatives is critical to avoid missing rare but clinically relevant subclones. These expectations underscore the broader significance of developing robust, real-time, label-free monitoring systems.

In this work, we address this gap by introducing a deep learning–based system capable of real-time analysis of unstained suspension cell cultures. Our model distinguishes between closely related leukemia cell lines, detects differentiation states induced by specific agents, and separates drug-induced morphological effects from gradual culture-time drift. This establishes a non-invasive framework for longitudinal drug testing, enabling continuous monitoring of living cells in their native state without staining, fixation, or disruption. By closing this methodological gap, our approach lays the foundation for autonomous, feedback-driven drug testing platforms in precision oncology.

2. Material and Methods

2.1. Cell Culture and In Vitro Treatment

The myeloid cell lines K562, HL-60, and Kasumi-1 (German Collection of Microorganisms and Cell Cultures GmbH, DSMZ, Braunschweig, Germany) were cultured in RPMI1640 media (Gibco, Waltham, MA, USA) containing 5% Penicillin/Streptomycin (Gibco) and 10% (K562) or 20% (HL-60, Kasumi-1) heat-inactivated fetal calf serum (Sigma-Aldrich, St. Louis, MO, USA). Cells were maintained at 0.2 × 10⁶ /cells per ml in T75 flasks and split every 3 to 4 days.

K562 cells were treated with 50 µM hemin (Sigma-Aldrich) for six days and with 5 nM phorbol myristate acetate (PMA, Sigma-Aldrich) for 48 h for induction of erythroid differentiation and megakaryocytic differentiation, respectively. Decitabine (DAC, Sigma-Aldrich) treatment was conducted with three 20 or 100 nM pulses in 24 h intervals and cells were harvested after six days, as previously described by Stomper et al. [13]. Cell numbers were counted by 0,4% trypan blue staining and viability was measured using the NucleoCounter system (ChemoMetec A/S, Allerød, Denmark).

2.2. Hemoglobin (HBA) ELISA

Hemoglobin a-subunit (HBA) levels in cell culture supernatants from leukemic cell lines were quantified using the Human Hemoglobin Alpha ELISA Kit MBS167177 (MyBioSource, Inc., San Diego, CA, USA) according to the manufacturer’s instructions. Supernatants and recombinant standards were assayed in duplicate, and measurements were performed in three independent biological replicates. Absorbance was read at 450 nm, and concentrations were calculated from the standard curve with adjustment for dilution factors.

2.3. Automated Cell Culture and Image Generation

For image generation cells were transferred to a 50 mL falcon (reactor) with a minimum concentration of 0.2 × 10⁶ cells/mL in 20 mL media. All preparations of the reactor were carried out under a laminar flow. The reactor containing the cells was assembled for automated cell culture and image generation in a custom-developed semi-automated AICE3 device (LABMaiTE, Freiburg, Germany) inside of an incubator as described in Figure 1A. The device comprises a peristaltic pump system and a microscope with a 20× objective with an integrated camera and is kept in an incubator at 37 °C and a 5% CO₂ atmosphere to ensure the correct conditions for cell culture during imaging. Cell suspension mixing was performed automatically by the integrated peristaltic pump of the AICE3 system, operating at a flow rate of 35 mL/min to ensure homogeneous distribution of cells and medium. At each step of image generation, five ml cell suspension were pumped from the reactor through a 0.2 µm Luer slide (ibidi GmbH, Gräfelfing, Germany) attached to the microscopic device. Cells were allowed to sediment on the slide for three minutes to ensure optimal focus before image generation. Images were acquired at a native resolution of 1920 × 1080 pixels with 24-bit RGB color depth (8 bits per channel). Using the integrated bright-field camera of the AICE3 system. At each step, four different images were generated and cell counts were documented as the mean of the detected cells over all four images. With this setup, image generation was performed for 1.5 h, resulting in 30 images per experimental run. Images without cells were used for background training. For longer experiments, half of the reactor was drained and refilled with fresh media about every 2.5 days and time between images was adapted to every 2 h. Cell imaging was performed independently at two laboratories using separate AICE3 setups, which introduced variation in illumination, focus, and culture conditions. Distinct cell batches from different passages (2–19) were included.

2.4. Image Processing

Roboflow [14] was used for annotation, splitting, and augmentation (flips, rotations). The annotation process was performed manually for each cell type and segmentation masks were created using Roboflow’s smart polygon tool. Annotation of training images was performed exclusively on mono-culture datasets, avoiding any mixed-cell populations. For each image, only two categories were distinguished: cell or non-cell (the latter including bubbles, debris, or other background structures, Supplementary Figure S1). An object was annotated as a cell if it displayed an intact cell body with clearly defined borders; objects without these features were labeled as non-cell. Annotation was conducted independently by three researchers experienced in cell culture. Although inter-rater reliability was not systematically quantified, the binary nature of the task and use of mono-cultures ensured a high degree of concordance across annotators.

2.5. Model Training and Validation

The data was split into training, validation, and test subsets (ratio 70:20:10) to facilitate effective model training, hyperparameter tuning, and performance evaluation and to ensure reliable and robust machine learning applications.

The models were trained on an NVIDIA DGX A100 using Ultralytics YOLOv8 environment [15]. YOLOv8 is a single-shot state of the art object detection algorithm with a high accuracy rate on COCO [16] and Roboflow. Different sized YOLOv8-models (n, s, m, l) and the transformer-based Realtime Detection Transformer (RT-DETR) were evaluated. YOLOv8 was chosen for all subsequent analyses. The overall detection framework, including integration of the semi-automated culture system, imaging pipeline, YOLOv8-based analysis, and interpretability components, is illustrated in Figure 1A. A more detailed depiction of the training and evaluation workflow is provided in Figure 1B. The code snippets used for training, validation, and prediction are listed in Supplementary Figure S2, as are the confidence thresholds used (objects with values below this threshold were discarded as false positives) and the remaining hyperparameters (image size, training epochs, confidence threshold) (Supplementary Table S1). To investigate the interpretability of the model, we applied both salience-based visualization and radiomic-based approaches.

Eigen-CAM was used to highlight the image regions contributing most strongly to predictions of the YOLOv8 model [17]. We used a public Eigen-CAM library for YOLOv8 (https://github.com/rigvedrs/YOLO-V12-CAM, accessed on 20 June 2024) on randomly picked cell crops. Gradients of the final convolutional layers were computed, and class-activation maps were generated to visualize the areas of highest model attention in each detection. These maps were overlaid on the original microscopy images to allow qualitative assessment of whether the model’s focus corresponded to biologically meaningful regions.

To complement this qualitative approach, we employed RedTell, a radiomics-based feature extraction tool for single-cell imaging data [12]. From each segmented cell, a set of morphological and texture descriptors was calculated, including size, elongation, solidity, perimeter ratios, and gray-level texture metrics such as variance and run-length emphasis. These features provided quantitative descriptors that could be related to known biological changes such as cell enlargement during differentiation, alterations in nuclear condensation, or irregular outlines associated with apoptosis.

In addition, we applied Orthogonal Matching Pursuit (OMP) for sparse feature selection within the radiomics feature space. OMP identifies the most discriminative subset of features that contribute to class separation, thereby enhancing interpretability and reducing redundancy. This allowed us to focus on a compact set of features with direct biological relevance.

3. Results

To explore whether deep learning (DL) can detect morphological differences in unstained cells, we used suspension cell lines cultured with the semi-automated AICE3 device. An overview of all baseline and application datasets is provided in

Table 1. Overview of the baseline datasets including name, description, abbreviation, and information per dataset including the number of images (# Images), average of annotations per image (Avg. ann./image), and the number of instances of corresponding classes (# classes).

Dataset	Description	# Images Avg. Ann./Image	# Classes
Single Class Cell Detection	Object detection based on K562 cells	436 15.5	K562: 6764
Multiclass Cell Detection	Testing for discrimination between myeloid cell lines (K562, HL-60 and Kasumi-1)	399 23.8	K562: 2682 HL-60: 3319 Kasumi-1: 3498
Cell culture time	Comparing fresh and 6 days-old K562	156 25.1	K562: 1924 K562d6: 1989
Differentiation Model	Using known substances (Hemin and PMA) to differentiate K562 cells	216 14.1	Hemin: 1112 K562: 1298 PMA: 631
Decitabine (DAC)	Applying 20 nM and 100 nM Decitabine on K562 cells	62 57.1	DAC 20 nM: 2005 DAC 100 nM: 1537
Decitabine + Differentiation Model	Decitabine-treated K562 added to the Differentiation Model independent of concentration	236 17.9	DAC: 1194 Hemin: 1112 K562: 1298 PMA: 631

and

Table 2. Overview of the validation datasets including abbreviation, name, description, and information about number of images.

Abbr.	Dataset	Description	Number of Images
LM	LABMaiTE K562 dataset	Other passage of K562 cells on a similar AICE system in Freiburg	275 images
Mixture	Mixture dataset	Mixing cells with different substance treatment in a 50:50 ratio	128 images
GC	Growth curve dataset	Two runs of K562 over nearly eight days in AICE system. Images were taken every two hours	716 images

.

3.1. Single-Class Cell Line Detection

We first tested the human chronic myeloid leukemia cell line K562 (Figure 2A). To build a robust dataset, we included not only in-focus cells but also images with out-of-focus cells, debris, bubbles, and background-only samples as negative controls (22 images), as exemplified in Supplementary Figure S1. The final dataset contained 436 images with an average of 15.5 annotations per image (range 0–54), yielding a total of 6764 annotations (Table 1).

All YOLOv8 detection models (s, n, m, l) achieved similar metrics, but larger models required longer training. To balance speed and accuracy, we selected YOLOv8-s for further analysis. In the test set, the model correctly identified 570 true-positive K562 cells (95.6%). Misclassifications were rare: 15 false positives (mainly bubbles or debris) and 11 false negatives (out-of-focus cells) (Figure 2B). Precision and recall were both above 97% (Figure 2C). Standard detection metrics, including mean average precision (mAP@[0.5:0.95]) and per-class average precision (AP), are summarized in

Table 3. Performance metrics across different datasets used for model development and evaluation. Summary of precision (P), sensitivity (Sens), specificity (Spec), and mean average precision at IoU thresholds of 0.5 (mAP@0.5) and 0.5:0.95 (mAP@0.5:0.95) for the detection and classification tasks across various datasets. Precision, sensitivity, and specificity are provided overall and per class.

Dataset	P	Sens	Spec	mAP@0.5	mAP@0.5:0.95
Cell Detection	97.4%	98.1%	-	98.9%	76.9%
Leukemia cell lines   HL-60   K562   Kasumi-1	94.6% 92.3% 95.5% 96.0%	97.7% 99.2% 96.2% 97.6%	97.3% 97.0% 98.2% 98.1%	98.3%	74.3%
Aging   K562   K562d6	95.6% 94.0% 97.2%	99.2% 100% 98.3%	95.8% 94.6% 96.9%	98.8%	74.9%
Differentiation Model   Hemin   K562   PMA	94.6% 96.9% 93.3% 93.5%	96.2% 98.4% 95.2% 95.1%	97.1% 98.0% 94.8% 98.5%	97.8%	77.7%
Decitabine   DAC20   DAC100	71.5% 77.9% 66.1%	78.4% 80.5% 76.2%	71.8% 69.4% 74.2%	82.6%	64.5%
Decitabine added to differentiation model   DAC   Hemin   K562   PMA	92.9% 94.3% 91.7% 89.2% 96.5%	94.5% 95.1% 96.8% 95.9% 90.2%	97.1% 97.8% 96.6% 94.5% 99.5%	97.3%	77.4%

.

To assess robustness and reproducibility, we validated the model in a second laboratory using an independent AICE3 device in Freiburg. K562 cells were cultured and imaged on this system, generating a dataset of 75 images with varying cell densities (Supplementary Figure S3). This dataset inherently included variation in illumination, focus, confluence, and culture conditions, as well as distinct cell batches (passages 2–19). Despite this technical and biological variability, the model reliably detected unstained cells. Precision was slightly reduced due to occasional misinterpretation of background structures; however, all cells were detected, and no false negatives occurred. These results demonstrate that the YOLOv8-based approach is stable, reliable, and accurate across independent laboratories and imaging setups.

To test real-time applicability, we compared latency and throughput on GPU and CPU environments. On the GPU, prediction required 5.4 ms, with preprocessing and postprocessing adding 0.9 ms, and an average of 33.3 ms for 75 images (>370 cells). CPU-based deployments were substantially slower: on a Kubernetes/Kubeflow environment preprocessing, prediction, and postprocessing took 14.1 ms, 2468.9 ms, and 38.4 ms, while on a local CPU they took 16.3 ms, 1301.9 ms, and 1.5 ms. These results show that GPU-based inference is preferable for real-time detection, although CPU-based execution remains feasible in laboratory settings without dedicated GPUs.

Finally, we compared YOLOv8 with the transformer-based RT-DETR architecture. RT-DETR achieved competitive accuracy but was slower and less consistent. YOLOv8 provided a superior balance of accuracy and real-time performance, which was critical for our setup, and was therefore chosen for all subsequent analyses.

3.2. Detection of Longitudinal Morphological Changes

Prolonged culture is expected to induce subtle morphological changes, independent of drug treatment. To separate such effects from substance-specific changes in a longitudinal drug-testing context, we analyzed the impact of culture time on K562 cells over six days. Cells were maintained under standard conditions without splitting or media change and imaged after six days, referred to as K562d6. The dataset comprised 156 images with 3913 annotations, averaging 25.1 cells per image (range 0–79). Five images contained only background. A subset of the baseline K562 dataset was used as reference. Using this setup, the YOLOv8-n model achieved high precision and sensitivity (>95%) and ~99% specificity in distinguishing fresh K562 cells from K562d6. The high mean average precision (mAP) scores further confirmed accurate cell detection (Table 3).

To test performance in a dynamic longitudinal experiment, K562 cells were continuously cultured for over seven days (190 h) in the AICE3 device. Images were acquired every two hours, generating 380 images in total. Across this time span, the proportion of cells classified as K562d6 gradually increased, reflecting the natural morphological drift associated with prolonged culture (Figure 2D). These results show that the AI model can detect subtle, time-dependent morphological changes in unstained cells that are not obvious to the human eye, allowing correction for culture-associated effects in longitudinal analyses. The validity of this detection is supported by the fact that the virtual mixing ratios of ‘young’ and ‘old’ cells closely mirrored the expected gradual culture-associated drift over time.

3.3. Multiclass Leukemia Cell Line Discrimination

We next extended the model to a multi-class setup by adding two morphologically similar acute myeloid leukemia cell lines, HL-60 and Kasumi-1, alongside K562. To maintain a balanced dataset, only a subset of K562 images was used, considering both image numbers and cell density. In total, the dataset comprised 399 images with an average of 23.8 annotations per image (range 0–139), resulting in 2682 annotations for K562, 3319 for HL-60, and 3498 for Kasumi-1. Among the tested architectures, YOLOv8-m performed best. High mAP values (mAP@0.5 = 98.3%; mAP@[0.5:0.95] = 74.3%) indicated strong overlap between ground-truth annotations and the model’s bounding box predictions. The model achieved a very high rate of true positives across all classes, with misclassifications occurring only rarely and mainly involving background (e.g., out-of-focus cells or bubbles). Only four K562 cells were incorrectly classified as HL-60. Average sensitivity and specificity exceeded 97%, and precision was 94.6% (Figure 3A,B). Detailed metrics are shown in Table 3. These results demonstrate that the trained model can reliably distinguish three morphologically similar leukemia cell lines based solely on unstained microscopic images.

3.4. Applying the Cell Line Recognition Model to a Cellular Differentiation Model

We next applied the recognition model to a previously described K562-based differentiation assay [13]. K562 cells were treated with hemin to induce erythroid differentiation and with PMA to induce megakaryocytic differentiation. In this model, not all cells fully differentiate; rather, they display gradual changes in morphology. As illustrated in Figure 4B, untreated, hemin-treated, and PMA-treated K562 cells exhibit clear morphological differences. Hemin treatment increased cell size and cytoplasmic opacity, while PMA treatment induced irregular morphology and greater granularity, in contrast to the baseline morphology of untreated K562 cells. The Differentiation Model dataset contained 216 images with 3041 annotations, averaging 14.1 annotations per image. Misclassifications were rare: three untreated K562 cells were incorrectly labeled as PMA-treated, and 15 background structures (debris or bubbles) were initially annotated as cells. Despite these few errors, YOLOv8-s achieved high performance, with precision, sensitivity, and specificity all around 95% (Table 3). These results show that the trained model successfully captured treatment-specific morphological differences and reliably distinguished untreated K562 cells from those treated with hemin or PMA.

3.5. Applicability of the Cell Differentiation Model to a Drug Testing Setup

We next tested the hypomethylating agent decitabine (DAC) (Figure 4A). In patients with acute myeloid leukemia (AML), DAC induces fetal hemoglobin expression, which serves as a dynamic biomarker of outcome. In vitro, DAC preferentially promotes erythroid over megakaryocytic differentiation [13]. We therefore introduced DAC-treated K562 cells as a new class in the training model.

We first assessed whether object detection could distinguish between treatment concentrations. The dataset included 2005 annotations from cells treated with 20 nM DAC and 1537 annotations from cells treated with 100 nM DAC. None of the models reliably separated these two concentrations (Table 3). Consequently, both concentrations were merged into a single “DAC” class, and 20 additional images with 1194 annotations were added.

The resulting dataset included four classes: untreated K562, hemin-treated, PMA-treated, and DAC-treated cells. In this setup, YOLOv8-m achieved high precision, sensitivity, specificity, and strong mAP scores (Table 3). To validate performance, we tested predefined mixing ratios of the four classes (Figure 4C,D). DAC cells were only detected in experiments where they were included (DAC–hemin/DAC–K562). In 50:50 DAC–hemin mixtures, hemin cells were identified with high precision but slightly underestimated (−10%), while DAC cells were substantially underestimated (−38%) and often misclassified as untreated K562. A similar pattern was observed in DAC–PMA mixtures, where DAC cells were frequently misclassified as untreated K562. In DAC–K562 mixtures, DAC cells were underestimated by ~25%, whereas untreated K562 cells were almost perfectly classified (98%). To provide orthogonal support for these differentiation assays, we measured hemoglobin induction by ELISA (Supplementary Figure S4). Hemin treatment caused a robust increase in hemoglobin concentration, consistent with erythroid differentiation. In contrast, PMA and DAC (20 nM or 100 nM, alone or in combination) produced little or no induction. These findings are consistent with previous reports [13] confirming robust hemoglobin induction in the hemin model. Although the ELISA does not provide single-cell resolution or proportional quantification of differentiation, it supports the differentiation capacity detected by the trained model.

3.6. Morphological Insights

To add interpretability to the model’s performance, we first applied Eigen-CAM to generate heatmaps highlighting image regions that contributed to classification of the three leukemia cell lines (Figure 5). As shown in Figure 5A, Eigen-CAM occasionally highlighted background regions, making its results not always conclusive.

To complement this, we performed classical radiomic feature extraction using RedTell, comparing conventional shape and texture descriptors with the deep learning model [12]. A total of 74 features were extracted per cell (examples in Figure 5B). To identify the most informative features, we applied Orthogonal Matching Pursuit (OMP), which iteratively selects the best feature combinations. This approach was tested on the three cell line datasets, resulting in a binary classification for each line. The selected variables are summarized in Supplementary Table S2.

For HL-60 cells, the model relied on intensity energy, RMS intensity, solidity, elongation, gray-level run-length matrix (GLRLM) variance, and perimeter–surface ratio, yielding 71% accuracy. For K562 cells, the most relevant features were perimeter and three gray-level emphasis measures, resulting in 82% accuracy. For Kasumi-1, eight features (Long-Run Low Gray-Level Emphasis/LRLGLE, minimum intensity, gray-level variance, diameter, Short-Run Low Gray-Level Emphasis/SRLGLE, maximum intensity, Low Gray-Level Run Emphasis/LGLRE, lmc12) were identified, achieving 78% accuracy. Precision–recall curves for all classes are shown in Figure 6. Overall, the extracted radiomic features provided some explanatory insight into morphological differences between leukemia cell lines. However, none of the explainable models reached the high accuracy of the deep learning approach.

4. Discussion

In vitro drug testing has become an important tool in cancer research. It enables high-throughput screening under controlled conditions and supports personalized medicine by linking therapeutic responses to individual molecular and genetic profiles. Conventional staining-based assays, however, are invasive, may alter physiology, and prevent sequential monitoring of dynamics. Label-free, morphology-based analysis offers a promising alternative, allowing real-time, non-invasive observation. Advances in DL and automated microscopy now make unbiased high-throughput phenotyping feasible, but detecting subtle drug-induced changes remains challenging.

Previous studies have shown that unstained cells can be detected [18,19,20], yet these were limited to cell groups with distinct morphologies. Here, we investigated whether DL models can discriminate unstained leukemia cells with highly similar morphology. AML cell lines provide a suitable test case: they grow in suspension, enabling automated culture in closed systems, and are increasingly used in drug-testing approaches [21].

Our models reliably detected unstained cells and distinguished them from non-cellular components. They also classified highly similar leukemia cell lines with high accuracy. While real-time detection usually requires GPU-based execution, we showed that even CPU-based systems can deliver rapid predictions, broadening applicability to routine laboratories without specialized infrastructure. YOLOv8 proved particularly effective due to its accuracy, efficiency, and real-time capability, while RT-DETR showed comparable accuracy but slower, less stable inference. At the time of study initiation (early 2023), YOLOv8 represented the state of the art. Since then, further refinements (e.g., TSD-YOLO, YOLOv12) and transformer-based segmentation models (e.g., Segment–Anything) have been introduced [22,23,24,25,26,27,28,29]. Although systematic benchmarking against these newer models was beyond our scope, they represent promising directions for future work.

In summary, these results lay the foundation for complex experiments with a straightforward, accessible, and fast readout, enabling measurement of subtle changes (e.g., specific responses to certain substances) in a drug screening setup in near real-time.

By eliminating pre-processing steps like staining and fixation, cytotoxic effects are avoided. This permits to keep the cells in a closed system, fostering a feedback loop that enhances experimental efficiency and allowing longitudinal experiments [30]. However, cells in cultivation carry out physiological processes naturally, such as metabolizing and consuming media over time. Additionally, cells undergo nutrient depletion and waste accumulation [31]. These effects can lead to subtle but significant morphological changes which may play a critical role in longitudinal drug testing experiments [32]. In longitudinal experiments, our AI model detected culture- and time-associated drift, demonstrating sensitivity to gradual changes not evident to the human eye. This ability is critical in drug testing, as it allows separation of temporal effects from drug-specific responses. Applied to differentiation assays, the model accurately classified erythroid (hemin) and megakaryocytic (PMA) induction, as well as decitabine treatment, with high precision. Importantly, it appeared sensitive to intermediate phenotypes, suggesting that the model captures differentiation as a continuum rather than as binary states—an important consideration for tracking dynamic cellular responses.

However, one of the key challenges in AI applications for biomedical imaging is the understanding how models make their decisions. While Eigen-CAM provided initial insights into the model’s focus, we observed that this technique is prone to background interference, with non-cellular regions occasionally highlighted. Such artifacts illustrate a general limitation of saliency-map approaches in microscopy, where illumination or background can confound interpretability. To address this, we complemented CAM analyses with RedTell-based radiomics, which yield biologically interpretable descriptors. Several of the most discriminative radiomics features correspond closely to known morphological characteristics of leukemia cells. For instance, measures of size and elongation reflect differentiation-associated shifts in cell geometry, while texture features (e.g., gray-level variance, run-length emphasis) align with cytoplasmic granularity and nuclear condensation. Solidity and perimeter ratios capture irregular cell outlines, often linked to stress or apoptosis. By connecting these descriptors to established morphological traits, we provide a logical bridge between the model’s data-driven outputs and the underlying biology.

Importantly, our YOLO-based model extends beyond these explainability approaches. While Eigen-CAM highlights image regions and RedTell captures predefined morphological descriptors, both remain limited in sensitivity and accuracy. In contrast, our deep learning framework not only achieves superior performance but also resolves subtle, drug-induced, and time-dependent morphological changes that were previously inaccessible with stain-free or radiomics-based methods. To our knowledge, this is the first demonstration of a deep learning system enabling such real-time, non-invasive monitoring of morphologically similar leukemia cell lines in a drug testing context.

Together, our results demonstrate the feasibility of robust, label-free, real-time monitoring of leukemia cells and highlights the potential of AI-driven morphology analysis to complement conventional assays. Furthermore, it lays groundwork for fully automated, feedback-driven drug testing platforms. Ultimately, such approaches could speed drug discovery and enable rapid, individualized treatment adaptation in precision oncology.

Beyond methodological advances, our results highlight potential applications aligned with standard expectations in cellular pathology. Accurate morphology-based detection can support diagnosis, treatment monitoring, and prognosis, for instance by stratifying patients based on drug sensitivity or remission dynamics. Continuous label-free monitoring could also contribute to disease surveillance, enabling early detection of relapse or treatment resistance. Importantly, by capturing subtle morphological adaptations, AI-driven models may provide insights into clonal evolution, which often underlies therapy failure. At the same time, reducing false negatives is essential in a clinical context, as undetected abnormal cells could delay treatment adaptation or miss emerging subclones.

Several limitations of this study should be acknowledged. A methodological limitation of our study is that it relied primarily on bounding box annotations, which, while effective for detection, do not capture detailed cell morphology. Segmentation-based approaches—now more accessible through tools such as Roboflow—could provide richer morphological information but will require additional computational resources and validation. In addition, our evaluation was restricted to two object detection architectures, YOLOv8 and RT-DETR, selected for their strong real-time performance at the time of the study. Although these delivered robust results, broader benchmarking against alternative or more recent models may further optimize performance. Third, the absence of a definitive ground truth remains a key limitation. Although image recognition provides relative quantification and pattern detection, we could not fully validate differentiation states (e.g., erythroid versus megakaryocytic) at the single-cell level, since confirmatory assays such as flow cytometry or immunophenotyping of the exact same cells were not included. Instead, we relied on supportive evidence such as cytospin preparations and hemoglobin ELISA, which confirmed differentiation trends but lack proportional single-cell resolution. Fourth, the biological scope of our work was intentionally narrow. Experiments were restricted to suspension leukemia cell lines and selected substances, reflecting a proof-of-concept design. While this demonstrates feasibility in a controlled setting, the transferability to other biological systems—such as adherent cells, primary patient samples, or co-culture environments—remains uncertain. Extending validation to these contexts will be essential for broader applicability in drug testing. Fifth, although imaging was performed in two independent laboratories on distinct AICE3 devices, ensuring some variability in illumination, focus, and cell batches, we did not systematically enforce leakage-safe splits by plate, day, or batch, nor did we provide per-batch confidence intervals. These factors could influence generalizability and should be addressed in follow-up studies. Finally, we did not perform systematic ablation experiments or extended benchmarking on diverse hardware platforms. While our GPU versus CPU comparison reflects practically relevant scenarios in standard laboratory environments, future studies should extend latency and throughput analyses to edge devices (e.g., laptops, Jetson boards, Coral TPUs) and current-generation GPUs such as NVIDIA Blackwell, which are optimized for real-time AI workloads.

5. Conclusions

Our study demonstrates that deep learning can resolve subtle, drug-induced, and time-dependent morphological changes in unstained leukemia cell cultures in real time. By avoiding invasive staining and fixation, this approach enables robust, label-free longitudinal drug testing in a closed system and lays the groundwork for automated, feedback-driven drug screening platforms. Beyond drug testing, the capabilities demonstrated here highlight broader potential applications of morphology-based AI models, including diagnosis, prognosis, and disease surveillance, where minimizing false negatives and capturing clonal evolution are of critical clinical importance.

Looking ahead, future work will expand applicability across additional biological systems, cell types, and technical contexts, including segmentation-based analyses for richer morphological detail and integration with complementary assays. A particularly promising direction is the incorporation of AI-driven models into feedback-loop systems for personalized medicine, where real-time monitoring of treatment responses could enable highly adaptive and individualized therapeutic strategies. Ultimately, such frameworks may advance precision oncology by providing a reliable and scalable platform for clinical decision-making.