Colonic Polyp Detection with Object Detection Models

Abstract

In recent years, deep learning has been applied more and more to medical image analysis. One such application of deep learning is the automated polyp detection in colonoscopy with the target of reducing miss rates. This study presents a comprehensive evaluation of nine state-of-the-art object detection models for colonic polyp detection: YOLOv8, YOLOv9, YOLOv10, YOLO11, YOLO12, YOLO26, RT-DETR, YOLO-World, and YOLOE. The models were evaluated on three publicly available datasets: CVC-ClinicDB, CVC-ColonDB, and ETIS-LaribPolypDB. All models were trained under standardized conditions using identical hyperparameters and data augmentation strategies to guarantee fair comparison. Performance was evaluated using multiple metrics: mAP@50, mAP@50–95, F1 score, precision, recall, inference time, and computational cost. YOLO11 demonstrated the best overall performance, achieving mAP@50 scores of 0.995, 0.944, and 0.978 on the three datasets respectively, while maintaining the fastest inference time of approximately 150 ms per image and the third-lowest computational cost at 21.3 GFLOPs. Cross-dataset generalization experiments revealed a significant loss of performance, with mAP@50 dropping by 20–40% when models were tested on an unseen dataset, highlighting the challenge of true generalization with limited datasets. Statistical analysis by polyp size showed that while all models achieved F1 scores exceeding 0.95 for large polyps, performance decreased to 0.60–0.85 for small polyps, indicating a limitation in detecting small lesions. The analysis of failure modes showed that missed detections, false positives and boundary errors constitute 60–75% of all failures, suggesting that domain adaptation of object detection models may be required.

1. Introduction

1.1. Colorectal Cancer

Gastrointestinal (GI) tract disorders have been linked to morbidity and healthcare expenses [1,2] leading to approximately 1 million deaths per year in Europe posing a significant public health concern. GI tract disorders, including colorectal cancer (CRC), are most common among the young and elderly, and it has been estimated that the number of people affected will only rise as the world population ages [1,2,3].

CRC is a common and dangerous type of cancer that affects the large intestine and/or rectum, often arising from polyps [3], and it remains a leading cause of mortality across both Eastern and Western countries. CRC ranks as the second leading cause of cancer-related mortality in the United States [4,5] and the most common type of GI cancer in Europe [1,2] with an estimated one in 20 people contracting it during their lifetime due to the Western diet and lifestyle and the increased life expectancy. Epidemiological data indicate a rise in CRC incidence among adults under 50 years, with prevalence already substantial and continuing to increase [6]. Over the past decade, colorectal cancer has seen the most rapid increase in incidence, with rates of new cases and fatalities doubling and currently growing at a yearly average of 4–5% [6].

Most CRCs develop progressively from colorectal polyps as shown by research [6]—especially adenomatous polyps—which means that prompt removal of these polyps can substantially reduce CRC incidence and lower mortality by as much as 70% [7]. This preventive approach is essential in reducing the growing incidence of CRC.

Survival from colon cancer is stage-dependent: early-stage disease is associated with survival rates exceeding 95%, whereas survival falls to below 35% for advanced disease in stages IV–V [8]. Five-year survival estimates vary by stage and by whether the tumor arises in the colon or rectum, with reported averages in the range of 48.6 to 59.4% [9].

Statistical projections suggested that by 2020, approximately 150,000 people would be diagnosed with CRC and that more than 50,000 would die of the disease [4]. While early detection can raise 5-year survival to approximately 70%, outcomes for late-stage remains as poor as 10% [1,2]. Hence, the importance of detecting it in its early stages by using screening techniques, such as colonoscopy [10], is clear. These numbers highlight the need for screening for early detection; moreover, removal of the CRC precursor polyps remains central to prevention [5]. However, conventional colonoscopy remains susceptible to human errors [5] as poorly conducted colonoscopies can compromise CRC prevention [7]. In particular, a lower adenoma detection rate during colonoscopy has been linked to higher rates of post-colonoscopy CRC and increased mortality [7]. Consequently, a clear clinical priority should be the improvement and standardization of colonoscopy quality to reduce the number of missed adenomas and polyps [7]. However, the accuracy of polyp detection remains another point of concern [7]. Polyps present a diverse array of sizes, shapes, colors, etc., leading to high false positive rates for traditional detection methods, which creates issues in the diagnosis process [11].

Despite colonoscopy still being considered the gold standard for CRC screening, the miss rate of polyps still remains an issue [12,13]. Over recent decades, a trend has emerged toward developing intelligent systems for medical use [12,13] but evaluation remains inconsistent [12]. Many intelligent approaches have also been proposed to aid cancer detection in other domains, such as breast cancer detection [14] or prostate cancer detection [15]. The shortage of publicly available annotated datasets has hindered objective comparison and assessment of whether algorithms reach clinically acceptable performance. An attempt has been made at tackling this issue [12] by the Automatic Polyp Detection Sub-Challenge within the Endoscopic Vision Challenge at the MICCAI conference in 2015. Eight different teams took part in the MICCAI 2015 Sub-Challenge on Automatic Polyp Detection in Colonoscopy [12], conducted as part of the Endoscopic Vision Challenge (http://endovis.grand-challenge.org (accessed on 5 March 2026)). A validation study comparing the performance of different polyp detection methods was conducted covering the different methodologies (ranging from methodologies based on hand-crafted methods to CNNs) proposed by participating teams [12]. The results indicate that machine learning approaches (specifically CNNs) have a superior performance [12] with some being close to clinically applicable 12]. A deeper analysis showed that the combination of a subset of the methods can outperform individual approaches, raising the question of a potential ensemble system [12].

Lately, artificial intelligence, including deep learning, has been applied more and more in medicine [7]. Within gastrointestinal endoscopy, computer-aided diagnosis for colonoscopy is the most intensively studied application, although many systems remain at a preclinical stage [7]. Polyp detection technologies based on AI are in the early stages of development. Compared with traditional statistical models or rule-based expert systems [5], deep learning approaches generally deliver significant improvements in detection performance and accuracy for most image-based target-detection tasks [5].

1.2. Neural Networks in Medical Applications

The potential application of deep learning algorithms for the analysis of biomedical images has been increasingly recognized by the research community, largely due to their ability to efficiently process complex and high-dimensional data [16,17] due to the recent advancements in both hardware and deep learning approaches [16]. In particular, segmentation and object detection models have demonstrated potential [18,19,20] in general imaging tasks [21,22,23,24] for biomedical analysis with a three-fold increase between 2018 and 2022 as the third most popular task in medical images [25]. In recent years, neural networks have been progressively integrated into medical applications [25,26,27]. This trend is further supported by the gradual replacement of traditional image recognition techniques with deep learning-based approaches [28,29,30].

Studies have shown the potential of object detection and segmentation models in biomedical contexts. For instance, YOLO and U-Net [19] models have shown promising results in nucleus detection/segmentation with fluorescence microscopy. Automatic tissue segmentation has been tackled with a combination of U-Net models [31]. Biomarkers have been identified in OCT scans through the segmentation of retinal layers and fluids using U-Net and U-Net++ [26].

Three categories can be delimited when considering object detection models: point-based, two-stage and single-stage detection models [32]. Among the large number of architectures proposed in the literature, the “You Only Look Once” (YOLO) [32] family of models is arguably the most widely recognized. Over time, the YOLO framework has evolved through numerous variants that either integrated new mechanisms designed either to improve overall performance or to address domain-specific tasks. The widespread adoption of YOLO models can be attributed to their single-stage detection paradigm, which provides a balance between detection speed and accuracy. This characteristic is particularly important for applications requiring real-time analysis, such as many tasks within the biomedical domain.

The applicability of YOLO architectures has been demonstrated across a wide range of medical imaging tasks. For example, YOLOv4 has been used for nucleus detection in breast histopathological images [20]. YOLOv5 has shown strong performance in the analysis of coronary OCT images [33]. In the automated detection of blood vessels, a modified YOLOv7 architecture has been shown to outperform several traditional approaches. Although some limitations remain, YOLOv8 and YOLOv9 have shown improvements in feature detection for diabetic retinopathy using retinal fundus images [34]. Furthermore, YOLO family of models have been highlighted the as the most frequently utilized architectures across a wide variety of medical imaging applications [16], including brain tumor detection, glaucoma diagnosis, pulmonary embolism detection, skin lesion detection, and kidney disease localization, among many others [16]. YOLO architectures have demonstrated broad applicability across various endoscopic imaging tasks. YOLOv5 object detection errors were investigated in endoscopic imaging of esophageal squamous cell carcinoma and dysplasia through systematic precision–recall analysis, providing a framework for understanding failure modes [35]. YOLO models combined with hyperspectral imaging were shown to be able to optimize esophageal cancer diagnosis, suggesting that domain-specific data modalities can enhance detection performance beyond standard RGB colonoscopy [36]. Spectrum-aided visual enhancement techniques have been evaluated with YOLO frameworks for esophageal cancer detection, indicating that preprocessing strategies targeting specific imaging artifacts may improve model robustness [37]. The YOLO family has also been recognized as one of the most performant approaches due to its robustness, high detection accuracy, and low computational requirements [16,38].

Regarding polyp detection for CRC prevention, several YOLO variations have been developed in the last decade. A YOLO-V3 model with integrated active learning was shown to reduce the rate of false positives in automatic polyp detection [39]. By integrating feature extraction and fusion modules in the YOLO-V3 architecture, it was shown to outperform other methods in the detection of small polyps [40]. A modified YOLO-V4 model with CSPnet network integrated into the architecture and integrated mish activation function, Diou loss function, and transformer block demonstrated higher performance than previous methods for real-time automatic polyp detection [41]. A real-time system for polyp detection was created based on the YOLO-V4 model employing multi-scale mesh, data augmentation and various activation functions [42]. A modified YOLO-V5 model with an integrated self-attention mechanism has shown increased performance in polyp detection [43]. A Scaled YOLO-V4 model has been shown to be the most suitable object detection algorithm for large-scale datasets such as SUN and PICCOLO in a large experimental study [44]. In another experimental study, including YOLO-V4, YOLO-V3, CenterNet, and EfficientDet, it has been shown that YOLO-V4 had the highest performance in automatic gastric polyp detection [45]. A modified YOLO-V4 incorporating a GAN architecture was shown to outperform U-Net in polyp detection on three publicly available datasets [46]. YOLO-based object detection has been integrated with the ABC algorithm for hyperparameter and activation function optimization in order to obtain lower computational cost [47]. The YOLO-V5 model obtained improved performance on the SUN and PICCOLO datasets [48] for real-time polyp detection. The YOLO-V8 model has been shown to outperform other state-of-the-art models when considering mean average precision on multiple publicly available datasets [5] for polyp detection. The PD-YOLO architecture, based on YOLO-V7 and incorporating the CBAM and SPD-Conv modules, has been proposed for colon polyp detection showing increased robustness and higher performance in detection on publicly available datasets [12]. A YOLO-V11 model combined with the Local Outlier Factor algorithm showed increased performance in polyp detection on five different publicly available datasets compared to previous YOLO-based methods [49]. A custom YOLO model with an integrated Wavelet-Enhanced Reverse Attention module has been shown to outperform the traditional YOLO-V11 model on three publicly available datasets and an aggregated dataset [38]. YOLO-SRPD, integrating SRGAN with ACmix, CBAM module and a YOLO-V5 backbone, has been proposed for polyp detection and demonstrated to outperform the traditional YOLO-V5 [50]. Several modified YOLO-V5 models with various integrated mechanisms (such as cross-stage networks, feature pyramid networks, squeeze-and-excitation) have been shown to also outperform the traditional YOLO-V5 in polyp detection [51]. A study including the YOLO-V8 to YOLO-V11 models has shown that the YOLO-V8 obtained the best performance on four publicly available colorectal datasets [3]. Various quantization levels have also been tested for polyp detection [52] using a YOLO-V4 model showing a high performance when considering the mean average precision obtained on the ETIS-LaribPolypDB and CVC-ClinicDB datasets. The plethora of YOLO variants that have been applied show a potential for real-time polyp detection during colonoscopy with clinical applicability.

In our work, we have selected a subset of the most recently developed object detection models, including the latest YOLO versions from YOLOv8-v12 and the recently released YOLO26 and alternative architectures such as RT-DETR, YOLO-World, and YOLOE were evaluated. The object detection models were evaluated on three publicly available colorectal datasets from the perspective of several performance metrics, including mAP@50, mAP@50–95, F1 score, precision and recall and computational cost.

Our contributions include: (1) a benchmarking of the latest object detection models including YOLOv8-v12, YOLO26, RT-DETR, YOLO-World, and YOLOE across three colorectal datasets, (2) an evaluation of performance considering multiple metrics including trade-offs between accuracy and computational efficiency in the clinical context of the data, (3) an aggregated statistical analysis to determine detection capability, distribution of failures and statistical ranking, and (4) insights for the deployment of automated analysis systems in clinical settings.

2. Materials and Methods

2.1. Datasets

Colonoscopy polyp datasets such as CVC-ClinicDB, CVC-ColonDB and ETIS-LaribPolypDB are widely used benchmarks for developing and evaluating object detection and segmentation methods. These datasets consist of manually annotated frames extracted from real colonoscopy videos and were assembled specifically to capture clinical variability in polyp appearance, imaging devices and acquisition conditions, making them suitable both for training deep networks and for testing cross-dataset generalization. Table 1 presents a short description of these datasets.

Table 1. Short description of the datasets used.

Dataset	Organs	Annotation Type	Image Count	Classes	Origin
CVC-ClinicDB [53]	Colon (colonoscopy still frames)	PNG mask	612	1 (polyp)	Computer Vision Center and Universitat Autònoma de Barcelona (Spain)
CVC-ColonDB [8]	Colon	PNG mask	380	1 (polyp)	Computer Vision Center and Universitat Autònoma de Barcelona (Spain)
ETIS-LaribPolypDB [54]	Colon	TIF mask	156	1 (polyp)	ETIS, ENSEA, University of Cergy-Pontoise, CNRS, Cergy (France)

CVC-ClinicDB [53] is a public colonoscopy polyp segmentation dataset released by the CVC (Computer Vision Center) group in collaboration with Hospital Clínic (Barcelona, Spain). It contains 612 frames (with a size of 348 × 288 pixels) extracted from colonoscopy videos (each paired with a pixel-wise polyp mask); it comprises 31 different polyps from 31 sequences from 23 patients. CVC-ClinicDB has been widely used as a training set in polyp detection/segmentation benchmarks and challenges (it was the training dataset in the MICCAI 2015 EndoVis polyp Sub-Challenge). CVC-ClinicDB contains no healthy mucosa images. Figure 1 shows an image from the CVC-ClinicDB dataset including the original image, segmentation mask and the processed bounding box.

Figure 1. An example from the CVC-ClinicDB dataset including the original image, segmentation mask and the processed red bounding box.

CVC-ColonDB [8] is another public colonoscopy dataset of still frames containing annotated polyps. It contains 300 polyp frames (with a size of 574 × 500 pixels) selected from colonoscopy videos (drawn from 13 polyp video sequences from 13 patients) to maximize viewpoint variability per polyp and is provided with ground truth polyp masks. It is commonly used as a training/benchmark dataset in polyp segmentation studies. Figure 2 shows an image from the CVC-ColonDB dataset including the original image, segmentation mask and the processed bounding box.

Figure 2. An example from the CVC-ColonDB dataset including the original image, segmentation mask and the processed red bounding box.

ETIS-LaribPolypDB [54] was used in the MICCAI 2015 Endoscopic Vision Challenge as the challenge test set. It was assembled by ETIS Lab and Lariboisière Hospital and contains 196 high-definition frames (each with a segmentation mask), collected from dozens of video sequences (comprises 44 different polyps from 34 sequences) and represents a diverse, challenging set of polyps from multiple patients, devices and larger image resolutions. Figure 3 shows an image from the ETIS-LaibPolypDB dataset including the original image, segmentation mask and the processed bounding box.

Figure 3. An example from the ETIS-LaribPolypDB dataset including the original image, segmentation mask and the processed red bounding box.

The original datasets provide pixel-wise segmentation masks for polyp regions. To enable object detection evaluation, these masks were converted to bounding boxes. For each image, connected component analysis was performed on the binary segmentation mask to identify distinct polyp regions. For each connected component, we computed the minimum axis-aligned bounding rectangle that fully encompasses all mask pixels by determining the minimum and maximum row and column coordinates of the foreground pixels. Each connected component was treated as a separate detection target with its own bounding box. For irregular or highly elongated polyp shapes, the axis-aligned bounding box naturally extends to cover the full extent of the mask, which may result in boxes containing substantial background tissue. This conversion process discards fine-grained shape information present in the segmentation masks and may introduce localization ambiguity (a common issue for the object detection task), particularly for polyps with complex morphologies. The converted bounding box annotations were used for all subsequent training and evaluation procedures.

2.2. Object Detection Models

Since its 2015 debut, the original YOLO (You Only Look Once) [32] changed real-time object detection by combining region proposal and classification into a single, end-to-end pipeline. YOLO partitions an image into a grid and, for each cell, predicts class probabilities together with bounding box coordinates. This design reduced inference time and enabled direct optimization of the full detection task. YOLO-v1 delivered real-time performance using a CNN backbone, but it struggled with small-object localization because each grid cell could predict at most two bounding boxes.

The YOLO family has diversified extensively since its conception, with successive releases pursuing better speed and/or accuracy and task-/domain-specific variants [55]. YOLOv2 introduced several important improvements to stabilize and accelerate training: higher input resolution, batch normalization, and anchor boxes adopted from Faster R-CNN [55,56,57]. YOLOv3 improved the small-object detection by using multi-scale predictions and a deeper backbone (Darknet-53) with feature pyramid networks [55,56,58]. YOLOv4 combined a set of architectural and training enhancements (CSPDarknet-53, PANet, Mish activation) and advanced data augmentations (Mosaic, CutMix) to increase both accuracy and inference speed [55,56,59].

The PyTorch-based YOLOv5 created by Ultralytics focused on usability and performance by enhancing the backbone, neck and head components and an optimizing anchor sizes through an auto-anchor learning mechanism during training [55,56,60]. YOLOv6 added an EfficientRep backbone and the RepOptimizer to boost efficiency [55,56,61], while YOLOv7 introduced the Extended Efficient Layer Aggregation Network (E-ELAN) and a trainable “bag-of-freebies” for improved learning dynamics [55,56,62].

YOLOv8 created by Ultralytics integrated a C2f module and transitioned to anchor-free detection to simplify the head and improve flexibility [55,56,60]. YOLOv9 integrated a Generalized Efficient Layer Aggregation Network (GELAN) and Programmable Gradient Information (PGI) to address information degradation prevalent in deep networks [55,56,63]. YOLOv10 enhances real-time throughput by eliminating non-maximum suppression (NMS) in favor of a dual assignment strategy and uses lightweight classification heads [55,56,64]. YOLOv11 improved feature extraction, spatial attention, and efficiency by integrating several modules including C3k2, SPPF, and C2PSA [55,56,60]. YOLOv12 introduced the attention mechanism while maintaining detection accuracy and speed; it included several modifications such as: integrating the Area Attention module, Residual Efficient Layer Aggregation Networks (R-ELAN), FlashAttention, MLP ratio adjustments, attention-convolution integration, Position Perceiver, and removal of positional encoding [55,56,60,65]. Comparative analyses indicate YOLOv12 outperforms several earlier YOLO variants (YOLOv6, YOLOv8, YOLOv10, YOLOv11) and RT-DETR variants in detection accuracy and speed [65]. Recent applications of YOLOv12 demonstrate its utility across varied domains, including UAV tracking [66], fruit detection [67,68], marine litter identification [69,70], and STD early detection [71].

Real-Time DEtection TRansformer (RT-DETR) is a notable non-YOLO object detection approach that removed the need for NMS through its end-to-end transformer design [68,72,73]. RT-DETR employs a hybrid encoder to separately handle intra-scale interactions and cross-scale feature fusion, and it uses a query selection mechanism to improve speed and accuracy. RT-DETRv2 further extends multi-scale detection capabilities through Selective Multi-Scale Sampling, dynamic data augmentation, and scale-adaptive hyperparameters [73].

Several branches extend the YOLO family toward new capabilities. YOLO-World, created from YOLOv8, integrated open-vocabulary detection via vision–language modeling, and uses a Re-parameterizable Vision–Language Path Aggregation Network (RepVL-PAN) and region–text contrastive objectives to enable increased zero-shot performance in diverse real-world settings [74]. YOLOE, built on YOLOv8 and YOLOv11, integrated prompt-based mechanisms to incorporate contextual cues for both detection and instance segmentation [75]. It supports text prompts (through the Re-parameterizable Region–Text Alignment), visual prompts (through the Semantic-Activated Visual Prompt Encoder), and prompt-free operation (through the Lazy Region-Prompt Contrast) to improve context-aware preprocessing and downstream performance.

The recently released yet unpublished YOLO26, created by Ultralytics [62], is an end-to-end object detector from Ultralytics that produces final detections directly by removing the non-max suppression step and the Distributed Focal Loss module. It introduces a hybrid MuSGD optimizer and the improved ProgLoss + STAL loss function offering training stability and small-object accuracy. The YOLO26 mAPs performance on the COCO dataset is reported to be at 640 px of ≈53.1 (YOLO26m), 55.0 (YOLO26L) and 57.5 (YOLO26x), and claims up to ~43% faster CPU inference versus earlier variants, making YOLO26 particularly suited for real-time resource-constrained detection tasks.

By design, YOLO variants are provided in multiple sizes: nano (n), small (s), medium (m), large (l), and extra-large (x). This allows users to choose an appropriate compromise between computational cost and detection accuracy (with larger variants having a higher detection accuracy but also computational cost). In our experiments, the small variant has been used.

2.3. Performance Evaluation Metrics

Relying on a single metric can be misleading because no single score fully captures all aspects of detector deficiencies [76,77]. To obtain a more comprehensive assessment, we therefore report multiple complementary metrics that can together evaluate detection accuracy, localization quality, and the balance between false positives and false negatives.

Precision quantifies the proportion of predicted positives that are correct and reflects a model’s tendency to avoid false alarms. It is computed as the ratio of true positives (TP) to the sum of true positives and false positives (FP):

$$Precision={\displaystyle \frac{TP}{TP+FP}}$$

where TP represents the number of correct detections and FP represents the number of incorrect detections when compared to the ground truth.

Recall quantifies the proportion of actual positives correctly identified and captures the model’s ability to minimize missed detections. It is computed as the ratio of true positives to the sum of true positives and false negatives (FN):

$$Recall={\displaystyle \frac{TP}{TP+FN}}$$

where TP represents the number of correct detections and FN represents the number of missed detections when compared to the ground truth.

F1 score provides a balanced summary of performance by taking the harmonic mean of precision and recall. It is particularly useful as it includes both false positives and false negatives in contrast to either precision or recall; thus, using it as an evaluation metric allows for both minimizing false alarms and missed detection:

$$F1={\displaystyle \frac{2\times Precision\times Recall}{\left(Precision+Recall\right)}}$$

Intersection over Union (IoU) evaluates localization accuracy by computing the overlap between the predicted and the ground truth bounding boxes:

$$IoU={\displaystyle \frac{Intersection area}{Union area}}$$

Mean average precision at IoU of 0.5 (mAP@50) is the cross-class mean of average precision (AP, the area under the precision–recall curve) computed using detections that meet the condition of having an IoU of at least 0.5 with the ground truth:

$$mAP@50={\displaystyle \frac{1}{\left|C\right|}}\sum _{c\in C}{AP}_c(IoU\ge 0.5)$$

where c is a class from the set of classes C.

Mean average precision across IoU from 0.5 to 0.95 (mAP@50–95) averages AP over multiple localization strictness levels, specifically at IoU thresholds from 0.5 to 0.95 with an increment of 0.05, thus penalizing detections that only loosely align with the ground truth:

$$mAP@50-95={\displaystyle \frac{1}{\left|C\right|}}\sum _{c\in C}\left({\displaystyle \frac{1}{\left|T\right|}}\sum _{t\in T}{AP}_c\left(IoU\ge t\right)\right)$$

where c is a class from the set of classes C, and t is a threshold from the set of thresholds T = {0.5, …, 0.95}.

3. Results

3.1. Performance Evaluation on Individual Datasets

All models were evaluated under the same conditions to ensure an unbiased comparison. The original datasets contained segmentation masks that have been converted to YOLO-type bounding boxes for evaluation. The original datasets were randomly split into 70% for training and 15%/15% for validation and testing, respectively. All reported results refer to the held-out non-augmented test partitions of the datasets.

Training was conducted for 50 epochs using a batch size of 16 on 640 × 640 pixel rescaled images. The AdamW algorithm was employed for optimization with an initial learning rate of 1 × 10⁻⁴ that was annealed down to 1 × 10⁻⁶ over the course of training; a weight decay of 5 × 10⁻⁴ provided L2-style regularization. To stabilize early training dynamics, the learning rate was warmed up for the first 10 epochs. A suite of data augmentations was applied to reduce overfitting and increase generalization: random HSV perturbations (hue ± 0.015, saturation ± 0.5, value ± 0.3), rotations up to ±10°, translations of up to 10% along each axis, random zooms up to ±20% and flips. Two images were blended into a single composite using mixup, and multiple images were combined to simulate more complex scenarios using mosaic augmentation; mosaic augmentation was turned off for the final 10 epochs to improve convergence stability. The standardized parametrization, including hyperparameter optimization and data augmentation, guarantees a fair comparison.

Our evaluation of the object detection models on the CVC-ClinicDB dataset shown in Table 2 indicates that YOLO11 achieved the highest detection accuracy with an mAP@50 of 0.995 demonstrating an almost perfect localization capability on this dataset. YOLO26 followed closely with an mAP@50 of 0.994, YOLO-World with 0.993 and YOLO12 with 0.992. The F1 scores obtained follow closely the conclusions from the mAP@50. On the stricter map@50–95, it was YOLO-World that actually managed to obtain the highest performance. Regarding the inference time, YOLOv11 excelled with the lowest inference time of ~150 ms. Although YOLO-World, YOLO12 and YOLO26 were extremely close in performance to YOLO11, they had a significantly higher inference time. The lowest scores for all metrics were obtained by the RT-DETR model. These results suggest that the enhanced feature extraction mechanism in YOLOv11 offers a better localization. Figure 4 shows a test image with the predictions obtained by each model showing that all models were capable of identifying the general location of the polyp with a low localization error; however, RT-DETR was unable to identify the entire extent of the polyp, YOLOE was unable to find a polyp at all, while YOLOv9’s and YOLO-World’s predictions included an overlapping false positive.

Table 2. Performance of object detection models on the CVC-ClinicDB dataset. Bold values represent the highest performance for each metric (column).

Model/Metric	mAP@50	mAP@50–95	Mean F1	Mean Precision	Mean Recall	Inference Time [ms]	Computational Cost [GFLOPs]
RT-DETR	0.889	0.656	0.866	0.848	0.885	706.5	103.4
YOLOv8	0.989	0.847	0.984	0.988	0.979	159.6	28.4
YOLOv9	0.992	0.830	0.979	0.969	0.990	176.0	26.7
YOLOv10	0.965	0.805	0.950	0.974	0.927	151.6	24.5
YOLO11	0.995	0.833	0.990	0.980	1.000	149.3	21.3
YOLO12	0.992	0.847	0.989	0.980	0.999	189.0	21.2
YOLOE	0.950	0.732	0.943	0.977	0.912	253.1	35.3
YOLOv8-Worldv2	0.993	0.866	0.984	0.979	0.990	194.4	32.6
YOLO26	0.994	0.814	0.989	0.990	0.988	174.6	20.5

Figure 4. Example from the CVC-ClinicDB dataset with the true annotations (left) and the predicted bounding boxes (right).

For the evaluation of the CVC-ColonDB dataset, presented in Table 3, RT-DETR managed to obtain comparable results to the YOLO models, but with considerably higher inference time and computational cost. The highest scores were obtained by the YOLOv9 model, with YOLO11, YOLO12, YOLO-World and YOLO26 having slightly lower performances (of about 5–10%) for all metrics. Nevertheless, YOLO11 remains the model with lowest inference time. Figure 5 shows a test image with the predictions obtained by each model showing that all models were capable of identifying the general location of the polyp with a low localization error.

Table 3. Performance of object detection models on the CVC-ColonDB dataset. Bold values represent the highest performance for each metric (column).

Model/Metric	mAP@50	mAP@50–95	Mean F1	Mean Precision	Mean Recall	Inference Time [ms]	Computational Cost [GFLOPs]
RT-DETR	0.970	0.712	0.907	0.897	0.917	671.8	103.4
YOLOv8	0.969	0.755	0.956	0.964	0.947	176.8	28.4
YOLOv9	0.991	0.733	0.991	1.000	0.982	197.9	26.7
YOLOv10	0.897	0.650	0.870	0.922	0.825	168.5	24.5
YOLO11	0.944	0.697	0.936	0.981	0.895	158.1	21.3
YOLO12	0.945	0.742	0.946	0.964	0.930	203.8	21.2
YOLOE	0.924	0.700	0.916	0.980	0.860	250.2	35.3
YOLOv8-Worldv2	0.951	0.684	0.935	0.979	0.895	218.5	32.6
YOLO26	0.938	0.620	0.877	0.891	0.863	180.2	20.5

Figure 5. Example from the CVC-ColonDB dataset with the true annotations (left) and the predicted bounding boxes (right).

For the most diverse dataset, the results obtained are rather inconclusive as the performances of models are metric dependent and no clear best can be found as shown in Table 4. Both YOLO11 and YOLO26 had a high performance for all metrics; yet, YOLO11 had a slightly lower inference time than YOLO26. RT-DETR failed to generalize on this more complex dataset. Figure 6 shows a test image with the predictions obtained by each model showing that all models are capable of generally identifying the location of the polyp; however, RT-DETR tends to assign multiple detections to the same location, while YOLOv9’s and YOLO12’s detections included false positives.

Table 4. Performance of object detection models on the ETIS-LaribPolypDB dataset. Bold values represent the highest performance for each metric (column).

Model/Metric	mAP@50	mAP@50–95	Mean F1	Mean Precision	Mean Recall	Inference Time [ms]	Computational Cost [GFLOPs]
RT-DETR	0.665	0.448	0.635	0.587	0.692	752.2	103.4
YOLOv8	0.927	0.608	0.892	1.000	0.805	178.3	28.4
YOLOv9	0.939	0.632	0.899	0.914	0.885	196.9	26.7
YOLOv10	0.907	0.632	0.880	0.917	0.846	178.8	24.5
YOLO11	0.978	0.696	0.938	0.960	0.918	171.1	21.3
YOLO12	0.981	0.620	0.923	0.959	0.890	193.4	21.2
YOLOE	0.973	0.669	0.961	0.961	0.962	241.2	35.3
YOLOv8-Worldv2	0.905	0.633	0.880	0.917	0.846	205.0	32.6
YOLO26	0.978	0.658	0.979	1.000	0.958	183.4	20.5

Figure 6. Example from the ETIS-LaribPolypDB dataset with the true annotations (left) and the predicted bounding boxes (right).

3.2. Statistics Across Datasets

An aggregated analysis was conducted to determine the ability of each model to accurately identify objects depending on their size. The results of each model were aggregated across datasets and split by the ground truth bounding box sizes. Four categories were delimited based on the distribution of ground truth bounding box sizes by image area percentage: small (<0.5%), medium (0.5–1%), large (1–5%) and very large (>5%). The predicted bounding boxes were matched to the ground truth ones with an IoU of 0.5 to compute the F1 score (false positives were counted as unmatched predictions, while false negatives as unmatched ground truth bounding boxes). Our aggregated analysis by object sizes allows us to identify if and which models struggle with certain object sizes.

Figure 7 clearly shows that polyp size can have an effect on performance. Small polyps (<0.5% of image area) posed the greatest challenge with YOLO9, YOLO12 and YOLOE dropping in performance, whilst the other models remained relatively stable across various sizes. RT-DETR had the lowest performance on the small polyps; however, its performance was lower than that of the other models on all sizes. Large and very large polyps remained within the YOLO models’ capabilities of detection, with F1 scores being generally over 0.9 (excepting YOLOE with 0.8 for very large polyps). The medium-sized polyps showed a slight drop in performance for all models except YOLO12 and YOLO26.

Figure 7. Detection accuracy (F1 score) of each model across all datasets split by object size.

An aggregated failure mode analysis was conducted to determine the most common detection error types for each model. Five failure mode categories were delimited: missed detections (unmatched ground truth bounding boxes above the IoU threshold), boundary errors (matched ground truth but below the threshold IoU), duplicate detections (multiple predictions matched with the ground truth bounding box), background false positives (unmatched predictions), and split detections (over-segmentation, when false positives are matched with the ground truth). The aggregated failure mode analysis includes all the test images of all datasets for each model.

Figure 8 shows that the RT-DETR obtained the largest number of errors, yet the number of boundary errors and missed detections are comparable to those of the other models. YOLO-World had the lowest number of errors of all models with no duplicate detections and a significantly lower number of false positive and boundary errors. YOLO11 had a comparable performance to that of the other YOLO models; however, it had a higher number of false positives.

Figure 8. Distribution of failure cases for each model across all datasets.

3.3. Cross-Dataset Generalization

To test the true generalization of the models within automatic polyp detection, we used the models trained on the CVC-ClinicDB and CVC-ColonDB datasets and tested them on the unseen ETIS-LaribPolypDB.

The performance of models trained on CVC-ClinicDB and tested on ETIS-LaribPolypDB are shown in Table 5 showing a considerable drop in performance for all models on all metrics. YOLO11’s mAP@50 dropped from 0.995 (CVC-Clinic test data) to 0.702 (ETIS-LaribPolypDB), a 29% relative decrease. The degradation pattern was consistent—F1 scores decreased by 30–40%, with precision suffering more than recall. This asymmetry suggests that models become overly specific to the characteristics of the training data, rejecting valid polyps in the unseen dataset. An example of this can be viewed in Figure 9 as most models are unable to identify the polyp. YOLO11 demonstrated relative resilience towards the data shift, maintaining the highest cross-dataset mAP@50 of 0.702 and an F1 of 0.667. This suggests that its architectural design, perhaps through better feature generalization or by being less prone to overfitting, transfers more effectively. YOLOv8-Worldv2, a variant designed for open-vocabulary detection, achieved slightly lower but comparable performance and the highest mAP@50–95 of 0.528 and recall of 0.613, indicating that its pre-training on diverse datasets (beyond medical images) makes it less anchored to domain-specific idiosyncrasies.

Table 5. Performance of object detection models trained on the CVC-ClinicDB dataset and tested on the unseen ETIS-LaribPolypDB dataset. Bold values represent the highest performance for each metric (column).

Model/Metric	mAP@50	mAP@50–95	Mean F1	Mean Precision	Mean Recall
RT-DETR	0.555	0.327	0.570	0.602	0.542
YOLOv8	0.669	0.491	0.614	0.730	0.530
YOLOv9	0.667	0.494	0.618	0.709	0.548
YOLOv10	0.626	0.437	0.588	0.653	0.536
YOLO11	0.702	0.511	0.667	0.789	0.577
YOLO12	0.631	0.449	0.577	0.626	0.536
YOLOE	0.564	0.384	0.492	0.696	0.381
YOLOv8-Worldv2	0.690	0.528	0.656	0.705	0.613
YOLO26	0.608	0.463	0.541	0.769	0.417

Figure 9. Predictions for each model trained on the CVC-ClinicDB dataset from the unseen ETIS-LaribPolypDB test dataset with the true annotations (left) and the predicted bounding boxes (right).

Training on CVC-ColonDB and testing on ETIS-LaribPolypDB yielded slightly worse generalization as shown in Table 6 likely due to CVC-ColonDB’s smaller size of only 380 images. F1 scores ranged from 0.38 to 0.52, barely above random detection adjusted for class imbalance and mAP@50 values ranged from 0.427 to 0.607, with most models under 0.60. The lower performance can be easily viewed in Figure 10 as well, where the models were unable to identify any polyp with the exception of RT-DETR that predicted multiple false positives. However, the same conclusions can be extracted; YOLO11 and YOLO-World are the models that managed to transfer the most knowledge and retain performance in this cross-dataset analysis.

Table 6. Performance of object detection models trained on the CVC-ColonDB dataset and tested on the unseen ETIS-LaribPolypDB dataset. Bold values represent the highest performance for each metric (column).

Model/Metric	mAP@50	mAP@50–95	Mean F1	Mean Precision	Mean Recall
RT-DETR	0.427	0.296	0.488	0.659	0.387
YOLOv8	0.548	0.348	0.490	0.688	0.381
YOLOv9	0.498	0.371	0.387	0.657	0.274
YOLOv10	0.590	0.446	0.460	0.852	0.315
YOLO11	0.581	0.429	0.459	0.841	0.315
YOLO12	0.516	0.358	0.463	0.606	0.375
YOLOE	0.509	0.350	0.462	0.652	0.357
YOLOv8-Worldv2	0.607	0.425	0.518	0.783	0.387
YOLO26	0.584	0.385	0.450	0.825	0.310

Figure 10. Predictions for each model trained on the CVC-ColonDB dataset from the unseen ETIS-LaribPolypDB test dataset with the true annotations (left) and the predicted bounding boxes (right).

4. Discussion and Conclusions

In this work, nine state-of-the-art object detection models were evaluated across three colonoscopy datasets. Through our analyses, we have identified that YOLO11 and YOLO-World are currently the best performing models for automated polyp detection.

YOLO11 and YOLO-World emerged as potentially the most appropriate models for further developments with mAP@50 scores of 0.995 on CVC-ClinicDB, 0.944 on CVC-ColonDB, and 0.978 on ETIS-LaribPolypDB, and with the fastest inference time of approximately 150 milliseconds. The results position it as the most promising candidate for real-time clinical integration. YOLO-World obtained mAP@50 scores of 0.993, 0.951, and 0.905 on CVC-ClinicDB, CVC-ColonDB and ETIS-LaribPolypDB respectively. The performance of YOLO-World was comparable to that of the YOLO11 model, excepting the ETIS-LaribPolypDB and the higher inference time of 15 to 50%.

The superior cross-dataset performance of YOLO11 and YOLO-World can be partially attributed to their distinct architectural characteristics. YOLO11 incorporates several refinements including the C3k2 module for enhanced feature extraction, the C2PSA module for improved spatial attention, and the SPPF module for multi-scale feature aggregation. These components may enhance the model’s ability to capture both local and global context, potentially leading to more robust feature representations that transfer better to dataset shifts. YOLO-World employs a different approach through prompt-based mechanisms and vision–language pre-training, with its Re-parameterizable Region–Text Alignment module enabling incorporation of textual context. Its training on diverse datasets beyond medical imaging may reduce overfitting to domain-specific characteristics. We hypothesize that YOLO11’s architectural balance between specificity and generalization, combined with YOLO-World’s exposure to broader visual scenarios during pre-training, contributes to their superior cross-dataset resilience.

On the CVC-ColonDB and ETIS-LaribPolypDB datasets, models’ precision–recall trade-off skewed towards precision, indicating a prioritization of minimizing false alarms over maximizing sensitivity. For screening colonoscopy, where the cost of a missed polyp (potentially cancer) may outweigh the burden of investigating a false positive, the development of algorithms specifically designed for polyp detection may warrant a reconsideration of this calibration.

The aggregated object size analysis showed that all models exhibited a loss of performance on small polyps. Given that small polyps may constitute a significant proportion of current miss rate, this limitation undermines the utility of the models. The aggregated failure mode analysis revealed that missed detections, false positives and boundary errors account for most of all failures across models. This finding suggests that domain adaptation may be required to reduce the errors of models before their integration into the clinical setting.

Addressing the limitation in small polyp detection is an important concern for clinical deployment, as small adenomas represent high-risk lesions whose miss rate must be minimized. Domain adaptation techniques may mitigate this limitation: (1) multi-scale training strategies that explicitly oversample or weight small objects during training to address class imbalance; (2) attention mechanisms specifically designed to enhance small-object feature representations, such as feature pyramid networks with additional high-resolution layers; (3) hybrid approaches combining object detection with segmentation to provide finer localization; (4) data augmentation strategies specifically targeting small polyps; (5) ensemble methods combining detectors optimized for different size ranges. Beyond architectural modifications, increasing dataset diversity to include more small polyp examples and employing focal loss or similar techniques to emphasize hard examples during training could also improve performance. Larger models may also benefit from the increased resolution at the cost of higher inference time.

Several factors may contribute to this brittleness of the object detection models in the cross-dataset analysis. Dataset shift encompasses differences in hardware (leading to possibly different image resolution, color balance), individual preparation quality, and possibly polyp characteristics (size distribution, morphology prevalence). The magnitude of performance collapse observed when training on CVC-ColonDB and testing on ETIS-LaribPolypDB reflects the combined effects of limited training set size (380 images from 13 sequences) and fundamental domain shift. The limited size of CVC-ColonDB (380 images from 13 sequences) may provide insufficient diversity for the models to learn generalizable representations, leading to a limited learning of only dataset-specific characteristics. However, dataset size alone does not fully explain the magnitude of performance collapse. Fundamental domain shifts between these datasets (as ETIS-LaribPolypDB was specifically designed to represent challenging cases) include: (1) differences in colonoscopy equipment and image acquisition protocols, leading to variations in color balance, contrast, and image quality; (2) variations in polyp morphology distributions; (3) differences in imaging resolution (CVC-ColonDB: 574 × 500 vs. ETIS-LaribPolypDB: variable high-resolution); (4) potential differences in patient preparation quality affecting appearance. The fact that models trained on the larger CVC-ClinicDB dataset (612 images) also exhibited substantial performance drops (though less severe) supports the hypothesis that domain shift, not just sample size, fundamentally limits generalization. This finding highlights the necessity of multi-center, multi-device training data for robust polyp detection systems.

These findings have immediate implications. First, dataset size and diversity dominate generalization more than architecture choice as all models struggled similarly despite architectural differences. Second, clinical validation protocols must include multi-site, multi-scanner testing to identify generalization failures before being released. Third, domain adaptation of object detection models may be required for clinical deployment.

Limitations

Several domain-specific adaptations of YOLO variants have been created for polyp detection; however, these employ older YOLO versions. Our objective was to benchmark the base architectures in their standardized form rather than domain-specific modifications. This approach allows for establishing baseline performance metrics that other researchers can reproduce, evaluating architectural improvements across YOLO generations under controlled conditions, and identifying which base architectures show the most promise for further domain-specific adaptation. The models we selected represent the current state-of-the-art of general-purpose object detection, and our findings provide insights into their transferability to the medical domain before custom modifications are applied.

Our evaluation is restricted to three public datasets (CVC-ClinicDB, CVC-ColonDB, and ETIS-LaribPolypDB), which may not capture the full diversity of clinical colonoscopy conditions, including variations in endoscopy equipment, patient populations, bowel preparation quality, and polyp morphology distributions encountered across real scenarios. The conversion of pixel-wise segmentation masks to bounding boxes may introduce localization artifacts, particularly for irregularly shaped polyps where the rectangular approximation includes substantial background tissue; nevertheless, this is a common issue for the object detection task. This conversion may contribute to the boundary errors observed in our failure mode analysis and potentially underestimates the true localization capabilities of the models. Our study evaluated only the small (s) variants of each architecture, leaving open questions about the scaling behavior and performance trade-offs of larger model variants (medium, large, extra-large), which may offer improved accuracy at the cost of increased computational requirements.

Our decision to employ uniform hyperparameters across all models was deliberate and stems from methodological considerations inherent to benchmarking studies. While different architectural families (particularly transformer-based models like RT-DETR versus CNN-based YOLO variants) may benefit from personalized training protocols (e.g., different learning rate schedules, augmentation strategies, or training durations), per-model optimization would confound architectural comparisons with optimization effort. Our standardized protocol represents a compromise enabling fair comparison of architectural capabilities under equivalent training resources. We acknowledge that architecture-specific optimization could potentially improve absolute performance for individual models, and future work exploring optimal training configurations for each architecture would provide valuable complementary insights.

The data splitting was performed randomly at the image level across all three datasets. We acknowledge that this approach carries a limitation: when consecutive frames from the same colonoscopy video sequence may be distributed across training and test sets, there is potential for data leakage due to high visual similarity between adjacent frames. This is particularly relevant for CVC-ClinicDB (comprising 31 sequences from 23 patients) and CVC-ColonDB (13 sequences from 13 patients), which are both derived from colonoscopy videos. While patient-level or sequence-level splitting would have provided stronger guarantees against information leakage, we note that our cross-dataset generalization experiments (Table 5 and Table 6) remain unaffected by this limitation, as ETIS-LaribPolypDB was held entirely separate during training and provides a robust test of true generalization under distribution shift.

The disproportionately high false positive rate observed for RT-DETR compared to YOLO variants, despite comparable missed detection and boundary error rates, likely reflects characteristics specific to transformer-based architectures. We hypothesize that several factors may contribute to this pattern. Transformer decoders employ learned object queries that directly predict object existence and location; if these queries are insufficiently constrained during training on small medical datasets, they may generate spurious detections in ambiguous regions. RT-DETR’s training may benefit from larger and more diverse datasets than the colonoscopy datasets we used (CVC-ClinicDB: 612 images), as transformers are known to require more data than CNNs. The end-to-end nature of RT-DETR eliminates non-maximum suppression, relying instead on the model to learn to suppress redundant detections—this mechanism may not be optimal on medical datasets of limited size. These factors suggest that transformer-based detectors like RT-DETR may require dataset-specific calibration or architectural modifications for optimal performance in medical imaging domains with limited training data.

The code that supports the findings of this work was written in Python 3.13 and is openly available at https://github.com/ArdeleanRichard/YOLO-Colonoscopy/ (accessed on 5 March 2026).