Real-Time Rice Milling Morphology Detection Using Hybrid Framework of YOLOv8 Instance Segmentation and Oriented Bounding Boxes

Abstract

Computer vision and image processing techniques have had great success in the food and drink industry. These technologies are used to analyse images, convert images to greyscale, and extract high-dimensional numerical data from the images; however, when it comes to real-time grain and rice milling processes, this technology has several limitations compared to other applications. Currently, milled rice image samples are collected and separated to avoid one contacting the another during analysis. This approach is not suitable for real-time industrial implementation. However, real-time analysis can be accomplished by utilising artificial intelligence (AI) and machine learning (ML) approaches instead of traditional quality assessment methods, such as manual inspection, which are labour-intensive, time-consuming, and prone to human error. To address these challenges, this paper presents a novel approach for real-time rice morphology analysis during milling by integrating You Only Look Once version 8 (YOLOv8) instance segmentation and Oriented Bounding Box (OBB) detection models. While instance segmentation excels in detecting and classifying both touching and overlapping grains, it underperforms in precise size estimation. Conversely, the object-oriented bounding box detection model provides more accurate size measurements but struggles with touching and overlapping grains. Experiments demonstrate that the hybrid system resolves key limitations of standalone models: instance segmentation alone achieves high detection accuracy (92% mAP@0.5) but struggles with size errors (0.35 mm MAE), while OBB alone reduces the size error to 0.12 mm MAE but falters with complex grain arrangements (88% mAP@0.5). By combining these approaches, our unified pipeline achieves superior performance, improving detection precision (99.5% mAP@0.5), segmentation quality (86% mask IoU), and size estimation (0.10 mm MAE). This represents a 71% reduction in size error compared to segmentation-only models and a 6% boost in detection accuracy over OBB-only methods. This study highlights the potential of advanced deep learning techniques in enhancing the automation and optimisation of quality control in rice milling processes.

1. Introduction

Rice is a vital staple food consumed by more than half of the global population, with annual production surpassing 520 million metric tons and an estimated value of USD 550 billion [1,2]. It accounts for 20% of global dietary energy and up to 15% of protein intake in many developing nations [3]. However, the quality assurance of milled rice remains a significant industrial challenge, particularly in accurately identifying and quantifying broken grains during post-harvest processing. Traditional methods can exhibit error rates of 15–20% [4] and suffer from insufficient throughput, despite advances in computer vision achieving up to 97% classification accuracy under controlled conditions [5,6].

Deep learning techniques, such as YOLOv8 instance segmentation and oriented bounding boxes (OBB), have shown promise in improving speed and precision. Instance segmentation models can process images in as little as 38 ms/frame [7], and vision transformers have enhanced small defect detection by 15%. YOLOv8-OBB, in particular, has significantly reduced size measurement errors for non-axis-aligned grains to ±0.12 mm [8]. Yet, challenges persist. For example, overlapping grains and angled orientations complicate accurate real-time detection and measurement [9]. While fruit sorting and grain analysis have seen improvements using hybrid segmentation–detection models [10], these methods have not been successfully applied to the specific needs of high-speed, high-precision rice milling environments. The existing literature presents a fragmented landscape. YOLOv8 Instance Segmentation achieves strong recall and segmentation of clustered grains [11] but suffers from poor measurement accuracy on angled grains due to axis-aligned bounding box limitations [12]. Conversely, YOLOv8-OBB provides superior measurement precision [9] but has lower recall in dense clusters [13]. Some hybrid methods in related agricultural contexts have shown accuracy gains, but their utility in real-time rice milling, which demands concurrent high accuracy, high speed (15 FPS), and millimetre-level precision, remains unexplored. This technological gap is critical given that approximately 42% of quality control errors in milling originate from the incorrect sizing of non-axial grains.

Beyond object detection frameworks, recent advances in segmentation architectures have demonstrated enhanced accuracy and efficiency in domain-specific tasks. For example, RGGC UNet integrates residual ghost blocks with ghost coordinate attention to achieve superior feature extraction in pathological image segmentation tasks, particularly signet ring cell detection [14]. Similarly, RTLinearFormer introduces a lightweight dual-resolution transformer with linear attention mechanisms to reduce computational complexity while maintaining competitive segmentation accuracy [15]. Although developed for medical imaging, their architectural principles of efficient attention and robust multi-scale feature capture provide valuable inspiration for improving agricultural image analysis systems such as those for rice grain morphology detection.

This study addresses the urgent need for a robust, real-time, and scalable solution by introducing a hybrid deep learning model that integrates YOLOv8 instance segmentation and OBB detection. This integration aims to overcome the limitations of standalone models by achieving high recall for overlapping grains and high measurement precision for angled grains. Economically, reducing the percentage of broken rice by just 1% could yield an estimated USD 5.5 billion in annual savings globally [16]. Additionally, post-harvest losses currently account for 15–30% of total rice production [17], highlighting the potential of such a system to improve food security and sustainability.

YOLOv8 was selected due to its superior speed–accuracy trade-off compared to YOLOv5. Ultralytics reports that YOLOv8 consistently achieves higher mAP while maintaining competitive inference speeds across all model sizes [18,19]. Structurally, YOLOv8 adopts a one-stage, anchor-free design with a decoupled detection head and the C2f feature extraction module, enhancing both small-object detection and inference efficiency [20]. Furthermore, application studies show that YOLOv8 achieves real-time inference speeds of approximately 50FPS on 1080p imagery [21]. This makes it well-suited for industrial rice milling scenarios requiring the rapid, accurate detection of small, overlapping grains.

In summary, the key contributions of this work include (1) the development of a novel hybrid YOLOv8 model tailored for real-time rice milling analysis, (2) the demonstration of significant performance improvements with 0.995 mAP and 0.10 mm MAE, surpassing existing models in both detection and measurement tasks, and (3) the deployment of an industry-ready implementation that automates the classification of rice grain types with 99.5% accuracy. These contributions mark a step forward in the use of AI for intelligent post-harvest quality control in the food industry.

2. Materials and Methods

2.1. Dataset Preparation

The study utilized a custom dataset of Oryza sativa grains collected from an industrial milling facility (Figure 1), comprising three quality classes: (1) Good Rice: whole grains (length ≥ 5.5 mm) with no visible cracks or defects; (2) Broken Rice: fragments (length < 5.5 mm) or grains with visible fractures; and (3) Brown Rice: unpolished or improperly milled grains retaining husk material. Prior to analysis, all the grains were sieved to remove dust and contaminants, ensuring consistent morphological quality for accurate data sampling and acquisition.

2.2. Sample Preparation

A custom design of Rice Inspection Bed (R.I.B) was constructed from a solid piece of Delrin cut to 600 × 200 × 18 mm with an anti-reflective matte black surface to minimize optical noise. The hopper was made from stainless steel and bolted to the rear of the R.I.B. The whole assembly was mounted on 4 aluminium legs but with the front being smaller in height than the rear (Figure 2). The setup is to simulate industrial milling conditions while enabling controlled image acquisition, featuring an adjustable incline (15°–30°) to regulate grain velocity. A variable speed vibrator (10–50 Hz) was integrated to control grain flow rates and dispersion patterns, with frequency adjustments strategically minimizing grain clustering to accurately replicate real-world milling scenarios of varying density.

2.3. Image Acquisition

Rice grain images were captured under standardized laboratory conditions using a GigE Vision CMOS camera (24 MP, global shutter) equipped with a 25 mm macro lens, providing high-resolution (4032 × 3024 px) real-time data acquisition at 15 fps for precise morphological analysis. The imaging system featured synchronized pulsed LED lighting with diffusers to ensure uniform illumination while minimizing shadows and glare.A total of 1000 images were captured, each containing, on average, 70 grains in random orientations. Native-resolution images were retained for quality validation but were downsampled to 640 × 640 px during preprocessing to optimize deep learning model training efficiency without compromising critical morphological features.

2.4. Annotation Pipeline

Image annotations were performed using Roboflow’s AI-assisted labelling platform for both instance segmentation masks (YOLOv8-Seg) and YOLOv8-OBB, with the instance segmentation employing polygon annotation with edge-snapping to precisely trace each grain’s pixel-wise boundaries (Figure 3a) while maintaining separate labels for overlapping grains, and the rotated bounding boxes utilizing a specialized orientation-aware tool ($\theta$ = ±45°) to fit minimum-area rectangles to grain morphology (Figure 3b), with angle validation achieved through principal component analysis (PCA)-based alignment, ensuring angular accuracy within a 2° tolerance. The dual annotation approach combined the granularity of pixel-level segmentation for accurate grain delineation with the geometric precision of oriented bounding boxes for reliable size and orientation measurements, creating complementary datasets optimized for their respective model architectures.

2.5. Image Processing Model Architecture

2.5.1. YOLOv8 Instance Segmentation

The YOLOv8 instance segmentation model was employed to detect and precisely segment individual rice grains, even in dense clusters or overlapping configurations. Utilizing its anchor-free split Ultralytics head architecture, the model generates binary masks for each detected grain, enabling pixel-accurate boundary delineation critical for morphological analysis. The segmentation head, built upon the CSPDarknet backbone, outputs both class probabilities and mask coefficients, allowing simultaneous grain classification (Good/Broken/Brown) and high-resolution mask generation at the native 640 × 640 input resolution, with its real-time processing capability (≥45 FPS on a V100 GPU) making it suitable for industrial deployment.

2.5.2. YOLOv8 Oriented Bounding Box Model

The YOLOv8-OBB variant was implemented to predict rotated bounding boxes that precisely conform to grain orientation, addressing the limitations of axis-aligned boxes in size measurement. The model extends standard YOLOv8 by adding an angle-prediction head ($\theta$ = ±45° range) to the regression outputs, generating minimum-area rectangles that optimally fit each grain’s major axis. This geometric adaptation reduces size estimation errors by 60% compared to traditional bounding boxes, with the angle prediction trained using a modified smooth L1 loss that accounts for circular periodicity (0° = 180°), particularly valuable for analysing elongated grains where the orientation significantly affects dimensional calculations

2.5.3. Hybrid Fusion Algorithm Approach

The hybrid fusion pipeline integrates YOLOv8 instance segmentation and OBB models through a three-stage process: First, the segmentation model detects and precisely segments all grains (including overlapping ones) by generating pixel-level masks; second, the OBB model processes these detected regions to predict optimally rotated bounding boxes that provide accurate grain dimensions irrespective of grain orientation; finally, a post-processing module merges both outputs by aligning mask boundaries with OBB orientations through affine transformations, while resolving conflicts through a weighted voting system that prioritizes OBB measurements for angled grains (>15°) and segmentation data for near-vertical grains, producing final classifications enriched with both morphological details (from masks) and precise size metrics (from OBB) in a unified output format. This synergistic approach combines the segmentation model’s superior detection capability (92.4% recall) with the OBB model’s measurement precision (±0.12 mm error), achieving a 12.3% improvement in overall accuracy compared to standalone models while maintaining real-time processing speeds (15 FPS).

The YOLOv8-Segmentation model contained approximately 11.2 million trainable parameters, while the YOLOv8-OBB variant comprised 10.8 million trainable parameters. The hybrid fusion pipeline leverages these pretrained architectures without additional heavy-weight layers, resulting in a combined parameter count of approximately 22.0 million when both models operate concurrently. This moderate parameter size ensures that the framework remains lightweight enough for real-time industrial deployment.

3. Experiments and Results

3.1. Experimental Setup and Evaluation

Real-time analysis was carried out using the rice inspection bed for both instance segmentation and the object-oriented bounding box model.

3.1.1. Instance Segmentation Evaluation Metrics

The YOLOv8 instance segmentation model achieved exceptional detection performance with perfect precision (1.00) at a 0.984 confidence threshold (Figure 4a) and near-ideal mask quality (F1-score = 0.99 at 0.408 confidence) across all grain (Figure 4b), maintaining precision > 0.8 (0.2–1.0 confidence range) and F1-scores > 0.8 (0.2–0.8 range) for the reliable segmentation of touching/overlapping grains (Figure 4c). However, its reliance on axis-aligned bounding boxes for size measurement (L × W) introduces inaccuracies with angled grains (Figure 4c), as the rectangular approximations fail to capture true morphological dimensions of angled grains. This is a limitation addressed by the hybrid model’s oriented bounding box integration. The mean Average Precision (mAP) is computed as shown in Equation (1), where N is the number of object classes and ${\mathrm{AP}}_i$ is the average precision for class i.

$$\mathrm{mAP}={\displaystyle \frac{1}{N}}\sum _{i=1}^N{\mathrm{AP}}_i$$

(1)

Precision and recall are computed as defined in Equation (2), where TP, FP, and FN denote true positives, false positives, and false negatives, respectively.

$$\mathrm{Precision}={\displaystyle \frac{TP}{TP+FP}},\mathrm{Recall}={\displaystyle \frac{TP}{TP+FN}}$$

(2)

3.1.2. Oriented Bounding Box Evaluation Metrics

The YOLOv8-OBB model achieved perfect precision (1.00) at a 0.987 confidence threshold (Figure 5a) and maintained robust performance across operational ranges (0.6–1.0), with consistently high F1-scores (Figure 5b) demonstrating excellent precision–recall balance for rotated detections. While the OBB model provided superior size and orientation measurements for angled grains, real-time analysis revealed detection gaps in dense clusters (Figure 5c) achieving only 88% recall, missing out some grains during the test, versus the segmentation model’s, with 99% detection accuracy, highlighting the complementary need for instance segmentation in full-grain coverage despite OBB’s geometric advantages. The Mean Absolute Error (MAE) is computed using Equation (3).

$$\mathrm{MAE}={\displaystyle \frac{1}{n}}\sum _{i=1}^n|{\text{Actual\_Length}}_i-{\text{Predicted\_Length}}_i|$$

(3)

The Relative Error (RE) is computed using Equation (4).

$$\mathrm{RE}=\left|{\displaystyle \frac{\mathrm{Actual}-\mathrm{Predicted}}{\mathrm{Actual}}}\right|\times 100$$

(4)

3.2. Comparative Analysis of Model Performance

The hybrid model demonstrated superior performance by combining YOLOv8-Segmentation’s exceptional detection capabilities (0.989 mAP) with YOLOv8-OBB’s precise geometric measurements (0.12 mm MAE), achieving the best overall results (0.995 mAP@0.5, 0.86 Mask IoU, 0.10 mm MAE) as shown in

Table 1. Performance comparison of rice morphology detection models.

Model	mAP@0.5	Mask IoU	Size MAE (mm)	Recall (Max)	Precision (Max)	F1-Score (Max)
YOLOv8-Seg	0.92	0.85	0.35	0.99	1.00 (0.984 conf)	0.99 (0.408 conf)
YOLOv8-OBB	0.88	–	0.12	0.98	1.00 (0.987 conf)	–
Proposed Hybrid	0.94	0.86	0.10	0.995	1.00	0.995

. Statistical analysis of the classification outputs confirmed the system’s ability to accurately distinguish between good rice (whole grains ≥ 5.5 mm), broken rice (fragments < 5.5 mm), and brown rice (unpolished grains) with 99.5% accuracy. When used together, the models delivered excellent detection of overlapping grains (99% recall via segmentation) (Figure 6 and Figure 7c) while maintaining sub-millimetre sizing precision (via OBB), outperforming standalone implementations by 2% in mAP and reducing size errors by 71%. The hybrid approach proved particularly effective in real milling conditions, where PR curves (0.974 mAP) showed optimal precision–recall balance across all grain classes. The performance improvement achieved by the hybrid model is computed using Equation (5).

$$\mathrm{Improvement}=\left({\displaystyle \frac{{\mathrm{MAE}}_{\mathrm{standalone}}-{\mathrm{MAE}}_{\mathrm{hybrid}}}{{\mathrm{MAE}}_{\mathrm{standalone}}}}\right)\times 100$$

(5)

The hybrid confidence interval (CI) is calculated using Equation (6).

$$\mathrm{CI}=\mathrm{Mean}\pm (1.96\times {\displaystyle \frac{\text{Std\_Dev}}{\sqrt{n}}})$$

(6)

4. Discussion

The hybrid model demonstrates superior performance by synergistically combining the strengths of both architectures: it achieves the highest detection accuracy (0.94 mAP@0.5, +2% over YOLOv8-Seg) through enhanced recall (0.99) for overlapping grains and perfect precision (1.00) in bounding box detection, while improving segmentation quality (0.86 Mask IoU) by incorporating OBB-refined boundaries for angled grains (>15°) and reducing the size measurement error by 71% (0.10 mm MAE vs. 0.35 mm) through geometric optimization. This balanced approach maintains robust performance in dense clusters (0.995 recall) while operating at real-time speeds (15 FPS), as the shared backbone architecture efficiently processes both segmentation masks and oriented bounding boxes, with instance segmentation resolving grain overlaps and OBB providing dimensionally accurate measurements, collectively addressing the fundamental limitations of each standalone model.

5. Conclusions and Future Work

5.1. Conclusions

The proposed hybrid model effectively integrates the complementary capabilities of YOLOv8 instance segmentation and oriented bounding box detection, delivering superior performance in rice grain analysis by simultaneously achieving high-precision detection (0.94 mAP@0.5) and accurate size estimation (0.10 mm MAE). This synergistic approach overcomes the individual limitations of each model, leveraging instance segmentation’s exceptional recall (0.995) for touching and overlapping grains while utilizing OBB’s geometric precision for dimensionally correct measurements. This approach ultimately provides a comprehensive solution that outperforms standalone implementations in both classification accuracy (+2% mAP) and size measurement reliability (71% error reduction). The model’s real-time processing capability (15 FPS) and robust performance across varying grain densities demonstrate its practical viability for industrial rice milling quality control applications, marking a significant advancement over conventional analysis methods.

5.2. Future Work

Future research directions will focus on extending the hybrid framework’s applicability to other major grains (wheat, maize) by adapting the annotation protocols and model architectures to account for varying grain morphologies and size distributions, while also optimizing the system for deployment on edge computing devices for instant quality checks to enable real-time, on-site quality control in milling facilities.