An Improved YOLO11n-Seg Method for RGB-Based Orange Fruit Instance Segmentation Toward Clean ROI Extraction for HSI-Assisted Observation

Accurate instance segmentation of oranges in complex orchard environments is crucial for obtaining clean regions of interest (ROIs). Coarse region extraction may include non-target pixels from leaves, shadows, background, and adjacent fruits, thereby increasing boundary pixel mixing in subsequent hyperspectral-assisted observation. This study proposes an improved lightweight YOLO11n-Seg method as an RGB-based visual front-end for cleaner single-fruit ROI extraction. Its contribution lies in the task-oriented integration of three complementary components: a Local Deformable Convolution Backbone (LDC-Backbone) for representing irregular and occluded fruit contours, a Boundary-Guided GSConv (BG-GSConv) module for efficiently fusing shallow boundary details with deep semantic features, and an ROI-Purity-Oriented Dice Boundary Loss for constraining mask integrity and boundary adherence. Evaluated on a complex orchard dataset, the improved model achieved a Mask mAP@0.5 of 0.962, a Mask mAP@0.5:0.95 of 0.692, a Box mAP@0.5 of 0.942, and an inference speed of 101 FPS with 3.20 M parameters. Background leakage analysis further showed that the proposed model reduced the inclusion of non-fruit pixels in extracted ROIs, supporting cleaner mask-based single-fruit region extraction. Preliminary ROI-based reflectance observation indicated that the reflectance curves obtained from the improved-model ROIs were closer to those of manually referenced pure ROIs than those obtained from the baseline extraction. These results suggest that the proposed method can serve as a real-time RGB-based front-end for cleaner single-fruit ROI extraction and later hyperspectral-assisted sampling. Complete closed-loop spectral quality modeling with paired RGB–HSI data remains a direction for future work.