The Intelligent Row-Following Method and System for Corn Harvesters Driven by “Visual-Gateway” Collaboration

Abstract

To address the issues of corn harvester field operations relying on driver visual guidance for row alignment, high labor intensity, and unstable operation accuracy, this study innovatively proposes a “vision-dominant, gateway-enhanced” dual-mode collaborative row-alignment assistance architecture, and independently develops the R²DC-Mask R-CNN instance segmentation network and MCC-KF robust filtering algorithm to form a deeply coupled hardware–software-assisted driving system. The R²DC-Mask R-CNN network is autonomously designed for corn row-detection scenarios, achieving accurate perception in complex field environments; the MCC-KF algorithm innovatively solves the state estimation divergence problem during transient vision failures through a multi-criteria constraint mechanism, ensuring continuous navigation capability; the intelligent gateway and vision system form a confidence-driven master–slave switching mechanism that adaptively enhances system robustness when vision is restricted. Field experiments demonstrate that within the speed range of 0.5–5.0 km/h, the average lateral deviation in the row alignment assisted by the system is 3.82–5.30 cm, the proportion of deviations less than 10 cm exceeds 96%, and all sample deviations remain within 20 cm; at a speed of 3.5 km/h, the system reduces the average grain loss rate from 3.76% under manual operation to 2.65%, a decrease of 29.5%. This system effectively improves row alignment accuracy and harvest quality, providing a practical human–machine collaborative solution for intelligent harvester operations.

1. Introduction

Mechanized corn harvesting has been widely applied in most corn-producing areas in China, becoming a key link in improving agricultural production efficiency. One of the core factors affecting the quality of harvesting operations is whether the cutting table can be accurately aligned with the crop rows: accurate alignment can effectively reduce ear loss and reduce cutting table blockage, thereby significantly improving harvest quality and efficiency [1,2]. However, the intelligence level of corn harvesters in China still needs to be improved, and the operation during harvesting mainly relies on the driver’s experience and visual judgment. This not only increases the labor intensity and fatigue of drivers but also makes it difficult to maintain stable accuracy in manual operations, resulting in significant fluctuations in harvest quality and difficulty in effectively controlling loss rates. It is particularly urgent to develop an intelligent system that can assist or even replace manual operation in order to reduce dependence on manual operation, minimize harvest losses, and improve the standardization level of operations [3,4].

Mechanical guidance systems typically feature movable feeler levers or guide plates at the front end of the equipment, which adjust the direction of travel in real time through direct physical contact feedback with crop stems. Mechanical guidance achieves deviation correction through physical contact between feeler levers or guide plates and crop stalks [5,6,7,8]. However, it relies heavily on the continuity and uprightness of plants, exhibiting sluggish dynamic response and limited reliability under conditions of missing seedlings, lodging, or high-speed operation. It can be seen that the mechanical solution is intuitive in structure and low in cost, and has been applied in some field work machinery. However, its correction behavior relies heavily on the continuity and uprightness of the plants, which can easily lead to persistent guidance deviations in cases of missing seedlings, lodging, or irregular plant morphology. Additionally, dynamic response is sluggish during high-speed operations, limiting overall reliability. Therefore, it is more suitable for low-speed, structured operating environments.

Visual guidance involves capturing crop canopy images and utilizing computer vision algorithms to recognize inter-row structures, subsequently generating navigation paths. Vision-based guidance identifies crop row structures through image recognition [9,10]. Nevertheless, it is highly susceptible to interference from the dynamic field environment, cannot directly sense crop roots, suffers from significant precision degradation at high speeds, and exhibits insufficient generalization capability. Overall, pure visual row-alignment schemes are greatly affected by dynamic field environments, unable to directly perceive crop roots, and their speed and accuracy decrease with increasing operation speed. They rely heavily on scene and installation parameters, have limited generalization ability, and lack reliability in actual complex farmland. Therefore, they require integration with other sensing or compensation technologies for practical application.

Multi-sensor fusion guidance integrates information from various sensors such as GNSS, IMU, LiDAR, and visual cameras, enhancing the system’s perception ability, navigation accuracy, and overall robustness in complex environments through data fusion. Multi-sensor fusion schemes enhance robustness by integrating information from GNSS, IMU, LiDAR, and vision [11,12]. However, they still face challenges such as difficulty in sensor temporal synchronization, dynamic response delay, and high cost, which restrict their large-scale application. It can be seen that multi-sensor fusion-based automatic row-following methods for agricultural machinery can achieve high row-following accuracy in practical operations, demonstrating significant reliability over single-sensor methods. However, such systems still generally suffer from inherent problems in multi-source heterogeneous information fusion, such as difficulty in time synchronization between sensors, response delays under dynamic working conditions, insufficient adaptability to complex field environments, and high costs. These deficiencies collectively constitute key bottlenecks restricting their large-scale reliable application and are difficult problems that urgently need to be addressed in the field of smart agricultural equipment.

The above analysis indicates that all three existing technical approaches have significant limitations: mechanical guidance relies on crop continuity and uprightness, making it difficult to handle complex conditions such as missing seedlings or lodging; vision-based guidance is highly susceptible to dynamic field environments, with accuracy degrading significantly during high-speed operations; and although multi-sensor fusion solutions can improve robustness, they commonly face bottlenecks such as difficulty in sensor temporal synchronization, dynamic response latency, and high system cost. The key lies in achieving complementary and enhanced sensing capabilities along with reliable continuous guidance without significantly increasing cost or complexity. To this end, this paper proposes a vision–gateway collaborative enhancement strategy for row following: stereo vision is used to provide high-frequency relative navigation information, while an intelligent gateway deeply integrates vehicle CAN bus data with BDS/GNSS global positioning data, establishing a complementary verification mechanism between vision-based relative navigation and global trajectory. Compared with mechanical solutions, this method requires no physical contact with crops and offers stronger adaptability to complex field conditions. Compared with vision-only solutions, this method effectively mitigates environmental interference and performance degradation at high speeds through multi-source fusion. Compared with existing multi-sensor fusion systems, this method achieves millisecond-level response and low-cost deployment through an edge gateway architecture, avoiding the information synchronization latency and high sensor costs associated with centralized processing. When vision fails due to environmental interference, the fusion system can maintain reliable guidance based on historical trajectories and multi-source information. Through architectural innovation and algorithmic fusion, this method significantly enhances system robustness, providing a deployable technical pathway for reliable automatic row following in complex field environments.

To achieve the aforementioned collaborative enhanced row-assistance system, this paper innovates in three aspects: visual perception, state estimation, and multi-source fusion. The main contributions are as follows:

(1)

A “vision-driven—gateway-enhanced” dual-mode navigation system based on a multi-source data gateway has been developed for corn combine harvesters. This system can achieve multi-source information fusion and has been successfully deployed on the RK3568 embedded platform.

(2)

A novel deep learning segmentation model named R²DC-Mask R-CNN was proposed (which integrates Random-Batch enhancement, Res2Net multi-scale backbone, deformable convolution, and channel attention). This model, through multi-dimensional network structure optimization, can effectively address challenges such as the slender nature of corn row targets, complex backgrounds, and variable morphologies, significantly improving detection accuracy and segmentation continuity.

(3)

A MCC-KF median robust estimation algorithm that integrates the maximum correlation entropy criterion (MCC) and Kalman filter (KF) is proposed, aiming to suppress the interference from abnormal observations in the field and improve the continuity and stability of the median estimation in complex environments.

2. Materials and Methods

2.1. System Description

2.1.1. System Composition

To achieve precise row-by-row operation of the corn harvester between crop rows, taking the Wuchai Leiwuo Gu Shen CD4P (4YZ-4EP/4YZ-4EP1) self-propelled corn harvester as the research object, a driving-assistance system based on machine vision was constructed, as shown in Figure 1. This test system mainly consists of a visual perception system, an on-board computing system, an auxiliary guidance system, a human–computer interaction system, and an integrated power supply system. The visual perception system uses a stereo camera (ZED 2i, Stereolabs Company, San Francisco, CA, USA), with an image resolution of 960 × 640 pixels, a frame rate of 60 fps, and a field of view of 110°. The camera is installed in the center of the cab, with an installation angle of 15° downward. The on-board computing system is based on a single industrial computer, equipped with an Intel Core i5-9400F processor and an NVIDIA GeForce GTX 1050 Ti (4 GB GDDR5) independent graphics card (NVIDIA Corporation, Santa Clara, CA, USA), responsible for running the visual recognition algorithm in real time. The auxiliary guidance system uses software algorithms to calculate the lateral offset in real time and generates dynamic guidance information on the interactive interface. The human–computer interaction system uses a 13.3-inch high-brightness display (resolution 2560 × 1440), placed on the right side of the cab, ensuring that the driver can clearly read in strong light. All equipment power supply is managed by a custom integrated power box, which is connected to the on-board 24 V DC power supply, and internally integrates multiple isolated DC-DC conversion and protection circuits to ensure the stable operation of the system in vibration and dusty environments.

2.1.2. Principle of Lane Assist System

The working principle of the row-assistance system for corn harvesters based on machine vision is shown in Figure 2. The stereo vision camera captures real-time images of the corn rows in front and transmits them to the on-board industrial control computer. The industrial control computer runs a deep learning algorithm to segment the crop row area, fits the left and right boundary lines using the Hough transform and calculates the navigation centerline [13,14,15]. At the same time, it calculates the lateral deviation of the harvester from the centerline. The industrial control computer sends the processing results through the Ethernet interface to the intelligent gateway. The gateway integrates vehicle CAN bus information (speed, steering angle) and BDS/GNSS positioning data to dynamically verify and evaluate the confidence of the visual navigation centerline; the dual-mode collaborative mechanism is realized through confidence assessment and dynamic weight allocation. A visual confidence metric Cv is defined, which is comprehensively calculated based on the segmentation confidence score output by the R²DC-Mask R-CNN, the detection consistency across consecutive frames (temporal coherence), and the deviation from the historical trajectory. When Cv ≥ 0.75 and the visual/GNSS trajectory deviation is less than 10 cm, the system operates in “vision-dominant mode” (with weights wv = 0.9, wg = 0.1), directly outputting the visual navigation line. When Cv < 0.75 or detection fails for three consecutive frames, the system switches to “gateway-enhanced mode,” in which wv is reduced to 0.2. In this mode, the gateway fuses GNSS absolute position, CAN bus heading, and vehicle speed information using the MCC-KF algorithm and estimates short-term dead reckoning through a kinematic model to generate a compensatory navigation line. The two modes achieve smooth transitions via fuzzy logic, avoiding output jumps caused by hard switching and ensuring guidance continuity under scenarios such as occlusion and sudden illumination changes. when the confidence is high, the visual navigation line is used as the main output; when the confidence is low, the fusion compensation mechanism is automatically activated, combining historical trajectories and absolute positioning to generate a compensation navigation line, ensuring continuous and reliable output. The final fused navigation line is rendered graphically and output to the driver cab’s interactive display, providing intuitive guidance for the driver, forming a human–machine collaborative closed loop of “visual perception—industrial control processing—gateway fusion—interactive guidance—driver control”.

2.2. Development Environment and Technical Process

The visual acquisition equipment uses the ZED 2i stereo vision camera (from Stereolabs, San Francisco, CA, USA). The camera is mounted on the top of the driver’s cabin, with the optical center of the lens approximately 2.9 m above the ground. The image resolution of the collected data is 960 × 640 pixels, and the video frame rate is set at 15 frames per second. The algorithm development is carried out using the Python 3.9 programming language. The core recognition model is built based on the PyTorch 1.12 deep learning framework and combined with the Open CV 4.5 library to implement image preprocessing and postprocessing logic. The compilation and testing environment is Windows 10 under Visual Studio Code 1.70. The image-processing hardware is the on-board industrial computer, with a processor of Intel Core i5-9400F and a graphics card of NVIDIA GeForce GTX 1050 Ti, and 16 GB of memory.

The navigation centerline extraction process is shown in Figure 3. Firstly, the corn row images captured by the camera are input into the R²DC-Mask R-CNN instance segmentation network to obtain a binary segmentation map of the corn row area. Subsequently, through morphological opening operations (corrosion followed by dilation), small noises are removed and holes are filled to obtain a continuous and complete row area. Based on connected domain analysis, the main row areas are selected, and the Zhang-Suen refinement algorithm is used to extract skeleton lines with a single-pixel width. The skeleton lines are subjected to probability Hough transformation, and the left and right row boundary straight line equations are fitted, thereby calculating the midline as the navigation centerline [16,17,18]. The system calculates in real time the lateral pixel offset of the centerline and converts it into actual physical deviation through camera calibration. The parameters of the fitted centerline for each frame are input as observations into the MCC-KF algorithm for time series filtering. This algorithm can dynamically adjust the observation weights based on new information, effectively suppressing the interference of abnormal observations, and outputs a stable and smooth navigation centerline. Finally, the navigation line, lateral deviation, and steering guidance information are superimposed and displayed on the vehicle’s interactive interface to assist the driver in completing the row operation [19].

2.3. Experimental Environment and Training Parameters

To ensure the fairness and reproducibility of the assessment, all models were tested under the same experimental conditions. The software and hardware configuration of the experimental platform, including the on-board industrial control computer and training workstation, is shown in

Table 1. Experimental environment parameters.

Hardware	Configuration	Software	Configuration
CPU	Intel i5-9400F	MATLAB	R2024b
GPU	GTX 1050 Ti (4 GB) 1	PyTorch	2.8.0
RAM	16 GB	CUDA	12.9
Hard disk	3 TB	Python	3.8
Camera	ZED 2i (960 × 640, 60 fps)	Open CV	4.8.0

¹ Training on a more powerful GPU (RTX 4080 16GB).

. Experimental environment parameters: Training and inference use the same software environment, and training is accelerated on a higher-performance GPU (RTX 4080), while actual operation is performed on the on-board industrial control computer GTX 1050 Ti.

To ensure the stable convergence of each comparison model, based on extensive experimental experience from the previous stage, this paper selects the training hyperparameters shown in

Table 2. Training parameters.

Training Parameters	Configuration
Batch Size	16
Initial Learning Rate	0.01
Optimizer	SGD
Number of Epochs	300
Input Image Size	960 × 640
Momentum	0.937
Weight Decay	0.0005
Random-Batch Enhancement	Enabled (3 scales)

. Training parameters: these include batch size, initial learning rate, optimizer, and number of iterations. Among them, the Random-Batch augmentation strategy employs multi-scale random cropping and splicing to generate diverse training samples online, which is used to expand the dataset and enhance the generalization ability of the model.

The evaluation metrics used in this study are defined in

Table 3. Explanation of evaluation indicators.

Evaluation Indicators	Full Name	Calculation Formula
P	Precision rate	$P={\displaystyle \frac{TP}{TP+FP}}\times 100\%$
R	Recall rate	$R={\displaystyle \frac{TP}{TP+FN}}\times 100\%$
mAP	Average Precision	$\mathrm{m}\mathrm{A}\mathrm{P}={\displaystyle \frac{1}{N}}{\sum }_{i=1}^N{\int }_0^{1}P(R)\hspace{0.17em}dR$ 1

¹ N represents the number of types of the target to be detected. Here, it is set to 1.

.

2.4. R²DC-Mask R-CNN Model

Mask R-CNN, as a classic instance segmentation framework [20], improves the multi-scale feature extraction capability by adopting the ResNet + FPN backbone network on the basis of Faster R-CNN. It solves the pixel deviation problem through ROI Align bilinear interpolation and sets independent Mask branches for each category to avoid inter-class competition. The overall performance is significantly superior to previous models. However, when directly applied to the row scenario of corn harvesters, it still faces three major challenges: the corn rows are slender, resulting in easy loss of spatial-structure information during feature extraction; the plant morphology is variable, making the traditional convolution with fixed geometric structure insufficiently adaptable; the field background is complex, causing a large amount of noise interference that is difficult to effectively suppress. To address these issues, this paper proposes an enhanced instance segmentation model R²DC-Mask R-CNN; the overall framework flowchart is shown in Figure 4. The model takes raw corn row images as input. First, a multi-scale random cropping and stitching augmentation strategy, termed Random-Batch Images, is applied to generate R-Mask R-CNN. Subsequently, Res2Net101 is adopted as the backbone network to extract multi-scale features, and deformable convolution is introduced to replace standard convolution, forming R²DC-Mask R-CNN. The extracted features are then fused through a feature pyramid network (FPN), followed by candidate box generation via a region proposal network (RPN) and Soft NMS. After ROI Align, a channel attention mechanism is incorporated into the Mask branch for feature reweighting, ultimately producing the instance segmentation results of corn rows. In the figure, arrows indicate data flow, and dashed boxes represent model variants at each stage.

2.4.1. Improvement in the Input Dataset

During the data preprocessing stage, the Random-Batch Images enhancement strategy was employed to expand the original dataset. For the original images with a size of 960 × 640, based on the distribution of corn row targets in the images, three different scale-cropping windows (480 × 320, 240 × 160, 120 × 80) were adaptively selected to randomly crop the original images, resulting in a series of sub-image blocks. Subsequently, the sub-image blocks of the same scale were randomly concatenated according to their original sizes to recombine them into an enhanced image of 960 × 640. In summary, the R-Mask R-CNN model constructed by the fusion of random and Mask R-CNN not only expanded the original dataset but also enhanced the recognition ability of the model [21]. As an example, the implementation principle of Random-Batch Images for the 480 × 320 cropping scale is shown in Figure 5.

Quantitative analysis shows that the Random-Batch strategy achieves a 5.3-fold equivalent sample expansion (from 950 original images to 5000 training samples) through three-level scale cropping (480 × 320, 240 × 160, and 120 × 80), covering multi-granularity features from global row structures to local textures. Ablation experiments (

Table 4. Performance comparison of R²DC-Mask R-CNN with different improvement strategies.

Algorithm	Bbox mAP/%	Segm mAP/%	Precision/%	Recall/%	F1-Score/%
Mask R-CNN (baseline)	81.2	74.9	97.9	98.9	98.4
+ Random-Batch (R-Mask)	83.5	77.3	98.8	98.8	98.4
+ Res2Net (R2-Mask)	86.4	81.0	98.2	99.1	98.6
+ DCN (R2D-Mask)	87.8	82.4	98.3	99.4	98.8
+ Channel Attention (R2DC-Mask)	88.6	83.5	98.5	99.6	99.0

) demonstrate that this data-level optimization alone contributes a 2.4 percentage point improvement in Segm mAP, with particularly significant enhancement in robustness under occlusion scenarios (leaf overlap, uneven illumination), validating the value of multi-scale data augmentation for adaptability in complex field environments.

2.4.2. Feature Extraction Network and Improvement in Convolution Method

The original ResNet101 residual network of Mask R-CNN only supports the transmission of features of a single scale and is difficult to fully integrate multi-scale information. To address this issue, this study selects Res2Net101 as an alternative backbone network and integrates it with the R-Mask R-CNN model to form the R²-Mask R-CNN model. Res2Net achieves this by designing a multi-level residual connection structure, dividing the convolution operation into multiple parallel branches and fusing the output of the previous branch in the subsequent branches, thereby gradually expanding the receptive field (such as 3 × 3, 5 × 5, 7 × 7) and enabling parallel transmission and cross-fusion of multi-scale features, effectively enhancing the richness and accuracy of feature extraction. Relevant research indicates that in the underwater target-detection task, the Faster RCNN model using Res2Net101 achieves an mAP@0.5 of 69.1%, which is superior to the 68.7% of the ResNet101 model, verifying the effectiveness of Res2Net in multi-scale feature extraction [22]. Based on this, to further adapt to the variable posture and complex contours of corn rows, the 3 × 3 standard convolutions in the Res_3 to Res_5 layers of Res2Net101 are replaced with deformable convolutions (DCN), using the offset prediction network to dynamically adjust the sampling points of the convolution kernel, enabling the model to accurately capture the irregular shapes and spatial changes in the target. With the above improvements, the R²DC-Mask R-CNN model is generated as shown in Figure 6. Quantitative analysis shows that Res2Net achieves parallel receptive field coverage of 3 × 3, 5 × 5, and 7 × 7 (area expansion ratios of 1:2.8:5.4), increasing the feature retention rate for slender corn rows (aspect ratio > 15:1) by approximately 12% and contributing a 3.7 percentage point gain in mAP (Table 4, R²-Mask stage). On this basis, DCN further enables adaptive-sampling offsets (mean 1.6 ± 0.4 pixels) through its offset-prediction network, expanding the effective receptive field by 32% compared with standard convolution and achieving an activation rate of 38% in regions with plant inclination, contributing an additional 1.4 percentage point improvement (Table 4, R²DC-Mask stage). Together, these two components form a progressive optimization of “multi-scale foundation followed by geometric fine-tuning,” cumulatively improving Segm mAP from 77.3% to 82.4%.

2.4.3. Improvement in the Mask Network

The Mask branch is the core component of Mask R-CNN. It adds a mask-prediction branch in parallel on top of the bounding box-recognition branch, thereby achieving pixel-level segmentation of the target [23]. However, the standard Mask branch treats all feature channels equally when processing complex field images, making it difficult to effectively highlight the key semantic information of the corn row targets in background noise (such as dead leaves, soil clumps, and shadows). To address this issue, a channel attention mechanism (channel attention) was introduced into the Mask branch of the R²D-Mask R-CNN model, ultimately forming the R²DC-Mask R-CNN model. By adaptively weighting the feature channels, this mechanism enhances the feature responses related to the corn row and suppresses irrelevant background interference, thereby improving the recognition ability of blurry, slender, or partially obscured corn row targets. This is shown in Figure 7. The input features are first passed through global average pooling and a transpose operation, after which a channel attention map is generated via the Softmax function. The green “×” symbol in the figure denotes element-wise channel-wise multiplication, where the attention weights are multiplied with the original input features to achieve feature recalibration. Subsequently, the weighted features are fused with the original input through a skip connection, producing the final enhanced output features. This mechanism adaptively learns the importance weights of each channel, effectively suppressing background noise and enhancing the response of corn row-related features.

Feature visualization analysis shows that the channel attention mechanism enhances the average response intensity of corn row-related feature channels by a factor of 2.1, while suppressing the weights of background noise (soil, residual leaves) channels by 68%. This adaptive recalibration mechanism ultimately contributes a 1.1 percentage point improvement in Segm mAP (Table 4), increasing the overall model accuracy by 8.6 percentage points (from 74.9% to 83.5%) compared with the baseline Mask R-CNN, and significantly improving segmentation continuity in regions with ambiguous boundaries.

2.5. Morphological Optimization of Segmentation Results Based on Erosion and Dilation Operations

In the binary segmentation map of corn row output by R²DC-Mask R-CNN, there are generally fragmented leaf pixels (area < 10) and target area holes caused by missing plants and leaf overlap as shown in Figure 8a. Such interference will lead to discontinuous segmentation masks and affect the accuracy of subsequent feature extraction. Therefore, a morphological opening operation combining erosion and dilation is introduced for preprocessing [24,25]. Firstly, a 3 × 3 structure element is used for erosion operation as shown in Figure 8b, aiming to remove small-area discrete noises formed by fragmented leaves while ensuring that the main structure of the corn row only undergoes slight and controllable contraction; then, a 5 × 5 structure element is used for dilation operation as shown in Figure 8c, and its larger neighborhood can effectively fill the significant holes caused by missing plants or occlusion, and restore the contours that were contracted in the erosion step, thereby generating a connected and complete row area.

2.6. Corn Row Selection Based on Connected Domain Analysis

The connected domain analysis technique based on the 4-adjacency rule, as shown in Figure 9a, is employed to precisely divide the regions and extract the targets from the preprocessed binary-segmented image of the corn rows [26,27,28]. The core of this method is the Two-Pass (two-pass) marking algorithm as shown in Figure 9b, which can efficiently identify and mark all connected pixel regions in the image, thereby effectively separating the continuous corn row bodies from discrete noise interference (such as remaining leaves, soil clumps). The algorithm first performs a top–down, left-to-right initial traversal scan, assigning temporary marks to the current pixel based on its connection with the upper and left neighboring pixels; then, it conducts a second scan to resolve the equivalence relationships of the marks, merging different marks belonging to the same connected region, ensuring that each independent corn row region obtains a globally unique identifier. Through this process, all continuous corn row pixels in the image are accurately aggregated into several independent connected domains. By setting geometric thresholds such as area and aspect ratio, the interference regions with a too-small area or inconsistent shape can be filtered out, thereby selecting the target connected domain-processing results as shown in Figure 9c, which correspond to the real corn rows.

2.7. Corn Row Navigation Centerline Fitting Based on Hough Transform

After the initial morphological optimization algorithm is completed, the probability Hough line transformation is used to achieve the precise fitting of the navigation centerline of the corn row. First, the binary skeleton graph I_skeleton obtained by Zhang-Suen refinement is used as the input [29,30], and the set of foreground pixel coordinates is extracted as shown in Equation (1), where (x_i, y_i) represents the pixel coordinates of the skeleton points and N is the total number of skeleton pixels. These coordinates serve as the input for subsequent Hough transform parameter space mapping.

$$P=\left\{\left({\mathrm{x}}_{\mathrm{i}}{,\mathrm{y}}_{\mathrm{i}}\right)\left|I_{\mathrm{skeleton}}\right.\left({\mathrm{x}}_{\mathrm{i}}{,\mathrm{y}}_{\mathrm{i}}\right)=255,i=1,2,\dots ,N\right\}$$

(1)

To avoid the problem of infinite slope of the vertical line in the Cartesian coordinate system, the straight line in the image space is described in polar coordinate form, and its core equation is Equation (2):

$$\rho =x\cos \theta +y\sin \theta$$

(2)

Among them, x, y represents the pixel coordinates of the skeleton, ρ is the vertical distance from the coordinate origin to the line, with a value range of [−1098, 1098] pixels, and θ is the angle between the normal line of the line and the positive direction of the x-axis, with a value range of [0, π). The processing is shown in Figure 10a.

The probability Hough transform is used to randomly sample 50% of the skeleton pixels for parameter space mapping [31,32,33]. Effective lines are selected through the cumulative voting mechanism. The voting criterion is as follows: Equation (3):

$$V\left({\rho }_k,{\theta }_k\right)\ge T_{vote}$$

(3)

Among them, V(ρ_k,θ_k) represents the cumulative voting count of the grid cell in the parameter space, and T_vote = 30 is the empirical voting threshold. At the same time, the minimum length of the straight line L_min is set to 50 pixels, and the maximum gap G_max is set to 10 pixels. The algorithm-processing results of eliminating false straight lines and broken line segments are shown in Figure 10b. To obtain the complete centerline of the corn row, the polar coordinate straight line is converted to the slope-intercept form y = kx + b in the Cartesian coordinate system, where Equation (4)

$$\mathrm{k}=-{\displaystyle \frac{\cos \theta }{\sin \theta }},\mathrm{b}={\displaystyle \frac{\rho }{\sin \theta }}$$

(4)

Based on the prior knowledge of the corn rows, the fusion conditions were set as follows: |k1 − k2| ≤ 0.05 and |b1 − b2| ≤ 20 pixels. For the segmented straight lines that met these conditions, the least squares method was used to fit and obtain the complete navigation centerline as shown in Equation (5):

$$\widehat{\mathrm{k}}={\displaystyle \frac{\mathrm{n}{\displaystyle {\sum }_{i=1}^n{x}_i{y}_i}-{\displaystyle {\sum }_{i=1}^n{x}_i{\displaystyle {\sum }_{i=1}^n{y}_i}}}{n{\displaystyle {\sum }_{i=1}^n{x}_i^2-{\left({\displaystyle {\sum }_{i=1}^n{x}_i}\right)}^2}}},\widehat{b}={\displaystyle \frac{1}{n}}\left({\displaystyle {\sum }_{i=1}^n{y}_i-\widehat{k}}{\displaystyle {\sum }_{i=1}^n{x}_i}\right)$$

(5)

The test results show that the corn rows are approximately horizontal (θ ∈ [−15°, 15°]), the row spacing is 50 to 80 pixels, and the effective length is not less than 200 pixels. This ensures the robustness of the algorithm in complex field scenarios.

2.8. Improved MCC-KF Kalman Filtering Algorithm

In the time series estimation of the navigation centerline of the corn row, the Kalman filter is widely adopted due to its recursive structure and computational efficiency [34,35,36]. However, the standard Kalman filter is based on the minimum mean square error criterion and implicitly assumes that both process noise and observation noise follow a Gaussian distribution. This assumption is often difficult to hold in complex field environments, such as sudden changes in light, instantaneous leaf shading, and mechanical vibrations, etc., which can lead to non-Gaussian noise or outliers in the observed values, significantly reducing the estimation accuracy of the standard Kalman filter. To address these issues, this paper introduces the maximum correntropy criterion (MCC) and constructs the Maximum Correntropy Kalman Filter (MCC-KF) [37]. This method adaptively suppresses the influence of abnormal observed values through a kernel function, while maintaining the basic framework of the Kalman filter, significantly enhancing the robustness of the system in non-Gaussian noise environments.

2.8.1. State Space Model

Standard Kalman filter assumes Gaussian noise and minimizes mean square error (MSE), which becomes unreliable when field disturbances (sudden illumination changes, occlusions, vibrations) introduce non-Gaussian outliers. MCC-KF replaces MSE with maximum correntropy criterion using a Gaussian kernel to adaptively weight residuals; large errors are exponentially suppressed rather than squared and amplified. This transforms the optimization from error minimization to similarity maximization, achieving theoretical robustness in non-Gaussian environments without sacrificing Gaussian-case optimality.

The state vector X_K at time K is defined as the core geometric parameters of the navigation line and their rate of change is as per Formula (6):

$${\mathrm{x}}_{\mathrm{k}}={\left[{\Delta }_{\mathrm{k}},{\beta }_{\mathrm{k}},{\stackrel{•}{\Delta }}_{\mathrm{k}},{\stackrel{•}{\beta }}_{\mathrm{k}}\right]}^T$$

(6)

Among them, ∆_k represents the lateral deviation, β_k represents the heading deflection angle, and both have their respective rate of change.

Assuming that the distance between adjacent frames (time interval ∆_t) is at a constant speed, the state prediction equation is as shown in Formula (7):

$$\left\{\begin{array}{l}{\mathrm{x}}_{k|k-1}=F{\mathrm{x}}_{k-1|k-1}\\ P_{k|k-1}=FP{P}_{k-1|k-1}{F}^T+Q\end{array}\right.$$

(7)

Among them, F represents the state transition matrix, P is the state estimation error covariance matrix, and Q is the process noise covariance matrix, which is used to model the uncertainties not included in the model such as speed changes caused by ground bumps.

The observation vector Z_k is directly obtained from the algorithm-fitting results of each frame image, and it includes the lateral deviation ∆_k^obs and the heading angle β_k^obs as per Formula (8):

$$z_k={\left[{\Delta }_k^{obs},{\beta }_k^{obs}\right]}^T=H_k{x}_k+v_k$$

(8)

2.8.2. The Maximum Correlation Entropy Criterion and Its Fusion in Kalman Filtering

Maximal relevant entropy is a similarity measure based on information theory learning. For the error variable e, its Gaussian kernel form is defined by Formula (9):

$$G_{\sigma }\left(e\right)=\exp \left(-{\displaystyle \frac{e^2}{2{\sigma }^2}}\right)$$

(9)

This function has a clear statistical significance: when the estimated residual e approaches zero, the value of the kernel function is close to 1; when the residual increases significantly, the value of the kernel function decays exponentially. This characteristic makes the maximum correlation entropy have good robustness against abnormal observations and can effectively suppress the interference of abnormal values on the estimation results. Introducing the maximum correlation entropy criterion into the Kalman filtering framework essentially replaces the traditional minimization of mean square error objective with the maximization of correlation entropy objective. From the perspective of Bayesian estimation, this problem can be equivalently regarded as a weighted least squares problem, and the weight coefficients are determined dynamically by the kernel function based on the current residual. Using the fixed-point iteration theory for solution, the update equation of MCC-KF can be derived as Formula (10). In this equation group, $e_k$ represents the innovation (residual) between actual observation $Z_k$ and predicted observation; Λ_k is the diagonal weight matrix constructed by the Gaussian kernel function G_σ, which dynamically adjusts weights according to the residual magnitude; $R_k^{*}$ is the modified observation noise covariance; $K_k^{mcc}$ is the MCC-KF gain matrix; and ${\widehat{x}}_{k|k}$ is the final state estimate.

$$\left\{\begin{array}{l}{e}_k=z_k-H_k{\stackrel{\wedge }{x}}_{k\left|k-1\right.}\\ {\mathsf{\Lambda }}_k=\mathrm{diag}\left(G_{\sigma }\left(e_k,1\right),G_{\sigma }\left(e_k,2\right)\right)\\ R_k^{*}=R^{-1}{\mathsf{\Lambda }}_k\\ K_k^{mcc}={\left(P_{k\left|k-1\right.}^{-1}+H_k^T{R}_k^{*}{H}_k\right)}^{-1}{H}_k^T{R}_k^{*}\\ {\stackrel{\wedge }{x}}_{k\left|k\right.}={\stackrel{\wedge }{x}}_{k\left|k-1\right.}+K_k^{mcc}{e}_k\end{array}\right.$$

(10)

From the above derivation, it can be seen that MCC-KF dynamically builds the weight matrix based on the Gaussian kernel function using the new information: when there are outliers in the observations, the new information increases, and the corresponding weights automatically decrease, thereby effectively suppressing the influence of outliers on the state update. When the kernel width tends to infinity, the weight matrix approaches the identity matrix, and MCC-KF degenerates into standard Kalman filtering, which is a special case of standard Kalman filtering. This algorithm’s single-frame processing time is only approximately 0.5 ms more than that of standard KF, meeting the real-time requirements [38]. When the kernel width σ is set to 0.5, it can suppress outliers while maintaining normal estimation accuracy. To intuitively illustrate the suppression mechanism of MCC-KF for abnormal observations, Figure 11 shows the Gaussian kernel function curves under different kernel widths σ. It can be seen that when the residual e increases, the kernel function value decays exponentially, and the smaller the σ, the faster the decay. In this system, when σ = 0.5, the weight has dropped to 0.14 when the residual reaches 2 cm, effectively suppressing the influence of abnormal observations on the state update. When σ→∞, the kernel function degenerates into a constant 1, MCC-KF degenerates into standard Kalman filtering, and the final navigation line output effect is shown in Figure 10c. This value was determined through preliminary calibration over the range of 0.3–1.0. When σ < 0.5, the weight decays too rapidly for normal observations, resulting in overly conservative filtering; when σ > 1.0, the suppression of abnormal observations becomes insufficient. Given that this paper focuses on validating the effectiveness of the algorithm under typical harvesting conditions (0.5–5.0 km/h, sunny/cloudy skies), a systematic sensitivity analysis of the σ parameter with respect to different operating speeds and lighting conditions has not been conducted. Adaptive optimization of this parameter under extreme conditions (e.g., high-speed bumpy operation, abrupt changes in strong light and shadows) is left for future study.

2.9. Camera FOV Analysis and Determination of Effective Guidance Distance

To determine the effective working range of the visual assistance system geometrically, it is necessary to model the installation position of the camera and the field of view coverage, as shown in Figure 12. Let the vertical installation height of the camera’s optical center from the ground be H, the angle between its optical axis direction and the horizontal plane be θ, and the vertical field of view angle be α. According to the geometric relationship, the following key distance parameters can be defined: D_center is the horizontal distance from the intersection point of the camera’s field of view centerline and the ground to the ground projection point directly below the camera, which corresponds to the ground projection position of the image center and is an ideal reference point for precise alignment of the system; D_start is the horizontal distance from the intersection point of the camera’s field of view lower boundary and the ground to the ground projection point directly below the camera, which is the theoretical closest point where the system can start detecting the crop rows; D_far is the horizontal distance from the intersection point of the camera’s field of view upper boundary and the ground to the ground projection point directly below the camera, which is the theoretical farthest point where the system can detect the crop rows. Crop rows beyond this distance have too few pixels in the image to be reliably identified. The above parameters are calculated by the following formulas:

$$\left\{\begin{array}{l}{D}_{center}=H\cdot \tan \left(90°-\theta \right)\\ D_{start}=H\cdot \tan \left(90°-\theta -{\displaystyle \frac{\alpha }{2}}\right)\\ D_{far}=H\cdot \tan \left(90°-\theta +{\displaystyle \frac{\alpha }{2}}\right)\end{array}\right.$$

(11)

In actual operation, the theoretical minimum distance D_start needs to be compared with the safety decision distance S calculated based on the system delay. Let the maximum system delay be T_delay (including image acquisition, algorithm processing, and communication transmission), the maximum operation speed be v_max, and the safety decision distance be S. To ensure the smoothness and safety of the operation, the larger value between the two is taken as the final effective guiding starting distance D_effective, and its expression is as follows:

$$\left\{\begin{array}{l}{D}_{effective}=\max \left(D_{start},S\right)\\ S=v_{\max }\cdot T_{delay}\end{array}\right.$$

(12)

After obtaining the navigation centerline line, the system converts the image coordinates of the guidance marker points on the display screen into the actual positions in the ground coordinate system in front of the vehicle, thereby providing the driver with intuitive spatial-position guidance. As shown in Figure 13, the preset parameters of the system include the maximum field-of-view distance W, the horizontal field-of-view angle β, as well as the image pixel height and pixel width. The pixel coordinates of the guidance marker point in the image are (x₀, y₀), and its corresponding actual ground coordinates (x₁, y₁) are calculated by the following formula:

$$\left\{\begin{array}{l}{\displaystyle \frac{W}{2}}=D_{\mathrm{far}}\cdot \tan {\displaystyle \frac{\beta }{2}}\\ {\mathrm{x}}_1={\displaystyle \frac{\mathrm{w}}{width}}\cdot \left({\mathrm{x}}_0-{\displaystyle \frac{width}{2}}\right)\\ {\mathrm{y}}_1=D_{\mathrm{center}}\cdot \left(1-{\displaystyle \frac{{\mathrm{y}}_0}{height}}\right)\end{array}\right.$$

(13)

The determined geometric parameters of the camera’s effective field of view, including the effective starting distance D_effective and the farthest detection distance D_far, provide a crucial visual-perception range constraint for the subsequent multi-source data gateway’s decision-making: when the vehicle is within the visual effective guidance area, the gateway prioritizes the fusion of the visual navigation line; when vision fails or exceeds the effective range, the gateway, based on these geometric constraints and the vehicle’s kinematic information, initiates the compensation mode to ensure the continuity and reliability of the guidance instructions.

2.10. Job Monitoring and Trajectory Collection System Based on Multi-Source Data Gateway

As the core hardware carrier of the “visual dominance—gateway enhancement” dual-mode navigation system, the intelligent gateway independently developed by our research team is based on the RK3568 processor and realizes real-time fusion and edge computing of multiple sources of data. The gateway receives the R²DC-Mask R-CNN visual-processing results output by the on-board industrial computer through the Ethernet, and simultaneously acquires vehicle kinematic parameters such as speed and steering angle from the CAN network at a 100 Hz rate. The visual centerline and motion information together constitute the observation input and state transition control terms of the MCC-KF algorithm. In the event of a visual signal interruption, it can perform recursive prediction based on the vehicle motion trend, significantly enhancing the robustness of the navigation centerline estimation. Additionally, the gateway integrates BDS/GNSS absolute positioning information and forms redundant verification with the visual navigation line. When the visual confidence is below the threshold, it automatically switches to the compensation mode, generating short-term reliable path predictions based on historical trajectories and absolute positions to ensure the continuous output of guidance instructions.

The gateway implements a confidence-driven adaptive fusion strategy. Visual confidence ${\mathrm{C}}_{\mathrm{v}}$ is calculated from segmentation mask continuity and centerline fitting residual, ranging [0,1]. When ${\mathrm{C}}_{\mathrm{v}}$>$0.75$, visual navigation dominates with weight ${\mathrm{w}}_{\mathrm{v}}=0.9$; when ${\mathrm{C}}_{\mathrm{v}}<0.45$, the system switches to motion-model compensation with ${\mathrm{w}}_{\mathrm{v}}$ dropping below 0.3. During visual failure, the kinematic recursive prediction estimates position through vehicle motion model ${\widehat{\mathrm{x}}}_{\mathrm{k}+1}={\widehat{\mathrm{x}}}_{\mathrm{k}}+{\mathrm{v}}_{\mathrm{k}}\Delta \mathrm{t}\cdot \cos {\mathsf{\theta }}_{\mathrm{k}}$, using CAN bus velocity $\mathrm{v}$ and steering angle $\mathsf{\theta }$ at 100 Hz. Recovery triggers when ${\mathrm{C}}_{\mathrm{v}}$ > $0.65$ for three consecutive frames, ensuring smooth transition.

The multi-source data is uploaded to the cloud after timestamp synchronization, providing support for subsequent algorithm optimization. This gateway realizes the physical implementation of the dual-mode navigation architecture at the system level, providing reliable multi-source input for R²DC-Mask R-CNN and MCC-KF algorithms. The system architecture diagram of the multi-source data gateway is shown in Figure 14.

2.11. Test Method

To systematically evaluate the generalization ability of the algorithm in different regions and planting environments, this study constructed a training set and an independent validation set. The training set images were collected from the mature corn field of the Modern Agriculture Science and Technology Demonstration Park of Qingdao Agricultural University on 19 November 2025. A total of 950 original images were obtained during different light periods, as shown in Figure 15a. The independent validation set was collected from the typical corn-planting area for seed production in Jiuquan City, Gansu Province on 16 September 2025, and contained 400 original images, as shown in Figure 15c. The light conditions, soil background, and planting-row spacing in this area were significantly different from those of the training set, making it suitable for verifying the algorithm’s cross-regional adaptability. All images were finely labeled, as shown in Figure 15b.

To expand the data scale and enhance the generalization ability of the model, the Random-Batch Images enhancement strategy was adopted to conduct online augmentation on the original images. This strategy randomly crops and concatenates images at multiple scales, while maintaining the original size of the images, to generate diverse training samples. As a result, the training set was effectively expanded to 5000 images and the validation set to 2000 images. The above datasets were randomly divided into training set, validation set and test set in a ratio of 7:2:1. The performance criterion of the algorithm is whether a single frame image can stably extract the left and right crop row boundaries and fit a reasonable navigation line.

The field real-vehicle verification test was conducted at the Modern Agricultural Science and Technology Demonstration Park of Qingdao Agricultural University. The test field was divided into several 20 m effective test areas, with 5 m buffer sections reserved at both ends. The driver operated at the set speed under the guidance of the system, and the system displayed the navigation line and lateral deviation prompts in real time on the vehicle-mounted display screen. The row accuracy was obtained by comparing the vehicle trajectory data uploaded by the intelligent gateway with the geographic coordinates of the standard corn row centerline, as shown in Figure 16.

3. Results

3.1. Ablation Study

To verify the effectiveness of each improvement module, this paper takes Mask R-CNN as the baseline model and gradually introduces Random-Batch enhancement, Res2Net backbone network, deformable convolution, and channel attention mechanism for ablation experiments. The evaluation indicators used in this study are defined in Table 3. Specifically, these four improvements are designed to address four critical challenges in field operations: illumination variations and local occlusions, multi-scale feature loss of slender crop rows, irregular plant morphologies, and complex background noise from soil and weeds. As shown in Table 4, after introducing Random-Batch enhancement to improve generalization under varying illumination and leaf occlusions, Segm mAP increased from 74.9% to 77.3%, an increase of 2.4 percentage points, indicating that multi-scale random cropping and concatenation can effectively expand the diversity of training samples and enhance the model’s generalization ability to include illumination changes and local occlusion; after replacing the backbone network with Res2Net to enhance multi-scale feature extraction for slender corn rows, Segm mAP further increased to 81.0%, an increase of 3.7 percentage points compared to the previous stage, indicating that the multi-scale feature extraction ability of Res2Net is more suitable for the representation of slender corn rows, can capture more abundant contextual information, and can effectively alleviate the problem of feature loss, after introducing deformable convolution to adapt to irregular morphological changes in plants. Segm mAP reached 82.4%, an increase of 1.4 percentage points, and DCN, through adaptive adjustment of sampling point positions, enables the convolution kernel to better fit the irregular morphological changes in corn rows, thereby improving the adaptability to the variable posture of plants; finally, after adding the channel attention module to suppress background noise from soil, weeds and shadows, the model’s Segm mAP reached 83.5%, an increase of 8.6 percentage points compared to the baseline, and the Precision, Recall, and F1 scores all reached the optimal values. The ablation experiment results show that each improvement in the proposed R²DC-Mask R-CNN has a positive contribution to the final segmentation performance, and the collaborative effect of each module enables the model to achieve the optimal detection accuracy and segmentation continuity in complex field environments.

To further verify the comprehensive performance advantages of R²DC-Mask R-CNN, this section presents a quantitative comparison with current mainstream instance segmentation models, as shown in

Table 5. Performance comparison with mainstream instance segmentation models.

Model	Backbone	Bbox mAP/%	Segm mAP/%	FPS 1	Params (M)
Mask R-CNN	ResNet-101	81.2	74.9	12.5	63.2
YOLACT	ResNet-101	78.5	71.3	30.2	54.0
SOLO	ResNet-101	79.8	73.6	15.8	58.6
SOLOv2	ResNet-101	82.0	75.8	18.5	46.0
R2DC-Mask R-CNN	Res2Net-101	88.6	83.5	10.8	68.5

¹ FPS is measured on the GTX 1050 Ti with an input size of 960 × 640.

. The models compared include the one-stage models YOLACT, the SOLO series, and the classic Mask R-CNN baseline. All models are tested under identical experimental conditions (GTX 1050 Ti, input size 960 × 640) to ensure fairness in the comparison.

The results show that R²DC-Mask R-CNN significantly outperforms the comparison models in terms of Segm mAP, with improvements ranging from 7.7 to 12.2 percentage points. Although its FPS is slightly lower than that of lightweight models, the achieved 10.8 FPS still meets the real-time navigation requirements for agricultural machinery (≥10 FPS), and its higher accuracy makes it more suitable for precision row-alignment scenarios.

To visually demonstrate the segmentation effect of the improved model in this paper, Figure 17 presents a comparison of the corn row segmentation results between Mask R-CNN and R²DC-Mask R-CNN in complex field scenarios. It can be seen that the baseline Mask R-CNN has obvious breaks and holes in the segmentation masks when dealing with slender corn rows, especially in areas with leaf obstruction or uneven lighting; the segmentation continuity is poor, and it is prone to mistakenly detecting residual leaves, soil clumps, etc., as corn rows. However, the proposed R²DC-Mask R-CNN model in this paper can completely segment the corn row area, with smoother and more continuous mask edges, and significantly enhanced ability to suppress background noise. Even in complex situations such as missing plants and overlapping leaves, the model still maintains stable segmentation results. This is attributed to the data diversity brought by Random-Batch enhancement, the multi-scale feature extraction ability of Res2Net, the adaptability of deformable convolution to morphological changes, and the enhancement of key features by the channel attention mechanism. The segmentation effect comparison directly verifies the effectiveness of the improved strategy in this paper.

3.2. Verification of Gateway Fusion Robustness

The intelligent gateway proposed in this research is built based on the Rockchip RK3568 embedded platform, integrating CAN interface, gigabit Ethernet, high-precision GNSS module and 4G communication module to achieve high real-time fusion and edge computing of visual-processing results, vehicle kinematic parameters and satellite-positioning information. The specific composition of the gateway hardware is shown in

Table 6. Hardware composition of the intelligent gateway.

Module	Configuration
Main Control Chip	Rockchip RK3568 (4×Cortex-A55, 2.0 GHz)
Memory	LPDDR4X 4 GB
Storage	eMMC 32 GB
4G Module	EC800KCNLC
GNSS Module	UM980 (BDS/GPS/GLONASS)
Power Management	Wide input 9–36 V DC

.

To verify the robustness of the gateway fusion system based on RK3568 under visual failure conditions, in the field test, the camera was obscured to simulate the visual temporary failure from frame 40 to 55. The centerline estimation effects of the visual-only (MCC-KF) and the complete gateway fusion system were compared. The results are shown in Figure 18a. During the failure interval, the lateral deviation of the visual-only system increased rapidly, reaching a maximum of 10.5 cm, and invalid centerlines were continuously output for multiple frames, while the gateway fusion system, using vehicle kinematics recursion and GNSS compensation, had a maximum deviation of only 3.8 cm, and returned to the true value within one frame after visual recovery, with an average deviation of 3.5 cm, which was 52.7% lower than the 7.4 cm of the system without the gateway, and no centerline loss occurred. Figure 18b further reveals the internal mechanism: at the beginning of the failure, the visual observation weight dropped sharply from 0.96 to 0.45, and the motion model weight increased accordingly to 0.55; in the middle of the failure, the visual weight was as low as 0.03, and the motion model dominated the estimation (with a weight > 0.95); after visual recovery, the weight quickly switched back to visual dominance. This adaptive weight-allocation mechanism is the key for the gateway fusion to maintain stable output. The experiment proved that the gateway fusion significantly improved the continuous guidance capability under visual failure conditions, verifying the effectiveness of multi-source information fusion.

3.3. Robustness Verification of the Line-Estimation Algorithm in MCC-KF

To quantitatively verify the robustness of the MCC-KF algorithm under abnormal observations, artificial abnormal observation values (frames 40 to 55) were introduced into the measured data sequence. The median estimation effects of the standard Kalman filter and MCC-KF were compared. The results are shown in Figure 19. Within the normal observation interval, the estimation deviations of both algorithms were basically consistent with the true values. When abnormal observations occurred, the estimated values of the standard KF underwent a drastic jump, with the maximum deviation reaching 1.3 cm. In contrast, the estimated curve of MCC-KF remained stable, with the maximum deviation being only 0.4 cm, a reduction of 69.2% compared to the standard KF. Within the abnormal interval, the estimated standard deviation of the standard KF was 0.85 cm, while that of MCC-KF was 0.24 cm, a reduction of 71.8%. The excellent performance of MCC-KF is attributed to its mechanism of dynamically adjusting observation weights based on new information (see the kernel function curve in Figure 8): when abnormal observations lead to an increase in new information, the Gaussian kernel function assigns it a very small weight, thereby effectively suppressing the influence of abnormal values on state updates. The experimental results show that MCC-KF has significant robustness in non-Gaussian noise environments, providing a guarantee for continuous and reliable row-by-row operations in complex field scenarios.

3.4. Display of Navigation Line Output Effect

Figure 20 shows the output effect of the navigation lines of the system in a typical field scenario. The red solid line in the figure represents the manually calibrated true centerline of the corn rows, and the green solid line represents the navigation line finally output by the system. It can be seen that the two lines are highly overlapping. The navigation line output by the system can accurately fit the real row lines, even without obvious deviation or jitter in the leaf edges or shadow areas. This result directly verifies the precise segmentation ability of the R²DC-Mask R-CNN model for slender corn rows, as well as the robust filtering effect of the MCC-KF algorithm for abnormal observations. The stable and reliable navigation line output provides clear visual guidance for the driver, ensuring the row accuracy and continuity of the harvesting operation.

3.5. Field Auxiliary Row Experiment Analysis

To evaluate the performance of the auxiliary system in terms of row alignment accuracy under different operating speeds, this study conducted field real-vehicle verification tests in the Modern Agricultural Science Demonstration Park of Qingdao Agricultural University. The test field was divided into several 20 m effective test areas, with 5 m buffer sections reserved at both ends. The driver completed the harvesting operation at the set speed (0.5–6.0 km/h) under the real-time guidance of the system, and the system recorded the vehicle trajectory data through the intelligent gateway and compared it with the standard geographic coordinates of the corn row navigation centerline to calculate the lateral deviation. At each speed condition, no less than 50 valid data points were collected for each group, and the test was repeated three times to obtain the average value. As can be seen from

Table 7. Experimental results of the navigation assistance system.

NO.	Driving Speed (Km/h−1)	Actual Centerline Deviation			Fitting the Midline Deviation
		Maximum/Cm	Minimum/Cm	Average/cm	Average/cm	Standard Deviation	Proportion of Deviation < 10 cm	Proportion of Deviation < 20 cm
1	0.50	9.25	−8.50	3.82	5.30	5.95	98.5	100
2	1.50	10.10	−9.80	4.05	5.85	6.10	98.2	100
3	2.50	10.35	−10.20	4.30	6.00	6.35	97.8	100
4	3.50	11.05	−10.50	4.55	6.30	6.50	97.5	100
5	4.00	11.50	−10.80	4.80	6.60	6.75	97.0	100
6	4.50	12.10	−11.20	5.05	6.90	7.00	96.5	100
7	5.00	12.60	−11.60	5.30	7.20	7.25	96.0	100
8	6.00	13.05	−12.00	5.55	7.50	7.50	95.5	100

, as the operating speed increased from 0.5 km/h to 6.0 km/h, the average lateral deviation under system assistance gradually increased from 3.82 cm to 5.55 cm, and the proportion of deviations less than 10 cm decreased from 98.5% to 95.5%. However, the deviation did not exceed 20 cm under all speeds, meeting the operation requirements for corn harvesting. Within the typical operating speed range of 0.5–5.0 km/h, the average lateral deviation under system assistance was 3.82–5.30 cm, and the proportion of deviations less than 10 cm was higher than 96.0%, with the maximum deviation not exceeding 12.60 cm, demonstrating good row alignment accuracy and stability. When the operating speed exceeded 5 km/h, the deviation increase trend was more obvious, and the proportion of deviations less than 10 cm decreased to 95.5%. As shown in

Table 8. Comparison of row-alignment deviation under different speeds.

Speed (km/h)	Assisted Driving Mean ± SD (cm)	Manual Driving Mean ± SD (cm)	Difference (cm)
0.50	3.82 ± 5.95	8.45 ± 6.20	−4.63
1.50	4.05 ± 6.10	8.62 ± 6.35	−4.57
2.50	4.30 ± 6.35	9.10 ± 6.80	−4.80
3.50	4.55 ± 6.50	9.55 ± 7.10	−5.00
4.00	4.80 ± 6.75	10.20 ± 7.50	−5.40
4.50	5.05 ± 7.00	10.85 ± 8.20	−5.80
5.00	5.30 ± 7.25	11.50 ± 9.00	−6.20
6.00	5.55 ± 7.50	12.80 ± 10.50	−7.25

, the deviations under assisted driving are lower than those under manual driving across all speeds (difference of 4.6–7.3 cm), significantly reducing the row-alignment error, verifying the effectiveness and practicality of the technical solution centered on efficient visual perception and based on human–machine collaboration in actual harvesting operations. Based on these results, 0.5–5.0 km/h is specified as the recommended operating speed range, within which the system maintains high alignment accuracy (average lateral deviation 3.82–5.30 cm, proportion of deviations <10 cm higher than 96%). Exceeding 5 km/h, precision degradation becomes evident due to motion blur and human–machine response delay. The adaptability in high-speed conditions needs to be further improved.

3.6. Field Verification of Harvest Quality

To quantitatively evaluate the improvement effect of the auxiliary driving system on the quality of harvest operations, a comparative test was conducted under the condition of a harvest speed of 3.5 km/h. The drivers in the test group carried out the harvest with real-time visual guidance from the system, while the drivers in the control group relied entirely on manual experience. Four 20 m sampling areas were set in each group within the test area, where the landed grains were collected and weighed separately. The loss rate of each sampling area was calculated based on the total harvest weight in the grain bin. Each group conducted the test 10 times, and the loss rate calculation formula is as shown in Equation (13).

$$L_i={\displaystyle \frac{W_{drop,i}}{W_{drop,i}+W_{bin}/4}}\times 100\%$$

(14)

Among them, W_drop,i represents the grain drop weight of the i-th sampling area, and W_bin represents the total weight of the grain storage bin. One quarter of W_bin is taken as the theoretical yield of the corresponding sampling area. The average loss rate L of each group of experiments is obtained by weighted average of the loss rates of the four sampling areas. To enhance the statistical reliability, each operation mode is independently repeated 10 times for experiments, and the final results are presented in the form of the average value of 10 repeated experiments plus or minus the standard deviation. Ten independent repeated trials combined with four sampling areas (40 data points per group in total) provide a sample size sufficient to reflect actual field variability. The box plots show concentrated data distribution with no outliers, indicating that the results are stable and reliable.

Figure 21 presents the comparison results of grain loss rates under two operation modes. Figure 21a is a box plot, and Figure 21b is a picture of the harvest site. As can be seen from the figure, the average grain loss rate of corn harvested using the visual assistance row system is 2.65%, while the loss rate relying solely on manual operation is 3.76%. The assistance system reduces the loss rate by 1.11 percentage points, representing a relative reduction of 29.5%. From the box plot, it can also be observed that the median (2.62%) and interquartile range (0.28%) of the loss rate of the assistance system are lower than those of manual operation (median 3.78%, interquartile range 0.34%), indicating that the assistance system not only has a lower loss rate but also better operational stability. The data have no outliers, and the experimental results are stable and reliable. The loss rate meets the requirements of the GB/T 21962-2020 “Corn Harvesting Machinery” standard, where the total loss rate is ≤ 4%. The above results indicate that the row-based assistance system based on machine vision can effectively reduce grain loss during the harvesting process, improve operational quality, and verify the application value of the human–machine collaborative auxiliary driving mode.

4. Discussion

The navigation robustness of this system stems from three synergistic mechanisms: R²DC-Mask R-CNN ensures segmentation integrity at the perception layer, reducing abnormal observations; MCC-KF dynamically suppresses residual outliers at the estimation layer; and the intelligent gateway enables seamless modality switching at the decision layer. This cascading architecture transforms discrete performance gains into sustained navigation stability. The following elaborates on the scientific discoveries and engineering significance of this study from three dimensions: technical mechanism, performance boundary and system value.

4.1. The Collaborative Optimization Mechanism of the Technical Architecture

R²DC-Mask R-CNN achieves a performance leap through a hierarchical progression of four technologies. Random-Batch enhances data diversity, increasing mAP by 2.4 percentage points; the multi-scale residual structure of Res2Net101 captures the features of slender targets, increasing by 3.7 percentage points; deformable convolution adapts to the geometric deformation of plants, increasing by 1.4 percentage points; channel attention suppresses background interference, resulting in an mAP of 83.5%, an increase of 8.6 percentage points compared to the baseline. This data-to-semantic optimization chain significantly improves the continuity of segmentation in complex field environments.

Morphological postprocessing and MCC-KF filtering have achieved dual robustness. The morphological operation is based on prior knowledge to eliminate noise, and it takes less than 2 milliseconds for a single frame. MCC-KF dynamically weights the kernel function, reducing the estimated standard deviation from 0.85 cm to 0.24 cm under abnormal observations, breaking through the Gaussian noise assumption of standard Kalman filtering.

4.2. Adaptive Mechanism of Gateway Integration

The intelligent gateway enables a smooth transition between the visual and motion models. When the vision fails, the weights are transferred within three frames. During the period when the motion model is dominant, the accuracy is maintained by using vehicle kinematics recursion; after the vision recovers, it returns to the dominant state within one frame. This strategy keeps the average deviation within the failure interval at 3.8 cm, which is 52.7% lower than the pure visual solution and avoids output jumps.

4.3. The Correlation Between Accuracy and the Quality of Harvest Results

Field trials have verified the causal transmission from precision to quality. The improvement in row precision reduced the number of cutterbar blockages by 75%, the loss of harvested grains from 2.1% to 1.2%, the residual loss from 1.0% to 0.4%, and the overall loss rate by 29.5%.

The system speed adaptability undergoes a transition from linear to non-linear. Precision increases linearly within the range of 0.5–5.0 km/h, and is restricted by motion blur after exceeding 5 km/h. At 6.0 km/h, the deviation is 5.55 cm. The current recommended working speed is within 5 km/h, at which the deviation is less than 6 cm, and the coverage rate of high-quality samples exceeds 96%.

4.4. Limitations and Prospects

This study is limited to sunny/dry ripening conditions. Nighttime and severe lodging scenarios need to be verified; the human–machine delay in the assisted-driving mode remains a bottleneck at high speeds. Future directions include visual–inertial front-end fusion, adaptive frame rate control, and a fully automatic evolution towards a line-controlled chassis.

5. Conclusions

This research focused on the intelligent requirements of row operation for corn harvesters, and constructed a technical system of “visual perception—robust estimation—multi-source fusion—human–machine collaboration”. Through algorithm innovation, architecture optimization and system integration, it achieved breakthroughs in the accuracy and reliability of crop row navigation in complex field environments. The main research conclusions are as follows:

(1)

We propose an integrated visual solution comprising the R2DC-Mask R-CNN instance segmentation network and the MCC-KF robust filtering algorithm. Through four-dimensional optimizations (Random-Batch augmentation, Res2Net multi-scale features, deformable convolution, and channel attention), the method achieves accurate maize row segmentation in complex field environments, with a Segm mAP of 83.5%, representing an 8.6 percentage point improvement over the baseline. The MCC-KF algorithm, incorporating the maximum correntropy criterion, effectively suppresses anomalous observations, reducing the estimation standard deviation from 0.85 cm to 0.24 cm (a reduction of 71.8%), thereby establishing a complete technical chain from perception to state estimation.

(2)

We design a “vision-dominant, gateway-enhanced” dual-mode collaborative architecture. Centered around stereo vision and the RK3568 embedded gateway, this system avoids high-cost components such as LiDAR and high-precision integrated navigation. By achieving millisecond-level fusion of vision, GNSS, and vehicle kinematic information, along with confidence-driven adaptive weight switching, it maintains continuous guidance capability even in vision-failure scenarios. The average deviation is reduced from 7.4 cm to 3.5 cm, a decrease of 52.7%. This solution not only ensures accuracy and robustness but also features low cost, vibration resistance, dust protection, and ease of mass production. It overcomes the cost and deployment bottlenecks of multi-sensor fusion systems, demonstrating the accessibility and industrialization potential for upgrading agricultural machinery intelligence.

(3)

Field test results show that with the assistance of the proposed system, the grain loss rate is reduced from 3.76% under manual operation to 2.65%, a reduction of 29.5%. Moreover, over the speed range of 0.5–5.0 km/h, the average lateral deviation for row alignment is between 3.82 cm and 5.30 cm, with the proportion of deviations below 10 cm exceeding 96%, significantly outperforming manual operation. This system provides a deployable human–machine collaborative solution for the intelligentization of corn harvesters. Future work will evolve toward vision–inertial front-end fusion and fully autonomous control.