Cut-Edge Detection Method for Rice Harvesting Based on Machine Vision

Abstract

A cut-edge detection method based on machine vision was developed for obtaining the navigation path of a combine harvester. First, the Cr component in the YCbCr color model was selected as the grayscale feature factor. Then, by detecting the end of the crop row, judging the target demarcation and getting the feature points, the region of interest (ROI) was automatically gained. Subsequently, the vertical projection was applied to reduce the noise. All the points in the ROI were calculated, and a dividing point was found in each row. The hierarchical clustering method was used to extract the outliers. At last, the polynomial fitting method was used to acquire the straight or curved cut-edge. The results gained from the samples showed that the average error for locating the cut-edge was 2.84 cm. The method was capable of providing support for the automatic navigation of a combine harvester.

1. Introduction

The invention of the harvester has led to an improvement in the production efficiency. However, during harvest the driver needs to continuously adjust those parameters such as speed, direction as well as cutting width [1,2], which inevitably increases operator fatigue. The invention of the automatic navigation technology of the harvester can effectively reduce the driver′s work intensity while improving the operating efficiency, which is of great significance [3].

The extraction of the navigation path is crucial for automatic navigation. Traditionally, the cut/uncut edge is often used as the reference for the harvester’s working path. The driver needs to steer the harvester along this cut-edge. Therefore, the technology of automatically detecting the cut-edge is one of the key technologies of the automatic navigation system of the harvester.

At present, Global Navigation Satellite System (GNSS), Light Detection and Ranging (LiDAR), and vision sensors are often used as main detection devices in automatic navigation systems, while the millimeter-wave radar, ultrasonic radar and infrared cameras are mainly used as the auxiliary part [4,5,6]. Now, agricultural machinery navigation systems based on the Real-time kinematic (RTK) GNSS have been adopted and are widespread [7]. However, the RTK GNSS costs a considerable amount and it can only provide navigation according to a predetermined path and can’t solve the external sensing problem. Therefore, other sensors are also needed in the navigation system to achieve the real-time environment detection. Although LiDAR is known for a long detection range and high accuracy [8], it is also high cost and know for its sparse point cloud. LiDAR is greatly affected by dust and straw debris, which means that it is not suitable to use in harvesting environments. The vision sensor is characterized by a high resolution, large amount of information and a low cost [9]. The machine vision is used to detect lateral and heading deviations of the agricultural machinery from the expected working path. This method can be applied for unknown environments, irregular fields, or environments where GNSS signals cannot be received.

Based on the above analysis, to meet the needs of small-scale rice fields and non-linear operations, this study used visual sensors as sensing devices. However, the following issues needed to be addressed in a complex field environment:

• the irregular cut-edge.
• great variability in relatively smaller areas caused by rice texture.
• dynamic changes in image brightness and color temperature.
• blurry images and weakened texture features caused by the harvester vibrations.
• the interference in the image.

To solve the above problems, this research, based on visual sensors, explored the cut-edge detection method in the work environment to make it more robust under field conditions.

Table 1. Current research status of the harvester vision navigation.

Method	Crop	Reference
Color segmentation	Alfalfa hay	M. Ollis [10,11], 1996
Texture segmentation	Grass	C. Debain [12], 2000
Grayscale segmentation	Corn	E.R. Benson [13], 2003
Stereo vision detection	Corn	F. Rovira-Más [14], 2007
Luminance segmentation	Wheat, corn	Z. Lei [15], 2007
Grayscale segmentation	Rice	M. Iida [16], 2010
Wavelet transformation	Wheat	Y. Ding [17], 2011
Color segmentation	Wheat, rice, rapeseed	Z. Tian [18], 2014
Color segmentation	Rice	W. Cho [19], 2014
Color segmentation	Wheat	M.Z. Ahmad [20], 2015
Point cloud segmentation	Wheat, rapeseed	J. Kneip [21], 2020

illustrates the research on the visual detection of the cut-edge, which could be classified into three groups: detection based on color, on texture and on stereo vision.

The stereo-vision-based method uses the height difference between the crops and the land to identify crop rows [22]. Although this method is not affected by shadows, it only works for tall crops and cannot effectively identify crop rows cut by a semi-feeder harvester.

The texture feature can be used to detect the cut-edge [23], but the vibrations of the harvester will blur the image and weaken the texture feature which would also be affected by light changes.

The color-based segmentation method uses the color difference between the cut and un-cut areas for segmentation, which is characterized by a small amount of calculation and high accuracy [24]. However, this method is greatly affected by light changes and shadows and is not suitable for crops with low color discrimination such as wheat.

Because the color difference between the cut and uncut areas is relatively stable during the rice harvesting, it was advisable to use color to segment the image. Current extraction methods have limitations such as the lack of robustness under a complex light and susceptibility to trees and roads. Therefore, improving the robustness of the color segmentation method and eliminating the interference factors were the focus of this study.

2. Materials and Methods

2.1. Image Collection

2.1.1. Image Collection System

The image collection system was installed on the combine harvester to capture and process the image in real time. The system was composed of vision sensors, mounting brackets, transmission cables and on-board computers.

The vision sensor was an Olympus EM1 Mark ii camera, which can provide five-axis body image stabilization to reduce the impact of the shaking caused by the combine harvester. The size of the image is 960 × 960 pixels and the collection frequency is 1Hz. The on-board computer was a Dell Precision 7530 mobile workstation.

Figure 1 illustrates the general layout of the system. The two cameras were installed directly above both sides of the header. The cameras were connected to the mounting bracket through a spherical head, which made it possible for them to adjust the shooting elevation angle. The camera near the cut-edge was turned on to cope with the combine harvester′s clockwise and counterclockwise operation. During the harvest, the camera was directly above the cut-edge. When controlling the steering angle of the vehicle, a driver only needed to keep the cut-edge in the center of the screen.

2.1.2. Prior Conditions for the Picture

The method proposed in this study worked only for the cases where the following prior conditions were met:

• The picture contained one or more cut-edges and there was only one cut-edge at the bottom of the picture.
• The target cut-edge started from the bottom of the screen and extended to the distance without a return.
• The target cut-edge was a single-valued function of row coordinates.

2.2. Grayscale Feature Factor Section

To improve the contrast between the cut and uncut areas, which is to reduce the intra-regional differences while increasing the inter-regional differences, it was necessary to compare the grayscale feature factors, and select the color space and components suitable for the rice harvesting scene [25].

The RGB images collected by the camera were converted to color spaces containing separate brightness components such as the HSV, YCbCr and the NTSC so that the effects of the light changes and shadows could be reduced [26]. Take Figure 2a as an example to view the different components in each color space. The results are shown in Figure 2b–d.

According to the statistical results, under the rice harvesting conditions, the Cr component in the YCbCr color space was characterized by the feature that its intra-regional variability was relatively low while its inter-regional difference was relatively high. It was therefore used as a processing project. The following studies were based on images gray-scaled by the Cr component in the YCbCr model.

2.3. Region of Interest (ROI) Extraction

To reduce the scanning range and the computation, and eliminate interference factors such as trees and roads, it was necessary to automatically obtain the region of interest (ROI). The ROI included the following characteristics:

• There was only one cut-edge starting from the bottom and extending to the top.
• There were only cut and uncut areas that existed.
• The region containing the target cut-edge was as small as possible.

The steps of creating the ROI included an end-of-row detection, target crop row selection and ROI extraction.

2.3.1. End-of-Row Detection

The end-of-row needed to be detected since there was a non-interest area, as well as many interference factors outside the end of the row.

The image was horizontally projected (Figure 3a) and accumulated pixels of each row according to formula (1):

$$Pi={\displaystyle \sum }_{1\le j\le n}P(i,j)$$

(1)

In formula (1), P(i, j) represents the pixel value of the i-th row and j-th column, and n is the number of the column of the image.

The projection result is shown in Figure 3b. The image shows good consistency in ROI. There was a clear valley boundary between ROI and non-ROI.

The projection result was shaped according to formula (2). The threshold was set to the average of the maximum and minimum values:

$$Pi\{\begin{array}{c}=1,Pi\ge (\max (\mathrm{P})-\min (\mathrm{P}))/2\\ =0,Pi<(\max (\mathrm{P})-\min (\mathrm{P}))/2\end{array}$$

(2)

In formula (2), Pi is the projection result of the i-th row, max (P) and min (P) are the maximum and minimum values in the projection result.

The effect after shaping is shown in Figure 3c. The area below the waveform is the vertical range of the ROI. Only the value of the ROI was retained. The result is shown in Figure 3d. The image is divided at the end-of-row and the detection result is shown in Figure 3e.

2.3.2. Target Crop Row Selection

If there were two uncut areas at the far end of the image (Figure 4a), the area to be processed needed to be selected. According to the prior conditions, there was only one cut-edge at the near end of the image, which could be used to decide whether the uncut area was on the left or the right side. Then, the crop rows at the far end were selected. Both the bottom and the top of the image were vertically projected. Since it was not a necessity to do an accurate selection of the target edge, which could be included in an approximate region, the number of projection rows was set to 100 to meet the need of reducing the effect of noise.

Figure 4b shows the result gained from the projection of the bottom, which, after shaping, turned into those data illustrated in Figure 4c. A threshold value of 20 was set according to experience to eliminate noise. Figure 4d shows the result when shorter fluctuations with periods less than 20 are removed. This method can separate cut/uncut areas.

The near-end image was binarized through projection, shaping and noise reduction. The result showed that the uncut area was on the right side of the image. The mutation could be regarded as the near-end feature point of the cut-edge.

The top of the image was processed and the result was illustrated in Figure 5a–c. It showed that there were uncut areas on both sides. According to Figure 4d, the target uncut area was on the right side. The left part was set to zero, and the final result was shown in Figure 5d, which contains the feature points of the distal boundary.

2.3.3. ROI Extraction

The prior conditions indicated that the cut-edge extended in one direction, so its approximate lateral range should have been between two characteristic points (Figure 6). Considering the computation and robustness, the scanning range was extended by 100 pixels (empirical value) to each side to obtain the initial ROI, as shown in the rectangular box in Figure 6a.

Because of the continuity of the field work, the difference between the two adjacent frames was small, and the frames were regarded as similar images. When the initial cut-edge was obtained, the ROI of the image was extended 100 pixels to both sides based on the cut-edge, which was taken as the ROI of the next image, as shown in Figure 6b.

2.4. Dividing Point Extraction

In this section, the dividing point was extracted in ROI in units of row. The vertical projection was used to eliminate the noise. The points in the ROI with the largest mean difference were found between the left and the right side of the image and were set as the dividing points.

2.4.1. The Vertical Projection

To eliminate the noise caused by the rice canopy texture as well as to reduce the variability within the region, it was necessary to vertically project the image according to formula (3):

$$Pj={\displaystyle \sum }_{i1\le i\le i2}P\left(i,j\right)$$

(3)

P (i,j) represents the pixel value of row i, column j. The process of cumulation starts at row i1 while ends at row i2.

The crops were projected in 1 row, in 10 rows, in 50 rows and in 200 rows, respectively, and the results were normalized to get Figure 7. The results indicated that the uncut area was the higher part of the graph. When the number of the projected rows climbed, the consistency of the cut area increased accordingly, while the accuracy was reduced. The number of projected rows was therefore set to 10 to realize the trade-off between the computational cost and the accuracy of discrimination.

2.4.2. Dividing Points Extraction

The process of extracting the dividing point was revealed in Figure 8. Projection was carried out on every 10 rows of pixels in the image. In the projection, the leftmost point within the ROI was taken. The average value of the pixels on each sides of this point were calculated, respectively. The difference between these two average values was obtained. The same calculation was applied to all points in the projection. The point with the greatest difference was the dividing point. Every 10 rows of pixels generated one dividing point, as shown in Figure 9.

2.5. Outlier Handling

Outliers with a large offset possibly appeared in the extracted results, so they needed to be detected and corrected. In this paper, the hierarchical clustering method was adopted. A minimum distance between the clusters was taken as the classification standard. The number of categories was set as 2. After several rounds of classification, all the outliers were separated.

The classification process is shown in Figure 10. During one process, a part of outliers could be extracted. The minimum distance between these outliers and the rest ones was calculated. The calculation did not stop until the value gained from the above process was less than the threshold. The distances between the normal points in the 20 images, which as revealed by the results, were always less than 50 pixels. Therefore, the threshold was set to 50.

Piecewise linear interpolation was carried out after removing the outliers. If the outliers were at the head or the end, the closest point was taken as the modified value of it.

Figure 11a shows two outliers gained from two processes of classification, which were represented as outlier 1 and outlier 2. Figure 11b shows the effect when the two outliers were corrected.

2.6. Edge Fitting

The results gained from the detection of the edge were fitted by the linear polynomial. If the R² was higher than 0.95, then the cut-edge could be regarded as a straight line. In this situation, the linear polynomial method was applied to describe the cut-edge. Otherwise, the quadratic polynomial method was adopted. The fitting effect is shown in Figure 12.

3. Results

The image collection system was installed on the combine harvester. The images were collected in the Erdaohe Farm in Heilongjiang Province, China. The rice fields were located at 134.124679 degrees east for longitude and 47.801431 degrees north for latitude. The driver manually drove the combine harvester. Vision sensors continuously collected images during the harvest. The working speed of the combine harvester was basically maintained at 1m/s.

3.1. Grayscale Feature Factor Comparison

The components of the cut and uncut areas in 20 images were normalized to calculate the variation coefficient of the two areas and the ratio of mean. The results are shown in

Table 2. Comparison of the different components.

Grayscale Feature Factor	Variation Coefficient of Cut Area	Variation Coefficient of Uncut Area	Ratio of Mean
HSV-H	0.2263	0.0974	0.8441
HSV-S	0.3778	0.3011	0.9291
NTSC-I	0.0463	0.0358	0.924
NTSC-Q	0.0391	0.0172	0.9844
YCbCr-Cb	0.0599	0.0346	0.9443
YCbCr-Cr	0.0235	0.0189	0.9169

. The Cr component in the YCbCr color space was characterized by the feature that its intra-regional variability was relatively low while its inter-regional difference was relatively high.

3.2. ROI Extraction

The ROI extraction experiment was carried out on 100 images. The success rate was 96%. The failure was caused by the inaccurate location of the end of the row.

3.3. Dividing Points Extraction

Twenty images were extracted for the dividing points. The deviation was calculated by comparing the manually marked dividing point with the predicted point. The error statistics for locating the cut-edge of one of the images are shown in Figure 13.

The formula (4) was adopted to convert the points [Xc, Yc, Zc] in the visual coordinate system into the points [X, Y, Z] in the world coordinate system. In this formula, θ is the angle between the camera and the horizontal plane and h is the height from the center of the camera to the ground:

$$\left[\begin{array}{c}X\\ Y\\ Z\end{array}\right]=\left[\begin{array}{ccc}1& 0& 0\\ 0& -sin\theta & cos\theta \\ 0& -cos\theta & -sin\theta \end{array}\right]·\left[\begin{array}{c}Xc\\ Yc\\ Zc\end{array}\right]+h\left[\begin{array}{c}0\\ 0\\ 1\end{array}\right]$$

(4)

According to the formula, the error in pixels is turned into an error in centimeters and the standard deviation of the error is calculated. The statistical results gained from 20 images are shown in

Table 3. Results of the dividing points extraction.

Index	Error in Pixels	Error in Centimeters	Standard Deviation
value	4.72	2.84	18.49

.

3.4. Outliers Detection

Twenty images with outliers were selected for clustering. According to the manual statistics, 110outliers were identified from 116 outliers in 20 images. The recall for outliers was 94.8%.

3.5. Cut Edge Fitting

The dividing points were fitted by the linear polynomial first. If the R² was higher than 0.95, then the cut-edge could be regarded as a straight line, and the calculation stopped. Otherwise, the cut-edge was regarded as a curved one and the quadratic polynomial fitting was applied. The fitting test was repeatedly carried out on 100 images, and the results are shown in

Table 4. Result of the edge fitting.

Fit Method	Linear Polynomial R² > 0.95	Quadratic Polynomial R² > 0.95	Quadratic Polynomial 0.75 < R² < 0.95
Amount	82	15	3

.

4. Discussion

The extraction of the ROI removed the interference of the non-targeted areas, trees and other factors in the distance. The Cr component of the YCbCr color model diminished the impact of the shadows and illumination changes. The vertical projection reduced the noise caused by the canopy holes between the plants. Under the prior conditions, the extraction of the ROI and the dividing points, the detection of the outliers as well as the fitting of the cut-edge were all realized.

Through analyzing the results, the main reason for the failure of the ROI extraction was the inaccurate judgment of the end of the row, which prevented the interference factors from being completely removed. Moreover, the error in the extraction of the dividing points was mainly caused by the interference of the spikes protruding near the cut-edge. The reason for the error of the outlier extraction was that the fixed threshold was incapable of handling all the outliers. Therefore, to maintain a certain precision, the threshold could not be set too low, which consequently, led to the result that 5.2% of the outliers were not being detected. Through applying the linear and quadratic polynomials, the R² of 97% of the cut-edges could reach more than 0.95. This indicated that it is advisable to use the linear and quadratic polynomials to describe most of the cut-edges.

The current method still has some inadequacy. First, the adopted method only compared the effect of the single component while the optimal combination of different components was not studied. The components will therefore be combined in different ways to find a relatively ideal combination, so that the contrast between the cut and uncut areas will be improved. Then, the negative impacts caused by shadows were not completely evaded. The failure of the extraction existed when the shadows covered too large an area. This shortcoming can be solved with a shadow removal method for rice fields. The method will be used to diminish the impact of the shadow when the shadow area is large. Finally, if the interference factor exists at the near end of the image then it cannot be eliminated. To improve robustness, the process of detecting obstacles can be added to remove near-end interference factors.

5. Conclusions

A cut-edge detection method based on machine vision was developed and evaluated under both laboratory and field conditions. The Cr component in the YCbCr color model was selected as the grayscale characteristic factor. In this way, the contrast between the cut and uncut areas was improved. This article also presented a method for automatically extracting the ROI, the success rate of which was 96%. A method to extract the dividing point was also presented, with an average error of 2.84 cm from the samples. The hierarchical clustering method was used to extract the outliers and the recall was 94.8%. The results show that the method is capable of providing support for the automatic navigation of a combine harvester.