Navigation Line Extraction Method for Broad-Leaved Plants in the Multi-Period Environments of the High-Ridge Cultivation Mode

Abstract

Navigation line extraction is critical for precision agriculture and automatic navigation. A novel method for extracting navigation lines based on machine vision is proposed herein using a straight line detected based on a high-ridge crop row. Aiming at the low-level automation of machines in field environments of a high-ridge cultivation mode for broad-leaved plants, a navigation line extraction method suitable for multiple periods and with high timeliness is designed. The method comprises four sequentially linked phases: image segmentation, feature point extraction, navigation line calculation, and dynamic segmentation horizontal strip number feedback. The a* component of the CIE-Lab colour space is extracted to preliminarily extract the crop row features. The OTSU algorithm is combined with morphological processing to completely separate the crop rows and backgrounds. The crop row feature points are extracted using an improved isometric segmented vertical projection method. While calculating the navigation lines, an adaptive clustering method is used to cluster the adjacent feature points. A dynamic segmentation point clustering method is used to determine the final clustering feature point sets, and the feature point sets are optimised using lateral distance and point line distance methods. In the optimisation process, a linear regression method based on the Huber loss function is used to fit the optimised feature point set to obtain the crop row centreline, and the navigation line is calculated according to the two crop lines. Finally, before entering the next frame processing process, a feedback mechanism to calculate a number of horizontal strips for the next frame is introduced to improve the ability of the algorithm to adapt to multiple periods. The experimental results show that the proposed method can meet the efficiency requirements for visual navigation. The average time for the image processing of four samples is 38.53 ms. Compared with the least squares method, the proposed method can adapt to a longer growth period of crops.

1. Introduction

The concept of precision agriculture has been widely accepted by the international community and has become a global research hotspot [1]. Automatic agricultural machinery navigation is a popular core technology in precision agriculture applications and represents the foundation of precision agriculture [2]. The collection of position and attitude measurement information is the first task to be addressed in agricultural machinery automatic navigation. The accuracy and reliability of the positioning and attitude measurement information are prerequisites for agricultural machinery to realise automatic navigation. Existing methods for collecting positioning and attitude information mainly include the global navigation satellite system (GNSS), inertial navigation system (INS), and machine vision (MV) navigation. GNSS navigation offers reliable and absolute coordinate and heading information, but the signal is easily obstructed by obstacles and interfered with by other radio frequency sources [3]. INS navigation offers accurate attitude data, but INS sensors are sensitive to changes in the electromagnetic field, vibrations, acoustics, temperatures, and other environmental factors; hence, they must be frequently calibrated to maintain their normal working conditions. MV navigation usually uses image sensors for data acquisition, such as charge-coupled devices (CCD) and complementary metal oxide semiconductors (CMOS). Although the nonstructural field environment tends to negatively affect the information collected by the image sensors, MV navigation is characterised by its low cost, good timeliness, the richness of information that can be processed, and its high scalability [4]. MV navigation accurately detects a navigation line, and the control system automatically guides the machine to move in the field according to the navigation line. A navigation line can generally be calculated according to crop row centrelines. Image segmentation, feature point extraction, and navigation line calculation are three important links in crop row centreline extraction.

In recent years, scholars, such as Yu et al. [5], Lin et al. [6], Kim et al. [7], and Adhikari et al. [8], have proposed extracting navigation lines, using deep learning methods. These methods have good adaptability and accuracy but require labeling and training on large-scale data sets, and they often require a large amount of computational resources and labor costs. In contrast, conventional extraction methods do not require training and have lower hardware requirements. Therefore, while ensuring availability, conventional methods still have certain cost advantages.

In the context of crop row centreline conventional extraction, the image segmentation is divided into three steps: grayscale, binarisation, and image morphological processing. The grayscale step is used to initially separate the crops from the background. The main methods of grayscale processing include the spectrophotometry method, RGB vegetation index methods, and a* component method. A spectrophotometry method combines an IR band-pass filter (>795 nm) and a CCD camera (the resolution of images was 640 pixels × 480 pixels) and uses the characteristics of spectral reflection difference between the crops and background for image segmentation [9]. RGB vegetation index methods usually use ordinary image sensors to obtain images and adjust the colour components after image normalisation to obtain grayscale images with clear backgrounds and target colours. The main vegetation indexes are excess green (ExG) [10], optimised ExG [11], G-R and G-B [12], MexG [13], etc. The methods described above mainly use normalisation to overcome the influence of different lighting conditions on feature extraction. The a* component method extracts the a* component from the device-independent CIE-Lab colour space. Spectrophotometry has higher hardware requirements than other methods. The RGB vegetation index method is widely used, but the grey images from RGB vegetation index methods generally have more noise. The a* component offers less noise, and it is more straightforward to operate when compared to the other two methods. Binarisation processing changes the grey values of all pixels to one of the two specified values through the threshold setting, further highlighting the crop contours and reducing the difficulty of image processing. At present, the mainstream binary processing methods are the maximum between the class variance algorithm (OTSU algorithm, hereafter referred to as OTSU) and the maximum entropy threshold algorithm [14]. Because OTSU can automatically use the variance to determine the threshold, it is more commonly used for binary image processing [15]. Image morphology processing uses basic mathematical morphology operation methods to eliminate noise or fill holes in binary images. Zhang et al. expanded the features of the binary image of a corn field to fill a crop gap and then filled the aperture with water injection. White dots with an area smaller than the threshold were deleted, and the white dots with an area larger than the threshold were retained [16]. García-Santillán et al. used open and majority operations to remove insignificant small blocks and false pixels from a corn binary image [17].

Crop line extraction represents the premise of the navigation line calculation. The positions of the feature points directly determine the positions of crop row lines. Therefore, crop feature point extraction is the most critical step in crop line extraction. The existing methods proposed by scholars for crop feature point extraction mainly include the horizontal strip method, corner point identification method, skeleton extraction method, and blob analysis method. In the horizontal strip method, Yu et al. divided the binary image into several horizontal stripes and determined the feature points by studying the number of white pixels on each horizontal stripe [18]. For the corner point recognition method, Zhai et al. calculated the three-dimensional coordinates of vegetation corner features based on feature point detection technology and binocular disparity ranging method, extracted crop row feature points based on the three-dimensional threshold, and established a crop row centreline detection algorithm [19]. For the skeleton extraction method, Diao et al. marked pending skeleton points satisfying a maximum square criterion in binary images of leeks and corn and then used a scanning operation to find the skeleton points closest to the centre of the traversed region as target skeleton points. Then, they used these marked skeleton points as the target features [20]. For the blob analysis method, Fontaine and Crowe identified and characterised neighbouring pixel regions with the same value within a wheat binary image. With an appropriate window size, crop row feature points with low heights could be properly located [21].

Generally, the navigation line calculation is divided into two steps: first, a ‘crop line straight line’ is obtained by straight-line fitting according to the crop feature points, and then a navigation line is calculated according to the crop line straight line. The common methods for straight-line fitting are the least squares method (LSM) and Hough transformation (HT). The LSM has the advantages of speed and high fitting accuracy and is especially suitable for single crop row recognition, but it is sensitive to various interference factors and lacks robustness to the presence of singular points. Scholars who have used LSM include Mao et al. [22] and Wang et al. [23]. The HT can detect any number of crop rows simultaneously, is less affected by noise, and has strong robustness. The extracted centreline has high accuracy, but it is difficult to reduce the processing time. Scholars who have used HT include Basso et al. [24], Winterhalter et al. [25], Xia et al. [26]., and Varela et al. [27].

First, the literature mentioned above mainly concerns crop row extraction methods for no-ridge or low-ridge cultivation environments, such as wheat and corn crops. In contrast, the high-ridge cultivation environment of broad-leaved plants has unique characteristics. For the convenience of expression, the high-ridge cultivation environment for broad-leaved plants is referred to in the following as a ‘high-ridge environment’. As shown in Figure 1a–c, the plant sizes of broad-leaved crops vary greatly on different field growth days after transplanting, as mainly reflected in the images; with the growth of the crops, connections between plants in rows and between crop rows gradually occur. As shown in Figure 1d, the high-ridge environment is prone to generating additional shadows owing to the influences of light conditions. The majority of scholars have only studied and analysed the two situations between plants in a row in the same period, i.e., contact situation or no-contact situation. At present, there is no research on a multi-period unified detection method for navigation lines calculated based on the row lines of broad-leaved high-ridge crops.

In addition, according to the actual situation of this study, there can only be one navigation line for the machine to base its walk on. Some scholars take the detection of multiple crop rows in the image as the goal, and such algorithms need to calculate navigation lines based on multiple crop rows. There are approaches proposed to detect only two crop rows and then to calculate navigation lines to improve the timeliness of the algorithm [11], but the ability of approaches proposed to adapt to a variety of environments requires further enhancement.

Finally, the number and height of horizontal strips as divided by the vertical projection method proposed by most scholars are fixed values. Thus, they generally cannot meet the requirements for multi-period detection. At present, no scholars using the vertical projection method have proposed a scheme for automatic adjustment of such values.

Aiming to extend the advantages of the existing methods mentioned above, this paper proposes a new navigation line extraction method focused on improving the adaptability of the algorithm to different periods of broad-leaf species in high-ridge environments while ensuring timeliness. The method consists of four sequential modules for image segmentation, feature point extraction, navigation line calculation, and dynamic feedback mechanism between two adjacent frames, respectively. In terms of image segmentation, the a* component of the CIE-Lab colour space is used to convert RGB images into grayscale images, and the crop and background are separated by combining OTSU and morphological processing methods. In terms of feature point extraction, an improved isometric segmented vertical projection method is used to extract crop row feature points. Then, an adaptive clustering method and dynamic segmentation point clustering method are successively applied to determine the feature point sets, and feature point sets are optimised using lateral distance optimisation and point-line distance optimisation. In the point-line distance optimisation process, a linear regression method based on the Huber loss function is used to fit points to obtain crop row centreline. Finally, the dynamic feedback mechanism between two adjacent frames is introduced to improve the adaptability of the algorithm to multiple crop growth periods. Therefore, the main contribution of this paper is a unified extraction method for broad-leaved crop multi-period navigation lines based on combining the adaptive clustering method and dynamic segmentation point clustering method, as well as the design and application of a feedback mechanism for automatically calculating the number of segmented horizontal strips for the next frame. In addition, we have further improved the robustness of the method by using a linear regression method with the Huber loss function.

2. Materials and Methods

After capturing a video using a CMOS camera, the static sample image is separated from the video. The method consists of the following parts: image segmentation, feature point extraction, navigation line calculation, and dynamic segmentation horizontal strip number feedback.

2.1. Image Acquisition

The image acquisition was performed in Panyang Town, Wuzhishan City, Hainan Province, the People’s Republic of China, and the coordinates of this location are (109.404223 E, 18.880396 N). The soil type of the field environment is sandy loam, and the crop variety is Guyin No. 4 Cigar. The planting method of crops in this study is the parallel ridge method, with four different high-ridge environment videos captured. The shooting description of the four videos is listed in Figure 1 and

Table 1. Shooting conditions of four different sample videos.

Sample Name	x Day of Crop Transplanting	Lighting Conditions	Graphic
A	3	Cloudy	Figure 1a
B	18	Cloudy	Figure 1b
C	33	Cloudy	Figure 1c
D	19	Sunny	Figure 1d

. The selected high-ridge environments were the 3rd day of crop transplanting under cloudy light conditions (sample A, Figure 1a), the 18th day of crop transplanting under cloudy light conditions (sample B, Figure 1b), the 33rd day of crop transplanting under cloudy light conditions (sample C, Figure 1c), and the 19th day of crop transplanting under sunny light conditions (sample D, Figure 1d).

As the weather is an uncontrollable factor, videos of a light control group were collected on two separate days. The crop growth conditions did not change much between transplanting day 18 and transplanting day 19, so they were regarded as the same crop growth condition. To improve the efficiency in obtaining the image samples, the image samples were obtained by video frame extraction: one image was extracted every 30 frames, and the extracted images were processed using a resampling method. The digital image resolution was 960 pixels × 544 pixels, and the image information was stored in the RGB colour space in the JPG file format with a bit depth of 24. The capture device was a CMOS machine vision camera. The model, company, and country of the CMOS vision camera are IMX334, LUORI, and China, respectively. The CPU was an Intel^® Core™ i7-8700 processor with a base frequency of 3.20 GHz. The programming language was Python, the computer vision and machine learning software library was OpenCV, the code editor was Visual Studio Code, and the program ran on the Windows 10 operating system.

2.2. Image Segmentation

2.2.1. Image Grey Processing and Binarisation

The main purpose of grey processing is to emphasise the area(s) of crops in the image while weakening the rest of the image. The CIE-Lab colour space is selected for this study and is independent of the equipment [28]. An image in the CIE-Lab colour space includes separate L*, a*, and b* components; among these, the a* component is sensitive to green information. An image is randomly extracted from the B sample video, and this image is used to explain the processing method. The original image of the cloudy environment is shown in Figure 1b. The colour space of an image is converted from the RGB colour space to the CIE-Lab colour space, and the a* component of the CIE-Lab colour space is separated. The value range of a* component is quantised to [0, 255], and the extracted a* component image is the desired grayscale image, as shown in Figure 2a. In the grayscale image, the target crop can clearly be distinguished from the background. The histogram shown in Figure 2b shows two evident peaks and one narrow trough. Therefore, the histogram is bimodal, and this distribution condition of the bimodal histogram is favourable for binarisation. In this study, OTSU is selected as the algorithm for image binarisation. OTSU does not require data training and learning and can automatically determine the appropriate threshold during the threshold segmentation process. Because it is simple to implement, it is widely used in image segmentation with various features. It can be seen from Figure 2a that the crop features appear as dark areas in the grayscale image. The binarisation processing method is shown in Equation (1) as follows:

$$B(i,j)=\left\{\begin{array}{c}0,p(i,j)<thresh\\ 255,p(i,j)\ge thresh\end{array}\right.$$

(1)

In the above, p (i, j) represent the pixel values of point (i, j) in the image, where i = 0, 1, 2, ..., W and j = 0,1, 2, ..., H. W represents the width of the image and H represents the height of the image. I represents the grey value of point (i, j) in the image; B(i, j) represents the grey value of the corresponding point (i, j) in the image after binarization; and the optimal threshold thresh is used to divide all pixels in the grayscale image into two parts: plant and background. The value of threshold is calculated by OTSU. It can be seen that the image segmentation method can effectively separate the crop rows and the background. In the binary image, the area with B(i, j) = 0 is the background (black), and the area with B(i, j) = 255 is the plant or noise point (white). The binary image obtained according to the above method is shown in Figure 2c.

2.2.2. Morphological Operations

In the above process, green weeds may introduce some noise, and uneven lighting conditions may cause obvious holes in crop row features in the binary image. A morphological opening operation can eliminate isolated small noises in the image, and a morphological closing operation can effectively fill the holes. Noise and holes can be removed by utilizing their respective advantages. The closing operation is first performed on the image, and then the opening operation is performed on the processed image. If the above operations are performed, the noise can be effectively removed and the holes can be effectively filled. Compared with the binary image shown in Figure 2c, it can be seen that the noise and holes in the morphological image shown in Figure 2d are significantly reduced. A few white areas on the edge of the crop row are removed, but the position of the crop row in the image remains the same overall, and the position of the centreline of the crop row remains almost unchanged.

2.3. Feature Points Extraction

Determining the feature points of crop rows is an important prerequisite for the extraction of the crop row centreline. As shown in Figure 1a–d, the actual crop planting method is ridge planting, so the spatial position relationship between each crop row is almost equal to the corresponding linear parallel relationship. In the same row, crops are planted on the top surface of the crop ridge in turn and are equidistant in the same straight-line direction. Therefore, it is best to choose the rhizome of each crop to characterise the crop row. However, the leaves of the crop block the rhizome part of the crop, so the camera cannot capture the rhizome part of each crop. Crop leaves generally exist in the cylindrical space along with the rhizome part as the central axis. Therefore, the edge points on both sides of the crop row need to be connected, and then the midpoint of the line should be extracted. The midpoint can be used as a feature point instead of the rhizome. The leaves of broad-leaved plants are irregular and large, so the feature points on the same plants can be added to improve the accuracy of the crop line centreline fitting. The details of the process are as follows.

(1)

The number of horizontal strips of this frame N_D is calculated according to the dynamic feedback mechanism between two adjacent frames. The specific process is as follows: N_D horizontal strips are obtained after isometric segmenting the image from bottom to top. The value of N_D is calculated according to the feedback mechanism at the last part of the processing of the previous frame. The rule of this mechanism is described in Section 2.5.

(2)

The top region is removed and the horizontal strip height D is calculated based on the number of horizontal strips N_D. As the crop rows at the top of the image are dense and cannot accurately represent the locations of the crop rows, the pixel values of all points in the top region of the image are set to 0. The points of the top region with pixel values set to 0 are denoted as P (i, j), where i = 0, 1, 2, ..., W and j = 0, 1, 2, ..., 100. The horizontal strip height D is calculated according to Equation (2).

$$D=\frac{H-100}{N_D}$$

(2)

(3)

The N_D horizontal strips are divided within the binary image, and all of the divided horizontal strips are numbered in a bottom-up order, for example, segment 0, segment 1, segment 2, ..., segment N_D − 1. In the image coordinate system u − v as shown in Figure 3, the upper boundary of the x-th segment horizontal strip is UppB = H − (x + 1)D, the lower boundary is LowB = H − xD, the left boundary is LefB = 0, and the right boundary is RigB = W. The strip from the original image is shown in Figure 3a, and the strip from binary image is shown in Figure 3b.

(4)

The vertical projection curve of the x-th horizontal strip is calculated and is denoted as the x-th vertical projection curve. The x-th horizontal strip described above is actually a binary image. f (i, j) represents the value of point (i, j) in the x-th horizontal strip. The process initialises x = N_D − 1 and finds the points with f (i, j) = 255 in the x-th horizontal strip, and accumulates the points with f (i, j) = 255 in the same column onto the column axis u to obtain the vertical projection curve as shown in Figure 3c. Let p(i) be the number of pixels with a pixel value of 255 in the i-th column of the binary horizontal stripes. The calculation of p(i) is shown in Equation (3), where i = 0, 1, 2, …, u; x = N_D − 1, …, 2, 1, 0.

$$p\left(i\right)=D-\sum _{\dot{J}=(D-1)x}^{Dx}min(255-f\left(i,j\right),1)$$

(3)

(5)

The average projected value avgp (x) of p(i) is calculated for all columns of the x-th vertical projection curve. A larger and higher peak area is identified as the crop characteristic area, and the crop characteristic area is simplified into a point. This point is the crop feature point. avgp (x) is calculated as follows:

$$avgp\left(x\right)=\frac{1}{\mathrm{W}}\sum _{i=1}^{w}p\left(i\right)$$

(4)

(6)

P_S and th_s (according to the following definitions) are calculated, and after a threshold m(x) is calculated according to Equation (5), the ordinate value Y_x of each feature point in the x-th horizontal strip can be calculated according to Equation (6). The image area threshold th_s is determined by repeated experiments, and P_S is the area ratio of the number of pixels with a grayscale value of 255 in the binary horizontal strip to the total number of pixels in the binary horizontal strip. In the x-th vertical projection curve, a horizontal line is set to pass through the curve, and the ordinate of the horizontal line is m(x), as shown in Figure 3c. When the crop is in the seedling stage, the crop’s leaves are small within the image and there are relatively few weeds, so selecting an appropriate m(x) can effectively avoid the influences of multimodal peaks on the feature points. When the crop is mature, a larger m(x) can filter out disturbances owing to the leaves and weeds of the crop. These processes above comprise a method for determining the ordinate values of feature points based on area threshold (hereafter ‘MAT’). m(x) is calculated as follows:

$$m\left(x\right)=\left\{\begin{array}{c}avgp\left(x\right), P_s<{th}_S\\ D-avgp\left(x\right),P_s\ge {th}_S\end{array}\right.$$

(5)

$$Y_x=UppB+m\left(x\right)$$

(6)

(7)

The value size relationship between p(i) and m(x) is analysed. The x-th vertical projection curve is simplified according to Equation (7), and a vertical projection simplified curve after the threshold is obtained as shown in Figure 3d.

$$B\left(i\right)=\left\{\begin{array}{c}0,P\left(i\right)<m\left(x\right)\\ 1,P\left(i\right)\ge m\left(x\right)\end{array}\right.$$

(7)

(8)

The abscissa value X_x is calculated for each feature point in the x-th horizontal strip. Each continuous data set with a B(i) value of 1 is extracted. The abscissa of the feature point is the average value of the abscissa u corresponding to the continuous data set. The noise judgment threshold th_n is used as the minimum width judgment indicator to identify the continuous data set. The noise judgment threshold is used to filter the noise interference of weeds, branches, and leaves. The value of th_n is determined based on experiments. If a cardinal number of the continuous data set is smaller than the value of th_n, the continuous data set is judged as noise interference and discarded. The equation for calculating the abscissa of the feature point is as follows:

$$p{\_mid}_c=\frac{1}{n_k}\sum _{u_1}^{u_2}i,n_k\ge {th}_n$$

(8)

In the above, p_mid_c represents the abscissa of the feature point numbered c in the horizontal strip of the x-th segment; u₁ is the starting point of the continuous data set; u₂ is the end point of the continuous data set; n_u is the number of the continuous data set; and n_u = u₂ − u₁. The final feature points are shown in Figure 3d and are distributed as shown in Figure 3e in the original image.

Steps (6)–(8) in fact represent the process of determining the abscissa of the feature points. This method is called the method of feature point extraction based on noise judgment threshold (hereafter abbreviated as MNJT).

(9)

The feature point set SET_FP = {(p_mid₀, Y_x), (p_mid₁, Y_x), (p_mid₂, Y_x), …, (p_mid_Q-1, Y_x)} is output. Q is the number of feature points in the x-th horizontal strip.

(10)

After setting x = x − 1, steps (4)–(9) are repeated in a loop until x < 0; then, the module program stops running.

Due to the fact that the performance of feature point extraction in this study is mainly determined by th_s and th_n, it is necessary to determine these two parameters. Twenty images are extracted from videos of samples A, samples B, samples C, and samples D, and feature point extraction is carried out according to the method in this article. For each sample, we observe and count whether the position of each feature point in each image is at the crop position in the image, then calculate the ratio of the number of feature points that are not in the crop position to the number of all feature points in the sample; the ratio is called the error rate of feature point extraction and is recorded as Rfpe. When the initial test (th_s, th_n) combination is (35%, 5), it has a good feature point extraction effect. Using this combination as the centre to expand the parameters for fine screening, the Rfpe of the four samples is shown in

Table 2. Statistical results of error rate for feature point extraction Rfpe, and its unit is %.

Samples	thn	ths	ths	ths	ths	ths	ths	ths
		25%	30%	35%	40%	45%	50%	55%
A	1	0.2086	0.2086	0.3194	0.3899	0.3081	0.4198	0.9800
	2	0.1827	0.1728	0.2927	0.3621	0.3049	0.4091	0.8591
	3	0.1302	0.1172	0.2410	0.3361	0.2982	0.4053	0.5156
	4	0.1060	0.0146	0.2170	0.3183	0.2266	0.3803	0.6399
	5	0.1339	0.1355	0.2448	0.3578	0.2687	0.4583	0.7911
	6	0.2589	0.2348	0.3699	0.4110	0.3604	0.4855	0.9188
	7	0.2854	0.3345	0.3960	0.4992	0.3843	0.5125	1.1292
B	1	0.0353	0.029	0.0211	0.0343	0.0303	0.0288	0.0308
	2	0.0225	0.0215	0.0274	0.0292	0.0294	0.0363	0.0383
	3	0.0242	0.0232	0.0263	0.0213	0.0331	0.0287	0.0307
	4	0.0218	0.02	0.0247	0.0342	0.0201	0.0199	0.0219
	5	0.0211	0.0201	0.0191	0.0199	0.0218	0.0259	0.0279
	6	0.0243	0.0233	0.0255	0.0335	0.0254	0.0284	0.0304
	7	0.0288	0.0278	0.0291	0.036	0.0324	0.0345	0.0365
C	1	0.0581	0.0535	0.0603	0.0593	0.0507	0.0557	0.0686
	2	0.044	0.0521	0.0565	0.0555	0.0464	0.0495	0.0579
	3	0.0424	0.0406	0.0501	0.0491	0.0432	0.0391	0.051
	4	0.0366	0.0393	0.0391	0.0381	0.0269	0.0272	0.0399
	5	0.0399	0.0315	0.037	0.036	0.0416	0.0328	0.0377
	6	0.0305	0.0304	0.0248	0.0238	0.0285	0.0277	0.0281
	7	0.0298	0.0274	0.0333	0.0323	0.0346	0.0355	0.0416
D	1	0.0303	0.0262	0.0232	0.0372	0.0358	0.0272	0.0325
	2	0.0251	0.0169	0.0206	0.0259	0.0286	0.0311	0.0316
	3	0.0227	0.0219	0.0179	0.0221	0.0242	0.0254	0.0304
	4	0.0246	0.0177	0.0213	0.0263	0.0259	0.022	0.0307
	5	0.0157	0.0186	0.0145	0.0239	0.0238	0.0194	0.0284
	6	0.0314	0.0266	0.0207	0.0297	0.0248	0.0178	0.0299
	7	0.0366	0.0345	0.0332	0.0285	0.0343	0.0268	0.0396

. It can be seen that when the combination is (30%, 4), (35%, 5), (40%, 6), (35%, 5), the Rfpe of the four samples is the smallest. Taking the average of the two parameters of the above four combinations to obtain (35%, 5), two parameters can be determined as follows: th_s = 35%, th_n = 5. In addition, when th_s = 35% and th_n = 5, Rfpe does not exceed 0.0240%, indicating that the feature point extraction method combined with this parameter has good adaptability to the four environments and the feature point extraction effect is very good.

2.4. Navigation Line Calculation

2.4.1. Feature Points Clustering

After obtaining the feature points, the next goal is to assign these points to the two crop rows at the centre of the image. To determine the feature points required for each fitted crop row centreline, we propose a method combining the adaptive clustering method and the dynamic segmentation point clustering method.

As shown in Figure 3e, when the crop leaves are large and uneven in shape, the above method may extract multiple feature points in the same crop. If multiple feature points represent the same crop, it will not only increase the number of calculations required for the fitting line but will also affect the straight-line fitting results. In addition, the feature point set extracted by the vertical projection method does not classify the feature points, so the line-fitting process cannot be conducted further.

In view of the above problems, the adaptive clustering method is used to cluster the multiple feature points in the same horizontal strip of the same crop into one feature point. The following step (1) is an explanation of the adaptive clustering method. The dynamic segmentation point clustering method is used to assign the clustered feature points to the corresponding feature point set. The following step (2) is an explanation of the dynamic segmentation point clustering method. The specific process is as follows.

(1)

In the clustering process, the horizontal strips from the feature point extraction step are used as units to be traversed. The horizontal strip is traversed from bottom to top, and each feature point is traversed sequentially from left to right in the horizontal strip. The number of feature points of the K-th horizontal strip is set as Q_K, and P_K,m (m = 0, 1, 2, …, n_K) represents the m-th feature point in the K-th horizontal strip. The distance between the m-th feature point and m + 1-th feature point in the K-th horizontal strip can be expressed as d_{K,m,(m + 1)}, where m = 0, 1, 2, …, n_K − 1. The average value of the distances between all adjacent feature points in the same horizontal strip can be expressed as d_K,avg, as shown in Equation (9). As shown in Figure 4, a clustering threshold th_div is established for comparison with the distance d_{K,m,(m + 1)} between each adjacent feature point. For the feature points in the K-th horizontal strip, when d_{K,m,(m + 1)} > th_div, the m value is recorded, and according to all of the recorded m values, the feature points can be divided into several feature point sets. The equations for calculating th_div and d_K,avg are as follows, and F is a coefficient used to calculate th_div:

$$\left\{\begin{array}{c}{th}_{div}=F{d}_{K,avg}\\ d_{K,avg}=\frac{1}{n_k}{\displaystyle \sum _0^{m+1}}{d}_{K,m,\left(m+1\right)}\end{array}\right.$$

(9)

Because the image crop rows are arranged in a vanishing point distribution, the average distance between adjacent feature points in the horizontal strip at the top of the image is less than the average distance between adjacent feature points in the horizontal strip at the bottom of the image. The fixed value of F cannot adapt to each horizontal strip, so it is necessary to set the value of F as a dynamic changing value. After a large number of experiments, an empirical calculation for the value of F is obtained as follows:

$$F_K=F_{K+1}-{\mathsf{\Delta }}_F$$

(10)

Here, F_K is the value of F of the K-th horizontal strip, and F_{K + 1} is the value of F of the K + 1-th horizontal strip. Δ_F is the difference parameter.

(2)

segmentation midpoint p_div is established. According to p_div, the adjacent feature point sets on the left and right sides (B and C sets in Figure 4, respectively) can be obtained. The left feature point set is denoted as Cleft, and the right feature point set is denoted as Cright. At the 0-th horizontal strip, p_div,0 is the rounded value of W/2. From the first horizontal strip to the last horizontal strip, the p_div of the current horizontal strip is set as the average point value of the set established based on the p_div values of all of the previous horizontal strips, and the split midpoint of the K-th horizontal strip is p_div,K. The calculations for Equation (11) are as follows:

$$p_{div,K}=\left\{\begin{array}{c}int\left(\frac{W}{2}\right), K=0\\ \frac{1}{K}{\sum }_{i=0}^{K-1}{p}_{div,i},K>0\end{array}\right.$$

(11)

The number of feature points Nc in the feature point sets (A, B, C, D in Figure 4) are respectively calculated, where Ncleft is the number of points in the left feature point set Cleft, and Nright is the number of points in the right feature set Cright. The feature points of Cleft and Cright are clustered separately according to the following same rules.

When Nc = 2, the distances between two adjacent points within the feature point set are calculated. If the distance is less than or equal to W/5, the abscissa of the set clustering point is the average of the abscissas of the two points inside the set. If the distance between two points in the feature point set is greater than W/5, these two points belong to two different crop rows. For the left feature point set, the abscissa of the set clustering point takes the larger value of the abscissa of the two points in the set. For the right feature point set, the abscissa of the set clustering point takes the smaller value of the abscissa of the two points in the set.

When Nc > 2, the abscissa of the clustering points in the set is the average of the abscissas of all points in the set.

The ordinate of the above cluster of points is consistent with the ordinate of the original feature points. Finally, two cluster feature points near the central crop row in the horizontal strip are obtained, as shown in Figure 4, i.e., point II and point III. Each horizontal strip is extracted according to the above rules. The extraction results for the original feature points are shown in Figure 5a. The left and right clustering points are assigned to the left and right clustering feature point sets, SetFit_Left and SetFit_Right, respectively, and the clustering results and classification results are shown in Figure 5b.

Due to the fact that the performance of the clustering method in this study is mainly determined by F and Δ_F, it is necessary to filter for these two parameters, extracting 20 images from the videos of samples A, samples B, samples C, and samples D, respectively. After extracting feature points according to the method in this article, the feature points are clustered. For each sample, we observe and count whether the position of each cluster point in each image is in the ideal cluster position, and then calculate the ratio of the number of cluster points that are not in the ideal position to the number of all cluster points in the sample. This ratio is called the clustering error rate, denoted as Rfa. When the combination is (1, 0.04), it has a good clustering effect. Based on this combination, the parameters are expanded for fine screening. The statistical results of clustering error rates for the four samples are shown in

Table 3. Statistical results of cluster error rate Rfa, and its unit is %.

Samples	ΔF	F	F	F	F	F	F	F
		0.7	0.8	0.9	1	1.1	1.2	13
A	0.01	0.5310	0.3916	0.3328	0.1407	0.0254	0.1417	0.2657
	0.02	0.4623	0.3273	0.2757	0.1354	0.0169	0.1668	0.2045
	0.03	0.3535	0.3191	0.2032	0.1763	0.0117	0.1963	0.2322
	0.04	0.2920	0.2572	0.1830	0.1279	0.0090	0.1625	0.2347
	0.05	0.3225	0.2660	0.1958	0.1541	0.0183	0.1916	0.2421
	0.06	0.4811	0.3042	0.2043	0.1855	0.0558	0.1319	0.2891
	0.07	0.5204	0.3447	0.2470	0.1385	0.1259	0.1423	0.2157
B	0.01	0.0234	0.0210	0.0331	0.0203	0.0301	0.0276	0.0252
	0.02	0.0193	0.0167	0.0245	0.019	0.0221	0.0253	0.0246
	0.03	0.0174	0.0154	0.0174	0.0163	0.0153	0.0232	0.0196
	0.04	0.0162	0.0153	0.0147	0.0186	0.0219	0.0174	0.0158
	0.05	0.0141	0.0146	0.0145	0.0138	0.0144	0.0141	0.014
	0.06	0.0238	0.0236	0.0187	0.0179	0.0233	0.0215	0.0173
	0.07	0.0308	0.0272	0.0193	0.0276	0.0255	0.0306	0.022
C	0.01	0.0368	0.042	0.0434	0.0429	0.05	0.0389	0.0401
	0.02	0.0294	0.0352	0.0357	0.0345	0.0408	0.0342	0.0339
	0.03	0.0256	0.0334	0.0317	0.0249	0.0334	0.033	0.0258
	0.04	0.0333	0.0318	0.0339	0.0301	0.0278	0.0255	0.0321
	0.05	0.0243	0.0239	0.0243	0.0245	0.0239	0.0230	0.0242
	0.06	0.0252	0.0315	0.0273	0.0258	0.0302	0.0273	0.0302
	0.07	0.0282	0.0401	0.0292	0.0314	0.0367	0.0305	0.0347
D	0.01	0.0124	0.0104	0.0074	0.0143	0.0096	0.0077	0.015
	0.02	0.0055	0.0052	0.0046	0.0094	0.005	0.0034	0.0055
	0.03	0.002	0.0019	0.0012	0.0018	0.0011	0.0016	0.002
	0.04	0.0071	0.0076	0.0062	0.0066	0.0027	0.008	0.0077
	0.05	0.0099	0.0142	0.0117	0.0139	0.0114	0.0092	0.0114
	0.06	0.0166	0.018	0.0161	0.0216	0.0182	0.0103	0.0156
	0.07	0.0202	0.0234	0.0216	0.0241	0.0192	0.0171	0.0196

. It can be seen that when the combinations are (1.1, 0.04), (1, 0.05), (1.2, 00.05), and (1.1, 0.03), the Rfa of the four samples is the smallest. Taking the average of the two parameters of the above four combinations to obtain (1.1, 0.0425), two parameters can be determined: F = 1.1, Δ_F = 0. 0425. Furthermore, when F = 1.1, Δ_F = 0.0425, Rfa of all four samples does not exceed 0.0300%, indicating that the clustering method with this parameter combination has good adaptability to the four environments and the clustering effect is very good.

2.4.2. Feature Point Optimisation and Linear Fitting

The clustering feature point sets of the left and right crop rows are determined. The next problem is simplified to how to obtain the corresponding linear equation for each crop row. Although the above three steps (feature point extraction, morphological processing, and clustering methods) can reduce interference to a certain extent, there may still be interference points in the set of clustering feature points. Therefore, the anti-interference ability of the above method needs to be further strengthened. To further strengthen the robustness of the method, it is necessary to further optimise the clustering feature points. The step of line fitting will be completed in the optimisation process for the clustering feature points and is described below.

(1)

Horizontal distance optimisation

In the pixel coordinate system, the main characteristic of the interference point is that its abscissa deviates from the abscissa of most feature point groups; thus, the interference point can be eliminated according to this characteristic. As the left and right cluster feature point sets (SetFit_Left, SetFit_Right) are generated from bottom to top according to the vertical projection method, in the ordinates of each point in the cluster feature point set, the ordinate value of the former point is always larger than the ordinate value of the latter point. The abscissa differences A of the two adjacent points P_m and P_{m + 1} in the cluster feature point set are calculated in turn, and the distance threshold th_hori is calculated according to the average value avg(A) of all of the differences. The size relationships between all of the differences A and distance threshold th_hori are compared. If A > th_hori, the point P_{m + 1} is removed; otherwise, the point P_{m + 1} continues to be retained in the cluster feature point set, and the value of P_{m + 1} is assigned to the value of P_m and continues to be iteratively calculated according to the above method. The left and right clustering feature point sets are optimised according to the above rules to obtain the left and right lateral distance optimisation point sets (Set_optimised_Left, Set_optimised_Right). The calculation method for the distance threshold th_hori is shown in Equation (12).

$${th}_{hori}=1.1\times avg\left(A\right)$$

(12)

(2)

Point-line distance optimisation based on the Huber loss function

The steps for the point-line distance optimisation are as follows. First, the straight-line fitting process is performed on the left lateral distance optimisation point set Set_optimised_Left, and the first fitting straight line L1 = k₁x + b₁ is obtained. Then, the distance optimization threshold th_dis is established, and the distances between all points of Set_optimised_Left and line L1 are traversed and compared with the distance threshold th_dis. If the distance is greater than th_dis, the point is removed; otherwise, the point remains in Set_optimised_Left. For a single set of data, the distance threshold th_dis is calculated as shown in Equation (13). The right lateral distance optimisation point set Set_optimised_Right is processed in the same way.

$${th}_{dis}=\frac{1.2}{m-1}\sum _0^m(\frac{\left|k_{1}i-j+b_1\right|}{\sqrt{k_1\left(k_1+1\right)}})$$

(13)

Here, m is the number of the point in Set_optimised_Left; the value is [0, m], i is the abscissa of the point, and j is the ordinate of the point. The data after eliminating the deviation points are linearly fitted again, and ultimately, the optimised straight line L2 (the second fitting straight line) is obtained. The straight line L2 is the centreline of the crop row.

Many studies have selected the LSM as the method for linear fitting [29]. However, the LSM has limitations. First of all, because the LSM gives the same weight to each point in the fitting process, it is only suitable for cases with high correlation. It is not suitable for cases where the data are discrete and there are singular values. In addition, as the LSM solves the residual sum of squares to achieve the minimum solution for the regression coefficient, it is easy to exaggerate the influence of singular values in the experimental data, and the statistical error increases; thus, the LSM is relatively lacking in robustness.

The basic idea of robust regression based on the M-estimator is to use an iterative weighted LSM to estimate the regression coefficient. The weight of each sample is determined according to the size of the regression residual, so as to achieve robustness. ρ(r_m) is the deviation function, which defines the cost function cost(L) of Huber-M estimation as follows:

$$cost(L)=\sum _0^m\rho \left(r_m\right)$$

(14)

If line L fits a straight line, k and b are calculated so that cost(L) takes the minimum value. ρ(r_m) is the Huber function, and the expression is as follows:

$$\rho \left(r_m\right)=\left\{\begin{array}{rr}\frac{{r_m}^2}{2},& r_m<C\\ C(r_m-\frac{C}{2}),& r_m\ge C\end{array}\right.$$

(15)

C is an adjustment parameter. In OpenCV, the fitLine function can be used to fit a point set to a straight line, and the function can be automatically adjusted to the optimal C value. Here, the system takes the distance type as Cv.DIST_HUBER, distance parameter as 0, radial accuracy as 0.01, and angular accuracy as 0.01. The output comprises four parameters: sinα, cosα, x₀, and y₀. sinα and cosα are the sine and cosine values of the inclination angle α of the fitted line, respectively, and x₀ and y₀, are the abscissa and ordinate values of the point (x₀, y₀) passing through the fitted line, respectively. Therefore, the equation of fitted straight line L can be expressed as follows:

$$y=\left(\frac{sin\alpha }{cos\alpha }{x}_0-y_0\right)+\frac{sin\alpha }{cos\alpha }x$$

(16)

2.4.3. Calculation of Navigation Lines Based on Fitted Straight Lines

After the fitting line equations for the left and right ridges are obtained, the angle bisector equation for the included angle of the fitting lines can be obtained according to the two straight line equations. The angle bisector equation represents the navigation reference line. The derivation equation for the slope of the angle bisector is as follows:

$$\left|\frac{k-k_1}{1+k_{1}k}\right|=\left|\frac{k-k_2}{1+k_{2}k}\right|$$

(17)

k is the slope of the navigation line; k₁ is the slope of the fitting straight line of the left ridge; and k₂ is the slope of the fitting straight line of the right ridge.

By determining the intersection of these two lines (left and right), we can obtain the equations for the two angle bisectors. As simplified by Equation (17), we obtain a calculation as follows:

$$\left(k_1+k_2\right)k^2+2\left({1-k}_1{k}_2\right)k-\left(k_1+k_2\right)=0$$

(18)

When Equation (18) has a solution, the solution is the slope of the navigation line. According to the Vieta theorem, when there are two solutions S₁ and S₂ for Equation (18), a calculation holds as follows:

$$S_1\times S_2=-\frac{\left(k_1+k_2\right)}{\left(k_1+k_2\right)}=-1$$

(19)

S₁ and S₂ are negative reciprocals of each other, and the relationship between the corresponding two navigation lines to be selected (L_NA1, and L_NA2) is a vertical relationship in a plane. According to the equations for two fitting straight lines (L_F1 and L_F2), the intersection point P_in of the two fitting straight lines can be obtained, and the corresponding two navigation line equations L_NA1 and L_NA2 can be obtained according to the two solutions of P_in and slopes (S₁ and S₂). The selection rules of the required navigation line equations are as follows:

The straight line between L_NA1 and L_NA2 located in the middle of the two ridges is selected. The selected straight line is the navigation line. Concretely, the method respectively calculates the intersection points of L_F1, L_F2, L_NA1, and L_NA2 with the u-axis in the pixel coordinate system u − v. If one of the straight lines L_NA1 and L_NA2 intersects the u-axis in the middle of the intersection of L_F1 with the u-axis and the intersection of L_F2 with the u-axis, the straight line is a navigation line.

2.5. Dynamic Feedback Mechanism between Two Adjacent Frames

The value for the number of horizontal strips is generally fixed in existing studies and is often adjusted according to the crop growth period or weed conditions. Because of the fixed number of horizontal strips, the height value is also a fixed value.

As shown in Figure 6, when the number of horizontal bands is fixed to 20, in images with small plants and large plant spacing, the area ratio of crop information to the image is small. After the image feature point optimization step, the extracted feature points for a single plant will be too few, and too few feature points will affect the linear fitting effect. In Figure 6, the blue line represents the left crop line, the red line represents the right crop line, and the yellow line represents the navigation line. The meaning of the straight line markings in subsequent field images is the same as that in Figure 6.

As shown in Figure 7, when the number of horizontal strips is fixed to 20, the probability of abnormal points appearing in images with larger plants and smaller plant spacing is high. During the feature point optimization process, abnormal points will be eliminated. If too many abnormal points are eliminated, the number of feature points used for line fitting will be too small, thereby affecting the line fitting effect. This problem can usually be solved by increasing the number of horizontal strips to increase the feature points of a single plant. However, for environments with large plants, blindly increasing the number of horizontal bars can lead to a larger number of individual plant feature points in a larger image of the plant, increasing processing time. Therefore, a fixed number of horizontal stripes cannot adapt to the environment for all periods, and dynamic adjustment of the number of horizontal stripes is necessary. The expected strategy is as follows: For environments with small plants, the number of horizontal bars should be increased, while for environments with large plants, an appropriate value for the number of horizontal bars should be determined. For environments with medium plants, the number of horizontal bars should be between the number of horizontal bars in environments with small plants and environments with large plants.

Twenty images each are extracted from sample A, sample B, sample C, and sample D videos. Using the horizontal bar number value Nv as a variable, the probability of abnormal feature points appearing in all images in each sample, expressed in Prob, is calculated. The image processing time of all images in the four samples is recorded and the average value, expressed in Thor, is calculated. The statistical results of the two indicators are shown in Figure 8. The Prob values of the four samples decrease as Nv increases. When Nv = 30, the Prob values of the C sample will stabilize at a lower-level range at the latest. Thor increases as Nv increases, and when Nv = 70, Thor increases significantly. Therefore, when the value range of Nv is within [30, 70], it can ensure both a small probability of occurrence of abnormal points and timeliness.

To ensure that each environment has an appropriate number of horizontal bands, this study introduces a self-created horizontal band dynamic quantity feedback mechanism; the feedback mechanism can automatically determine the number and height of the appropriate horizontal bands for each environment. The main idea is to form a closed-loop feedback mechanism between the output process of the previous frame and the calculation process of the number of horizontal bands in the next frame.

The following strategies are used to keep the value of Nv within the range of [30, 70]: when there are too few feature points in the previous frame, the number of horizontal segments in the next frame will be increased after calculation according to this mechanism to ensure that there are enough feature points for straight-line fitting in the future; when there are too many feature points in the previous frame, the number of horizontal bars in the next frame will be reduced after calculation according to this mechanism to ensure the timeliness of the program. The specific process of this mechanism is as follows.

(1)

The algorithm initialises a variable (NUMBER) representing the number of horizontal strips, and the algorithm starts with the initial NUMBER = 30 in the first frame.

(2)

After the image processing mentioned above, the number of left feature points (num_left) and number of right feature points (num_right) of this frame image can be obtained.

(3)

The average value of num_left and num_right is taken, and denoted as avg_feature.

(4)

The number of horizontal strips required to divide in the next frame image (denoted as num_next) is calculated according to the following feedback mechanism Equation (20). At each round, NUMBER= num_next, and the next frame image processing commences and segments the horizontal strips according to NUMBER. This loop continues until the termination command is issued.

$$\left\{\begin{array}{c}num\_next=-2\times avg\_feature+70,ifavg\_feature\le 10\\ num\_next=-1.5\times avg\_feature+70,if10<avg\_feature<20\\ num\_next=30,ifavg\_feature\ge 20\end{array}\right.$$

(20)

The num_next values of 20 images are recorded from sample A, sample B, sample C, and sample D, and the average value is calculated to obtain the experimental results in

Table 4. Statistical results of the average value of num_next.

Sample Naming	Sample A	Sample B	Sample C	Sample D
The average value of num_next	40.45	36.15	34.25	37.45
Crop size	Small	Medium	Large	Medium
Lighting conditions	Cloudy	Cloudy	Cloudy	Sunny

. A comparison of data from samples A, B, and C shows that the average value of num_next is within the expected range [30, 70]. The average value of num_next decreases because the crop size increases, meeting the requirements of the expected strategy. In addition, by comparing the data of samples B and D, the average values of num_next of the two samples are not significantly different, so changes in lighting conditions will not affect the effectiveness of this mechanism.

In summary, the flowchart of the entire detection algorithm is shown in Figure 9.

3. Results and Discussion

3.1. Image Preprocessing

Figure 1d shows the original image of the sunny environment. High ridges in a sunny environment are easily affected by light conditions and generate many shadows. Figure 10a is the grey image obtained by extracting the a* component from the original image of the sunny environment. The grey image reflects the overall location of the crops. Figure 10b shows the binary image obtained after using the image segmentation method proposed in this study to process Figure 10a. The binary image shows that the contour of the crop row extracted by this method is complete. Combined with the analysis shown in Figure 10 for the cloudy environment, it can be seen that the image segmentation method proposed herein can adapt to both cloudy and sunny light conditions.

An ideal binary image is noiseless. The less noise, the better the accuracy of the crop row detection and the better the reduction of the calculation amount for the subsequent process. From a qualitative perspective, the binary image generated by the a* component has very little noise. From a quantitative perspective, the noise generally exists as small and isolated connected areas in a binary image, so this type of connected areas can be used to identify the noise in a binary image. An isolated connected area of less than 50 pixels in the binary image is extracted as a noise area. We extracted 40 images from the videos of samples B and C, respectively, and identified the connected area with an area of less than 50 pixels per image. The numbers and total areas of the identified connected area are recorded, and the average, maximum, and minimum corresponding to the numbers and total areas of the connected noise regions are shown in

Table 5. Noise analysis results.

Lighting Conditions	Number of Connected Noise Areas			Total Area of Connected Noise Areas
	Average	Maximum	Minimum	Average	Maximum	Minimum
Cloudy environment	10	25	2	67	153	2
Sunny environment	48	71	29	293	496	113

.

Compared with the number of total pixels in a single image (960 × 544 pixels = 522,240 pixels), all values of the indicators in Table 5 are very small. Therefore, using the grayscale method and binarisation method described in this paper can effectively separate the crops and the background in an image. The final generated binary image has very little noise, thereby meeting the requirements for use.

3.2. Accuracy Verification Test

As the navigation line is estimated from the crop row lines, verifying the accuracy of the navigation line can be converted to verifying the accuracy of the crop rows. To obtain a more convincing verification of the accuracy of this research method, the performance of the linear fitting method in this study and of the LSM method [16] is analysed from two perspectives: qualitative and quantitative. We extracted 20 images from the videos of samples A, B, C, and D, respectively. The two types of crop row extraction methods analysed are different from the linear fitting method, and the other parts use the same method described herein. The proposed research method uses the point-line distance optimisation based on the Huber loss function described above to fit the feature point sets, whereas the method for comparison directly uses the traditional LSM straight-line fitting method to fit the feature point sets. For the convenience of expression, the proposed research method based on point-line distance optimisation based on the Huber loss function is referred to below as HUBERP, and the method based on LSM for comparison is referred to as LSMC.

As shown in Figure 11a–h, in the images of the different growth periods, the inter-plant relationships for crops in the same crop row and the inter-row relationships for different rows are different, and the sizes of the plants are also inconsistent. Further details are provided below.

(1)

The leaf area of sample A is small. The single plant form can be clearly distinguished among plants with no shadows. There are almost no weeds, but the number of leaf surfaces is small.

(2)

The leaf area of sample B is slightly larger with a small amount of shadow, and the leaf surfaces of the crops are connected between plants to a certain extent. There are small weeds.

(3)

The leaf area of sample C is large with many shadows, and the leaf surfaces of the crops are connected between plants and rows to a certain extent. There is a large area of weeds.

(4)

The leaf area of sample D is slightly larger and has many shadows, and the leaf surfaces of crops are connected between plants to a certain extent. There are small weeds.

(5)

From a qualitative perspective, i.e., by comparing the crop row extraction results for the three crop growth periods and two light conditions, it can be seen that HUBERP has strong adaptability. The fitted straight lines detected in this study and shown in Figure 11 are all distributed in the position of the crop row. The straight lines obtained by HUBERP (solid line) have little deviation from the manually marked crop row lines (white dashed line), and the results from HUBERP are basically consistent with expectations. In contrast, only part of the straight lines (solid lines) obtained by the LSMC fit well with the manually marked crop row lines (white dotted lines). Therefore, HUBERP can meet the requirements for crop row line identification in selected environments during different growth periods and different light conditions, whereas the LSMC cannot meet the requirements for the early identification of crop growth.

There are two reasons for the large deviation of the partial line fitting of the LSMC. First, as shown by the black dotted line in Figure 11a, when extracting feature points, there is a large plant spacing between the plants in the same crop row of sample A. When traversing to a certain horizontal strip, the two crop row areas in the middle of the image have no crop features, causing the algorithm to incorporate by error the feature points (deviation points) of the adjacent crop rows into the fitting point set. The feature points of the adjacent crop rows originally do not belong to the fitting point set. Second, when the plant size is large as shown in Figure 11e, there is a connection phenomenon (blue circled in Figure 11e) between adjacent crop rows. This leads to the feature point extraction and the program identifying two crops in different crop rows as one. These two aspects above will introduce outliers into the set of fitted points.

When using the LSMC, outliers usually affect the fitting, because of the square of the offset. HUBERP removes the influence of outliers using an M-estimated robust regression. As shown in Figure 11a–f, this research method can effectively suppress the influence of weeds and soil clods (marked by circles) on the recognition of crop row lines in various periods of broad-leaved plants to a certain extent. Compared with the LSMC, HUBERP can be applied to field environments with different crop growth periods; moreover, as shown in Figure 11c,d,g,h, HUBERP can be applied to field environments with different light conditions. To sum up, HUBERP is more robust than LSMC and is adaptable to different periods and lighting conditions.

The central approach of the quantitative evaluation standard is to compare lines drawn by an expert human with fitted straight lines. We call the lines drawn by the expert human reference lines. To strictly evaluate the similarity between the two, it is necessary to evaluate both the distance and the angle. As shown in Figure 12, the assumption is that the line L_F1 is the left-fitting line and line L_R1 is the corresponding reference line. Line L_R1 intersects with the upper and lower boundaries of the image at T₁ and B₁, respectively. Line L_F1 intersects the upper and lower boundaries of the image at T₂ and B₂, respectively. The deviation angle θ between the fitted straight line and reference line reflects the accuracy of the angular deviation. θ_L represents the deviation angle between L_F1 and L_R1. d₁ represents the distance between T₁ and L_F1, and d₂ represents the distance from B₁ to L_R1. k_F1 and b_F1 are the slope and intercept of L_F1, respectively, and k_R1 and b _R1 are the slope and intercept of L_R1, respectively. The angle θ_L is calculated as follows:

$${\theta }_L=arctan(\left|\frac{k_{R1}-k_{F1}}{1+k_{R1}{k}_{F1}}\right|)$$

(21)

θ_L distinguishes between positive and negative values and can represent the relative positions of L_F1 and L_R1, thereby indicating the angular deviation value; θ_absL is the absolute value of θ_L, and can indicate the degree of the angular deviation. The distance d₁ from T₁ to L_F1 and distance d₂ from B₁ to L_R1 are calculated according to a distance equation from a dot to a straight line. L_R1, L_F2, T₃, T₄, B₃, B₄, θ_R, θ_absR, d_3, and d₄, as shown in Figure 12, are the relevant parameters of the right lines, and the meanings and calculation methods for the parameters above are similar to those on the left. The comprehensive evaluation indicator of the distance deviation of each image is represented by lineComp, and is calculated from the average value of d₁, d₂, d_3, and d₄, i.e., lineComp = (d₁ + d₂ + d₃ + d₄)/4. lineComp represents the central tendency of the distance deviation for each image. The maximum value of the average value of d₁, average value of d₂, average value of d₃, and average value of d₄ of all images in a single sample is represented by dmax, i.e., dmax = max [avg(d₁), avg(d₂), avg(d₃)_, avg(d₄)]. dmax reflects the maximum average distance deviation of each sample. The angular deviation comprehensive evaluation index angComp is calculated from the mean values of θ_L and θ_R, i.e., angComp = (θ_L + θ_R)/2. Corresponding angComp_abs is calculated from the average value of θ_absL and θ_absR, i.e., angComp = (θ_absL + θ_absR)/2. angComp and angComp_abs represent the angle deviation value of the overall images and the degree of deviation in the overall images, respectively.

The distance deviation analysis proceeded as follows. Two methods (HUBERP and LSMC) were used to obtain two distance deviation indicators (lineComp and dmax) of each sample for four sample statistics. Figure 13a shows the average value for each image in each sample. Figure 13b shows the statistical data of each sample’s dmax. The two distance deviation indicators from the two methods for sample A are quite different, but those for the other samples are not very different. In sample A, the lineComp and dmax values of HUBERP are much smaller than those of the LSMC, indicating that the distance deviation of HUBERP in the early period of crop growth is smaller than that of the LSMC. In the samples from other periods, the difference between the two distance deviation indicators as measured by the two methods is not as evident.

The angle deviation analysis was based on two sub-indicators (rateREC and avgANG). The two angle indicators (angComp, angComp_abs) of each image in each sample were judged separately. The judging rules were as follows: if the value of the angle indicator (angComp or angComp_abs) was within the available range, the value was an available value; if not, it was an unavailable value. Assuming that the number of available values of an angle index set was denoted as avNUM and the total number of elements in the set was denoted as cNUM, then the efficiency of the angle index was the ratio of avNUM to cNUM, denoted as rateREC. The available range was [−5, 5]. Without excluding the unavailable value, the average value of the angle indicator was calculated and denoted as avgANG. The rateREC and avgANG values were respectively calculated for the two angle indicators. The statistical results for rateREC and avgANG are shown in

Table 6. Statistical results of rateREC and avgANG.

Method	Sample Name	rateREC(%)		avgANG(°)		Df
		angComp(°)	angComp_abs(°)	angComp(°)	angComp_abs(°)	Df1	Df2
HUBERP	A	100	100	0.20	1.92	−2.72	−9.79
HUBERP	B	100	100	−0.35	1.49	0.05	0.12
HUBERP	C	90	90	2.96	4.81	1.44	0.06
HUBERP	D	100	100	0.24	0.88	0.10	−0.09
LSMC	A	50	35	−2.92	11.71	/	/
LSMC	B	100	100	−0.30	1.37	/	/
LSMC	C	85	85	−1.52	4.75	/	/
LSMC	D	100	100	−0.14	0.97	/	/

.

From the perspective of rateREC, it can be seen that the rateREC values of the two angle indicators of the four samples as measured by HUBERP are greater than or equal to those from the LSMC. HUBERP is generally better than LSMC from the perspective of rateREC, especially in the early crop growth period corresponding to sample A.

From the perspective of avgANG, it can be observed from the concrete values of avgANG that all of avgANG values of HUBERP belong to the available range [−5, 5] referred to above. However, the angComp_abs of sample A of the LSMC (11.71°) does not belong to the available range. To compare the two methods more intuitively and quantitatively, we calculated the difference between the absolute values of the same angle indicator of HUBERP or LSMC in the same sample (denoted as Df). For example, in Table 6, |0.20| − |−2.92| = 0.20 − 2.92 = −2.72. The closer the value of Df to 0, the smaller the absolute value of the angular deviation indicator between HUBERP and the LSMC. If Df is less than 0, the indicator’s absolute value in HUBERP is smaller relative to that in the LSMC. The Df of angComp and Df of angComp_abs are denoted as Df1 and Df2, respectively.

For sample A, the values of Df1 and Df2 deviate from 0 and are less than 0. The HUBERP angle deviation is large relative to that of the LSMC, and the HUBERP angle indicator is smaller. Notably, the angComp_abs values of sample A in the LSMC are large, indicating that when using the LSMC, the angular deviation of sample A is large overall. Thus, HUBERP adapts to the environment corresponding to sample A, whereas the LSMC does not.

For sample B and sample D, both Df1 and Df2 are close to 0. Therefore, the difference between the angular deviations of the two methods is not large, and both methods are adapted to the environments of sample B and sample D.

For the C sample, although the value of Df1 is far from 0 and greater than 0, the value of Df1 (shown as 1.44°) falls within the valid range [−5, 5], and owing to the large leaves of the single broad-leaved crops, there is an error between the reference line and actual crop row centreline; thus, 1.44° is acceptable. Df2 approaches 0 and is greater than 0. Therefore, both methods are suitable for the sample C conditions.

The analysis of the two evaluation indicators (avgANG and Df) shows that HUBERP has a small angular deviation in early crop growth. In other periods, the difference in the angle deviation as measured by the two methods is not evident.

In summary, compared with the LSMC, HUBERP can adapt to longer crop growth periods. Especially in the early stages of crop growth, HUBERP has significant advantages in distance deviation and angle deviation, and can effectively improve the multi-period adaptability of the algorithm.

3.3. Timeliness Verification Test

Under the same conditions, four samples were processed five times using the above two methods (LSMC and HUBERP), and the CPU time taken by the program during each test process was recorded.

In each test of each sample, the test times for the first and last images were removed, and the average value of the remaining data was calculated. The obtained value was the average test time for each sample of each method in each test, denoted as avg_t1.

The avg_t1 data obtained from five tests were averaged, and the obtained value represented the average testing time for each sample of each method (denoted as avgt). The avgt data were used to evaluate the timeliness of each sample of each method.

The avgt data of the four samples were averaged, and the obtained value represented the average testing time of each method (denoted as AVGT). This was used to evaluate the timeliness of each method. The timeliness verification test results are shown in Figure 14.

By analysing the principles of the two methods, it was found that HUBERP needs to determine the weight of the data samples according to the size of the regression residuals, whereas the LSMC does not need to determine the weight of data samples; additionally, HUBERP needs to perform two straight-line fittings. Therefore, the time complexity of HUBERP is higher than that of the LSMC, inevitably leading to lower efficiency. It can be seen from Figure 14 that AVGT (HUBERP) = 38.53 ms > AVGT (LSMC) = 37.57 ms. When the size of the image meets the needs of the visual navigation industry, although the efficiency of HUBERP is inevitably reduced, the average time consumed by the two methods is not significantly different and can meet the timeliness requirements for visual navigation.

By observing the avgt values of samples A, B, and C in the two methods, it can be seen that the corresponding avgt values gradually increase with an increase in the crop growth time. This is because, with the growth of crops, the area proportion of the crops in the image increases, resulting in the increase of data processing required for preprocessing, feature point extraction, and optimisation. However, in the same sample, the maximum difference of avgt values corresponding to the two methods is less than 3 ms, indicating that HUBERP has similar timeliness to LSMC in different crop growth periods. By observing the avgt values of samples B and D in the two methods, it can be seen that in the same sample, the maximum difference in the avgt values corresponding to the two methods is less than 2 ms, indicating that HUBERP has the same timeliness as the LSMC under different light conditions.

To further validate the timeliness of the method proposed in this paper, the method proposed by Diao Zhihua et al. (denoted as Dmethod) was compared with the method proposed in this paper (HUBERP) in terms of time consumption [30]. The evaluation index was AVGT, and the statistical results are shown in Figure 15.

The above data in Figure 15 show that the time consumption of HUBERP is lower than that of Dmethod.

According to the inductive research of Yang et al., most algorithms based on using machine vision to extract navigation lines take more than 100 ms [31]. Without considering the factors of differentiation in the image resolution and hardware, an average test time of less than 100 ms is used as the standard to evaluate whether the algorithm has high timeliness. As shown in Figure 14, the average test time avgt of each sample image in the proposed method (HUBERP) does not exceed 41 ms. Therefore, when the hardware used in the test algorithm is assembled into the agricultural vehicles, the proposed method can meet the requirements of high timeliness in practical applications while using the current image resolution.

In summary, under the premise of meeting the efficiency requirements of visual navigation, HUBERP has timeliness similar to that of the LSMC and can adapt to a longer crop growth period than the LSMC.

4. Conclusions

The purpose of this article is to develop a multi-period unified detection method for navigation lines calculated based on the row lines of broad-leaved high-ridge crops with high timeliness. This method is expected to improve the automation of machines in field environments with a high-ridge cultivation mode for broad-leaved plants.

The proposed method includes four sequential aspects: image segmentation, feature point extraction, navigation line calculation, and dynamic feedback mechanism between two adjacent frames. First, the original image is obtained in the RGB colour space. After the image is converted into the CIE-Lab colour space, the a* component is extracted to obtain a grayscale image. The OTSU method and morphological processing are combined to separate target and background features. Then, the improved isometric segmented vertical projection method is used to identify the crop feature points. An adaptive clustering method and dynamic segmentation point clustering methods are extensively used to determine the feature point set of the final clustering. We have dynamically processed the thresholds of the above two methods (the adaptive clustering method and the dynamic segmentation point clustering method). The feature point set is optimised using the lateral distance and point line distance optimisation methods. Under the guidance of M-estimation robust regression, a linear regression method based on the Huber loss function is applied to the feature point fitting process of the point-line distance optimisation. The navigation line equation is calculated according to the equation of the slope of the angle bisector and the positional relationship of the angle bisector. Finally, the dynamic feedback mechanism between two adjacent frames is introduced—the mechanism links the image processing of the front and back frames and automatically adjusts the number of horizontal strips.

The feedback mechanism has been tested, and the experiment proves that the average value of num_next is within the expected range [30, 70], and the number of horizontal strips in different periods of the environment can be dynamically adjusted in order to improve the algorithm’s ability to adapt to multiple periods. Four samples were selected to test the performance of the proposed method. In the early stages of crop growth, HUBERP has advantages over LSMC in terms of distance, angle deviation, and the ability to adapt to longer crop growth periods. The average processing time of the four sample images in this study is 38.53 ms and meets the timeliness requirements of visual navigation. The timeliness of our algorithm is higher than that of existing algorithms.

Compared with LSM, the linear regression method used in this paper has not made a breakthrough in real-time performance, and it will be our next research content. In the future, the navigation line extraction method will become the core part of embedded navigation systems and will be used for visual navigation operations of unmanned agricultural vehicles in a high-ridge cultivation environment for broad-leaved plants.