Path Planning and Control System Design of an Unmanned Weeding Robot

Abstract

Aiming at the demand by unmanned farms for unmanned operation in the entire process of field management, an unmanned plant protection robot for field management was developed based on a platform comprising a traditional high-clearance spray rod sprayer, integrated unmanned driving technology, image recognition technology, intelligent control technology, and precision operation technology. According to the agricultural machinery operation mode, agricultural machinery path planning, linear path tracking, and header path tracking algorithms were developed. Based on the overall structure and working principle of the chassis, the robot control system, steering control system, and operation control system were set. Based on the YOLOv5 image recognition algorithm, the crop–weed recognition model was developed. After 6000 rounds of training, the accuracy, recall, and mean average precision of the model were 87.7%, 84.5%, and 79.3%, respectively. Finally, a field experiment was carried out with the unmanned plant protection robot equipped with a complete system. Results show that the average lateral error of the robot is 0.036 m, the maximum lateral error is 0.2 m, the average root mean square error is 0.053 m, the average velocity error is 0.034 m/s, and the average root mean square error of velocity is 0.045 m/s when the robot works in a straight line. In weeding operations, the area ratio of weedy zones to field is 25%, which saves 75% of the herbicide compared to that dispensed in full spraying mode. The unmanned plant protection robot designed in this study effectively achieves machinery’s autonomous operation, providing valuable insights for research in unmanned farming and autonomous agricultural machinery.

1. Introduction

With the development of driverless, artificial intelligence, and intelligent control technologies, intelligent perception, decision-making, and control technologies designed for agricultural production have also begun to be applied to agricultural production. The application of new technologies not only improves the efficiency of agricultural production but also greatly reduces the labor intensity of practitioners, opening up a new direction for the research of agricultural production tools [1,2,3,4].

As a typical agricultural field management production tool, plant protection machinery is mainly used for weeding, fertilization, pesticide spraying, crop growth monitoring, crop pest identification, farmland information collection and so on. Regarding unmanned plant protection robots, a driverless system and intelligent operation system should be added to traditional plant protection machinery to realize the entire unmanned plant protection operation.

The existing unmanned system includes path planning and path tracking, and path planning includes both global path planning [5,6] and local path planning [7,8]. The global path-planning method used in this paper is the AB line operation algorithm. Compared to other control algorithms, this method is simple and versatile. The commonly used path-tracking methods include path tracking based on kinematics [9,10], pure path tracking based on proportion integration differentiation (PID) [11,12], improved pure path tracking [9,13,14,15,16], and Model Predictive Control (MPC) control [17,18,19], among others.

Since the method based on a kinematic model is used to linearize the model at a certain speed, the stability and robustness of the control system become worse when the speed changes. The pure tracking method is the most commonly used and mature path tracking algorithm, but under different speed and curvature conditions, the choice of forward distance directly determines the adjustment accuracy of the control system. MPC and other control methods based on a dynamic model rely heavily on the dynamic model of the controlled object, and how to establish an accurate dynamic model has always been the difficulty in achieving intelligent control of agricultural machinery, so the algorithm is still in the exploratory stage.

Intelligent operation system includes weed identification [20,21,22], pest identification [23,24], target spraying [25,26], and robot weeding [27,28,29]. With the development of convolutional neural network technology, target detection and semantic segmentation have been widely used in agricultural production. The target detection method is fast in recognition, but it cannot obtain an accurate contour of the object. Semantic segmentation can obtain the pixel-level contour of the object, but it requires high computational power and is difficult to apply to an embedded system.

The above-mentioned research on the unmanned plant protection technology of agricultural machinery has laid the foundation for the development of intelligent agricultural machinery. In the present work, we designed the navigation control, robot control, and operation control systems of an unmanned plant protection robot based on the existing navigation, control, and intelligent operation technologies of domestic unmanned agricultural machinery. Our goals were to test the feasibility of an unmanned plant protection robot through field experiments and to provide reference for researching unmanned agricultural machinery operation systems.

2. Materials and Methods

2.1. Composition of Weeding Robot

An unmanned weeding robot mainly includes four parts: a navigation system, a motion control system, intelligent working parts, and a data transmission unit (DTU). The navigation system is responsible for robot motion path planning and path tracking control, and the motion instructions are sent to the motion control system. The motion control system converts the motion command to the action of the robot through the electro-hydraulic control system, including speed regulation and clutch control. Intelligent working parts are the core of autonomous operation of robots, including weed identification, spraying control, and operation monitoring. The DTU is responsible for transmitting the robot’s operation information to the remote-control platform of agricultural machinery. The robot is shown in Figure 1, and the main performance parameters are in

Table 1. Main technical parameters of robot.

Technical Indicators	Value
Engine power (kW)	15.4
Length × width × height (m)	4.1 × 2.2 × 2.54
Spraying width (m)	2
Ground clearance (m)	1.2
Turning radius (m)	3.5
Tank volume (L)	500
Speed (km·h−1)	0–5.6
Autopilot accuracy (cm)	±2.5
Weed identification rate (%)	>80

.

The chassis of the weeding robot was adapted from the 3WP-500G high-pressure chassis sprayer, manufactured by Shandong Huasheng Zhongtian Machinery Group Co., Ltd. (Linyi, China). The weeding robot’s chassis is equipped with fully hydraulic power steering, hydrostatic transmission drive system, cable-type continuously variable transmission, throttle regulator, and a braking device. The original steering wheel of the high-pressure chassis sprayer has been replaced with the servo steering wheel (CES-T) produced by Shanghai Huace Navigation Technology Co., Ltd. (Shanghai, China). The servo steering wheel consists of two parts: the fixed part, which is the cylindrical motor stator connected to the robot’s chassis, and the rotating part, comprising the motor rotor, steering wheel, and spline sleeve. The upper part of the spline sleeve is securely attached to the rotor with screws, while the lower end is connected to the robot’s steering column through gear meshing.

Figure 2 illustrates the controllers used on the weeding robot, namely the T506 edge computing box and the ZQWL IO expansion module. The T506 edge computing box is manufactured by Shenzhen Tuwei Information Technology Co., Ltd. (Shenzhen, China). It runs the Ubuntu 18.04 operating system, supports 4G/5G connectivity, features a fanless design for passive heat dissipation, and boasts a lightweight and modern appearance. It offers a variety of IO interface options and includes a bottom bracket for easy on-site installation. The ZQWL IO expansion module is provided by Shenzhen Zhiqian Internet of Things Electronic Technology Co., Ltd. (Shenzhen, China). It features an 8-channel serial port relay controller capable of handling 8-channel switching value collection and 8-channel relay output. The control board is equipped with RS232, RS485, and CAN communication interfaces, enabling switch acquisition and relay output control through custom protocols.

2.2. Path Planning

Path planning is used to generate the target trajectory of a robot and guide it to complete tasks independently according to farmland information and operational requirements of the plant protection robot. In plant protection robots, the path contains elements of point P, line L, and curve C, represented mathematically as:

$$P={\left[\begin{array}{ccc}x& y& h\end{array}\right]}^{-1},$$

(1)

where P is the point in the plane, x the longitude in UTM coordinates, y the latitude in UTM coordinates, and h the heading of the robot body at that point.

$$L=[\begin{array}{cc}{P}_1& P_2\end{array}],$$

(2)

where L is the line, P₁ the starting point of the line, and P₂ the end point of the line.

$$C=\left[\begin{array}{cccc}{P}_1& P_2& \cdot \cdot \cdot & P_{\mathrm{n}}\end{array}\right],$$

(3)

Curve C is composed of a series of points, P means the points in the curve, and 1, 2, …, n means the sequence of points in the curve.

Path planning is divided into linear path planning and header path planning. The linear path is a series of parallel lines. The calculation method is:

$$L=L_0+\left[\begin{array}{c}\cos (\alpha -\frac{\pi }{2})\\ \sin (\alpha -\frac{\pi }{2})\\ 0\end{array}\right]\times \left[\begin{array}{cc}w& w\end{array}\right],$$

(4)

where L is the new line, L₀ the reference line, w the width of the line, and α the angle between the line and the x axis

As shown in Figure 3, the reference line A₀B₀ is determined before operation. According to the width of the field and the width of the machine operation, the corresponding parallel operation lines A₁B₁, A₂B₂, …, A_nB_n are generated. During the operation, the robot opens the operating component from point A₀ to point B₀ along the straight line A₀B₀ and then closes the operating component. The target path is switched to the multi-segment curve B₀B₁ for tracking. After reaching point B₁, the operating component is opened, and the operating component is driven to point A₁ along the straight line B₁A₁. The above steps are repeated until the robot reaches point B_n and the operation is completed.

Header path planning uses the Dubins path to plan the turning path, as shown in Figure 4. According to the different paths of the steering mode, steering can be divided into six modes: RSR, LSL, RSL, LSR, RLR, and LRL (where R denotes a right turn, S a straight line, and L a left turn) [30]. For agricultural robots, because they operate according to a parallel line, the heading difference between before and after turning is 180°, and RSL and LSR do not meet the operating conditions. In this paper, we only solved the four steering modes RSR, LSL, RLR, and LRL. In addition, we selected the minimum path length scheme as the target path.

To facilitate the path transition, the entry and exit points of the solved path are extended by 2 m along the tangent direction to enhance the smoothness of the path connection. Four Dubins paths become SRSRS, SLSLS, SRLRS, and SLRLS. Each path is divided into five sections, and each section is composed of arcs or straight lines. The above path is a multi-segment continuous function. For the processor, it is necessary to discretely sample the arcs and lines and represent the turnaround path as a set of points. The path is composed of straight lines and arc curves, so the straight lines and curves are sampled separately.

The linear sampling method is expressed as:

$$P_{\mathrm{i}}=\left[\begin{array}{c}{x}_A+id\cos (\alpha )\\ y_A+id\sin (\alpha )\\ \alpha \end{array}\right],$$

(5)

(x_A, y_A) is the starting point of the line, d the sampling distance, α the angle between the line and x axis, and i the sampling sequence.

The curve discrete method is expressed as:

$$P_{\mathrm{i}}=\left[\begin{array}{c}{x}_C-R\cos (\sigma -i\ast {\Delta }_{\sigma })\\ y_C-R\sin (\sigma -i\ast {\Delta }_{\sigma })\\ \sigma -\pi /2-i\ast {\Delta }_{\sigma }\end{array}\right],$$

(6)

(x_C, y_C) is the arc center, R the arc radius, σ the angle between the arc starting point and arc center line and x axis, Δ_σ the sampling angle, and i the sampling sequence.

As shown in Figure 5, the header path is divided into three parts: straight line segment A₀B₀, circular arc segment B₀B₁, and straight-line segment B₁A₁. When the robot moves along the A₀B₀ straight line during operation, and when the front end of the robot crosses the straight line L₃, which is 2 m from straight line L₁, the head adjustment path B₀B₁ is calculated. After the calculation is completed, the target path is switched to the head adjustment path. A 2 m overlap exists between the header path and the linear path, which can ensure the smooth transition of the path during the driving process of the robot. When the end of the robot crosses the line L₁ formed by B₀B₁ from inside the field, the working parts are closed. When the end of the robot crosses line L₂ 1 m from line L₁ from the outside of the field, the operating component is opened.

2.3. Path Tracing

The path-tracking algorithm is designed based on the pure path-tracking model. As shown in Figure 6, O is the origin of the world coordinate system, C the vehicle steering center, and E the vertical foot of the vehicle steering center C to the target straight path AB. In the world coordinate system:

$$\overrightarrow{OE}=\frac{\overrightarrow{AB}\cdot \overrightarrow{AC}}{{\overrightarrow{\left|AB\right|}}^2}\overrightarrow{AB}+\overrightarrow{OA},$$

(7)

In the equation below, G is the preview point, v is the ground speed of the robot, and t is the preview time. It can be seen that:

$$\overrightarrow{OG}=\mathrm{vt}\frac{\overrightarrow{AB}}{\left|\overrightarrow{AB}\right|}+\overrightarrow{OE},$$

(8)

In the equation below, ∠α is the angle between the target point G and body axis CH in the body coordinate system. According to the pure path-tracking model algorithm, it can be seen that:

$$\frac{\overrightarrow{\left|CG\right|}}{\sin (2\alpha )}=\frac{R}{\sin (\frac{\pi }{2}-\alpha )},$$

(9)

The available turning radius R is:

$$R=\frac{\left|\overrightarrow{CG}\right|}{2{\left|\overrightarrow{HG}\right|}^2},$$

(10)

According to Ackerman steering geometry, the robot front wheel steering angle δ is:

$$\delta =\arctan (\frac{L}{2R}),$$

(11)

The lateral error E_lat is:

$$E_{\mathrm{lat}}=\left|\overrightarrow{CE}\right|=\frac{\Vert \overrightarrow{AB}\times \overrightarrow{AC}\Vert }{\Vert \overrightarrow{AB}\Vert },$$

(12)

The principle of header path tracking is shown in Figure 7. A pure path-tracking model based on preview point search is adopted. Its control principle is similar to that of linear tracking. The preview point G is obtained by the curve path search method and then the steering angle δ of the front wheel calculated. The process is as follows.

(1) Curve path AB is searched to find the closest point P_e to the P_c using Euclidean distance:

$$\underset{e=i}{\min }(\Vert \overrightarrow{C{P}_i}\Vert )(i\in 0\dots n),$$

(13)

(2) The preview distance is calculated according to the vehicle speed v and preview time t. Curve path EB is searched, and the closest point P_g with distance vt from point P_e is found:

$$\underset{g=i}{\min }(\Vert \overrightarrow{P_i{P}_e}\Vert -\left|vt\right|)(i\in e\dots n),$$

(14)

Additionally, the vehicle steering angle δ is calculated using Formulas (9)–(11).

2.4. Speed Control System

The robot transmission route is shown in Figure 8. The engine output shaft is connected to the input shaft of the hydrostatic electrodeless transmission (HSET) electrodeless transmission through the belt, and the output shaft of the HSET electrodeless transmission is connected with the gearbox through the coupling. The two output shafts of the gearbox distribute the power to the front and rear axles of the vehicle body through the transmission shaft. The front axle of the vehicle body distributes the power to the left and right front wheels of the vehicle body through the differential. The structure of the rear axle of the vehicle body is similar.

Speed control is realized by a cascade PID controller. As shown in Figure 9, the outer loop is a speed proportional integral (PI) controller, and the inner loop is a proportion differentiation (PD) controller to control the opening of the HSET valve. The PI controller is expressed as follows:

$$\tilde{Q}\left(k\right)+=K{p}_{\mathrm{v}}\left(E_{\mathrm{v}}\left(k\right)-E_{\mathrm{v}}\left(k-1\right)\right)+K{i}_{\mathrm{v}}{E}_{\mathrm{v}}\left(k\right),$$

(15)

where $\tilde{Q}$ is the target opening of the HSET valve, Kp_v and Ki_v are the proportional and integral parameters of the PI controller, respectively, and E_v is the speed error—which is the difference between the target speed and the actual speed v—as follows:

$$E_{\mathrm{v}}\left(k\right)=\tilde{v}\left(k\right)-v\left(k\right),$$

(16)

The HSET valve PD expression is:

$$M(k)=K{p}_{\mathrm{q}}{E}_{\mathrm{q}}\left(k\right)+K{d}_q\left(E_{\mathrm{q}}\left(k\right)-E_{\mathrm{q}}\left(k-1\right)\right),$$

(17)

$$E_{\mathrm{q}}(k)=\tilde{Q}(k)-Q(k),$$

(18)

M is the motor voltage, Kp_q and Kd_q are the proportional and differential parameters of the PD controller, Q is the actual position of the HSET valve, and E_q is the angle error of the HSET valve.

2.5. Steering Control System

The robot steering system, shown in Figure 10, mainly includes an electric steering wheel, hydraulic steering valve, oil tank, hydraulic pump, cooler, and front and rear steering hydraulic cylinders. The hydraulic pump is connected to the engine to provide power for the steering system. When the steering wheel turns right, the hydraulic oil outputs from the R port of the steering valve drive the front steering hydraulic cylinder to move left, and the front wheel turns right. The rear steering hydraulic cylinder is in series with the front steering hydraulic cylinder and drives the rear wheel to turn left. When the robot body turns left, the principle is the same as that of turning right, and the direction is opposite.

Wheel steering control is realized by the PID algorithm. The input is the target angle of the wheel, the output is the position of the motor, and the feedback is the actual angle of the wheel. To achieve better control of the wheel angle, the front wheel steering angle is measured via the method of integrating the kinematic estimation and transfer model of the steering system, expressed as:

$$\widehat{\delta }\left(k\right)=\mathrm{G}{\delta }_a\left(k\right)+\left(1-\mathrm{G}\right){\delta }_b\left(k\right),$$

(19)

G is the weight parameter used for front wheel steering angle estimation, δ_a is the front wheel steering angle through the hydraulic steering system transfer model, and δ_b is the front wheel steering angle obtained via kinematics estimation. According to the research [31], the relationship between steering wheel angular velocity and front wheel steering angle is expressed as:

$${\delta }_a(k)=\tilde{\delta }(k-1)+{\displaystyle \int \left(G_{\mathrm{h}}{\omega }_s\left(k\right)+\mu \right)}\mathrm{dt},$$

(20)

where G_h is the transfer coefficient of the hydraulic steering system, ω_s the angular rate of the steering wheel motor, and μ the angular rate noise caused by hydraulic leakage.

$${\delta }_{\mathrm{b}}\left(k\right)=\arctan (\frac{\dot{\varphi }\left(k\right)L}{2v\left(k\right)}),$$

(21)

where $\dot{\varphi }$ is the angular rate of rotation along the direction of the robot body, v the speed of the reference point of the robot body, and L the axial distance.

2.6. Dataset Production

The dataset used was collected by a vehicle camera installed at the bottom of the high-altitude gap sprayer; the camera was placed 80 cm above the ground. The data were collected on 27 August 2020 at the Baima Experimental Base of the Nanjing Institute of Agricultural Mechanization, Ministry of Agriculture and Rural Affairs, China. The onboard camera was a Ruierwei model USBFHD01M with a maximum resolution of 1920 × 1080. A total of 436 images were obtained via frame sampling and clipping.

The rectangular box-labeling tool in Labelme (v4.5.10) was used to label the crop objects in the image, where corn is labeled as 1 and weeds as 0. The labeled data were divided into a training set and a verification set in a ratio of 8:2. Owing to the small size of the dataset, the data were repeated twice. In addition, the images were enhanced using random contrast adjustment, random brightness adjustment, random left–right flipping, and random noise addition according to the probability of 50%. The original image sizes were adjusted to 640 × 320 pixels to facilitate easier training.

2.7. Model Building and Training

YOLOv5 was used in the present work for weed identification. The network includes three modules: feature extraction, feature aggregation, and target prediction layers [32]. The feature extraction layer is mainly used to extract feature information at different levels from an image. The feature aggregation layer is used to enhance the detection ability of the network for objects of different scales. The target prediction layer is used to associate the feature information of the image with the shape and position information of the object.

The model training platform configuration was an Intel core i7 10700 CPU with a main frequency of 4.8 GHz and 16 GB of memory and an Nvidia RTX 2060 GPU with 11 GB of memory. The environment was the Ubuntu 20.04 operating system, and the programming language was Python 3.9.0. The stochastic gradient descent method was used to train the model. The batch size was set to 8 and the momentum factor to 0.9. The other parameters used the default values of YOLOv5. The confidence threshold was set as 0.5, and the evaluation indexes selected were precision, recall, and mean average precision (mAP).

$$Precision=\frac{TP}{TP+FP},$$

(22)

$$Recall=\frac{TP}{TP+FN},$$

(23)

$$mAP=\frac{A{P}_1+A{P}_2}{2},$$

(24)

The training results of the model are shown in Figure 11. In the first 3000 rounds, the jitter of each index is relatively intense, and the training parameters tend to be stable in the range 3000–6000 rounds. After 6000 rounds, the accuracy of the model increases and the recall rate and mAP value decrease. Therefore, the model output after 6000 rounds of training was selected as the weed identification model in the present work, with an accuracy, recall rate, and mAP of 87.7%, 84.5%, and 79.3%, respectively.

Figure 12 displays the identification results of corn and weeds based on the YOLOv5 convolutional neural network model. It is evident that the model accurately identifies both corn and weed targets in complex field images, and the segmentation results closely align with the actual values.

2.8. Weeding Decision Method

The spraying control flowchart is shown in Figure 13. The camera installed at the bottom of the machine automatically obtains an image of the working area with a length of 2 m and a width of 1 m, obtains the contour and position information of crops and weeds through the crop–weed identification network, and then calculates whether there are weeds in the spraying area of each nozzle. If there are weeds in the area, the nozzle is opened for t_k, and if not, no action is taken. The solenoid valve opening time t_k can be calculated by the solenoid valve dead time t_dead, the distance len from the camera to the solenoid valve, and the vehicle speed v. Here, t_dead = 2 s and len = 1 m:

$$t_{\mathrm{k}}=t_{dead}+\frac{len}{v}$$

(25)

2.9. Spray System

The spray system of the plant protection robot is shown in Figure 14. The spraying module includes the tank, pump, valve group, distributor, electronic control nozzle, and other components which can adjust the pressure and flow of the spraying as well as control each nozzle to open and close independently. The electrical system is composed of a distributed input/output (IO) module and wiring harness. Through the controller area network (CAN) bus, the navigation system, the robot control, and working systems are connected, and the working state of the working parts is adjusted based on the control instructions.

3. Results and Discussion

To test the feasibility of the proposed plant protection robot unmanned operation system, a field experiment was carried out at the Baima Experimental Base of the Nanjing Institute of Agricultural Mechanization, Ministry of Agriculture and Rural Affairs, China in August 2021. The row spacing of the tested maize was 50 cm, the plant spacing 25 cm, and the plant height 15–25 cm. The main component of herbicide is mesotrione, which is safe for corn at the 3–5-leaf stage. The experiment was carried out twice. The first area was 60 m long, 6 m wide, and consisted of three working paths. The second area was 50 m long, 8 m wide, and consisted of four working paths. During the operation, the robot traveled and operated according to the planned path, setting a straight-line speed of 3.6 km/h. During operation, the lateral deviation of the robot and the opening and closing of the solenoid valve were recorded using a GCAN-401 data recorder.

Figure 15 shows the statistical data of the lateral error of the operating path of the plant protection robot. When the robot works in a straight line, the average lateral error of the robot is 0.036 m, the maximum lateral error 0.2 m, and the average root-mean-square error of lateral is 0.053 m, the average velocity error is 0.034 m/s, and the average root mean square error of velocity is 0.045 m/s.

Figure 16 shows the results of the weeding operation, in which green represents the weedy zone of the test field and the red is the clean zone. As determined statistically, the area of the test field is 720 m² and the area of the weedy zones in the test field is 180 m². The area ratio of weedy zones to field is 25%, which saves 75% of the herbicide compared that dispensed in full spraying mode. The area of weedy zones in the blue frame accounted for 62.3% of the weedy zones in the field, and the area ratio of weedy zones in the blue frame to field was 51.7%, which was approximately 2.05 times of that not in the blue frame. It can be concluded from the operational effectiveness diagram that weeds tend to cluster in the field environment. In other words, weed distribution in the field is uneven, with certain areas having a significantly higher ratio of weed distribution area to land area compared to others. This analysis suggests that guiding robot applications based on weed identification can lead to a substantial reduction in pesticide usage.

4. Conclusions

In this paper, we have designed a weeding robot by harnessing the power of both intelligent control technology and deep learning. To achieve the goal of unmanned weeding assignments, we have developed automatic path-planning and path-tracking algorithms. To enable autonomous driving, we have implemented speed control and steering wheel systems using PID algorithms. For enhanced intelligent operation, we have designed a weed identification system based on YOLOv5. To achieve precise weeding, we have also created an electronic spraying system.

According to the results of field experiment, the average lateral error of the robot is 0.036 m, the maximum lateral error is 0.2 m, the average root mean square error is 0.053 m, the average velocity error is 0.034 m/s, and the average root mean square error of velocity is 0.045 m/s when the robot works in a straight line. In the weeding operations, the area ratio of weedy zones to field is 25%, which saves 75% of the herbicide compared to that dispensed in full spraying mode.

In the future, we plan to delve deeper into the vehicle body dynamics model of the robot to enhance path tracking accuracy. We will also create a comprehensive weed dataset to improve weed identification accuracy under various working conditions. Additionally, we aim to incorporate more sensors, such as laser radar, ultrasonic waves, and gyroscopes, to further enhance the robot’s positioning and control accuracy.