Vision-Based a Seedling Selective Planting Control System for Vegetable Transplanter

Abstract

Seedling transplanting is an important part of vegetable mechanized production in modern agriculture. After the seedlings are cultivated on a large scale by the nursery tray, they are planted into the field by the transplanter. However, unlike manual transplanting, transplanter is unable to judge the status of seedlings in the hole during seedling planting, which leads to problems such as damaged seedlings and empty holes being picked in the same order and planted into the field, resulting in yield reduction and missed planting. Aiming at this problem, we designed a seedling selective planting control system for vegetable transplanter which includes vision unit, seedling picking mechanism, seedling feeding mechanism, planting mechanism, pneumatic push rod unit, limit sensor, industrial computer and logic controller. We used asymmetrical light to construct visual identification scenes for planting conditions, which suppresses environmental disturbances. Based on the intersection operation of mask and image, a fast framework of tray hole location and seedling identification (FHLSI) was proposed combined with FCM segmentation algorithm. The vision unit provides the transplanting system with information on the status of the holes to be transplanted. Based on the information, planting system chooses the healthy seedlings for transplanting, improving the survival rate and quality of transplanting. The results show that the proposed visual method has an average accuracy of 92.35% for identification with the selective planting control system of seedlings and improves the transplanting quality by 15.4%.

1. Introduction

When seedlings are subjected to transplanter operation, empty holes, damaged seedlings and other unsuitable features can easily occur in seedling trays due to factors such as transport and seed germination [1]. The transplanter is unable to judge the state of the seedlings to be transplanted in trays as humans do, avoiding damaged seedlings and empty holes, etc., and selecting the healthy seedlings for transplanting, thus causing problems such as missed plantings and reduced survival rates. For this reason, it is particularly necessary to design selective planting control systems with autonomous sensing capabilities in combination with information sensing and computer control technologies to achieve selective transplanting of seedlings in trays under operational conditions.

Improving the planting quality of transplanters has been a hot topic of research in the industry [2,3,4], especially detection and identification of seedlings, which is beneficial for grading and assessing quality and further ensuring survival rates after planting [5]. Wen et al. [6], aiming at the problems such as missed planting and low survival rate of pepper seedling transplanting, developed seedling selection system based on machine vision used to identify missing and broken seedlings and other seedling morphologies. Suo et al. [7] used machine vision to quickly grade the quality of apple seedlings and proposed an apple seedling segmentation method based on BlendMask and ResNet-101 which can effectively replace manual measurement and effectively divide apple seedlings. Tabinda et al. [8] developed a machine vision system based on RealSense for comprehensive monitoring of seedlings; point cloud clustering and suitable algorithms are applied to obtain the segmentation of three-dimensional seedling models to realize the measurement of non-destructive plant growth parameters. McGuinness et al. [9] proposed an algorithm for measuring seedling diameter, seedling height and root sprawl characteristics using machine vision system combined with image processing technology; the processing time for the whole seedling is about 30 milliseconds, and the experiments show that there is a good correlation between seedling characteristics measured by machine vision systems and artificial processes. Zhang et al. [10] proposed an automatic detection method for late emergence of cucumber plug seedlings based on point cloud processing. Automatic detection of late transplanting was achieved, and the proposed grading coefficients could effectively describe the growth of seedlings with a success rate of 95% for the automatic detection method. Wang et al. [11] proposed a non-destructive monitoring method for the growth process of transplanted seedlings. A Kinect camera was used to obtain color and depth images of the transplanted seedlings, and pixel matching was performed on the color and depth images. Based on the seedling outline and depth values obtained from image segmentation, the pixel coordinates of the leaf center and its corresponding depth were calculated to achieve localization, and a robust seedling index evaluation model was established based on the obtained plant height and leaf area of the transplanted seedlings. Benoit et al. [12] proposed a method for numerical validation of image processing algorithms specifically for plant root segmentation, which offers the possibility of assessing the impact of these parameters on the performance of any segmentation algorithm in an infinite number of virtual plant populations. Jin et al. [13], aiming at the problem of seedling injury during the transplanting process, developed a low-loss transplanting robot based on machine vision. Tong et al. [14] proposed a method for grading the quality of seedlings based on machine vision. The quality evaluation of single seedlings was achieved by dividing the overlapping leaves through a watershed algorithm, and the accuracy of quality grading reached 98%. Hong et al. [15] adopted the Census transformation and truncated gradient fusion method, and the cost aggregation step adopted the multi-scale cost merging split tree algorithm to realize the detection of farmland boundary information before planting. In the current research, a large number of researchers have introduced machine vision techniques [16,17,18], and a lot of research has been done on the identification of seedlings. However, the absence of effective application of the recognition information to agricultural production is still a pressing issue.

This paper develops a seedling selection and planting control system for vegetable transplanters based on the idea of imitating human transplanting in order to solve the problem of low planting quality due to the inability of the transplanter to select suitable seedlings. Transplanter first identifies the seedlings and then selectively plants them according to the identification information. Frist, we designed seedling picking mechanism and seedling feeding mechanism. The asymmetric light source was used to construct the visual recognition scene, which suppresses environmental disturbances. Then, we proposed a fast framework of tray hole location and seedlings identification (FHLSI) and applied the FHLSI algorithm to identify the whole row of seedlings in trays. Planting operations were identified into two categories: a. characteristics suitable for planting (Healthy Seedlings) and b. characteristics unsuitable for planting (Damaged Seedlings, Empty Holes and Others). Finally, the identification decision information was transmitted to the PLC, which controls the action logic of the seedling picking and feeding mechanism for selective planting. The system enables selective planting under working conditions and can provide technical support for the intelligent upgrading of transplanters.

2. Materials and Methods

Seedling is the object of transplanter’s work. In this paper, the seedling tray used by transplanter contains 128 holes, 8 trees in a single row. According to the seedling tray, we designed key features such as seedling picking and seedling feeding mechanisms combined with the logic control based on vision; the action logic of the mechanism is controllable.

2.1. Seedlings and Seedling Tray

The seedlings are cultivated by nursery factory using 128-hole seedling tray. The variety is Shouyan 905 tomato, as shown in Figure 1a. Mechanized transplanting requires seedlings to be 35–45 days of age, when the roots are well developed and can tightly wrap around the nursery substrate to form a bowl [19,20]. The stem is stout with 3–4 leaves and the crossing of leaves in adjacent holes does not interfere with the picking action [21]. The morphology of the seedling is shown in Figure 1b. Figure 1b lists the morphological parameters that need to be considered for mechanized transplanting, mainly including seedling width, height and stem diameter. This paper considers the identification of seedlings; therefore, leaf height and stem height of seedling need to be considered.

2.2. Seedling Picking Mechanism and Principle

In order to enable the transplanter to perform selective planting during operation, the picking cycle of the transplanter should include seedling identification time and the transplanting should be more efficient than manual. Based on these two requirements, a seedling picking mechanism with multiple end-effectors which can take up to four seedlings at a time has been constructed, but the planting mechanism can only plant one seedling at a time. In order to ensure the normal operation of the mechanism, the action time t₀ of the seedling picking mechanism and the planting time t of the planting mechanism should satisfy t₀ ≤ t. The relationship between the planting time and the seedling picking time T is shown in Equation (1). In logical design, waiting time should be included in the seedling picking time; the seedling picking time T consists of two parts: 1. the action time t₀, 2. the waiting time t_d. During the waiting time, the seedling identification can be carried out.

$$T=\{\begin{array}{l}t\begin{array}{cc}& \end{array}(\lambda =1)\\ \lambda t(1\le \lambda \le 4)\end{array}$$

(1)

where t is the planting time, T is the seedling picking time, and λ is the quantity of identification-healthy seedlings.

The end-effector is shown in Figure 2a. Seedling arrangement in working conditions is shown in Figure 2b. In order to provide space for the end-effector to operate, in this paper, seedling picking is carried out in an interval picking mode, the principle of which is shown in Figure 2c. The first time, the seedlings are picked in their initial position, and the second time, the initial position is relocated by the slide of the end effector to complete the picking of the whole row of seedlings.

2.3. Seedling Feeding Mechanism and Principle

In mechanical transplanters, the actions of picking and planting seedlings are closely linked and do not provide identification time. This paper proposes a seedling feeding mechanism, which is a new mechanism that connects the picking and planting mechanisms as shown in Figure 3. The buffer box provides a buffer space for the planting of four seedlings, and the limit plates at different positions can be opened according to the control signal.

2.4. Logic Execution Function Design

The timing of the mechanical action is fixed and cannot be flexibly controlled. In this paper, the action mechanism is improved with selective logic control based on the structure of the action mechanism. This is achieved by adding a cylinder pusher (the cylinder pusher is equivalent to a movement control switch). The logic control system is constructed to control the timing of each action, and the cylinder parameters of each functional component are selected in

Table 1. Function parameters.

Parameter	Picking Cylinder	Feeding Cylinder
Cylinder bore/mm	25	16
Maximum open/mm	28	/
Travel/mm	/	150
Maximum push/kg	6	8
Piston rod thread/mm	M5	M5
Maximum clamp/kg	3	/
Speed range/mm·s−1	/	50–750

.

The control principle of the seedling selective planting control system is shown in Figure 4. The PC, as the upper computer, runs the image processing algorithm to process the image information of the seedling trays and provide the PLC with selective decision signals; the PLC, as the lower machine, is mainly used for the action regulation of the picking mechanism and the feeding mechanism. In the IO port of the PLC, the X1–X8 addresses are assigned to the cylinder control terminals of the picking mechanism and the feeding mechanism, respectively. The whole system does not rely on a fixed time sequence and is logically controlled for selective planting based on the image information of the seedling characteristics.

3. Seedling Selective Planting System

The principle of the selective planting system is shown in Figure 5. A machine vision unit was constructed to obtain image information of the seedlings to be transplanted. A recognition framework was designed to determine the characteristics of the seedlings in the tray, and based on the image information, the system made the selective planting decision. The total number of seedlings to be transplanted was identified and the feeding time T_feeding was obtained. Its value fluctuates in the range (λ,4t), where λ is the number of seedlings to be transplanted identified by the vision unit.

3.1. Image Acquisition Sense

In order to suppress the interference of background information with image acquisition, this paper uses asymmetric light to fill the seedlings with light. Asymmetric light is a special light source that emits a light path which can be controlled by reflectors and lenses, providing good light convergence, controlled scattering and uniform illumination compared to ordinary light sources. According to the structure of the transplanter, the light source is mounted below the seedling picking mechanism, as shown in Figure 6, creating a scene where the light path is tangential to the curved surface of the tray. The front row of seedlings to be identified is located at the intersection of the light path and the camera field of view. The rear row of seedlings has only weak scattered light, which suppresses the interference of the background seedlings and obtains a better image of the seedlings, which is easy to recognize and process.

3.2. Image Preprocessing

To improve the effectiveness of image recognition, the images were compressed, cropped and masked prior to recognition. The original image size captured by the camera was 1920 × 1080. To reduce the processing time of the image, some of the background information in the image was cropped and the original image was compressed to a size of 331 × 181.

During the transplanting process, the image information captured each time contained the eight seedlings to be transplanted in the front row. The position of each seedling was relatively fixed. Therefore, during recognition, mask processing was used to generate 16 ROI regions corresponding to the stem height and leaf height regions of each of the eight seedlings in the front row in the same 331 × 181-sized mask image, thus obtaining the location information of the eight seedlings in the tray. Subsequent image processing identified the stems and leaves of the eight seedlings and determined the characteristics of the seedlings.

3.3. Image Segmentation Based on FCM

Image segmentation is the basis for the subsequent recognition process, which divides the image into different feature parts and obtains the target. With the mask pre-processing in Section 2.2, the ROI region in the seedling image was divided and this part is segmented according to the mask image.

FCM is a commonly used fuzzy theory-based clustering algorithm [22] which maximizes the similarity of data objects within classes and minimizes the similarity of data objects between classes, with the objective function shown in Equation (2):

$$C(U,V)={\displaystyle \sum _{k=1}^n{\displaystyle \sum _{i=1}^c{u}_{ik}^m{d}_{ik}^2(x_k,v_i)}}$$

(2)

where c is the number of clusters, u_ik(i = 1, 2, 3,…, c; k = 1, 2, 3,…, n) is the degree of x_k’s membership of the i-th cluster, the value range is (0,1); the parameter m is the weighted index, which determines the ambiguity of the clustering result, and in practical applications, it is usually taken as 2; d_ik(x_k,v_i) represents the Euclidean distance of x_k to the i-th cluster center v_i as shown in Equation (3):

$$d_{ik}(x_k,v_i)=\parallel x_k-v_i\parallel$$

(3)

The FCM is an improvement on the traditional C-means cluster delineation which achieves sample delineation of cluster centers based on the attribution of image pixels and determines the membership degree and cluster center parameters for each cluster after initializing the image cluster center.

$$u_{ik}={\left({{\displaystyle \sum _{j=1}^c\left(\frac{d\left(x_k,v_i\right)}{d\left(x_k,v_j\right)}\right)}}^{\frac{2}{m-1}}\right)}^{-1}$$

(4)

$$v_i=\frac{{\displaystyle \sum _{j=1}^c{\left(u_{ik}\right)}^m{x}_k}}{{\displaystyle \sum _{k=1}^n{\left(u_{ik}\right)}^m}}$$

(5)

According to Equations (4) and (5), the membership degree of each pixel to the cluster center in the image is solved with u_ik and the new cluster center v_i, and the objective function is converged to achieve the optimal by updating and iteration.

3.4. Visual Identification of Seedlings

Through image segmentation, information about the stems and leaves of seedlings is extracted and the status of the seedlings is determined based on the morphology of the stems and leaves. In this paper, a fast framework of tray hole location and seedlings identification (FHLSI) is proposed by calculating the ratio P of the pixel area of each mask ROI region to the area of the mask ROI region in the image as a threshold value. Based on the value of P, the seedlings are classified into two categories, A and B, where category A denotes the presence of a feature suitable for transplanting and the judgement output is 1, and category B denotes the presence of a feature unsuitable for transplanting and the judgement output is 0. In this paper, 50 samples are randomly selected for calculation based on the morphology of tomato seedlings, and the recognition parameters are provided as shown in

Table 2. Function parameters.

Type	Leaf ROI (PLeaf)	Stem ROI (PStem)	Real Image
A	0.5 ≤ P ≤ 1	0.08 ≤ P ≤ 0.2	Healthy Seedling
B	0 ≤ P ≤ 0.5	0 ≤ P ≤ 0.08	Empty Hole
		0.2 ≤ P ≤ 1	Damaged Seedling

.

The fast framework of tray hole location and seedlings identification (FHLSI) is mainly divided into 7 steps:

• Step 1: The seedling image and preprocessing such as image cropping and compression are read;
• Step 2: Image mask operation, ROI region extracted;
• Step 3: Parameters such as c, m, and v_i initialized, and the image feature segmented;
• Step 4: The membership of the cluster center u_ik and cluster center v_i updated;
• Step 5: It is determined whether the optimal conditions or the maximum number of iterations are met, and one of the two is met, then Step 6 is executed; otherwise, return to Step 4 follows;
• Step 6: The threshold P of each ROI region calculated, the seedlings and labels identified;
• Step 7: Seedling selective plan identified and output.

The FHLSI flow is shown in Figure 7.

4. Results and Discussions

In order to verify the design of the seedling selective planting control system and the FHLSI method, the seedlings identification test and the experiment of seedling selective planting control system were carried out.

4.1. Seedling Identification Test

The characteristics of the experimental seedlings in trays are shown in Section 2.1. In all, 15 trays of seedlings were randomly selected, totalling 1920 seedlings, and in order to verify the identification performance of the FHLSI method, Empty Hole and Damaged Seedling categories were manually added. The number of features of the four types in the nursery tray was counted. Among them, the number of Healthy Seedlings (HS) was 970, the number of Empty Holes (EH) was 500, the number of Damaged Seedlings (DS) was 420, the number of Others (OT) was 30, and the four types of characteristics in the tray are shown in Figure 8.

4.1.1. Result of Seedling Identification

The test results are shown in

Table 3. Identification results and accuracy.

Characteristics	Actual Class	Predict Class	Accuracy/%
HS	970	945	97.4
EH	500	407	81.4
DS	420	398	94.8
OT	30	23	76.7

.

Figure 9 shows the typical characteristic identification process and selectivity results in the input sample image set.

4.1.2. Discussion

The method of FHLSI was used to judge the characteristics of seedlings in the tray. The Healthy Seedling identification accuracy rate was 97.4%, Empty Hole identification accuracy rate was 81.4%, Damaged Seedling identification accuracy rate was 94.8%, and Others identification accuracy rate was 76.7%.

During the test, image segmentation response time was ≤ 0.5 s. Compared with the system operation time, the response time was almost negligible. The system transplanting frequency was 60 plants/min, and the execution time of single seedling picking operation was 4 s. At the same time, the response time was also affected by the PC performance. The PC configuration used in the study was the following: CPU was i5-7200, GPU was RX560.

As can be seen from Figure 10a, the FHLSI algorithm can accurately identify the four characteristics of seedlings, which can provide reliable information for selective planting control system. Analysis findings, the accuracy of Empty Holes and Others can be further improved in the process. In the Empty Holes identification, the false identification rate of identifying them as Healthy Seedlings was 11.6%, which would cause missing planting due to the crossing of branches and leaves in the tray. In the identification of Others, the false identification rate of identifying Others as Damaged Seedlings was 16.7%, mainly due to the special definition of Others. Usually, the Others, such as the stem bending of the seedling’s growth, the short stem in the seedlings of the same age, etc., rarely occur in the hole, and the stem position is off the center of the hole. This type is not easy to be picked up on by the manipulator, resulting in the failure of seedling planting. In the future research work, semantic segmentation [23] and other methods can be considered to improve the accuracy of this type of identification.

In order to make machine vision recognition results operational, the FHLSI converts the seedlings of visual recognition into two types of results: A. transplanting, B. not transplanting. The results as shown in Figure 10b. The accuracy of class A recognition is 97.4%, and the accuracy of class B recognition is 87.2%. Based on sample characteristics, the weighted average accuracy is 97.4 × (970/1920) + 87.2 × (950/1920) = 92.35%.

According to the experimental analysis, the main causes of misidentification were the following: 1. In category A identification, the hollow seedlings were misidentified as Healthy Seedlings mainly due to the crossover of adjacent seedlings; 2. In category B identification, there was crossover interference of leaves between adjacent seedlings, and some of the stems of the seedlings were bent, causing shading and covering so that the original category B was misidentified as category A seedlings.

In view of the above, further research work could consider fusing CCD sensing with multi-sensor data such as photoelectric sensing and proximity switches to synthesize seedling conditions and provide reliable and selective decision making for rice transplanters.

4.2. Experiment of Seedling Selective Planting

An experiment was carried out to verify the performance of the seedling selective planting control system. In total, 10 trays of seedlings were randomly selected for the experiment; the characteristics of the seedlings were the same as in Section 2.1. The transplanting experiment was carried out in two groups and the number of seedlings fed and transplanted was counted manually; the results of the experiment are shown in Figure 11.

4.2.1. Result of the Selective Planting Experiment

The evaluation index of the experiment results is planting qualified rate, which is defined as follows:

$$Q=\frac{Z_q}{Z_o}\times 100\%$$

(6)

where Q is planting qualified rate, Z_q is quality qualified number, Z_o is seedling feeding number.

The results are shown in

Table 4. Planting results.

Planting Status	Test Number	Seedling Feeding Number	Quality Qualified Number	Planting Qualified Rate/%
Group 1	1	128	102	79.6
	2	128	97	75.8
	3	128	105	82.0
	4	128	99	77.3
	5	128	100	78.1
Average:				78.5
Group 2	1	103	95	92.2
	2	108	101	93.5
	3	112	106	94.6
	4	109	104	95.4
	5	114	107	93.8
Average:				93.9

. The average planting qualified rate was 78.5% for Group 1 with the selective planting control system switched off and 93.9% for Group 2 with the selective planting control system switched on.

4.2.2. Discussion

Figure 12 shows the comparison of the experimental results. With the selective planting control system, the transplanting qualified rate increased by an average of 15.4%, which shows that the selective planting control system can effectively improve the quality of transplanting.

After planting, the statistical data of the planting characteristics are shown in Figure 13 where R-HS is the remaining Healthy Seedlings in the tray; R-EH is the remainig Empty Holes in the tray; R-DS is the remaining Damaged Seedlings in the tray; R-OT is the remaining Others in the tray.

In group 1, Empty Hole and Damaged seedlings are planted in order, which is the main reason for the planting qualified rate decline; In group 2, a large number of Empty Hole and Damaged seedlings are kept in the tray and not planted, reducing the proportion of EH, DS and OT characteristics. As can be seen from Figure 13, the Seedlings Selective Planting Control System provides a selective dividing line for planting according to the characteristics. Identifying the quality of seedlings provides a basis for transplanter. How to further improve the effect of selective planting, the following two points need to be considered:

• The selective planting system is an interactive system of seedlings, mechanical and visual control, has the agronomic integration attributes of agricultural machinery. Therefore, the growth morphology of seedlings is an important factor in improving the effect of selective transplanting [24], screen the combination of seedlings parameters suitable for selective planting, such as seedling age (leaf stretch), stem mechanical properties (moisture content), etc., and reduce the systematic misjudgment caused by leaf crossing.
• The misidentification of HS by selective planting system will cause waste of resources. The seedlings waste rate indicator should also be considered in the design and optimization of the system, as defined as follows:

$$W_H=\frac{R_H}{R}\times 100\%$$

(7)

where W_H is the seedlings waste rate; R_H is the quantity of remained healthy seedlings in the tray; R is the quantity of all remained characteristics in tray.

According to the definition of W_H, the W_H value index is only effective for the selective planting system, and the W_H value of the system designed by this paper is 4.25%. In further research work, the W_H value of the system should be further reduced. In the experiment, it is found that the picking method, and the misidentification of the visual unit are cause the WH value to rise. Therefore, on the basis of improving the accuracy of algorithm recognition, optimized the seedling picking trajectory, improved the success rate of seedling extraction, and the comprehensive improve the benefits of the selective planting system.

5. Conclusions

• Based on the idea of humanoid selective transplanting, selecting suitable seedlings for transplanting and improving the efficiency of transplanting, this paper constructs a seedling selective planting control system applicable to vegetable transplanters. The seedling picking mechanism and seedling feeding mechanism are designed, and the selective planting scheme is proposed by combining machine vision and logic control technology.
• A fast framework of tray hole location and seedling identification (FHLSI) is proposed for the fast and accurate identification requirements of selective planting under transplanter working conditions. The visual recognition scene suitable for transplanting conditions is constructed using asymmetric light and the crossover of the CCD camera’s field of view, effectively suppressing background interference. A selective planting control system for seedlings was designed based on the intersection of mask and image operations combined with the FCM segmentation algorithm.
• The test results show that the proposed visual identification method achieves an average accuracy of 92.35% under working conditions, with a 15.4% improvement in transplanting quality with the seedling selective planting control system.