Dynamic Measurement of Portos Tomato Seedling Growth Using the Kinect 2.0 Sensor

Abstract

Traditionally farmers monitor their crops employing their senses and experience. However, the human sensory system is inconsistent due to stress, health, and age. In this paper, we propose an agronomic application for monitoring the growth of Portos tomato seedlings using Kinect 2.0 to build a more accurate, cost-effective, and portable system. The proposed methodology classifies the tomato seedlings into four categories: The first corresponds to the seedling with normal growth at the time of germination; the second corresponds to germination that occurred days after; the third category entails exceedingly late germination where its growth will be outside of the estimated harvest time; the fourth category corresponds to seedlings that did not germinate. Typically, an expert performs this classification by analyzing ten percent of the randomly selected seedlings. In this work, we studied different methods of segmentation and classification where the Gaussian Mixture Model (GMM) and Decision Tree Classifier (DTC) showed the best performance in segmenting and classifying Portos tomato seedlings.

1. Introduction

Multiple types of sensors have been used for 3D digitalization [1]. In this work, we used the Microsoft Kinect 2.0 sensor, which is a low-cost device that allows acquiring colored 3D point clouds, combining a color and a depth camera. The point clouds acquired with the depth sensor are defined in cartesian coordinates (x, y, z); these coordinates represent the surface of the scanned object or area. Depth-based systems have been used to solve neuro-rehabilitation problems [2,3], video conferencing system [4], facial recognition [5,6], extraction and recognition of human body movements [7,8,9,10], search, localization, and detection of objects [11,12,13,14], navigation [15,16,17], robotics [18,19,20,21], reconstruction [22,23,24,25], modeling of objects or surfaces [26,27,28,29,30,31,32], and plant monitoring [33,34,35,36,37,38,39,40,41,42].

The motivation of this study is to solve real problems in greenhouses in central Mexico, using low-cost equipment, without modifying their conditions, adapting the prototype and methodology to reduce time, fatigue and predict the production of crops with the least error employing a portable prototype by optimizing and comparing the different stages of the proposed methodology.

The main contribution of this work is to present a method for monitoring seedlings in greenhouses for decision-making by reducing the use of water and agrochemicals. The results show that the proposed methodology predicts the final yield of the Portos tomato seedling crop better than an expert. In addition, the solution is low cost, at around €300, and highly efficient in determining seedling growth within the 15th day of germination.

Considering that agricultural ecosystems are complex, this paper proposes a suitable approach for such systems. In [43], Dobrota et al. identified the errors and limitations found in the state-of-the-art mainly produced by environmental factors such as the position of the sun. Other frequent errors are mathematical operations and the use of small samples. Our work analyzed these factors carefully during the seedling monitoring, pre-processing, and processing of scanned data and the amount of sample used. The results demonstrate the efficiency of our prototype and methodology.

In this work, we propose the ScanSeedling v1.0 prototype. This prototype mimics an expert visual monitoring. It takes the same representative sample as the expert and scans one tray at a time. The Kinect is positioned on the top of the trays, vertically oriented facing the ground. The Kinect can be positioned at two different heights due to the change in the trays’ size. Greenhouses normally use trays made of rigid plastic or soft plastic. Rigid plastic trays are larger than the soft ones, so the Kinect must be positioned according to the tray-type on the z-axis. We used the $(x,y)$ axes to adjust the Kinect in the center of the trays. Figure 1 shows the seedling scanning prototype using rigid trays. We optimized the computational algorithms in this prototype for monitoring Portos tomato seedlings. We decided to analyze this type of seedling because its cultivation represents high production costs, making it a priority seedling for entrepreneurs.

According to the National Institute of Statistics and Geography (INEGI), around three million tons of Portos tomatoes are produced in Mexico, representing approximately US$750 millon per year, the seventh-largest agricultural production in Mexico. According to Victor Villalobos, the former general of the Inter-American Institute for Cooperation in Agriculture (IICA), as stated in the National Agricultural Planning of 2017–2030, published by the ministry of the interior (SEGOB), it is necessary to increase agricultural production by at least $70\%$ by the year 2050 in a sustainable manner, avoiding crop losses, improving area utilization, and using less water. Finally, he proposes using technology and innovation to reduce the existing gap. On the other hand, in the same planning document, Martín Piñeiro, a special advisor to the director general of the Food and Agriculture Organization of the United Nations FAO, places Mexico as an agricultural world leader exporter.

The visual process of growth monitoring is tedious, time consuming, and error prone. The wear and tear are more noticeable in seasons when the greenhouse has a high production demand. We developed this work in six steps: (1) Concept analysis, (2) analysis of applied research, (3) design and implementation of a laboratory prototype, (4) implementation of applied research, (5) proposal and testing of a prototype in a simulated environment, and finally (6) implementation of the prototype in a natural environment. These steps allowed us to produce a low-cost and functional prototype. In addition, we propose a real-time procedure to automate the visual monitoring of the production of Portos tomatoes that achieves results in less time. Figure 2 shows the methodology proposed in six stages:

• 3D seedling scanning
• Tray point cloud extraction
• Seedling segmentation
• Morphological features extraction
• Seedling classification into four categories
• Output monitoring using a graphical user interface (GUI) that shows the classification results and provides output exporting capabilities.

Figure 2. The methodology is divided into six stages: 3D seedling scanning, tray point cloud extraction, seedling segmentation, height and development features extraction, seedling classification into four categories, and output monitoring.

Figure 2. The methodology is divided into six stages: 3D seedling scanning, tray point cloud extraction, seedling segmentation, height and development features extraction, seedling classification into four categories, and output monitoring.

The rest of paper is divided into four sections. Section 2 describes the related works. Section 3 describes the materials and methods in three subsections: 3D segmentation, morphological feature extraction, and seedling classification. Section 4 presents the results, Section 5 includes a discussion, and Section 6 presents the conclusions.

2. Related Works

There have been multiple approaches for seedling monitoring; Among the most used, we can find phenotyping based on laser scanning, root growth analysis, and plant growth analysis. Two-dimensional/three-dimensional data are used in controlled or uncontrolled conditions to obtain morphological characteristics spatio-temporally, with one or several viewing angles of the plant, and with one or several acquisition systems [44,45,46,47].

2.1. 2D Dynamic Measurement Systems

Nowadays, the methods using 2D data remain the basis for image analysis applied to plant growth, such as measuring and searching for characteristics. However, measurements from 2D images are often inaccurate. The work of Spalding and Miller [48] described four research trends applied to plant growth measurement: (1) Basic image analysis [49,50,51,52], (2) morphometrics [53,54], (3) kinematics [55,56], and (4) 2D versus 3D comparisons [57,58].

The works of [40,41] are carried out in natural environments. Ref. [40] detected and sprayed the plant based on volume, reducing the use of agrochemicals. In [36], the authors detected and extracted stems using stereo vision systems, arguing that stems are closely related to plant growth. In contrast, [41] calculated the seedling’s height using, stereoscopic images by converting them into disparity images to determine the depth and then obtain and segment the regions of interest. In this work, we performed the Portos tomato seedling scanning in a natural daylight environment avoiding scanning the seedling when the sun is above the sensor to avoid interference. Our prototype and the methodology proposed in this paper allows efficient and effective monitoring of the development of the Portos tomatoes seedling, one of the most expensive crops to produce in Mexico.

Currently, neural networks are used to solve many problems, including agriculture [59,60,61,62,63]. Zhang et al. [63] proposed a method to monitor the growth of lettuce in greenhouses, using images and convolutional neural networks (CNNs). They trained a model to obtain the relationship between images and growth-related traits. The authors showed that the values estimated by the CNN agree with the real measurements; they proved that this technology is a robust tool in monitoring lettuce growth in greenhouses. Neural networks’ advantage over classical methods is that they can learn from the data without manual optimization, and the segmentation can obtain highly effective results. However, we selected classical methods to reduce hardware and annotation costs. Still, however, the use of images and classical methods increase the error in monitoring any plant compared with CNNs; for this reason, we present a prototype based on 3D monitoring. We compared and optimized the methods used at each stage to reduce the need for high-cost technology.

2.2. 3D Dynamic Measurement Systems

Three-dimensional sensing methods for plant phenotyping have increased in recent years [64,65], the following works present biological applications using 3D sensing, derived traits, and different measurement techniques. The methods can be divided into six different areas or techniques: (1) Laser triangulation LT [66,67], (2) structure from motion SfM [68,69], (3) structured light SL [70,71], (4) time-of-flight ToF [72,73,74,75], (5) light field $LF$ [75,76], and (6) terrestrial laser scanning TLS [77]. These techniques allow the evaluation of plant features such as seedling size, leaf width and length, volume, and development at the plant and organ level. We selected a time-of-flight (ToF) sensor for its accuracy in monitoring open environments. Unlike other sensors such as laser scanning, or structure from motion (SfM) methods, ToF has a faster processing time. Refs. [72,73] use an unmanned aerial vehicle (UAV), and [74,75] use LiDAR systems. Moreover, ToF systems are commonly inexpensive and accessible for most greenhouses in Mexico.

DiFilippo and Jouaneh [78] proposed an automatic 3D measurement system and investigated the performance of three different models of the Kinect, obtaining significant results. Our work offers a low-cost, optimized, robust, and scalable scanning prototype with real-time results. In addition, we obtained seedling growth and development to determine estimated crop yield. The low cost and fast acquisition of the Kinect make it a promising tool for field studies, especially in small stature vegetation, even though it shows limitations in daylight [34].

The works presented in [33,35,37,38,39] propose methodologies in controlled environments obtaining multiple views of the seedling and, in some cases, using rotating platforms. In the point cloud processing stage using state-of-the-art classical techniques, it is necessary to pre-process the data to obtain the 3D model of interest. The pre-processing can include stages such as background removal, outlier removal, and denoising [64]. Once the model is obtained, the processing is divided into three stages: Segmentation, feature extraction, and classification. In the first stage, according to [64], the segmentation is a critical phase in which there is no standard method that can be used because segmentation depends on the quality of the 3D data and seedling physiology. Gibbs et al. [79] propose an automatic methodology for data acquisition and retrieval of 3D plant models. The segmentation stage suggests a hybrid clustering algorithm that combines the algorithm’s principal component analysis (PCA) and modified automatic divisive hierarchical clustering (DIVFRP). Their proposal obtained better accuracy compared with K-Means, Iterative Principal Direction Divisive Partitioning (IPDDP), and spectral clustering. In our proposal, we compared and optimized K-Means and Gaussian Mixture Model GMM, and quantitatively demonstrated that GMM obtains better accuracy for plant segmentation.

The second stage entails the extraction of characteristics from the point clouds. For this task, Paturkar et al. [64] divide the approaches into three groups, including individual plant trait measurements, whole-plant measurements, and canopy-based measurements. Nguyen et al. [80] present a methodology for extracting the plant height, the number of leaves, the leaf size, and the internode distances using point clouds. First, they use Euclidean clustering to segment the plant into smaller parts. Then, they use eigenvalues to group the seedling features to extract the features. This work proposed Algorithm 1 to extract the height seedling and mean shift clustering to determine development.

Algorithm 1 Height Seedling

Require: Data X where $x_i\in {\Re }^d;i=1...n$

Ensure:$w_H\in {\Re }^d$

$m_{j_{(x,y,z)}}\leftarrow min\left(x_i\right)$

$m_{k_{(x,y,z)}}\leftarrow max\left(x_i\right)$

procedure Projection($m_j$,$m_k$)

if $m_{j_{(x,y)}}$!=$m_{k_{(x,y)}}$ then

$m_{x_{\left(z\right)}}\leftarrow m_{j_{\left(z\right)}}$

return $m_{x_{\left(x\right)}}\leftarrow m_{k_{\left(x\right)}}$

$w_H=${$x\in X \backslash dist(m_{x_{(x,z)}},m_{k_{(x,z)}})$ is minimum}

The third stage involves classification. Guoxiang and Xiaochan [33] performed 3D point cloud reconstruction maps of the greenhouse-grown tomato plant to verify the accuracy of the extracted morphological characteristics. Furthermore, they argued that the correlation between the real values of the measurements can be calculated directly from the reconstructed 3D point cloud. In our work, we implemented, optimized, and compared five classifiers quantified using precision and recall to classify tomato seedlings into four classes: Three developmental stages and one failure.

3. Materials and Methods

This section presents a methodology for extracting, segmenting, and classifying scanned seedlings in a greenhouse. The main procedure that involves the processing of the point clouds and obtaining the morphology of the seedling to determine its growth is divided into four stages: Tray extraction of the point cloud, seedling segmentation, morphology features extraction and seedling classification; see Figure 3.

Currently, in the central region of Mexico, seedling growth is visually monitored by trained personnel based on experience. On the 15th day of sowing the seed, the expert randomly selects $10\%$ of the crop trays to classify the seedling into four stages or categories: The first stage corresponds to the seedling whose growth is normal to the time of germination with average height of 6 cm as shown in Figure 4a; the second stage is the seedling whose germination occurred days after the first stage with size ranges around 4 cm in height as shown in Figure 4b; the third stage is the seedling that germinated after the second stage, and it will grow outside of the estimated harvest period; its height is less than 3 cm as shown in Figure 4c; and the fourth stage corresponds to seedlings that did not germinate.

The tomato seedling cultivation process is divided into five stages: (1) Seed sowing, (2) storage in germination chamber for approximately 65 h, (3) tray extended on the greenhouse, (4) germination day count 15, and (5) harvesting.

Table 1. Cultivation process stages. Stage 1: Seedling; stage 2: Storage in germination chamber; stage 3: Tray extended; stage 4: Germination day count; stage 5: Harvesting.

		Stage 1	Stage 2	Stage 3	Stage 4	Stage 5
ID	Tray	Seeding	Germ. Cham.	Extended	Germ. Count	Harvest
A1	179	May 20	May 20	May 23	June 4	June 12
B2	721	May 20	May 20	May 23	June 4	June 12
C3	176	June 19	June 19	June 22	July 5	July 19

shows the five stages of the cultivation process of three different productions that we monitored for this work. Figure 5 shows the Portos tomato seedling at 15 days after sowing. The trays show irregularities because not all the seedlings have the same growth level, thus complicating monitoring and production decisions. The A1 and B2 crops differ because a new sowing technique was tried in A1. This change in sowing technique did not affect the harvest or the methodology proposed in this work.

The monitoring or counting to determine the production is performed 15 days after sowing. In this work, we monitored $10\%$ of each crop; the total sample corresponds to 107 trays on three productions. Each tray had 72 cavities for a total of 7704 seedlings, of which $70\%$ was taken for training and $30\%$ for testing. We discard the outer trays for testing because they tend to have better development since they do not compete for sunlight.

Ambient temperature is vital in seedling development, so it is crucial to control the greenhouse environment. If the temperature drops outside the greenhouse, the humidity rises inside. Conversely, the moisture drops inside if the temperature rises outside the greenhouse. In central Mexico, during the summer season, the outside temperature varies on opposite poles. During the day, the temperature increases, and during the night, the temperature drops. The heat of the day allows an accelerated development of the seedling with low humidity while the humidity increases during the night. Therefore, we controlled the conditions inside the greenhouse. Figure 6 shows the temperature and humidity changes from 17 May to 24 May 2021. The green line represents the temperature, the purple line is the humidity, the horizontal line in mustard color is the high alarm, and the horizontal line in brown color is a low alarm. These climatic parameters play an important role in production.

3.1. 3D Segmentation

This section describes the tray extraction and tray cavities segmentation from the rest of the image. First, we extracted the planes using the point clouds for two main reasons: To model the digitized environment in real time, separating objects with the (x,y)-plane, and reducing the computational cost and processing time. Then, we projected the point cloud to the (x,y) plane; this projection is lossless and specifies an extraction polygon with predefined coordinates to process the information of interest. The prototype design ScanSeedling v1.0 allows scanning trays in the same position. Once a tray is extracted $(x,y,z)$, we applied and compared two segmentation algorithms: Gaussian Mixture Model (GMM) and K-Means. We chose these two unsupervised methods because they provide an advantage over other methods when the number of clusters is known; in this case, if we know the number of tray cavities in advance. The user provides this information from the GUI, allowing the initialization of the segmentation with n seeds, increasing the efficiency during seedling segmentation.

K-Means [81,82,83] is a technique that groups the data into K clusters. Elements that share similar features will be clustered and isolated from points that do not share similar features. To know if the data belong to the same group K-Means uses the distance between the data, and similar observations will have a smaller distance between them. We use a Euclidean distance as a measure, although there are other functions as Mahalanobis distance, Manhattan distance, Chebyshev distance, and others.

Gaussian Mixture Models (GMMs) is a probabilistic clustering model and classification method based on the probability density of the data [83,84,85]. GMMs consist of K multivariate Gaussian distributions know as mixture components, where each component k has its own parameter set $\{{\mu }_k,\sum {.}_k\}$ where ${\mu }_k$ is the mean value vector and $\sum {.}_k$ is the covariance matrix of the multivariate Gaussian. Each component of the mixture also has an associated mixing weight $0\le {\pi }_k\le 1$, where ${\sum }_{k=1}^K{\pi }_k=1$. In a graphical representation, the joint distribution is given by $p(x,z)=p\left(z\right)p(x\mid z)$ and the marginal distribution of x is the sum of the joint distribution over all the possible states of z. GMM is given in Equation (1).

$$\sum _{k=1}p\left(z\right)p(x\mid z)=\sum _{k=1}^K{\pi }_{k}N(x\mid {\mu }_k,\sum {.}_k)$$

(1)

3.2. Morphology Features

Once the seedling was segmented and the dataset created, we extracted the seedling features. This section presents the methodology of seedling height and feature extraction. The development and size of the seedling determine its stage and complement each other to achieve its classification. No matter how the seedling developed, if its height is equal to or similar to the second stage’s height, it is considered to be in the second stage.

3.2.1. Seedling Height

The agriculture process fills the tray cavities with a substrate; the seed is sown, and the agrolite is placed. Agrolite is a volcanic rock that contains water in its molecular structure; it aids decompaction, aeration, and moisture retention in seedling production. Cotyledons are the first leaves of plants; they are distinguished from the other secondary leaves of the seedling by their size, and they contain nutrients such as oil starch and can digest the albumen.

The height calculation consists of three stages: First, we normalized the points that correspond to agrolite; second, we identified the maximum height of the seedling, where the highest point commonly corresponds to the leaves because these protrude with respect to the stem; finally, we adjusted the normal to the $(x,y)$ plane of the highest point. This adjustment in all cases generated the distance in a straight line from the agrolite to the maximum height found. See Algorithm 1, where X is a point cloud, $m_j$ and $m_k$ are the minimum and maximum points on the point cloud, and $m_{x_{\left(x\right)}}$ receives the value of the z-axis, to obtain the projection from the highest point to the lowest point in a straight line.

3.2.2. Seedling Development

Once the height of the seedling is obtained, we segmented it, including leaves and stems. The number of leaves is unknown and varies from one seedling to another. We used the mean shift method for computing the seedling development. Mean shift is a non-supervised technique that does not need to be fed by the user. This algorithm increases the procedure’s efficiency. It can be defined as the decomposition of the sapling to analyze its homogeneous regions, which share spatial distribution and intensity characteristics. The algorithm clusters a dataset of dimension d by associating each point with the mode or peak of the probability density function. The authors of [86] present a detailed evaluation of the algorithm.

3.3. Classification

Using the morphology features identified in the previous section, we evaluated the following classifiers: Neighborhood Components Analysis (NCA), a linear model with stochastic gradient descent (SGD), a Decision Tree Classifier (DTC) or Decision Trees (DTs), a Support Vector Machine (SVM-Lineal), and a Radial Basis Function (SVM-RBF), SVM-Polynomial.

3.3.1. Neighborhood Components Analysis

This paper presents an unsupervised nearest neighbor method based on spectral clustering and manifold learning. The goal is to find a group of nearby samples that belong to an object and predict the label. The k-nearest with Euclidean distance used in this work considers the morphological data of the seedlings, which are uniformly sampled. The Neighborhood Components Analysis (NCA) is used in [87] to choose the significant features from a huge dataset. The NCA goal is to learn an optimal linear transformation matrix of size $(n,m)$, where n is the components, and m represents the features. The goal is to learn a projection A which maximizes the samples of the probability that $p_i$ is correctly classified (Equation (2)).

$$argmax\left(A\right)\sum _{i=0}^{N-1}{p}_i$$

(2)

where N is the number of samples and $p_i$ is the probability of the sample being correctly classified Equation (3).

$$p_i=\sum _{j\in C_i}^{N-1}{p}_{ij}$$

(3)

$C_i=\{j\mid y_j=y_i\}$ is the set of points in the same class as sample i, A is a linear transformation matrix, $p_{ij}$ is the softmax over Euclidean distances shown in Equation (4).

$$p_{ij}=\frac{exp(-\parallel A_{x_i}-A_{x_j}{\parallel }^2)}{{\sum }_{k\ne i}exp-(\parallel A_{x_i}-A_{x_k}{\parallel }^2)},p_{ii}=0$$

(4)

where each point j in the training set has probability $p_{ij}$ of assigning its label to a point i that decays as the distance between points $(i,j)$ increases.

3.3.2. Stochastic Gradient Descent

Stochastic Gradient Descent (SGD) is used on multi-class classification problems such as text classification and natural language processing of large-scale and sparse machine learning using one versus all [88]. SGD computes the confidence score, the signed distances to the hyperplane for each classifier and chooses the class with the highest confidence. There is a set of training examples $(x_1,y_1),\dots ,(x_n,y_n)$ where $x_i\in R^m$ is an arbitrary input or weight vector and $y_i\in R$, $y_i\in \{-1,1\}$ is a scalar output or the sum of squared residuals for classification. The algorithm can be divided into five steps.

• Compute the gradient of the function. The goal is to learn a linear scoring function concerning each feature. If the model has more features like $x_1$ to $x_n$, take the partial derivative of y with respect to each of the features.
• Update the gradient function by plugging in the parameter values.
• Calculate the step sizes for each feature as $size=gradient \ast rate.$
• Calculate the new parameters as $ne{w}_{param}=ol{d}_{param}-size.$
• Repeat steps 3 to 5 until the algorithm converges, i.e., the gradient is almost 0.

We used a learning rate of $0.01$ to avoid taking long steps down the slope.

3.3.3. Decision Tree Classifier

Decision Trees (DTs) are a non-parametric supervised learning method used for classification. The DT classifier predicts the value of the target variable using the data features through decision inferred rules [89]. It finds the features that offer the most significant entropy reduction assuming the sub-trees remain balanced. Given the feature vectors $x_i\in R^n$, $i=1,\dots ,l$ and a label vector $y\in R^l$, a decision tree groups samples with the same labels by recursive partitions through the feature space. $P_m$ represents the data at node m with $N_m$ samples. For each candidate split $\theta =(j,t_m)$ consisting of a feature j and threshold $t_m$ and partition the data into sub-tree left $P_m^{left}\left(\theta \right)$ (Equation (5)) and sub-tree right $P_m^{right}\left(\theta \right)$ subsets (Equation (6)).

$$P_m^{\mathrm{left}}\left(\theta \right)=\{(x,y)\mid x_j<=t_m\}$$

(5)

$$P_m^{\mathrm{right}}\left(\theta \right)=P_m \backslash P_m^{\mathrm{left}}\left(\theta \right)$$

(6)

DTs compute a loss function $H\left(\right)$ to measure the quality of the splitting of the candidate node m, solving the choice of the same based on the classification Equation (7).

$$G(P_m,\theta )=\frac{N_m^{\mathrm{left}}}{N_m}H\left(P_m^{\mathrm{left}}\left(\theta \right)\right)+\frac{N_m^{\mathrm{right}}}{N_m}H\left(P_m^{\mathrm{right}}\left(\theta \right)\right)$$

(7)

$${\theta }^{\ast }={\mathrm{argmin}}_{\theta }\mathrm{G}(P_m,\theta )$$

(8)

where ${\theta }^{\ast }$ minimises the impurity until allowable depth on subsets $P_m^{\mathrm{left}}\left({\theta }^{\ast }\right)$ and $P_m^{\mathrm{right}}\left({\theta }^{\ast }\right)$ while $N_m<mi{n}_{samples}$ or $N_m=1$. If node m obtains a classification into values $0,1,\dots ,K-1$, Equation (9) is applied to determine the classification criteria.

$$p_{mk}=\frac{1}{N_m}\sum _{y\in P_m}I(y=k)$$

(9)

where $p_{mk}$ is the proportion of class k observations in node m. If m is a terminal node, the prediction probability for this region is set to $p_{mk}$; where k is the class and m is the node. The common impurity metrics are gini, entropy and misclassification. To avoid overfitting, we used the minimal cost-complexity pruning algorithm, parameterized by $\alpha \ge 0$, to measure the cost-complexity of a given tree. Equation (10).

$$R_{\alpha }\left(T\right)=R\left(T\right)+\alpha \mid \tilde{T}\mid$$

(10)

where $\mid \tilde{T}\mid$ is the number of terminal nodes in T and is defined $R\left(T\right)$ as the total misclassification rate of the terminal nodes.

3.3.4. Suppport Vector Machine

Support Vector Machine (SVM) is a set of supervised learning methods for binary classification. Its application has been extended to multi-class classification and regression problems. In a $p-dimensional$ space, a hyperplane is defined as a flat affine subspace of $p-1$ dimensions. In three-dimensional space, a hyperplane is a two-dimensional subspace, a conventional plane. SVM is a quadratic programming problem $QPp$, separating support vectors from the rest of the data. The $QPp$ solver is used in a scale between $O(n_f\times n_s^2)$ and $O(n_f\times n_s^3)$, f is the number of features, and s is the number of samples.

In this work, we evaluated a multi-class SVM as one-vs-rest classification. We evaluated three different kernels: Lineal $\langle x,x^{{}^{\prime }}\rangle$ [90], RBF exp $(-\gamma \parallel x-x^{{}^{\prime }}{\parallel }^2)$ [91] and polynomial ${(\gamma \langle x,x^{{}^{\prime }}\rangle +r)}^d$ [92], where d is specified by parameter $degree$ and r by $coef\left(0\right)$, and $\gamma$ must be greater than 0.

4. Results

This work mimics the visual monitoring performed by a qualified expert to classify the seedling into four growth stages. We used three different crops to train and test the algorithms. This section is divided into three subsections: Segmentation, morphological features extraction, and classification.

4.1. 3D Segmentation

This section describes two stages: Tray extraction and tray cavity segmentation. When the system starts, the user provides the tray size which is used to determine the extraction polygon, and the system automatically adapts the extraction and segmentation. Figure 7a shows 72 cavity trays; Figure 7b shows the point cloud in the same tray with $12\times 6$ cavities captured using the Kinect V2.0 sensor. The color in the seedlings indicates the height, where the blue color represents the highest and red the lowest. In the point cloud shown in Figure 7c, the red-colored boxes indicate empty cavities where the seed did not germinate, the yellow-colored boxes show cavities with late seedling germination, and the green boxes show seedlings with low growth. These three boxes represent production failures. There is no way for the seedling in the red, yellow, and green boxes to achieve the seedling development categorized as stage first.

Once we extract the trays from the point cloud, we used K-Means, and GMM methods to segment the cavities of the trays. Figure 8a shows the segmentation result using the K-Means algorithm and Figure 8b shows the segmentation performed with the GMM technique. Each segmentation is stored and processed in $(x,y,z)$. Each color change represents a segmented cavity; due to graphic limitations of the programming language, in both images, the colors are repeated.

We optimized and compared K-Means and GMM segmentation techniques; see

Table 2. Segmentation Comparison., F1-score, average precision (AP), and Area under the ROC curve (AUC).

Method	F1-Score	AP	AUC
KMeans	0.7361	0.7766	0.6137
GMM	0.8474	0.8434	0.7117

. As ground truth, we used a hand-labeled dataset to compare both segmentation techniques. The GMM technique shows better performance for segmenting Portos tomato seedlings in 72 cavity trays; its average precision (AP) is $0.8434$, F1-Score $0.8474$, and area under the ROC curve (AUC) equals $0.7117$ against the average precision AP $0.7766$, F1-Score $0.7361$, and AUC of $0.6137$ of K-Means.

Once the trays were segmented, and the seedlings were extracted, we constructed the seedling dataset, labeled it, and divided it into three growth stages and failures. Figure 9 shows an example of each class of seedlings that form the training and test sets segmented with the GMM technique. Figure 9a shows a seedling in the first stage for its height and development; the red color points correspond to the agrolite, the cotidellons, the first and second true leaf. Figure 9b shows a second stage seedling. However, its height is similar to Figure 9a; its development does not correspond to a first stage seedling. Figure 9c shows the seedling in the third stage with small height; the cotyledons and the first true leaf can be observed. Figure 9d shows a cavity; the color changes correspond to agrolite and the part of the substrate where the seed did not germinate.

4.2. Morphology Features

/hlDuring the first 15 days after sowing, the seedlings pass through three stages. During the first stage, they should have cotyledons and their first true leaf. Apart from the cotyledons during the second stage, their second true leaf should be emerging. Finally, in the third stage, the seedlings are about to germinate or have their cotyledons in development. In Figure 10 the agrolite is shown in red color, the leaves marked with number one correspond to cotyledons, number two are the first true leaf, and number three is the second true leaf.

Algorithm 1 presented previously describes how to extract the seedling height without agrolite. First, the agrolite points are normalized; then, we obtained the highest seedling point and assigned the $(x,z)$ coordinates to the normalized points, applying a threshold of 4.15 cm to cut the seedling and eliminate any agrolite that might exist. Finally, the distance between both points defines the height of the seedling. The green vertical line represents height as shown in Figure 10. This process ensures that the agrolite and the scanned part of the tray are removed. The cavity is discarded if the outliers in the set are $\{\varnothing \}$, counting it as a failure.

Figure 11a shows point cloud outliers resulting from the extraction, once the Portos tomato seedling is extracted. The next stage is to extract the seedling development. We used mean shift to obtain the development by segmenting each characteristic of the seedling. It segments $n-1$ features, where n are the segmented clusters corresponding to the leaves minus the stem. Figure 11b shows the segmentation of four leaves and stems. The points segmented with orange and red color correspond to cotyledons; in brown and green, we show the first segmented genuine leaf, and the red points represent the stem. To reduce seedlings’ over-segmentation error, we discarded distances greater than $2.5$ cm from the stem centroid to the centroid of each segmented leaf.

We optimized the thresholds to extract the height features and seedling development at 4.15 cm and 2 cm. We obtained these values by varying the thresholds in an interval. For example, we varied the height threshold from $1.5$ cm to $6.5$ cm. For the leaf size threshold, we varied from $0.5$ cm to $5.5$ cm; in both cases, with increments of $0.1$ cm. We used precision and recall to quantify the optimal threshold.

4.3. Classification

The last step of our method computes the developmental stage and height features using classifiers. Then, the classifiers are used to obtain the estimated crop yield. Finally, we obtain the classification report of each classifier, the weighted mean, and the model’s accuracy.

According to greenhouse quality standards, we performed the monitoring process of the greenhouse tomato seedling on the 15th day after planting.

Table 3. Crop production accuracy by growth stage of each classifier against the expert’s crop production accuracy.

Classificator Crop Production %
Stage	ID	NCA	SGD	DTC	Lineal	RBF	Poly	Expert
Third	A1	6.75	2.18	8.33	2.02	5.95	5.36	13.19
	B2	8.53	0.99	8.93	2.25	6.75	8.93	10.69
	C3	9.72	8.33	9.92	8.73	9.92	10.52	8.75
Second	A1	13.1	9.72	13.49	13.49	11.9	13.29	8.08
	B2	13.69	8.93	14.09	14.09	12.7	13.49	19.16
	C3	17.86	12.5	19.25	13.69	14.48	16.27	18.13
First	A1	68.06	70.24	70.83	69.84	68.45	67.66	70.40
	B2	72.42	72.42	70.22	71.23	71.43	70.83	61.40
	C3	63.1	64.09	63.1	64.09	62.5	63.29	62.43
Fails	A1	6.94	7.14	7.14	6.35	6.35	6.35	8.33
	B2	3.97	3.97	4.37	1.45	2.38	3.17	8.75
	C3	5.16	4.96	5.36	0.4	4.96	5.16	10.69
15th day Prod.	A1	87.90	82.15	92.66	83.34	86.51	86.51	91.67
	B2	94.64	82.34	93.23	87.57	90.87	93.26	91.25
	C3	90.67	84.92	92.26	86.5	86.91	90.08	89.34
Final Prod.	A1	-	-	-	-	-	-	93.06
	B2	-	-	-	-	-	-	91.67
	C3	-	-	-	-	-	-	94.44

shows the accuracy of the six algorithms used to classify the three growth stages and the failures. Our results are compared against the expert’s accuracy, where A1, B2, and C3 are the three Portos tomato crops that were monitored. Once the crop is harvested, the expert obtains the final yield by counting the occupied cavities in the tray. The DTC classifier has the best accuracy of the six classifiers. In addition, it has the best accuracy in predicting the final yield against the expert.

The model accuracy corresponds to the correct prediction count for each crop compared against the hand-labeled dataset. According to model accuracy, see

Table 4. Model accuracy comparison of multiple classifiers.

Model Accuracy (%)
Classificator	A1	B2	C3	Average
NCA	94.84	98.61	95.83	96.42
SGD	89.29	86.31	89.88	88.47
DTC	99.80	99.60	97.62	99.00
SVM Lineal	89.68	89.02	86.90	88.53
SVM RBF	92.86	93.25	91.87	92.66
SVM Poly	92.86	96.43	95.24	94.84

, $DTC$ is the best classifier with $99\%$ correct predictions. In contrast, SGD and SVM Lineal are the worst classifiers, with an accuracy below $90\%$. Our system can achieve human-level performance in predicting the final yield of Portos tomato crops.

An error analysis showed that the expert estimates crop production with a variation of $\pm 2\%$. To obtain the production, the failures are subtracted from the accuracy of the model of each classifier. The crop production or final production is obtained by a visual count of cavities containing substrate, agrolite, and, in some cases, third-stage seedlings.

Finally, Table 4 shows a summary of the model accuracy of six classifiers for the three crops analyzed. $DTC$ proved to be the best classifier with an average accuracy of $99\%$ for classifying Portos tomato seedlings into four classes.

Table 5. Results for the three test crops named A1, B2, C3 and a weighted average.

Classificator Accuracy (%)
Metrics	ID	NCA	SGD	DTC	Lineal	RBF	Poly
A1 Crop
Precision	A1-0	0.9	0.71	1.00	0.73	1.00	0.79
	A1-1	1.00	1.00	1.00	0	0.94	0.9
	A1-2	0.77	0.71	0.99	0.63	0.72	0.74
	A1-3	0.99	0.95	1.00	1.00	0.97	1.00
Recall	A1-0	0.97	1.00	1.00	0.89	0.92	0.92
	A1-1	0.79	0.26	0.98	0	0.70	0.63
	A1-2	0.97	0.72	1.00	1.00	0.88	0.99
	A1-3	0.96	0.99	1.00	0.99	0.97	0.96
F1 Score	A1-0	0.93	0.83	1.00	0.80	0.96	0.85
	A1-1	0.88	0.41	0.99	0	0.80	0.74
	A1-2	0.86	0.72	0.99	0.77	0.79	0.85
	A1-3	0.98	0.97	1.00	0.99	0.97	0.98
B2 Crop
Precision	B2-0	0.95	0.77	0.96	0	0.92	0.94
	B2-1	0.96	0.71	1.00	0	0.71	0.83
	B2-2	0.97	0.56	0.99	0.54	0.80	0.89
	B2-3	0.99	0.93	1.00	1.00	0.99	1.00
Recall	B2-0	0.91	0.91	1.00	0	0.55	0.73
	B2-1	0.93	0.11	0.98	0	0.74	0.98
	B2-2	0.97	0.63	1.00	1.00	0.90	0.96
	B2-3	1.00	1.00	1.00	0.98	0.99	0.98
F1 Score	B2-0	0.93	0.83	0.98	0	0.69	0.82
	B2-1	0.95	0.19	0.99	0	0.72	0.90
	B2-2	0.97	0.60	0.99	0.70	0.85	0.93
	B2-3	1.00	0.97	1.00	0.99	0.99	0.99
Metrics	ID	NCA	SGD	DTC	Lineal	RBF	Poly
C3 Crop
Precision	C3-0	0.81	0.86	0.84	1.00	0.89	0.93
	C3-1	0.91	0.74	0.96	0.64	0.76	0.76
	C3-2	0.95	0.88	0.95	0.86	0.89	0.95
	C3-3	0.98	0.93	1.00	0.92	0.96	1.00
Recall	C3-0	0.96	0.93	1.00	0.07	0.93	0.96
	C3-1	0.89	0.76	0.91	0.80	0.91	0.96
	C3-2	0.91	0.64	0.98	0.70	0.74	0.83
	C3-3	0.98	1.00	0.98	1.00	0.98	0.99
F1 Score	C3-0	0.88	0.89	0.92	0.14	0.91	0.95
	C3-1	0.90	0.75	0.93	0.71	0.83	0.85
	C3-2	0.93	0.74	0.97	0.77	0.81	0.89
	C3-3	0.98	0.97	0.99	0.96	0.97	0.99
Weighted Average
Precision	A1	0.96	0.90	1.00	0.85	0.94	0.94
	B2	0.99	0.85	1.00	0.80	0.94	0.97
	C3	0.96	0.90	0.98	0.88	0.92	0.96
Recall	A1	0.95	0.89	1.00	0.90	0.93	0.93
	B2	0.99	0.86	1.00	0.85	0.93	0.96
	C3	0.96	0.90	0.98	0.87	0.92	0.95
F1	A1	0.95	0.88	1.00	0.86	0.93	0.93
	B2	0.99	0.84	1.00	0.82	0.93	0.96
	C3	0.96	0.89	0.98	0.85	0.92	0.95

shows results for the three test crops named A1, B2, and C3.

We used precision and recall to evaluate the performance of the classification algorithms. Class 0 represents the failures, class 1 corresponds to the third stage, class 2 is the second stage, and class 3 represents the first stage. Figure 12 shows the precision for each class and the micro-average precision-recall of the four classes. Visually we can determine which algorithm performs best. The best performing algorithms for classifying the three stages and seedling failures are observed close to the [1,1] point of the quadrant, starting at the [0,1] point and ending at the [1,0] point. In contrast, the worst ranking results are far from the [1,1] point and, in some cases, often end at the origin. The yellow color represents the micro-average precision-recall of the four classes. The blue curve represents class 0, cyan corresponds to class 1, orange is class 2, and purple is class 3. DTC obtains the best micro-average precision-recall with area $=1.00$. At the same time, the worst micro-average precision-recall is from GSD with area = $0.90$.

On the other hand, the sensor viewing angle and trays are parallel during the seedling scanning. This angle favors the correct extraction of height and seedling development features. However, this viewing angle generates occlusion of most seedling stems, especially the first stage seedling. Algorithm 1 is used to obtain the height effectively.

5. Discussion

This prototype aims to test the correct functioning of the Kinect in a natural greenhouse environment, detect the ideal parameters to position the Kinect in the three axes $(x,y,z)$, and improve the results of the expert to determine the growth and development of the Portos tomato seedling. It is important to remember that the monitoring of the seedling is performed in one single stage, on the 15th day after planting. Therefore, the monitoring objectives can be divided into two phases: The first objective is to decide agrochemical application and water use, and the second objective is to predict final crop production.

In this project, we showed that it is possible to scan and monitor the seedlings during daylight using a Kinect sensor in natural greenhouse conditions. After testing the Kinect sensor, we found four situations where the Kinect reduces its acquisition capabilities:

• When the Kinect is facing the sun.
• When the sunlight is covering the sensor.
• When the scanning zone includes light and shadow.
• When overgrown plants occlude the others.

Considering the complexity of agricultural ecosystems, the methodology proposed in this work to extract the morphology in Portos tomato seedlings is compelling. The expert’s knowledge allows us to evaluate the proposed algorithms on three different Portos tomato crops. Figure 13 shows the normal distribution of Portos tomato seedling height test. It is shown that the measurement values were linearly related to the manual measurement values. Figure 13a shows culture A1, Figure 13b presents culture B2, and finally, in Figure 13c, we can see culture C3. The relative weighted average error at day 15 monitoring in three crops on our proposal against expert results is $\pm 1.96\%$.

The expert’s final production weighted average error was $\pm 2.30\%$ and our system weighted average error was $\pm 1.38\%$. Therefore, it is quantitatively demonstrated that our proposal reduced the error in predicting the final production of Portos tomatoes. The time to scan the seedling is about 0.2 s, the time to extract the tray from the point cloud is about $0.3$ s, the time to segment a tray of 72 cavities is $0.4$ s, and the total time to obtain height and development of the tomato seedling is $0.15$ s.

The labeling of the data used to train and test the algorithms was performed using software developed by the authors. Our system automatically scans the seedling, then extracts the tray using the polygon given by the user, then it segments the seedling using the Gaussian Mixture Models (GMMs) and extracts the seedling morphology, development, and height. Finally, it classifies the seedling into four categories using a Decision Tree Classifier (DTC). The GMM segmentation, DTC classification algorithms, and GUI are mounted in a local server. The server saves historical data of crop monitoring. Once the production monitoring process is finished, the GUI gives the end-user the option to download the monitoring report as a .csv or .pdf file.

We used python language to implement, optimize, and compare the algorithms. The Kinect is configured using the Robot Operating System (ROS) embedded into the python GUI. We used a mid-range laptop with a 7th gen i3, 4 cores, and 8 GB of RAM.

The use of the depth sensor in an outdoor environment required its calibration [93,94]. Therefore, we performed the sensor calibration in an indoor environment controlling the illumination received by the pattern, mainly in the back of the pattern and the front of the Kinect sensor. We used a standard 3 cm calibration pattern and collected images of the pattern at $0.60$ m and $1.2$ m distance from the sensor. As a result, we got a re-projection error for color, depth, and both concerning $0.15$, $0.08$, $0.17$ pixels, respectively.

The innovation of any technology development project is measured using the TRL levels proposed by the National Aeronautics and Space Administration (NASA). The project presented here entails six stages, described in the introduction section. These stages allow positioning the technological development in nine TRL maturity levels. Our prototype complies with the legal standards of Mexican agronomy and has been tested in a natural greenhouse environment; therefore, the level that corresponds to this development is the TRL-5 maturity level. The prototype and GUI (front end) and computational algorithms (back end) work accurately without modifying greenhouse conditions.

According to the manufacturer’s specifications, the seed used in the three crops was labeled in March 2020 with $90\%$ germination. We labeled the datasets by hand with the help of greenhouse experts. Our method optimized the segmentation and classification for the specific task achieving a segmentation with an average precision of $84.34\%$, and a classification accuracy of $99\%$.

We compare our work with those of Yang et al. [95] and Ma et al. [96] as they are the most similar to our research. In [95] is described the difficulty of finding a quantitative comparison criterion because the approaches focus on different tasks, while in [95,96], they scanned the seedling, placing the sensor over the seedling; [96] performed the scans outdoors, while in [95] and our proposed approach they were performed inside a greenhouse under natural conditions. Our work has three main differences concerning [95,96], namely the background and/or tray extraction, the seedling segmentation, and the morphology extraction. In [96] the authors extract the bottom according to a depth threshold and segment the seedling; the results are inaccurate if the crop is very dense. In [95], they propose to use FPFH to describe each point and employ K-means to cluster the bottom and crop points, extracting the bottom and the seedling tray. In our proposal, the user feeds the GUI with the type of tray to be processed, the point cloud is projected to the $(x,y)$ axes without losing information, and a polygon is used for extracting the tray. This process reduces the computational cost processes.

The segmentation stage in [96] knows in advance the sizes of the pots and crops; in [95], the authors use the VCCS and LCCP-based algorithms, while we propose an automatic segmentation comparing the performance of K-means and GMM. The last comparison is according to the morphology extraction. In [96], the authors include the color indices, and [95] included leaf area and canopy breadth. Our methodology extracts seedling development using mean shift, by stem count and the number of first true leaves and cotyledons. Finally, the height and development information allows us to classify the seedling according to its growth stage. In summary, our approach is based on robust algorithms and relies on a single pre-set parameter to extract the tray. This is because greenhouses in Mexico use trays with different characteristics such as height, length, width, number of cavities, and shape. Hence, the proposed procedure works for any tray type and makes it a general approach.

6. Conclusions

Seedling monitoring was carried out in central Mexico in late spring and early summer. These tests do not need repeating at other times of the year due to the internal conditions of the greenhouse. The methodology proposed in this work offers a new approach to seedling measurement in greenhouses. Our system uses different algorithms for each stage: ROS for scanning as shown in Figure 1; tray extraction using the proposed method and seedling segmentation using the GMM method as described in Section 3.1; morphology extraction using Algorithm 1 and seedling classification using the DTC algorithm, as shown in Section 3.3. Our results reveal that the optimized algorithms and the comparison allow us to obtain a predictive model for the harvest with less error than specialized personnel and in less time. Our system is affordable and easy to transport and install without modifying the greenhouse conditions.

Future works intend to propose a mechatronic system that allows scanning the totality of the crop trays. A remaining challenge in this proposal is the computational time that is considerably increased by using a method such as mean shift to segment each cotyledon, leaf, and stem. We plan to detect the development at a significantly shorter computational time. In addition, the algorithms and GUI will be scaled and determine the production of multiple tomato varieties and seedling types such as corn, carrots, onions, or watermelon, to name a few.