A Comparative Study of Vetiveria zizanioides Leaf Segmentation Techniques Using Visible, Infrared, and Thermal Camera Sensors in an Outdoor Environment

Abstract

A camera vision system is a fast and effective approach to monitoring leaves. It can be used to monitor plant growth, detect diseases, and conduct plant phenotyping. However, due to the outdoor environment of plants, it becomes challenging to use. This paper addresses the problems of Vetiveria zizanioides leaf segmentation by comparing different camera types and segmentation techniques. Visible, no infrared filter (NoIR), and thermal cameras interfaced in an embedded device were used to capture plants during the day and at night. Several popular thresholding techniques and the K-Means algorithm were employed for leaf segmentation, and their performance was measured using Recall, Precision, and F1 score. The comparison results show that the visible camera achieved the best performance on daytime images, with the highest Recall of 0.934 using Triangle thresholding, the highest Precision of 0.751 using K-Means (K = 3), and the highest F1 score of 0.794 using Multi-Otsu thresholding. For nighttime images, the highest Recall of 0.990 was achieved by the thermal camera using Isodata and Otsu thresholding, the highest Precision of 0.572 was achieved by the NoIR camera using K-Means (K = 3), and the highest F1 score of 0.636 was achieved by the NoIR camera using K-Means (K = 3). To compare the leaf segmentation performance of the thresholding techniques and the K-Means algorithm between our image dataset and the well-known plant image dataset, we also evaluated the methods using the Ara2012 image dataset. The results showed that K-Means (K-3) achieved the best performance. The execution time of K-Means was about 3 s, which was longer than the thresholding techniques. However, it is still acceptable for the real-time plant monitoring system.

1. Introduction

Image processing approaches have been used in a range of applications. One of the challenging and interesting applications is plant monitoring, where camera systems are employed to capture information of plants for further tasks, such as leaf detection and counting [1,2,3,4,5], growth monitoring [6,7,8,9], leaf classification [10], disease detection [11,12,13], stress monitoring [14,15], and phenotyping [16,17,18].

A leaf segmentation technique is commonly used as the preliminary process, which extracts the leaf area from the background in the image. Once the leaf area is detected, image analysis of the leaf object can be carried out. The most common leaf segmentation techniques are thresholding-based approaches, watershed, random walker, K-Means, artificial neural networks (ANNs), and color-index-based approaches.

HSV color thresholding was used in [19] for rice plant segmentation as the initial stage of plant height measurement. Color thresholding using the CIELAB color model was used in [20] for vine leaf segmentation from real environment images. HSI color segmentation was employed in [9] to extract leafy vegetables using a Kinect sensor. In [21], thresholding using a new color model was used to measure the leaf area of lettuce plants. Otsu thresholding based on the hue component was used for leaf area measurement in [22]. A software tool for estimating the leaf area was developed in [23] using RGB color thresholding to extract the leaf from the background.

The watershed algorithm has been used for rosette plant leaf segmentation [16] with a Dice score greater than 90%, cotton leaf segmentation [24] with a correct rate of 98%, and vegetable leaf segmentation [25]. The random walker algorithm was used in an interactive tool for leaf annotation [26] with a Dice of 97%. The robust random walker algorithm was proposed in [27] for leaf segmentation under different conditions with an F-measure of 98%. The K-Means algorithm was used for extraction of paddy leaf images for detecting leaf disease [12], conducting tomato leaf segmentation [28] with an F1 score of 98%, and detecting defective regions in leaves [29]. Deep learning was used in [2] for leaf counting. Mask R-CNN was used in [10] for leaf segmentation with a misclassification error of 1% and classification against complex backgrounds. A convolutional neural network (CNN) was used to detect and recognize leaf diseases [13]. Leaf segmentation based on color indexes, including normalized difference index (NDI), excess green minus excess red index (ExGR), and color index of vegetation extraction (CIVE), was addressed in [30] with segmentation rates of 80.7%, 80.9%, and 81.4%, respectively.

Even though the previously described leaf segmentation techniques showed a high performance, they used a visible camera (RGB camera). Thus, they worked in the daytime. Furthermore, most of them were not real-time systems. Other cameras, such as an infrared camera, are also employed. The NoIR camera, which is a standard camera without the infrared filter, thus allowing the infrared spectrum (around 880 nm) to be captured by the sensor, was used in [3,4,31,32]. A near-infrared (NIR) camera was used in [5] for leaf phenotyping. A stereo infrared camera that worked in the NIR spectrum (700 nm to 1400 nm) was used in [7] for leaf growth modeling. To the best of our knowledge, no such works have used a thermal camera for leaf segmentation. The typical applications of a thermal camera in plant monitoring are for plant canopy temperature measurement [33,34,35,36], plant water status monitoring [34,37], and leaf stomata conductance measurement [14,38]. A technique related to leaf segmentation was proposed by [36], in which a visible camera is combined with a thermal camera. In that system, the visible camera was used for leaf segmentation in order to define the region for temperature measurement by the thermal camera.

The thresholding technique is a simple, effective method for leaf segmentation, thus it is suitable for implementation on an embedded device for real-time application. However, the algorithm is sensitive to lighting changes. Therefore, the segmentation performance under different lighting should be adequately evaluated.

In this paper, we present the results of an experiment on techniques for Vetiveria zizanioides leaf segmentation using three camera sensors: a visible camera (standard RGB camera), a non-infrared filter (NoIR) camera, and a thermal camera. We evaluated the performance of several popular image segmentation techniques implemented with the three cameras. The objective was to find the best solution for a low-cost embedded camera system for real-time implementation of leaf monitoring.

The main contributions of this study are as follows:

• The image acquisition system employs low-cost camera systems suitable for real-time implementation.
• The evaluated image segmentation techniques are fast-computation algorithms suitable for implementation on the embedded device.
• The leaf segmentation performance of camera types and segmentation techniques was compared in an outdoor environment during the day and at night.
• This is the first work to investigate the feasibility of using a thermal camera for leaf segmentation.
• The image dataset used for evaluation comprised real images captured from the natural outdoor environment.

The rest of the paper is organized as follows: Section 2 presents the materials and methods. Section 3 presents the result and discussion. Section 4 covers the conclusion.

2. Materials and Methods

2.1. Image Data Collection

The image datasets were prepared using images captured by the multi-camera system in the outdoor environment. The multi-camera system contains 3 camera sensors: visible, NoIR, and thermal, as shown in Figure 1. Each camera is connected to a Raspberry Pi 3 Model B+ for image processing and data storage. The visible camera, on the right side of the figure, uses a Sony IMX219 camera sensor with an image resolution of 8 megapixels. The NoIR camera system, on the left side, consists of a camera module with a 5-megapixel Omnivision 5647 sensor (without an infrared filter), and a pair of infrared LEDs. The thermal camera, in the center, is a Seek Thermal CompactPro with a resolution of 320 × 240 pixels, connected to the Raspberry Pi module via a USB interface, as shown in the figure; its vertical position is above both the visible and NoIR cameras. The image data were also sent to Google Drive every 5 min for easy data storage and access. The images were captured continuously for a whole day (day and night). The proposed camera systems provide a low-cost solution, where the prices of the Raspberry Pi module and visible, NoIR, and thermal cameras are about USD 45, USD 7, USD 28, and USD 384, respectively.

The plant used for image data collection was Vetiveria zizanioides, planted in polybags and put in an outdoor environment (yard) during image collection. Photographs of the plants and the environment taken by the visible, NoIR, and thermal cameras are shown in Figure 2, Figure 3 and Figure 4, respectively. Due to the arrangement of the three cameras (as shown in Figure 1), the image viewpoint of each camera is different. Therefore, to make a fair comparison, we employed image warping to transform the images taken by the NoIR and thermal cameras to the match the reference image taken by the visible camera. In addition, to optimize the cameras’ resolution and the computation time of the image processor, all images were resized to 600 × 400 pixels. It is noted that even though the resized images of the thermal camera are larger than the original ones, it will not affect the performance significantly for the following reason. A thermal camera captures images based on the thermal energy emitted by the objects; thus, the parts of things appear in uniform color (as shown in Figure 4) rather than the precise color as with the visible camera. Therefore, resizing an image to a higher resolution does not change the detail of objects significantly. Figure 3a,b depict the original and transformed NoIR images, respectively. Figure 4a,b depict the original and transformed thermal images, respectively. The figures show that the viewpoints of the NoIR and thermal images are similar to that of the visible image, in the sense that the objects and their orientation in the images are almost the same. For instance, the leaves, polybags, and background objects appear in nearly the same position or area in Figure 2, Figure 3b and Figure 4b. Since the objects’ appearances in the observed images (visible, NoIR, and thermal images) are almost the same, we may evaluate the segmentation performance fairly.

2.2. Image Segmentation Techniques

2.2.1. Thresholding Technique

Thresholding is a simple technique for separating objects from the background by introducing a threshold. Pixels are assigned to the foreground if their intensity is lower than the threshold; otherwise, they are set as background. There are about 40 thresholding algorithms, as investigated in [39]. In this work, we evaluated 9 thresholding algorithms: Otsu, Multi-Otsu, Yen, Isodata, Li, Local, Minimum, Mean, and Triangle. Short explanations of the algorithms are described in the following.

The cumulative distribution function (F) is defined as:

$$F(g)={\displaystyle \sum _{i=0}^{g}p(i)},$$

(1)

where p(g) is the probability mass function, and g is the intensity value in the image (g = 0, ..., 255). The mean of foreground and background can be expressed as a function of threshold level T as:

$$m_f(T)={\displaystyle \sum _{g=0}^{T}gp(g)},$$

(2)

$$m_b(T)={\displaystyle \sum _{g=T+1}^{255}gp(g)}.$$

(3)

The variance of foreground and background can be expressed as a function of threshold level T:

$${\sigma }_f^2(T)={\displaystyle \sum _{g=0}^T{\left(g-m_f(T)\right)}^{2}p(g)},$$

(4)

$${\sigma }_b^2(T)={\displaystyle \sum _{g=T+1}^{255}{\left(g-m_b(T)\right)}^{2}p(g)}.$$

(5)

Otsu thresholding [40] finds an optimal threshold T_opt by maximizing the between-class variance of foreground and background, and is given as [39]:

$$T_{opt}=\arg \max \left\{\frac{F(T)\left(1-F(T)\right){\left(m_f(T)-m_b(T)\right)}^2}{F(T){\sigma }_f^2(T)+\left(1-F(T)\right){\sigma }_b^2(T)}\right\}.$$

(6)

Multi-level Otsu (Multi-Otsu) thresholding [41,42] is an extension of Otsu thresholding, where M-1 optimal thresholds (Topt_₁, …, Topt__M-1) are calculated for multi-class (M classes) separation.

Yen thresholding [43] finds an optimal threshold T_opt by maximizing the entropic correlation, and is given as [39]:

$$T_{opt}=\arg \max \left\{C_b(T)+C_f(T)\right\},$$

(7)

where

$$C_b(T)=-\log \left\{{\displaystyle \sum _{g=0}^T{\left(\frac{p(g)}{F(T)}\right)}^2}\right\},$$

(8)

$$C_f(T)=-\log \left\{{\displaystyle \sum _{g=T+1}^{255}{\left(\frac{p(g)}{1-F(T)}\right)}^2}\right\}.$$

(9)

Isodata thresholding [44] finds an optimal threshold T_opt, which is defined as [39]:

$$T_{opt}=\underset{n\to \propto }{\lim }\frac{m_f(T_n)+m_b(T_n)}{2}.$$

(10)

Li thresholding [45,46] finds an optimal threshold T_opt, by minimizing the information theoretic distance, which is given as [39]:

$$T_{opt}=\arg \min \left[{\displaystyle \sum _{g=0}^{T}gp(g)\log \frac{g}{m_f(T)}+}{\displaystyle \sum _{g=T+1}^{255}gp(g)\log \frac{g}{m_b(T)}}\right],$$

(11)

where

$${\displaystyle \sum _{g\le T}g={\displaystyle \sum _{g\le T}{m}_f(T) \mathrm{and} }}{\displaystyle \sum _{g\ge T}g={\displaystyle \sum _{g\ge T}{m}_b(T)\hspace{0.17em}}}.$$

(12)

In Local thresholding [47], an optimal threshold is calculated for each pixel by considering the mean value of its neighbors. The optimal threshold at pixel-(x,y) T_opt(x,y) is defined as:

$$T_{opt}(x,y)=m_{w\times w}(x,y)-C.$$

(13)

where m_wxw(x,y) is the local mean value over a window size w at the neighbors of pixel-(x,y), and C is a constant.

Minimum thresholding [48] finds an optimal threshold T_opt as the valley between two maxima of the histogram, which is defined such that the following formula is satisfied:

$$y_{T_{opt}}>y_{T_{opt}-1} AND y_{T_{opt}}\le y_{T_{opt}+1}.$$

(14)

where y_g is the number of pixels with intensity value g.

Mean thresholding [48] finds an optimal threshold T_opt as the mean of the intensity value, which is calculated as the integer part of the following:

$$\frac{{\displaystyle \sum _{g=0}^{255}g{y}_g}}{{\displaystyle \sum _{g=0}^{255}{y}_g}}.$$

(15)

Triangle thresholding [49] finds an optimal threshold T_opt in the histogram based on the Triangle method, as illustrated in Figure 5. The algorithm locates a point A by maximizing the distance between the triangle line and the histogram, then the optimal threshold T_opt is defined by adding a fixed offset Ofs to A.

2.2.2. K-Means Segmentation

The K-Means algorithm [50] is a machine learning technique that is used to cluster a dataset into K classes. The algorithm divides the dataset into classes such that there is maximum distance between classes and minimum distance within the class [28]. The algorithm is as follows:

• Set the number of classes (=K).
• Initialize K cluster centers from the dataset randomly.
• Calculate the distance of each pixel to each cluster center using a distance function.
• Classify each pixel to the nearest cluster, i.e., the closest distance.
• Recalculate the cluster center using the pixels belonging to the cluster.
• Repeat Steps 2 to 5 until there is no change in the cluster center.

2.3. Performance Measurement of Leaf Segmentation

Leaf segmentation performance was measured using the standard metrics: Recall, Precision, and F1 score. The metrics are defined using true positive (TP), false negative (FN), and false positive (FP). TP indicates that the extracted leaf pixel was correctly identified as a leaf. FN indicates that the leaf was not extracted. FP indicates that the extracted leaf pixel was wrongly identified as a leaf. Recall, Precision, and F1 score are defined as follows:

$$\mathrm{Recall}=\frac{TP}{TP+FN},$$

(16)

$$\mathrm{Precision}=\frac{TP}{TP+FP},$$

(17)

$$\mathrm{F}1\hspace{0.17em}\mathrm{score}=\frac{2(\mathrm{Recall})(\mathrm{Precision})}{\mathrm{Recall}+\mathrm{Precision}}.$$

(18)

The summation of TP and FN in (16) represents the ground-truth leaf pixels. Thus, Recall represents the portion of ground-truth leaf pixels present in the extracted leaf pixels in the segmented image [1]. The summation of TP and FP in (17) represents the extracted leaf pixels. Thus, the Precision represents the portion of extracted leaf pixels in the segmented image that matches the ground-truth leaf pixels [1]. The F1 score combines Recall and Precision and represents the mean harmonic between them.

In addition to TP, FN, and FP, the true negative (TN) is also computed to evaluate the confusion table composed of TP, FN, FP, and TN. TN indicates that the extracted non-leaf was correctly identified as a non-leaf object.

According to (16)–(18), the F1 score does not consider TN like Matthews’s correlation coefficient or Cohen’s kappa. However, it is noted here that the F1 score is selected for measuring performance rather than Matthews’s correlation coefficient or Cohen’s kappa because our objective is to extract the leaf (foreground) from the image. Thus, we emphasize the TP more than TN. Furthermore, the F1 score is commonly used for evaluating the leaf segmentation performance, as described in Section 1.

2.4. Image Dataset

The image dataset, collected as described in Section 2.1, consisted of images captured by visible, NoIR, and thermal cameras. There was a total of 672 images containing:

• 133 images taken by the visible camera during the day (06:00–17:00 h);
• 275 images taken by the NoIR camera during the day and at night (06:00–05:00 h (next day)); and
• 264 images taken by the thermal camera during the day and at night (06:00–05:00 h (next day)).

It is noted here that all three cameras captured the same scene at the same time interval, where each scene consists of four or five Vetiveria zizanioides plants. The data loss during transmission from the camera modules on the site to the cloud storage causes differences in the number of images captured by the NoIR and thermal cameras. This loss is affected by several problems, such as the camera module’s image acquisition error, the internet connection problem, and the error while accessing the Google Drive cloud storage. Since the visible camera only captured images during the day, the number of captured images is half those of the NoIR and thermal cameras.

Since the image data are time-stamped in intervals of 5 min, the image intensity during a whole day can be plotted, as shown in Figure 6. The blue, red, and green lines represent the intensity of visible, NoIR, and thermal images, respectively. It is clearly shown in the figure that at night, the intensity of the visible image is zero (no objects captured), and the image intensity varies during the day due to variations in sunlight (for visible and NoIR images) and temperature (for thermal images).

Since infrared LEDs are the light source for the NoIR camera, the camera can capture objects at night. However, due to the low power of the LEDs, only objects near the camera are captured. Therefore, the image intensity is low, as shown in Figure 6.

Figure 6 also shows that there is a significant difference in image intensity between daytime and nighttime images with the NoIR camera. This can be observed in Figure 7, which shows NoIR images at 13:00 h and 19:00 h. In the thermal image, the difference is insignificant, indicating that the temperature did not drop abruptly at night. This situation can be explained by observing Figure 8, which shows thermal images at 13:00 h and 19:00 h. In the images, brighter colors (orange and white) represent higher temperatures than dark colors (blue and purple). As shown in Figure 8, in the night image, some background parts are orange, indicating high temperature. This condition produces images with medium intensity as shown in Figure 6.

2.5. Ground-Truth Images

Ground-truth images were prepared manually, labeling the leaves using image editor software (Microsoft Paint). The ground truth for visible and NoIR cameras was prepared from visible and NoIR images, respectively. However, due to the different characteristics of a thermal camera, its ground-truth images are challenging to prepare from thermal images; instead, we used the ground-truth images from the visible images. Fortunately, this arrangement complies with our objective to evaluate the feasibility of a thermal camera for leaf segmentation rather than leaf temperature measurement.

Examples of ground-truth images for visible and NoIR cameras are shown in Figure 9a,b, respectively. In the figures, green represents the leaf object, while other colors represent the non-leaf objects and background.

2.6. Method of Leaf Segmentation Performance

The segmentation algorithms were implemented on a Raspberry Pi 3 Model B+ powered with: Broadcom BCM2837B0, Cortex-A53 64-bit SoC @ 1.4GHz, and run with the Raspberry Pi OS operating system. The algorithms were written in Python using the OpenCV and Scikit-image libraries.

The comparison of leaf segmentation performance is divided into five parts:

• Comparison of camera types and segmentation algorithms.
• Comparison of camera types only (performance measurement of algorithms is averaged or maximized).
• Comparison of segmentation algorithms only (performance measurement of camera types is averaged or maximized).
• Comparison of execution time.
• Comparison of segmentation algorithms using the well-known tested images.

3. Results and Discussion

3.1. Segmentation Results

Examples of leaf segmentation results of visible, NoIR, and thermal images taken during the day are illustrated in Figure 10, Figure 11 and Figure 12, respectively. In each figure, panel (a) shows the original tested image and panel (b) shows the leaf segmentation results using Isodata, Li, Local, Mean, Minimum, Otsu, Multi-Otsu, Triangle, Yen, K-Means (K = 2), and K-Means (K = 3), respectively. Recall (Rec), Precision (Prec), and F1 score (F1) are shown at the bottom of each image. The white and black colors in the figures represent the foreground (extracted leaf pixels) and background, respectively.

Figure 10 shows that when given a visible image depicted in Figure 10a, most of the algorithms achieve high Recall, except the images in Figure 10d,h,l, which achieved lower Recall. Higher Recall is indicated by many white pixels in the leaf object. The lower Recall for the image in Figure 10d was caused by Local thresholding using the local window to segment the image, resulting in some parts of the leaves not being extracted. Multi-Otsu thresholding (Figure 10h) and the K-Means algorithm (Figure 10l) use three classes in the segmentation, and they fail to extract some parts of the leaves when the color intensity is not uniform, thus they achieved lower Recall.

Figure 10 shows that most of the algorithms achieved medium Precision. This can be observed by the presence of white on non-leaf objects, which indicates false positive results. As false positives increase, Precision will decrease. The unbalance of Recall and Precision results yields a medium F1 score value for most algorithms. It is noted that Local thresholding had the worst F1 score value.

Observing Figure 10, it is found that due to the nature of the backgrounds, the wall (upper left side in the image) and concrete floor (lower right side in the image and white gap below the black fence) produce false positives with most algorithms, except Multi-Otsu and K-Means (K = 3). This condition suggests that bi-level segmentation techniques cannot separate these particular backgrounds from the foreground (leaf). They can be detected as backgrounds by multi-level segmentation techniques such as Multi-Otsu and K-Means (K = 3). However, the multi-level segmentation techniques tend to extract some parts of leaves as the background, as shown in Figure 10h,l, thus producing lower Recall. From this discussion, it is clear that our image dataset offers a challenge in achieving a high F1 score. It is worth noting that this challenge allows our aim to evaluate the thresholding techniques and the camera types for a suitable real-time leaf segmentation application. Similar results were obtained for the NoIR images shown in Figure 11. However, the performance of NoIR images was lower than that of visible images, as can be seen by the higher color contrast of the visible image in Figure 10a than the NoIR image in Figure 11a.

The thermal images in Figure 12 appear different from the visible and NoIR images. Due to the thermal camera’s resolution and characteristics, the extracted leaves are condensed rather than dispersed as in Figure 10 and Figure 11. Furthermore, the edge and thin parts of the leaves are not extracted. Therefore, the Recall, Precision, and F1 scores are lower compared to visible and NoIR images. Figure 12 also shows an interesting phenomenon: the plants’ polybags contribute to false positive results, which is opposite to the visible and NoIR cameras. It is also noted in Figure 12i that the Triangle thresholding failed to extract the leaves.

3.2. Comparison of Leaf Segmentation Performance

3.2.1. Comparison of Camera Types and Segmentation Algorithms

Figure 13 shows the segmentation performance of the visible camera during the day. The figure shows that the segmentation performance (Recall, Precision, and F1 score) of 11 algorithms with the visible camera varied. The highest Recall of 0.934, Precision of 0.751, and F1 score of 0.794 were achieved by Triangle thresholding, K-Means algorithm (K = 3), and Multi-Otsu thresholding. Local thresholding, as predicted from one example image discussed in Section 3.1, achieved the lowest performance.

Table 1. Normalized confusion tables of segmentation results using visible camera (daytime).

Isodata		Li		Local		Mean
TP	FN	TP	FN	TP	FN	TP	FN
22.27%	6.15%	25.52%	2.90%	17.28%	11.14%	24.54%	3.88%
FP	TN	FP	TN	FP	TN	FP	TN
17.40%	54.18%	18.52%	53.06%	27.75%	43.83%	18.11%	53.47%
Min		Otsu		Multi-Otsu		Triangle
TP	FN	TP	FN	TP	FN	TP	FN
25.59%	2.83%	22.22%	6.20%	19.26%	9.16%	26.48%	1.94%
FP	TN	FP	TN	FP	TN	FP	TN
18.76%	52.82%	17.39%	54.19%	5.19%	66.39%	24.96%	46.62%
Yen		K-Means (K = 2)		K-Means (K = 3)
TP	FN	TP	FN	TP	FN
23.63%	4.79%	21.39%	7.03%	20.33%	8.09%
FP	TN	FP	TN	FP	TN
18.07%	53.51%	17.75%	53.83%	5.51%	66.07%

shows the confusion tables of segmentation results using the visible camera, where TP, FN, FP, and TN for 11 algorithms are expressed in the normalized form. The normalization is calculated by dividing the numbers of TP, FN, FP, and TN with the total number of samples. The table shows that Triangle thresholding achieved the highest TP and lowest FN. This result conforms with the highest Recall as described previously. Multi-Otsu and K-Means (K = 3) algorithms achieved the lower FP that resulted in the higher Precision as explained previously. The tables also show that higher TN was achieved by Multi-Otsu and K-Means (K = 3). This suggested that for the given tested images, non-leaf objects can be extracted properly by considering the multi-class segmentation.

Figure 14 and

Table 2. Normalized confusion tables of segmentation results using NoIR camera (daytime).

Isodata		Li		Local		Mean
TP	FN	TP	FN	TP	FN	TP	FN
19.69%	5.25%	21.18%	3.77%	13.21%	11.74%	20.54%	4.41%
FP	TN	FP	TN	FP	TN	FP	TN
22.49%	52.56%	24.57%	50.48%	34.46%	40.59%	23.51%	51.54%
Min		Otsu		Multi-Otsu		Triangle
TP	FN	TP	FN	TP	FN	TP	FN
21.08%	3.87%	19.67%	5.28%	14.68%	10.27%	22.85%	2.10%
FP	TN	FP	TN	FP	TN	FP	TN
24.52%	50.54%	22.46%	52.59%	11.77%	63.28%	33.18%	41.87%
Yen		K-Means (K = 2)		K-Means (K = 3)
TP	FN	TP	FN	TP	FN
12.15%	12.80%	19.70%	5.25%	14.85%	10.10%
FP	TN	FP	TN	FP	TN
16.11%	58.94%	22.33%	52.72%	11.96%	63.09%

show the segmentation performance and normalized confusion tables of segmentation results using the NoIR camera during the day, respectively. Figure 14 shows that most algorithms had similar performance to the NoIR camera (daytime). Significantly, Precision was very close, except for Yen thresholding. Similar to the visible camera, Triangle thresholding achieved the highest Recall. It is notable that Multi-Otsu thresholding and K-Means (K = 3) achieved the highest F1 scores. This suggests that by employing three classes/clusters, both methods can balance Recall and Precision, thus increasing the Precision of the NoIR camera (daytime). These results are confirmed by Table 2, where normalized confusion tables in the figure show that Triangle thresholding achieved the highest TP and lowest FP, and Multi-Otsu and K-Means (K-3) achieved the highest TN and lowest FP, similarly to Table 1.

Figure 15 and

Table 3. Normalized confusion tables of segmentation results using NoIR camera (nighttime).

Isodata		Li		Local		Mean
TP	FN	TP	FN	TP	FN	TP	FN
1.46%	15.68%	10.65%	6.50%	7.21%	9.94%	10.69%	6.45%
FP	TN	FP	TN	FP	TN	FP	TN
5.50%	77.35%	11.38%	71.48%	11.18%	71.68%	11.39%	71.46%
Min		Otsu		Multi-Otsu		Triangle
TP	FN	TP	FN	TP	FN	TP	FN
0.31%	16.84%	1.39%	15.75%	8.95%	8.20%	2.94%	14.21%
FP	TN	FP	TN	FP	TN	FP	TN
4.90%	77.95%	5.46%	77.39%	17.26%	65.59%	3.91%	78.95%
Yen		K-Means (K = 2)		K-Means (K = 3)
TP	FN	TP	FN	TP	FN
4.05%	13.10%	16.07%	1.08%	9.15%	7.99%
FP	TN	FP	TN	FP	TN
6.67%	76.19%	76.26%	6.60%	7.07%	75.78%

show the segmentation performance and normalized confusion tables of segmentation results using the NoIR camera at night, respectively. Figure 15 shows that the performance of algorithms with the NoIR camera (nighttime) is low. It can be seen from Figure 7b that the leaf objects partially appeared due to the low power of infrared LEDs used in this experiment. This condition produced a lower TP for most algorithms, as shown in the normalized confusion tables in Table 3. However, the approach to using a NoIR camera for night measurement is promising due to the low cost and small size of the infrared LEDs embedded in the camera module. Further, it is non-visible light; thus, it does not disrupt the natural conditions in the plant environments at night.

It can be noted from the figure that K-Means (K = 2) achieved high Recall but low Precision and F1 score. From the experimental results, the high Recall was caused by an incorrect segmentation method that produced a wide white section (foreground), thus producing very high Recall and very low Precision. The results suggest that Recall and Precision should be considered when evaluating segmentation performance.

Figure 16 and

Table 4. Normalized confusion tables of segmentation results using NoIR camera (daytime and nighttime).

Isodata		Li		Local		Mean
TP	FN	TP	FN	TP	FN	TP	FN
10.15%	10.71%	15.66%	5.20%	10.07%	10.80%	15.38%	5.48%
FP	TN	FP	TN	FP	TN	FP	TN
13.59%	65.54%	17.66%	61.47%	22.27%	56.87%	17.17%	61.97%
Min		Otsu		Multi-Otsu		Triangle
TP	FN	TP	FN	TP	FN	TP	FN
10.20%	10.66%	10.10%	10.76%	11.68%	9.18%	12.42%	8.44%
FP	TN	FP	TN	FP	TN	FP	TN
14.24%	64.89%	13.56%	65.58%	14.65%	64.49%	17.85%	61.28%
Yen		K-Means (K = 2)		K-Means (K = 3)
TP	FN	TP	FN	TP	FN
7.91%	12.96%	17.80%	3.06%	11.86%	9.00%
FP	TN	FP	TN	FP	TN
11.16%	67.97%	50.57%	28.57%	9.40%	69.74%

show the segmentation performance and normalized confusion tables of segmentation results using the NoIR camera over a whole day, respectively. Figure 16 shows that similar to the visible camera, the performance of 11 algorithms with the NoIR camera (daytime and nighttime) also varied. The figure shows that the Multi-Otsu and K-Means (K = 3) achieved the highest Precisions, however, the values are lower than the visible camera. Table 4 shows that TP values are lower and FN values are higher compared to the ones in Table 1 (visible camera). These results are affected by the condition at night, where due to the low power of infrared LEDs, the leaf objects partially appeared in the images, as discussed previously.

Figure 17, Figure 18 and Figure 19 and

Table 5. Normalized confusion tables of segmentation results using thermal camera (daytime).

Isodata		Li		Local		Mean
TP	FN	TP	FN	TP	FN	TP	FN
22.54%	5.68%	20.28%	7.93%	15.43%	12.79%	21.34%	6.88%
FP	TN	FP	TN	FP	TN	FP	TN
41.08%	30.70%	35.62%	36.16%	39.49%	32.29%	36.88%	34.90%
Min		Otsu		Multi-Otsu		Triangle
TP	FN	TP	FN	TP	FN	TP	FN
13.02%	15.20%	22.59%	5.63%	19.10%	9.12%	0.43%	27.79%
FP	TN	FP	TN	FP	TN	FP	TN
25.07%	46.72%	41.20%	30.59%	30.72%	41.06%	4.90%	66.88%
Yen		K-Means (K = 2)		K-Means (K = 3)
TP	FN	TP	FN	TP	FN
23.97%	4.25%	21.41%	7.19%	16.68%	11.92%
FP	TN	FP	TN	FP	TN
45.54%	26.24%	37.10%	34.30%	24.62%	46.78%

,

Table 6. Normalized confusion tables of segmentation results using thermal camera (nighttime).

Isodata		Li		Local		Mean
TP	FN	TP	FN	TP	FN	TP	FN
23.50%	0.23%	23.36%	0.37%	12.62%	11.11%	22.96%	0.77%
FP	TN	FP	TN	FP	TN	FP	TN
56.25%	20.03%	52.75%	23.52%	41.56%	34.71%	46.23%	30.04%
Min		Otsu		Multi-Otsu		Triangle
TP	FN	TP	FN	TP	FN	TP	FN
0.41%	23.32%	23.50%	0.23%	23.07%	0.65%	22.98%	0.75%
FP	TN	FP	TN	FP	TN	FP	TN
5.49%	70.78%	56.40%	19.87%	47.71%	28.56%	48.58%	27.69%
Yen		K-Means (K = 2)		K-Means (K = 3)
TP	FN	TP	FN	TP	FN
23.47%	0.26%	23.46%	0.30%	17.69%	6.06%
FP	TN	FP	TN	FP	TN
55.41%	20.86%	53.76%	22.49%	28.38%	47.86%

and

Table 7. Normalized confusion tables of segmentation results using thermal camera (daytime and nighttime).

Isodata		Li		Local		Mean
TP	FN	TP	FN	TP	FN	TP	FN
23.02%	2.94%	21.83%	4.12%	14.01%	11.94%	22.16%	3.80%
FP	TN	FP	TN	FP	TN	FP	TN
48.72%	25.32%	44.25%	29.79%	40.54%	33.51%	41.59%	32.45%
Min		Otsu		Multi-Otsu		Triangle
TP	FN	TP	FN	TP	FN	TP	FN
6.67%	19.29%	23.05%	2.91%	21.10%	4.85%	11.79%	14.17%
FP	TN	FP	TN	FP	TN	FP	TN
15.20%	58.84%	48.86%	25.19%	39.28%	34.76%	26.91%	47.14%
Yen		K-Means (K = 2)		K-Means (K = 3)
TP	FN	TP	FN	TP	FN
23.72%	2.24%	22.53%	3.43%	17.23%	8.73%
FP	TN	FP	TN	FP	TN
50.51%	23.53%	46.19%	27.86%	26.67%	47.37%

show the segmentation performances and normalized confusion tables of segmentation results using thermal camera during the day, at night, and over a whole day, respectively. As shown in the figures, a significant difference between Recall and Precision was found for most algorithms. It can be seen in Figure 8 and the previous discussion in Section 3.2 that thermal images tend to yield condensed segmented images, thus they will achieve high Recall and low or medium Precision. The figures also show that the Minimum and Triangle thresholding almost fail on nighttime images (Figure 18 and Table 6) and daytime images (Figure 17 and Table 5).

3.2.2. Comparison of Camera Types

To find the effectiveness of camera types in segmentation, we compared the performance of each camera based on the average and maximum values of the metrics of all segmentation algorithms. The comparisons of average and maximum Recall, Precision, and F1 score for each camera type are given in

Table 8. Average and maximum Recall for each camera type.

Camera Type	Average Recall			Maximum Recall
	Daytime	Nighttime	Whole Day	Daytime	Nighttime	Whole Day
Visible camera	0.794	NA	NA	0.934	NA	NA
NoIR camera	0.761	0.405	0.574	0.917	0.937	0.868
Thermal camera	0.647	0.831	0.740	0.863	0.990	0.926

,

Table 9. Average and maximum Precision for each camera type.

Camera Type	Average Precision			Maximum Precision
	Daytime	Nighttime	Whole Day	Daytime	Nighttime	Whole Day
Visible camera	0.663	NA	NA	0.751	NA	NA
NoIR camera	0.568	0.329	0.443	0.599	0.572	0.575
Thermal camera	0.419	0.418	0.418	0.493	0.505	0.494

and

Table 10. Average and maximum F1 score for each camera type.

Camera Type	Average F1 Score			Maximum F1 Score
	Daytime	Nighttime	Whole Day	Daytime	Nighttime	Whole Day
Visible camera	0.584	NA	NA	0.794	NA	NA
NoIR camera	0.466	0.333	0.396	0.560	0.636	0.599
Thermal camera	0.319	0.282	0.301	0.405	0.383	0.394

, respectively. A comparison of daytime, nighttime, and the whole day was also conducted, and the results are given in the tables. For straightforward reading, the highest values in the tables are given in gray.

Table 8 shows that the visible camera achieved the best performance for average and maximum Recall on daytime images with values of 0.794 and 0.934, respectively. The thermal camera achieved the best performance for average and maximum Recall on daytime (whole day) images with values of 0.831 (0.740) and 0.990 (0.926), respectively.

Table 9 shows that the visible camera achieved the best performance for average and maximum Precision on daytime images, with values of 0.663 and 0.751, respectively. The NoIR camera achieved the best performance for average and maximum Precision on whole day images, with the values of 0.443 and 0.575, respectively, and maximum Precision of 0.572 on nighttime images. The thermal camera achieved the best performance for average Precision on nighttime images with a value of 0.418.

Table 10 shows that the visible camera achieved the best performance for average and maximum F1 score on daytime images with values of 0.584 and 0.794, respectively. The NoIR camera achieved the best performance for average and maximum F1 score on daytime (whole day) images, with values of 0.333 (0.396) and 0.636 (0.599), respectively.

From Table 8, Table 9 and Table 10, we can conclude that the visible camera is the best choice for daytime images. When considering Recall only, the thermal camera is the best choice for nighttime and whole day images. When considering F1 score, the NoIR camera is the best choice for nighttime and whole day images.

3.2.3. Comparison of Segmentation Algorithms

To find the effectiveness of the segmentation algorithm, we compared the performance of each algorithm based on the average and maximum values of the metrics for the three cameras. The comparisons of average and maximum Recall, Precision, and F1 score for each algorithm are given in

Table 11. Average and maximum Recall for each segmentation method.

Segmentation Method	Average Recall			Maximum Recall
	Daytime	Nighttime	Whole Day	Daytime	Nighttime	Whole Day
Isodata	0.795	0.538	0.662	0.814	0.990	0.903
Li	0.828	0.803	0.796	0.897	0.984	0.862
Local	0.667	0.275	0.476	0.847	0.532	0.539
Mean	0.820	0.796	0.795	0.864	0.968	0.871
Min	0.739	0.319	0.486	0.896	0.621	0.731
Otsu	0.794	0.536	0.661	0.816	0.990	0.904
Multi-Otsu	0.659	0.747	0.697	0.698	0.972	0.836
Triangle	0.622	0.570	0.511	0.934	0.968	0.527
Yen	0.732	0.613	0.645	0.863	0.989	0.926
K-Means (K = 2)	0.773	0.963	0.876	0.791	0.988	0.885
K-Means (K = 3)	0.643	0.639	0.622	0.717	0.744	0.676

,

Table 12. Average and maximum Precision for each segmentation method.

Segmentation Method	Average Precision			Maximum Precision
	Daytime	Nighttime	Whole Day	Daytime	Nighttime	Whole Day
Isodata	0.575	0.279	0.403	0.650	0.454	0.472
Li	0.593	0.506	0.522	0.702	0.545	0.571
Local	0.479	0.168	0.319	0.597	0.324	0.347
Mean	0.592	0.520	0.532	0.689	0.546	0.570
Min	0.538	0.277	0.366	0.699	0.545	0.571
Otsu	0.575	0.276	0.401	0.649	0.454	0.471
Multi-Otsu	0.598	0.496	0.513	0.731	0.503	0.538
Triangle	0.411	0.299	0.290	0.655	0.480	0.332
Yen	0.514	0.367	0.400	0.669	0.457	0.474
K-Means (K = 2)	0.570	0.379	0.456	0.630	0.462	0.477
K-Means (K = 3)	0.603	0.539	0.534	0.751	0.572	0.575

and

Table 13. Average and maximum F1 score for each segmentation method.

Segmentation Method	Average F1 Score			Maximum F1 Score
	Daytime	Nighttime	Whole Day	Daytime	Nighttime	Whole Day
Isodata	0.460	0.238	0.321	0.558	0.295	0.325
Li	0.468	0.398	0.406	0.578	0.489	0.476
Local	0.375	0.122	0.241	0.462	0.233	0.257
Mean	0.469	0.410	0.413	0.574	0.489	0.478
Min	0.434	0.252	0.308	0.574	0.489	0.476
Otsu	0.460	0.235	0.319	0.558	0.294	0.324
Multi-Otsu	0.581	0.436	0.455	0.794	0.546	0.553
Triangle	0.314	0.205	0.208	0.508	0.319	0.244
Yen	0.418	0.335	0.341	0.562	0.372	0.359
K-Means (K = 2)	0.459	0.240	0.324	0.546	0.302	0.332
K-Means (K = 3)	0.585	0.509	0.496	0.792	0.636	0.599

, respectively.

Table 11 shows that on daytime images, the best performance for average Recall was achieved by Mean thresholding, with a value of 0.820, and for maximum Recall Triangle thresholding performed best, with a value of 0.934. K-Means (K = 3) achieved the best performance on nighttime and whole day images, with average Recall values of 0.963 and 0.876, respectively. The best performance on nighttime images for maximum Recall was achieved by Isodata and Otsu thresholding, with a value of 0.990. Yen thresholding achieved the best performance for maximum Recall on whole day images with a value of 0.926.

Table 12 shows that K-Means (K = 3) performed best in Precision for all conditions. The average Precision values for daytime, nighttime, and whole day images are 0.603, 0.539, and 0.534, respectively. The maximum Precisions for daytime, nighttime, and whole day images are 0.751, 0.572, and 0.575, respectively.

Table 13 shows that K-Means (K = 3) performed best on the F1 score for most conditions. The average F1 score values for daytime, nighttime, and whole day images are 0.585, 0.509, and 0.496, respectively. The maximum Precision values for nighttime and whole day images are 0.636 and 0.599, respectively. The maximum Precision of 0.794 for daytime images was achieved by Multi-Otsu.

From Table 11, Table 12 and Table 13, we can conclude that K-Means (K = 3) is the best choice when considering F1 score and Precision. In the case of Recall, there is no single algorithm that outperforms under all conditions.

3.2.4. Comparison of the Execution Time

The execution time of 11 segmentation algorithms is given in

Table 14. Execution time of segmentation algorithms.

Segmentation Algorithm	Isodata	Li	Local	Mean	Minimum	Otsu	Multi- Otsu	Triangle	Yen	K-Means (K = 2)	K-Means (K = 3)
Execution time (s)	0.174	0.210	4.992	0.175	0.342	0.183	0.205	0.167	0.176	2.463	3.483

. Most of the thresholding algorithms, except Local thresholding, have a fast execution time of about 0.2 s. Since Local thresholding computes the thresholding throughout the local windows, the computation time is longer. The execution time of K-Means, an iterative approach, is longer than that of the thresholding approach; it took 2.463 s for two clusters (K = 2) and 3.483 s for three clusters (K = 3). With regard to its application to plant monitoring, the execution time is still acceptable for real-time implementation.

3.2.5. Comparison of Segmentation Algorithms Using Well-Known Image Dataset

To verify the performance of segmentation techniques (thresholding and K-means) used in this work for leaf segmentation, we evaluated 11 segmentation algorithms using a well-known image dataset, namely Ara2012 [16,17,51], consisting of 120 Arabidopsis thaliana plant images. Examples of tested images are shown in Figure 20.

Figure 21 and

Table 15. Normalized confusion tables of segmentation results of Arabidopsis thaliana plant.

Isodata		Li		Local		Mean
TP	FN	TP	FN	TP	FN	TP	FN
32.70%	1.05%	33.02%	0.72%	24.39%	9.36%	32.73%	1.01%
FP	TN	FP	TN	FP	TN	FP	TN
15.49%	50.76%	17.81%	48.44%	25.24%	41.02%	15.48%	50.78%
Min		Otsu		Multi-Otsu		Triangle
TP	FN	TP	FN	TP	FN	TP	FN
31.87%	1.88%	32.68%	1.07%	30.35%	3.39%	30.41%	3.34%
FP	TN	FP	TN	FP	TN	FP	TN
12.95%	53.30%	15.35%	50.90%	9.27%	56.98%	10.94%	55.31%
Yen		K-Means (K = 2)		K-Means (K = 3)
TP	FN	TP	FN	TP	FN
32.24%	1.51%	32.82%	0.92%	32.47%	1.28%
FP	TN	FP	TN	FP	TN
18.20%	48.05%	14.26%	52.00%	3.49%	62.76%

show the segmentation performance and normalized confusion tables of segmentation results using the Ara2012 dataset. Most algorithms achieved high Recall, Precision, and F1 score, which are higher than that of the image dataset used in our experiments. These results show that our image datasets are more complicated than Ara2012. The normalized confusion tables in Table 15 show that FP is low and FN is very low, thus resulting the higher Recall and Precision. TP and TN of most algorithms were almost the same with higher values. Similar to previous results in Section 3.2.1, Local thresholding showed the worst performance.

Figure 21 shows that K-Means (K = 3) achieved the highest F1 score of 0.917, while values of Recall and Precision are 0.952 and 0.931, respectively. The second highest F1 score was achieved by Multi-Otsu with a value of 0.813, while values of Recall and Precision are 0.894 and 0.845, respectively. Referring to Section 3.2.1 and Section 3.2.3, K-Means (K-3) and Multi-Otsu also show the best performance, even though it is lower than for the Ara2012 dataset. This result implies that leaf segmentation using multi-clusters provides a better solution for achieving high Recall, Precision, and F1 score. Unfortunately, due to the complicated backgrounds in our image dataset, this approach cannot achieve high performance.

Furthermore, by comparing our image dataset and the Ara2012 dataset, we can observe an interesting result, as described in the following. Comparing F1 scores of the Otsu and Triangle thresholding techniques between our dataset and the Ara2012 dataset, it was seen that Otsu thresholding shows better performance than Triangle thresholding in our dataset, however, the Triangle thresholding shows better performance than Otsu thresholding in the Ara2012 dataset. This result could be seen from the image histograms shown in Figure 22, where Figure 22a,b are the image histograms of our dataset and the Ara2012 dataset, respectively. It is clear from the figures that the Triangle thresholding works better when the shape of the histogram is bi-modal, as shown in Figure 16. The histograms also show evidence that our image dataset consisted of complicated images for leaf segmentation, especially when considering bi-level segmentation.

4. Conclusions

A comparative study of leaf segmentation methods for Vetiveria zizanioides was conducted. The objective was to find the best camera type and technique for leaf segmentation in an outdoor environment using an embedded camera system. Three segmentation metrics, Recall, Precision, and F1 score, were used to measure the segmentation performance. The results show that the visible camera performed best on daytime images, while the NoIR camera was suitable for nighttime images. The K-Means algorithm achieved the maximum Precision and F1 score in all conditions; meanwhile, no single segmentation algorithm showed superior Recall in all conditions.

Based on the results, a technique for combining camera types and segmentation techniques will be developed in future work for better performance of the leaf segmentation system.