The Use of the Combination of Texture, Color and Intensity Transformation Features for Segmentation in the Outdoors with Emphasis on Video Processing

: Segmentation is the ﬁrst and most important part in the development of any machine vision system with speciﬁc goals. Segmentation is especially important when the machine vision system works under environmental conditions, which means under natural light with natural backgrounds. In this case, segmentation will face many challenges, including the presence of various natural and artiﬁcial objects in the background and the lack of uniformity of light intensity in di ﬀ erent parts of the camera's ﬁeld of view. However, today, we must use di ﬀ erent machine vision systems for outdoor use. For this reason, in this study, a segmentation algorithm was proposed for use in environmental conditions without the need for light control and the creation of artiﬁcial background using video processing with emphasizing the recognition of apple fruits on trees. Therefore, a video with more than 12 minutes duration containing more than 22,000 frames was studied under natural light and background conditions. Generally, in the proposed segmentation algorithm, ﬁve segmentation steps were used. These steps include: 1. Using a suitable color model; 2. Using the appropriate texture feature; 3. Using the intensity transformation method; 4. Using morphological operators; and 5. Using di ﬀ erent color thresholds. The results showed that the segmentation algorithm had the total correct detection percentage of 99.013%. The highest sensitivity and speciﬁcity of segmentation algorithm were 99.224 and 99.458%, respectively. Finally, the results showed that the processor speed was about 0.825 seconds for segmentation of a frame.


Introduction
Performing segmentation operations in accordance with the desired purpose has different complexities. In principle, segmentation operations in agriculture and horticulture are more complex than other sectors. This complexity is because of crowded backgrounds with various objects. In applications such as site-specific spraying and combat weeds, segmentation is the first step in the design of machine vision systems [1][2][3][4]. The segmentation involves various steps depending on the complexity of the background of the image and may include a combination of several methods. Therefore, the programmer skill is very important in this field. Generally, the conventional methods of objects with different colors. 2. Camera movement in gardens needs for different operations, such as site-specific spraying, for this reason, the frames have no good quality. Therefore, the purpose of this study is to develop a segmentation algorithm for working in a completely natural environment both in terms of light and in terms of backgrounds using video processing, with emphasis on the segmentation of apples on trees.
In the last years horticulture has been one of the most important research objects in many universities in the world. We have found that the main works are directed to the recognition of fruits, counting, detection of plants, monitoring of irrigation, etc.

Materials and Methods
Each machine or computer vision system for development needs to have different stages, such as the filming stage, the stage of analysis, and so on. In this study, as same as other machine vision system, the steps have been designed to train the system. Some of these steps can be filming, examining different color models, extracting various texture features, employing different morphological operators and using the intensity transformation method.

Data Collection
In this study, a digital camera (DFK 23GM021 specification, CMOS, 120 f/s, Imaging Source GmbH, Bremen, Germany) was used to filming from apple gardens in Kermanshah province-Iran. In Table 1, shows the related details of one of the videos from these gardens. As observed, there is a video more than 12 minutes that contains more than 22,000 frames that has been recorded in different days, times of the day, and weather conditions. Since the capability of performing of the segmentation system in different light intensities is an essential principle, the video was provided in full natural light conditions throughout the day with a completely natural background. Some light intensities were 398 lux, 1096 lux, 692 lux, 1591 lux, 1923 lux, 894 lux, 2010 lux, 918 lux, 798 lux, 493 lux and 579 lux. We collected several films of orchards, but we used only 12 min (22,001 frames) of them for the training of the algorithm. We had filming in four stages of ripening, including Unripe (20 days before maturity), Half-ripe (10 days before maturity), Ripe and Overripe (10 days after maturity), that combined them for the training of algorithm.
The distance from the trees was between 0.5 and 2 m, the speed was around 1 m/s, and the viewing angle was nearly parallel to the ground. The camera was manually held, simulating a low-medium height flight of a drone. With the system described, which has a horizontal viewing angle around 80o an apple about 7 cm would be observed with a size of 20 pixels at a distance about 3 m over the trees.
Apple variety was Malus Domestica L., var. Red Delicious. We had filming in 4 stages of ripening include Unripe, Half-ripe, Ripe and Overripe.

Various Color Models
An image in different color models has different colors. In fact, different objects in one image, in each color model, will have different colors. This feature can be used to distinguish between different Agriculture 2019, 9, 104 4 of 14 background objects and apples. For this investigation, 17 color spaces were investigated [10,11], which are shown in Table 2.

Extraction of Texture Features
Intuitively, the texture of a region can be described by roughness and softness. In fact, different regions in one image can be formed from a very rough to very soft modes. Mathematically, there are several methods for describing the texture. One of these methods is the texture features based on the gray level co-occurrence matrix (GLCM) extracted from the position of the pixels with same values. In fact, this method presents an average of the entire area in which the texture is examined. Therefore, this study is not applicable because here it is necessary to examine the texture of all pixels. Another method is to measure the spectral range of the texture based on the Furrier spectrum. This spectrum describes periodic or nearly periodic two-dimensional patterns in an image. The Furrier spectrum performs spectral measurement in a polar coordinate system (i.e., based on radius and angle), since spectral properties are interpreted by describing the spectrum in polar coordinates as a simple function of S (r, θ). In this function S is the spectral function and r and θ are variables of the polar system. Therefore, the function of S (r, θ) can be considered as two one-dimensional functions of Sθ(r) and Sr(θ) for each direction θ and each frequency r. Sθ(r) for the constant values of θ shows the behavior of the spectrum along the radius, while Sr(θ) for the constant values of r shows the behavior of the spectrum along a circle with the center of origin [10]. This method, like the previous method, provides the mean value for the entire area. In the third method, textural descriptors are applied to the entire image pixels, and finally, the results are also observed intuitively. Therefore, in this study, the texture features of local entropy, local standard deviation and local range were investigated.

Application of Morphological Operators
Outdoor operations under natural light with complex backgrounds are particularly sensitive, as unpredictable noise and effects can make it difficult to achieve the desired goal. One of the most important methods for removing these noise and unpredicted factors is the use of morphological operators. These operators include a wide range of operators, such as opening, closing, filling holes, deleting border pixels, removing objects with pixels less than threshold values, thinning, thickening, and others. In the proposed segmentation algorithm, opening, closing, filling holes and deleting objects with a number of pixels less than 100 were used at different stages. This threshold value was selected with trial and error and with the consideration of not removing the apple pixels.
The process of mathematical morphology in computational terms, consists of moving all the pixels of the image from left to right and from top to bottom in order to find isolated pixels, which are considered noise [12]. This noise is eliminated by applying erosion and dilation with the following equations: The operation open, disappears the fine points or fine structures and the operation close, fill the black holes of a certain size.

The Importance of Using Intensity Transformation
In segmentation, we are looking for methods that eliminate background objects and prevent the removal of target object pixels. The use of the intensity transformation method by limiting the pixel intensity variation in the desired range provides more differences between different objects. Therefore, in this study, a part of the segmentation operation was performed by changing the intensity from 0 and 1 to 0 and 0.6 with applying the threshold of 95. Since image were in uint8 data class pixels were multiplied in 225.

Different Stages in the Elaboration of Segmentation Algorithm
In Figure 1 shows the main steps in creating a segmentation algorithm. As observed, there are 11 main stages in this algorithm. In segmentation, we are looking for methods that eliminate background objects and prevent the removal of target object pixels. The use of the intensity transformation method by limiting the pixel intensity variation in the desired range provides more differences between different objects. Therefore, in this study, a part of the segmentation operation was performed by changing the intensity from 0 and 1 to 0 and 0.6 with applying the threshold of 95. Since image were in uint8 data class pixels were multiplied in 225.

Different Stages in the Elaboration of Segmentation Algorithm
In figure 1 shows the main steps in creating a segmentation algorithm. As observed, there are 11 main stages in this algorithm.

The Most Suitable Color Model for the First Stage of Segmentation
In figure 2 shows a sample image in six different color models. As observed, objects in different color models have different colors. The most suitable color space for segmentation is the color space with a minimum number of colors and the display of all the objects in the image, because there is the possibility of using threshold or thresholds with a very high accuracy. These images show that the worst color model is the LCH because it shows almost all the objects of the image in white. Other color models expect Luv have shown different objects with a large number of colors, which causes applying the threshold be difficult. The Luv color model has been able to represent various objects in the image with almost three colors. In fact, in this image, the leaves are shown in purple color, which led to performing a part of the segmentation based on this feature. Finally, using trial and error, it was determined that if all the pixel components in the image of the Luv color model are more than 115, then those pixels are related to the background and should be deleted.

The Most Suitable Color Model for the First Stage of Segmentation
In Figure 2 shows a sample image in six different color models. As observed, objects in different color models have different colors. The most suitable color space for segmentation is the color space with a minimum number of colors and the display of all the objects in the image, because there is the possibility of using threshold or thresholds with a very high accuracy. These images show that the worst color model is the LCH because it shows almost all the objects of the image in white. Other color models expect Luv have shown different objects with a large number of colors, which causes applying the threshold be difficult. The Luv color model has been able to represent various objects in the image with almost three colors. In fact, in this image, the leaves are shown in purple color, which led to performing a part of the segmentation based on this feature. Finally, using trial and error, it was determined that if all the pixel components in the image of the Luv color model are more than 115, then those pixels are related to the background and should be deleted. Figure 3 illustrates the results of applying the three features of texture of local range, local entropy and local standard deviations. As observed, the images extracted from the local range and local standard deviation methods are very similar expect that the edges of the objects in the image resulted from the local area is darker. The images from these two methods represent more objects compared with the local entropy method. Therefore, finally, the image resulted from applying the texture feature of local range was studied as the target image to apply another step of the segmentation. In fact, this image converted into a binary image, and then segmentation was performed by applying the threshold 1. This threshold means that if the image pixels have a value equal to 1, those pixels belong to the background and should be deleted.   Figure 3 illustrates the results of applying the three features of texture of local range, local entropy and local standard deviations. As observed, the images extracted from the local range and local standard deviation methods are very similar expect that the edges of the objects in the image resulted from the local area is darker. The images from these two methods represent more objects compared with the local entropy method. Therefore, finally, the image resulted from applying the texture feature of local range was studied as the target image to apply another step of the segmentation. In fact, this image converted into a binary image, and then segmentation was performed by applying the threshold 1. This threshold means that if the image pixels have a value equal to 1, those pixels belong to the background and should be deleted.

Intensity Transformation Performance in the Third
Step of Segmentation Figure 4 shows an image of the intensity transformation performance. Figure 4a shows the main studied image. As observed, this image has various objects such as green leaves in the shade, green leaves in the sun, soil, green plants in the shade, green plants in the sun, tiny branches, thick branches, tree trunks, and others. Figure 4b shows the image segmented in the two previous steps by color and texture methods. As observed, most of the relevant branches and trunks of trees remained without any change. Figure 4c shows the image of the intensity transformation. Eventually, by applying thresholds 95 on image Figure 4c, the image shown in Figure 4d was obtained. By comparing this image and the image of Figure 4(b), it is clear that many parts of the trunk and branches have been deleted.

The Performance of Segmentation Algorithm in Different Modes of Ordering Color, Texture, Intensity Transformation Methods
One of the innovations of this research is to arrange the sequence of different segmentation methods. Figure 5 shows three different sequences of texture, color, and intensity transformation methods. Figure 5a shows the original image. Figure 5b shows the image segmented before applying the color thresholds with the sequence of the texture method, the color method and the intensity transformation method. As observed, the segmentation accuracy is very low, and many of the relevant apple segments have been deleted, while many background pixels are remained. Figure 5c shows the segmented image of Figure 5a with the sequence of the intensity transformation method, the texture method and the color method. This sequence of methods has better performance than the previous one, but in general, it has a low accuracy. Figure 5d shows the segmented image of the segmentation algorithm with the sequence of the color method, the texture method and the intensity transformation method. As observed, the algorithm has a very good performance. In fact, by using this sequence, a large part of the background was removed and the pixels of the apple were not deleted.

Intensity Transformation Performance in the Third
Step of Segmentation Figure 4 shows an image of the intensity transformation performance. Figure 4a shows the main studied image. As observed, this image has various objects such as green leaves in the shade, green leaves in the sun, soil, green plants in the shade, green plants in the sun, tiny branches, thick branches, tree trunks, and others. Figure 4b shows the image segmented in the two previous steps by color and texture methods. As observed, most of the relevant branches and trunks of trees remained without any change. Figure 4c shows the image of the intensity transformation. Eventually, by applying thresholds 95 on image Figure 4c, the image shown in Figure 4d was obtained. By comparing this image and the image of Figure 4b, it is clear that many parts of the trunk and branches have been deleted.

The Performance of Segmentation Algorithm in Different Modes of Ordering Color, Texture, Intensity Transformation Methods
One of the innovations of this research is to arrange the sequence of different segmentation methods. Figure 5 shows three different sequences of texture, color, and intensity transformation methods. Figure 5a shows the original image. Figure 5b shows the image segmented before applying the color thresholds with the sequence of the texture method, the color method and the intensity transformation method. As observed, the segmentation accuracy is very low, and many of the relevant apple segments have been deleted, while many background pixels are remained. Figure 5c shows the segmented image of Figure 5a with the sequence of the intensity transformation method, the texture method and the color method. This sequence of methods has better performance than the previous one, but in general, it has a low accuracy. Figure 5d shows the segmented image of the segmentation algorithm with the sequence of the color method, the texture method and the intensity transformation method. As observed, the algorithm has a very good performance. In fact, by using this sequence, a large part of the background was removed and the pixels of the apple were not deleted.

Applying Thresholding Function to Complete the Segmentation Process
After completing the first part of the segmentation, which involves applying thresholds using different methods and the sequence of the methods in the segmentation algorithm, it is necessary to implement the second part of the segmentation to complete the segmentation process due to the presence of small objects in the background. Due to the sensitivity of the work, in this study, a thresholding function related to RGB color space channels were used for final segmentation in a comprehensive segmentation algorithm by exact study of frames and considerations so as to not remove apple pixels. After survey different images in light different conditions such as shadow and sunny modes as well as various objects on the trees, 10-color threshold for function training were select. In fact, this function is based on pixel. Each pixel survey individually and the values of RGB color space components compare with 10-color threshold. This function has two outputs: 0 and 1. When the output is 0, it means that pixel is related to background and when the output is 1 it means that pixel is related to apples. These thresholds have been shown in Table 3. Figure 6 shows two sample images for displaying the performance of a number of thresholds. The target objects have been shown with bold blue lines. Other objects that are left in the images and not in the right image were removed by other thresholds. Table 3. Different thresholds to remove background pixels remaining from previous steps.  Table 3, (c): The image before applying threshold, (d): The image after applying threshold 5 in Table 3. Table 4 shows the average percentage of background pixels removed by each segmentation method in comprehensive segmentation algorithm. This table shows that the highest percentage of  Table 3, (c): The image before applying threshold, (d): The image after applying threshold 5 in Table 3. Table 4 shows the average percentage of background pixels removed by each segmentation method in comprehensive segmentation algorithm. This table shows that the highest percentage of background pixels removed is related to the method of threshold in Luv color space with value 36%. As this table shows, no single method can do segmentation operations alone. Therefor a combination of different method is need for segmentation with high accuracy. The combination of different segmentation techniques and their arrangement can be considered as an innovation. Table 5 shows the confusion matrix of thresholding function. As this table shows the error of this segmentation method is less than 0.8. Table 4. The average percentage of background pixels removed by each segmentation method.

Main Segmentation Methods The Average Percentage of Background Pixels Removed by Each Method
The use of threshold in Luv color space 36 The use of texture feature 26 The use of morphological operators 23 The use of thresholding function 15  Table 6 shows the confusion matrix and the percentage of the total detection of the proposed segmentation algorithm. As observed, objects in the images are divided into two classes of apple and background objects. This table shows that 324 samples out of 42,750 apple samples are mistakenly located in the background objects class by the segmentation algorithm, so the segmentation algorithm has 0.758% error in this class. This algorithm also mistakenly classified 691 samples of the objects in the background with the total members of 60125 in the apple class. This leads to a 1.15% error in segmentation algorithm for this class. Finally, the percentage of total detection of the segmentation algorithm is 99.013%. This accuracy is very good for this sample number, which proves the algorithm was configured properly.

Performance of Segmentation Algorithm
To evaluate the performance of segmentation algorithm, three criteria of sensitivity, specificity and accuracy were used. Based on definition, sensitivity expresses the wrong placement of the samples of the studied class and the specificity expresses the wrong placement of the other class samples in the studied class. Finally, the accuracy is the percentage of total placement of the correct samples in their classes. These three criteria are expressed using Equations 1 to 3. TP is the number of samples of each class that are correctly classified; TN is the number of samples on the main diameter of the confusion matrix minus the number of samples of the studied classes. FN is the sum of the horizontal samples of the class examined minus the number of samples of the studied class. Finally, FP is the sum of the vertical samples of the studied class minus the number of samples of the same class [13]. In Table 7 shows the results of the performance criteria of segmentation algorithm. Based on this table, it is shown that the highest sensitivity is related to the apple class with the value of 99.242 percent and the highest specificity is related to the class of background objects with a value of 99.458 percent. In Figure 7 shows a pseudo code of segmentation algorithm. In pseudo code explain final segmentation algorithm in 13 stages.

The Speed of the Segmentation Algorithm
The system used to analyze, run the segmentation algorithm and detect background objects and apples was a laptop with Intel Core i3 CFI processor, 330M at 2.13 GHz, 4 GB of RAM-4GB and

7.
Results showed that the percentage of total detection of segmentation algorithm was 99.013%. 8.
The highest sensitivity was related to apple class with the value of 99.242% and the highest specificity was related to the class of background objects with a value of 99.458%. 9.
The results showed that the processor speed was about 0.825 seconds for the segmentation of a frame.
For future work, a fruit recognition system should be implemented and vegetables to improve recognition functionality and flexibility for wider use.
The process should be improved by extending its functions to process and recognize more variety of different fruit images. Besides that, a texture-based analysis technique could be combined with the existing three features analysis technique on the system in order to gain better discerning of different fruit images.