Development and Evaluation of a Small-Scale Apple Sorting Machine Equipped with a Smart Vision System

Abstract

One of the most important matters in international trades for many local apple industries and auctions is accurate fruit quality classification. Defect recognition is a key in online computer-assisted apple sorting machines. Because of the cavity structure of the stem and calyx regions, the system tends to mistakenly treat them as true defects. Furthermore, there is no small-scale sorting machine with a smart vision system for apple quality classification where it is needed. Thus, the current study focuses on a highly accurate and feasible methodology for stem and calyx recognition based on Niblack thresholding and a machine learning technique using k-nearest neighbor (k-NN) classifiers associated with a locally designed small-scale apple sorting machine. To find an appropriate mode, the effects of different numbers of k and metric distances on stem and calyx region detection were evaluated. Results showed the effectiveness of the value of k and Euclidean distances in recognition accuracy. It is found that the 5-nearest neighbor classifier and the Euclidean distance using 80 training samples produced the best accuracy rates, at 100% for stem and 97.5% for calyx. The significance of the result is very promising in fabricating an advanced small-scale and low-cost sorting machine with a high accuracy for the horticultural industry.

1. Introduction

Apples are one of the most popular fruits worldwide with an annual production of over 80 million metric tons. Thus, the apple industry, including sorting systems, is as important as the production of apples and the supply chain in the international trade market of fruit. In general, a key subsystem of an advanced fruit sorting system consists of a computer vision system (CVS). By now, the developments in computer vision have empowered a range of applications not only for fruit defect detection, but also for quality classification. On the other hand, the classification of fresh fruit has shown more complexities that should be resolved. Tian et al. [1] reviewed research that applied computer vision technology in agricultural automation and concluded that this technology has wide engagement potential in the future works of various agriculture-related matters. Moreover, computer-assisted fruit classification presents many challenges either due to similarities in color or characteristics [2,3,4]. The key problem of apple defect detection is identifying characteristics of defects such as size and location [5]. One of the challenges in computer vision-based apple sorting is no obvious distinction between the stem-end/calyx and real defect. The stem end/calyx appears in binary images as a dark object and may be confused with true defects. If a proper image processing method is not developed, the inspection system will tend to mistakenly sort the fruits, thereby causing a decrease in the accuracy of the classification system.

In order to solve this problem, researchers have applied numerous methods [3]. Some researchers attempted to design and construct a system to orientate the fruit in such a way that this region would not exist in the images [6,7,8,9]. Others used several algorithms for the identification of these regions. For instance, Cheng et al. [10] applied a near-infrared and mid-infrared dual camera system for online stem end/calyx recognition. Employing blob analysis and blob labeling to identify true defects from calyxes, the accuracy of the system to recognize both defects and stem ends/calyx was estimated at 92% in the range of 700–750 nm wavelengths. In another work, Ekramirad et al. [11] used NIR hyperspectral imaging for defect detection of apples in the wavelength of 900–1700 nm and a gradient tree boosting (GTB) ensemble classifier to achieve accuracy of up to 97%. However, the use of this system in industrial sorters was hindered by the high cost of the imaging equipment. Mohana and Probhakar [12] introduced shape features for stem and calyx region recognition and compared Fourier, Radon, and multifractal approaches for shape description. The stem end/calyx was segmented by multi-thresholding and differentiated from defects using multiple classifiers such as SVM, ANN, and the linear discriminant classifier (LDC). The best classification results were achieved by the SVM classifier, reaching accuracy rates of 98% and 94% for the stem region and calyx, respectively. The authors claimed that this method can be carried out offline and can be used as one of the steps in apple sorters. A hyperspectral imaging system for distinguishing the stem end and calyx from bruises was employed by Xing et al. [13]. Using this approach, accuracies of 100% for Golden Delicious and 98.33% for Jonagold apples were obtained. In this method, there was no fruit rotation, and the feeding system had a low speed. Mizushima and Lu [14] used a radius function for the stem detection of apples. The radius function is a function of the number of contour points and the distance between the contour points and the centroid [15]. Evolution-constructed features were utilized by Zhang et al. [16] for automated stem and calyx detection from images acquired in the infrared spectrum. Results showed that bruises and blemishes were differentiated from the stem/calyx at an accuracy of 94%. Differences in the 3D shape information of the stem/calyx and defects for automated stem/calyx detection were used by Jiang et al. [17]. Using the shape-from-shading method (SFS), 2D-NIR images of apples were extended to 3D reconstruction, and an overall accuracy of 90.15% was achieved. Wen and Tao [18] used a NIR/MIR imaging system for distinguishing the stem/calyx from defects. NIR is sensitive to true defects and the stem/calyx regions while the MIR camera is only sensitive to the stem/calyx. The stem ends and calyxes were classified with recognition rates of 98.86% and 99.34%, respectively. Penman [19] used blue light for the determination of concavities such as the stem and calyx on the apples and analyzed the reflection patterns that depend on the shape and orientation of apples. When the stem and calyx were near the center of view of the fruits, the results were better than when they were near the perimeter. Moallem et al. [20] considered two cases of stem location within the image inside and outside the apple and used a morphological method and a Mahalanobis distance classifier in a computer-based apple grading algorithm. The final decision for stem detection was made by combining the results of the two methods. They used k-means clustering method (k = 2) and Cb components of YCbCr color space for calyx detection. Results of detection for stem outside the apple and calyx was good, but the similarity between stem region and defects led to incorrect results in the case of stem inside the apple. In this experiment, the evaluation was based on one image per apple, meaning that more efforts would be necessary for real-time experiments involving multiple images of apples, in addition to improving the recognition in cases when the stem end is inside the apple. Sofu et al. [21] used the geometrical and color features of the stem, calyx, and actual defects to distinguish these areas in an online apple grading system. They do not report distinct results of stem/calyx detection, but the overall sorting rate results of their system were 79–89% for speeds of 0.2 and 0.05 m/s, respectively. Color, structural and textural features for discriminating between stem/calyx and true defects as well as RELIEF algorithm were used by Zhang et al. [22] for feature selection. A weighted RVM classifier is used for classification of candidate regions into stem/calyx or defects. Overall accuracy result was 95.63%. Zhang et al. [23] used a structured light system for reconstructing the 3D concave shape of the stem/calyx region. NIR structured light was used because it reflects the property of the material and is independent of color variation. The system correctly identified stem/calyx at an average accuracy of 96.8%. Lv et al. [24] used salient target detection by convex hull center priori and Markov adsorption chain and Otsu method thresholding for segmentation of red Fuji apple. The proposed method showed a better result than k- mean clustering method and region growing method. In another work, Lv et al. [25] used contrast limited adaptive histogram equalization (CLAHE) algorithm and R-B color difference for edge detection and segmentation of bagged green apple images to overcome problems of highlighted area caused by natural light reflection. By this approach, the more complete fruit region of apples was obtained. Beyaz [26] used a webcam and LabView platform for analyzing color change of damaged area on the Mondial Gala, Mitch Gala and Jonagold apple varieties. The method could determine the damages caused by finger touch force. Results showed that the analysis of images can be used for determining damaged area by changes in color of this region. Unay and Gosselin [27] proposed an approach based on statistical, textural and shape features, and introduced them into LDC, nearest neighbor, fuzzy nearest neighbor, SVM, and AdaBoost classifiers. Results showed that SVM presented the best performance, with a recognition accuracy of 99% for stem and 100% for calyx. Xia et al. [28] applies a fringe projection system to identify the stem and calyx of apples based on their concave shape. A 3D surface of the front half of an apple reconstructed and followed by threshold segmentation and a 2D convex hull technique. By this approach they could identify stem and calyx with 98.93% accuracy. Some researchers used Convolutional Neural Network (CNN) based on deep learning technology for image classification of food products. Liu et al. [29] stated that results obtained by CNN have better performance than traditional machine learning algorithms and used CNN as a depth feature extractor for food material such as apple. Defective apples by this approach could be classified by the rate of 96.5%. In another work, Wu et al. [30] also proposed an automatic detection method for stem and calyx detection based on laser induced light backscattering imaging and CNN algorithm and they obtained better results than conventional machine learning algorithms. Li et al. [31] developed a lightweight convolutional neural network so-called WearNet, for surface scratch detection and achieved precise results with a smaller model size and higher speed detection, in comparison to other commonly used CNN algorithms.

Most of the experiments carried out for stem and calyx recognition were offline and did not involve real-time sorting experiments. Furthermore, the high cost of the pieces of equipment involved prohibits their adoption in industrial scale. However, it is still a challenging task to accurately detect the apple defect area on the automatic sorting line [32]. Transition and feeding speed are other important factors that also affect the deployment of the approaches described in real-time applications. Hence, the absence of a comprehensive and practical apple sorting system is observed.

The objective of this research was to evaluate an online identification of stem/calyx of apple based on the nearest neighbor, k-nearest neighbor classifiers with different values of k and three metric distances, and segmentation by Niblack thresholding. Results were compared to obtain the best classifier and metric distance for an online apple sorting machine.

2. Materials and Methods

2.1. Image Acquisition System

The imaging system consisted of a VGA color camera with a resolution of 640 × 480 pixels. A lighting chamber with the dimensions of 51 × 35 × 40 cm (L, W, H) was used. Apples were conveyed by a biconical roller conveyor into the imaging chamber. The conveyor length in the field of view of the camera was 51 cm. So, the image resolution in the x-direction was calculated as:

$$resolution in the x direction=\frac{510}{640}=0.797\frac{mm}{pix}$$

(1)

The x-direction was the direction of forward movement of the conveyor. A pair of 10 white colored power LEDs with the color temperature of 6000–6500 k was used as a lighting source. This wavelength range was chosen to produce a maximum difference between the intensity of sound and defected area of apple surface. The light sources were arranged on two sides of the lighting chamber. The LEDs were placed in front of the chamber wall, and the light was reflected from the wall into the chamber. Also, a concave shape was made of calk paper and placed in front of the light source to act as a diffuser; this is, for light to be diffused in a more uniform way. A schematic of the light source arrangement is shown in Figure 1. Apples were rotated while moved forward by the biconical roller so that all sides of apples could be inspected by the imaging system. The biconical roller conveyor is also shown in Figure 2.

2.2. Stem and Calyx Recognition

The proposed system for stem and calyx recognition consisted of the training and the testing phase. In the training phase, images of apples were acquired offline by manually placing apples in different directions in the field of view (FOV) of the camera and saved individually in a personal computer.

Apples were framed in the FOV in such a way that different positions of stem/calyx in the images could be evaluated. In order to assess the effect of the number of training samples on the results, two training subsets consisting of 40 and 80 samples were created. In the testing phase, color RGB images were acquired at a frame rate of 30 fps and processed in real-time by the image processing system. For the classification of stem/calyx, two methods of NN and k-NN, using values of k from 3 to 5, were applied under three different metric distances.

2.3. Classification Method

Nowadays, classification techniques are one of the main approaches for food quality evaluation. Computer vision simulates human thinking by the application of advanced classification system. This method helps in complex, fast and consistent decision-making [33]. In a classification problem, chosen features of objects are measured and compared with a set of previously set criteria in order to sort such objects in a set of limited classes.

Nearest neighbor (NN) is one of the statistical classification methods that use statistical properties of observations from training samples to classify new objects. The NN is a non-parametric method that assigns a sample into a class. The class of a new sample is determined by measuring the distance to the nearest training set, which is called metric distance. The k-NN is an extended method of the NN by taking k nearest points into account. When using this classifier, several design choices must be evaluated. The most suitable number of neighbors of k and measuring distances should be defined in order to obtain the best predictions. Choosing a high number of k results in a linear classifier while choosing a low number of k results in a nonlinear classifier. Furthermore, the design towards acquiring suitable training data is an important issue. The notion of distance is one of the bases for classification. There is not an appropriate distance for all the problems; instead, the distance is specific to every classification problem. The right choice of distance is one of the most decisive steps for determining the object classes. In this study, three metric distances were used: maximum, sum (also known as Manhattan distance) and Euclidean. The Euclidean distance or Euclidean metric is the ordinary straight-line distance between two points in Euclidean space. The Euclidean distance between points of p and q is the length of the line segment connecting them $\left(\overline{pq}\right)$ as Equation (2) (Figure 3).

$$d\left(\mathrm{p},\mathrm{q}\right)=\mathrm{d}\left(\mathrm{q},\mathrm{p}\right)=\sqrt{{\left({\mathrm{q}}_1-{\mathrm{p}}_1\right)}^2+{\left({\mathrm{q}}_2-{\mathrm{p}}_2\right)}^2+\dots +{\left({\mathrm{q}}_{\mathrm{n}}-{\mathrm{p}}_{\mathrm{n}}\right)}^2}=\sqrt{{\displaystyle \sum }_{\mathrm{i}=1}^{\mathrm{n}}{\left({\mathrm{q}}_{\mathrm{i}}-{\mathrm{p}}_{\mathrm{i}}\right)}^2}$$

(2)

Taxicab geometry is a form of geometry in which the distance between two points is the sum of the absolute difference of their coordinates. The Manhattan distance, also known as rectilinear distance, city black distance or taxicab metric, is defined as the sum of lengths of the projections of the line segment between the points onto the coordinate axes (Equation (3)). The difference between Euclidean and Manhattan distances is shown in Figure 4.

$$\mathrm{d}={\displaystyle \sum }_{\mathrm{i}=1}^{\mathrm{n}}\left|{\mathrm{X}}_{\mathrm{i}}-{\mathrm{Y}}_{\mathrm{i}}\right|$$

(3)

The k-nearest neighbor algorithm is one of the most popular classifiers used in food products sorting problems. It assigns an object to a category that is the most representative among the k-nearest samples of that object. Nearest samples are found by metric distance measure using the distance information of the nearest samples. Performance of k-NN highly depends on the k parameter and size of training set. Both of which should be large enough for optimal performance and small enough to avoid computational overload [34].

2.4. Feature Extraction

Feature extraction computes the feature vectors in the feature space from the input image. Necessary or appropriate properties features are extracted from the input image to distinguish images of different classes and, thus, prevent feeding of unnecessary information into the machine learning. The goal of classification is the definition of which feature to use. Feature values of different classes should have significant differences. In this particular classification study, different types of shape descriptors were used. A shape descriptor is a feature vector based on particle analysis measurements. In this research, the range of pixel-based size of stem and calyx regions was defined by multiple samples. Firstly, the algorithm defined the size of candidate regions, and only those candidate regions whose size fell within the range were introduced onto classifier. By doing this, the effect of size on the shape description was eliminated. Shape description was based on shape features that are invariant to scale changes, rotation, and mirror symmetry. Six features were used for shape description in particle classification algorithms for stem and calyx definition. These features are described below.

Feature 1 was Circularity of the sample. This feature is a function of perimeter and area of the sample. The circularity of a circle is 1. The circularity ratio of the calyx is higher than that of the stem.

Feature 2 was Elongation of the sample. This feature is especially useful for stem definition. Feature 3 was Convexity of the sample. Convexity is defined as the ratio of the convex hull over that of the original contour. Feature 4 was the presence of holes in the samples. Because stem/calyx regions and defects appeared in images as a hole, this feature is useful for the determination of these regions in the image.

Feature 5 was the spread of the sample, while Feature 6 was Slenderness of the shape.

2.5. Segmentation of Image by Niblack Thresholding

Images of apples were fed to the training phase algorithm. In order to evaluate the effect of the training set sample size, two training sets, containing 40 and 80 samples, were prepared. The proposed system contained the following steps: pre-processing, classification algorithm optimization, class definition, and training. In the image pre-processing step, the background was removed by setting the region of interest (ROI). In the next steps, the detection algorithms were applied to the ROI space instead of the whole image. The cropped RGB image was converted into a binary image by thresholding, so as to clarify the stem and calyx region or other dark regions with low intensity. Local thresholding was applied instead of general thresholding for segmentation. The threshold $T\left(x,y\right)$ is a cut-off grey-level value used to convert an image into a black-and-white image, according to Equation (4),

$$\mathrm{b}\left(\mathrm{x},\mathrm{y}\right)=\{\begin{array}{c}0 \mathrm{if} \mathrm{I}\left(\mathrm{x},\mathrm{y}\right)\le \mathrm{T}\left(\mathrm{x},\mathrm{y}\right)\\ 1 \mathrm{otherwise}\end{array}$$

(4)

where b (x, y) is the resulting binary image and I (x, y) is the intensity of a pixel in location (x, y) in the image. In this approach, images were converted into smaller sub-images and the threshold was defined for each of them based on local properties such as the variance range or the surface fitting parameters of neighbor pixels. One of the most efficient local thresholding methods is the Niblack thresholding algorithm. In this method, a threshold is defined for each region by moving a rectangle window across the grey-scale image. The threshold computed by the mean (m) and standard deviation (s) for all pixels in the rectangular region is:

$${\mathrm{T}}_{\mathrm{Niblack}}=\mathrm{m}+\mathrm{k}*\mathrm{s}=\mathrm{m}+\mathrm{k}\sqrt{\frac{1}{\mathrm{NP}}{\sum }^{}{\left({\mathrm{P}}_{\mathrm{i}}-\mathrm{m}\right)}^2}=\mathrm{m}+\mathrm{k}\sqrt{\frac{{\sum }^{}{\mathrm{P}}_{\mathrm{i}}{}^2}{\mathrm{NP}}-{\mathrm{m}}^2}$$

(5)

where NP is the number of all pixels in the image, T is the threshold value, m is the mean of $P_i$ pixels, and k is a constant that depends on the extent of noise in the background and is equal to −0.1 [35]. The local mean in Equation (5) could be defined by the convergence image of I($x_0$,$y_0$). The value of I($x_0$,$y_0$) from the primary image was computed by Equation (6):

$$I\left(x_0,y_0\right)={\displaystyle \sum }_{x=1}^{x_0}{\displaystyle \sum }_{y=1}^{y_0}f\left(x,y\right)$$

(6)

The convergence image of the local mean in the rectangular region could be defined by Equation (7):

$$\mathrm{m}\left({\mathrm{x}}_0,{\mathrm{y}}_0\right).\mathrm{N}=\mathrm{I}\left({\mathrm{x}}_0+\Delta \mathrm{x},{\mathrm{y}}_0+\Delta \mathrm{y}\right)+\mathrm{I}\left({\mathrm{x}}_0-\Delta \mathrm{x}-1,{\mathrm{y}}_0-\Delta \mathrm{y}-1\right)-\mathrm{I}\left({\mathrm{x}}_0-\Delta \mathrm{x}-1,{\mathrm{y}}_0+\Delta \mathrm{y}\right)-\mathrm{I}\left({\mathrm{x}}_0+\Delta \mathrm{x},{\mathrm{y}}_0-\Delta \mathrm{y}-1\right)$$

(7)

where $m\left(x_0,y_0\right)$ is the local mean value in the rectangular region with the coordinate center $\left(x_0,y_0\right)$; $(y_0-\Delta y,x_0-\Delta x)$ is the top left corner coordinate; $(y_0+\Delta y,x_0+\Delta x)$ is the down right corner coordinate; and N is the number of pixels in the rectangle, which is computed by:

$$\mathrm{N}=\left(2\Delta \mathrm{x}+1\right)\left(2\Delta \mathrm{y}+1\right)$$

(8)

It is well-known that local thresholding leads to more accurate results than universal thresholding. However, results depend on the rectangular window size. When the rectangle size is smaller, more accurate results are obtained, although at the expense of longer processing times. The images were converted to a binary image whereby dark objects appeared as black regions. The dark objects were separated from bright ones, while objects touching the ROI were deleted. In the binary image, all of the dark regions except the stem/calyx region were removed. The remaining object in the image was a dark region belonging to the stem or calyx. Two methods of NN and k-NN were appraised for classification. The number of k was varied from 3 to 5 in order to assess the effect of k. The maximum, Manhattan, and Euclidean distance metrics were selected one by one. Two classes of stem and calyx were introduced. Then, in the last step, the recognition algorithm was trained. Apples were placed in different positions on the conveyor so that the system could be trained to locate the stem/calyx located in many situations in the images, such as the center and near the periphery of the apple.

3. Results and Discussion

The results of training were set by k-NN, k = 3, and Euclidean metric distance, as shown in Figure 5. Evaluations of 80 training samples showed that the minimum area of the calyx region was 80 pixels, and the maximum area of the stem region was 250 pixels. Thus, to reduce the computational time of the process, only the regions whose areas were greater than 80 pixels were evaluated. The classification results of stem and calyx detection are shown in

Table 1. Classification rates for stem and calyx detection by NN, 3-NN, 4-NN, and 5-NN; 40 training samples and 80 training samples; and 3 metric distances: Euclidean (Eu), maximum (max), and Manhattan (sum).

Classification Method		Metric Distance
		Euclidean		Maximum		Sum
	Training set	40	80	40	80	40	80
NN	stem	100	100	100	100	100	100
	calyx	75	85	72.5	81.25	75	85
3-NN	stem	100	100	100	100	100	100
	calyx	80	92.5	75	88.75	80	91.25
4-NN	stem	100	100	100	100	100	100
	calyx	80	92.5	75	88.75	80	91.25
5-NN	stem	100	100	100	100	100	100
	calyx	80	97.5	75	93.75	80	96.25

.

As can be seen in Table 1, the accuracy of the system for stem classification is high, and it can detect the stem completely by all classification methods and metric distances. One of the reasons is the special shape of the stem region, which facilitates its discrimination from true defects. Detection accuracy of the calyx was lower than the stem. The calyx shape and the pixel-based area of this region is more similar to true defects, and thus the system incurs more errors differentiating the calyx from true defects. When the calyx falls near the periphery of the apple boundary in the image, the error rate can be even higher. By increasing the number of training samples, the accuracy of the system in calyx classification using all classification methods and metric distances could be increased. The accuracy was maximized by the k-NN with k = 5 and Euclidean distance.

The number of training samples has an important effect on the performance of the classification algorithms. Using an insufficient number of training samples, the classification accuracy would not have significantly increased, even using the strongest algorithm. The results of the calyx classification by four types of classifiers are shown in Figure 6. Using 40 samples for training, an increase in k had a negligible effect on the detection accuracy; however, when the training set was doubled, increasing k led to a noticeable improvement in the classification rate. The accuracy was the highest at k = 5 and using the Euclidean distance. Classification rates by Euclidean and sum metrics were generally higher than the maximum distance metric, although significant differences were not observed between the Euclidean and sum distances. In this research, the poorest performance was observed for the maximum metric distance. Simultaneous effects of the k number and training set on calyx classification are shown in Figure 7. By increasing k and the training samples, accuracy was improved. For k = 3, this increase was significant, although the classification performance was not improved by increasing k from 3 to 4. By increasing k, the classification rate by Euclidean distance was higher than the other two metric distances.

All the classifiers could completely detect the apple stem. Thus, an identification score was defined as a criterion for evaluating the different situations in stem detection. The classification score is an indicator for identifying a class of a given sample using the classifier. In general, if the identification score is higher than 800, it is considered as appropriate, and if higher than 900, it is desirable. The highest value of the identification score was assumed to be 1000. Analysis of variance was done on the identification scores by assessing the simultaneous effect of the classification algorithm, metric distance, and training number. The results (

Table 2. ANOVA for the effect of classifier, training sample size, and metric distance on stem identification score.

Source	Type III Sum of Squares	df	Mean Square	F	Sig.
Method	224,748.909	3	74,916.303	2.438	0.021
Number of training samples	316,714.909	1	316,714.909	10.306	0.000
Metric distances	147,470.576	2	73,735.288	2.399	0.060
Classification methods × training samples	362,626.61	3	120,875.354	4.504	0.002
Classification methods × metric distances	420,985.857	6	70,164.309	2.615	0.013
Training samples × metric distances	159,939.660	2	79,969.830	2.980	0.053
Classification methods × training samples × metric distances	509,843.426	6	84,973.904	3.167	0.004
Error(s)	7,898,122.091	257	30,731.993
Total	1.034 × 108	264
Corrected Total	8,587,056.485	263
a. R Squared = 0.961 (Adjusted R Squared = 0.957)

) show that metric distance had no significant effect on the stem identification score, while the effects of training sample size and classification method were significant at 1% and 5% levels, respectively. The analysis of variance showed that the interaction effect of the metric distance and the number of training samples was not significant on stem identification score, although interactions were significant. The interactions between training sample size and classification method were significant at the 1% level; between metric distance and classification method at the 5% level; and the three-way interaction at the 1% level. The comparisons between the means of these three factors are shown in

Table 3. Comparison of means for stem classification scores produced by different classification methods, training sample sizes, and metric distances.

	Mean	Standard Deviation
Classification method
NN	698.712	200.660
3-NN	766.287	170.801
4-NN	766.621	173.498
5-NN	777.030	171.299
Training sample size
40	714.606	166.956
80	789.719	187.770
Metric distance
Euclidean	770.693	167.833
Maximum	718.806	200.560
Sum	766.988	165.513

. Results showed that the 4-NN classifier, Euclidean metric distance, and 80 training samples led to the best performance, although the differences between the three classifiers were only slight.

The poorest result was obtained by using 40 training samples, the NN classifier, and the maximum distance. Detection rates of the maximum distance were lower than the other two metrics in all conditions. In addition, k = 5 produced maximum accuracy. However, by increasing k, the computational time was increased; therefore, a compromise should be attained between accuracy and speed detection for the stem/calyx. Further increasing the value of k showed that the accuracy does not change significantly, but the computational time increases.

The results show that the success of the algorithm in identifying the stem/calyx region highly depended on training classifier. Local thresholding is another determinant of the classification accuracy. By this approach, the segmentation of the candidate region can be more accurately performed. Figure 8 shows the segmentation of the stem and calyx regions using clustering and inter-variance as the global thresholding approach and Niblack as the local thresholding approach. As can be seen in Figure 8, the regions segmented by clustering and inter-variance were smaller than the exact shape of the stem and calyx. One of the advantages of this method is the accurate shape extraction by local thresholding. The main disadvantage of using local thresholding is its high computational load, particularly for the online classification process. Nevertheless, by precisely defining the search window and other related parameters, this computational time could be optimized in this study.

In this study, the combination of local thresholding and machine learning led to more precise results than previous works. The results were comparable to other studies that utilized complicated and high-cost equipment such as near-infrared structural lighting [23,36] and hyperspectral imaging [37]. Unlike when using a thermal imaging camera, uniform temperature distribution is unnecessary in this methodology. Moreover, noises due to uneven lighting have no problematic matter compared to hyperspectral imaging. In the study, the uniform illumination caused the local thresholding to increase the precision of binarization. Here, up to 97% accuracy was obtained for calyx recognition and all of stem end regions either with or without the stem in the images were successfully recognized (100%). The results were improved both in precision and computational time in comparison to other documented studies. For example, the recognition rate reported using the SVM classifier and monochrome camera with a 750 nm filter was 100% for stem and 99.5% for calyx, but the system only captured one view of each fruit [27]. Indeed, this reduces the value of quality control since other views of fruit are significantly unobserved. Zhang et al. [23] used NIR-coded structural light and obtained a precision of 90.25%, Yu et al. [38] used hyperspectral imaging and obtained a precision of 95%, and Wu et al. [30] used laser-scattering spectroscopic images and CNN algorithms and obtained a precision of 92.5%. This shows that the results add value to quality control, discrimination accuracy, computational time, and the development of low-cost and small-scale fruit sorting machines.

The cost of a machine vision system mainly depends on the cost of imaging equipment. In this research, a conventional VGA camera was used whose imaging system is significantly more cost-effective in comparison to other performed research. Moreover, the use of an appropriate algorithm and a designed conveying system eliminated having to use expensive cameras and a complicated lighting system. The cost of the conveying system was also comparable to the complex equipment used by some researchers in

Table 4. The most important mechatronic components applied by different researchers in high-tech apple sorting systems (2010–2022).

Authors	Year	Conveying Type	Imaging Type	Lighting Type	Pattern Recognition Type
Xiaobo et al. [39]	2010	cone shape roller conveyor	three-color CCD cameras	N/A	N/A
Unay et al. [40]	2011	N/A	monochrome camera	N/A	N/A
Baranowski et al. [41]	2012	N/A	hyperspectral and thermal cameras	N/A	N/A
Shive Rame and Anand Singh [42]	2012	N/A	N/A	N/A	multiclass SVM
Suresha et al. [43]	2012	N/A	N/A	N/A	SVM
Mohana et al. [44]	2013	N/A	N/A	N/A	k-NN classifier
Mizushima and Lu [45]	2013	N/A	CCD color camera	eight LED lights	linear SVM and Otsu method
Sasnjak et al. [46]	2013	roller conveyor	N/A	N/A	N/A
Zhang et al. [16]	2013	N/A	N/A	N/A	ECO features and GA
Mendoza et al. [47]	2014	N/A	N/A	quartz tungsten halogen light source	N/A
Toylan and Kuscu. [48]	2014	roller conveyor	complementary metal oxide semiconductor color camera	four fluorescent lamps	a method based on multicolor space
Zhang et al. [36]	2015	N/A	multispectral vision system	two pairs of visible LED light sources and NIR LED light sources	RVM classifier
Sadegaonkar and Wagh [49]	2015	roller conveyor	N/A	N/A	N/A
Zhang et al. [50]	2015	N/A	hyperspectral monochrome CCD camera	N/A	SPA and a binary PLS-DA
Vakilian and Massaah [51]	2016	shielded conveyor belt	CCD color camera	N/A	Gabor filter and NN classifier
Sofu et al. [21]	2016	two channels roller conveyor	two CCD video cameras	N/A	N/A
Keresztes et al. [52]	2016	conveyor belt	infrared line scan camera	four 20 W DC tungsten halogen spots	N/A
Chio and Chen [8]	2017	revolving tray	N/A	N/A	N/A
Wu et al. [30]	2020	N/A	laser-induced backscattering imaging system	semiconductor laser	convolutional neural networks (CNN)
Fan et al. [53]	2020	black fruit cup conveyor	two commercial RGB cameras	two light- emitting diode (LED) strips	convolutional neural networks (CNN)
Henila and Chithra [54]	2020	N/A	CCD camera	N/A	fuzzy cluster-based thresholding (FCBT) method
Shurygin et al. [55]	2022	table with rubber roller	hyperspectrometer BaySpec OCI-F	tungsten- halogen lamps	random forest (RF) classifiers
Tang et al. [56]	2022	N/A	near-infrared industrial camera	adjustable ring light source	U-Net

.

4. Conclusions

In this research, a combination of Niblack local thresholding and machine learning techniques was used for online stem/calyx detection in a small-scale and low-cost apple-sorting machine. Niblack thresholding produces precise segmentations of apple images, allowing the extraction of the stem/calyx region. The recognition performance is directly dependent on the segmentation process. A precise stem/calyx shape extraction leads to more precise feature extraction and classification. Both the true defects and the stem/calyx regions were identified in this segmentation step. By removing the true defects, the tested classifiers were trained by binary images of the stem/calyx. When the calyx region fell near the boundary of the apple in the images, the accuracy of detection was lower. The 5-nearest neighbor classifier and the Euclidean distance using 80 training samples produced the best accuracy rates, at 100% for stem and 97.5% for calyx. By eliminating many pre-processing algorithms and having precise detection of the stem/calyx region shapes, the computational time was reduced, which makes it possible to use this method in online processes. The required time for image processing operations was 201 ms, and the computing time for stem and calyx detection was 60 ms by a PC Intel(R) Core (TM) 2 Due CPU. In this system, five apples could be classified per second. While the determination of the best classifier is still a challenging task, researchers have attempted different types of classifiers. In this study, the combination of local thresholding with k-NN produced encouraging results. The combined algorithms led to increased accuracy in detection, which would also be appropriate for online detection in real-time sorting processes by commercial equipment on a small scale. As a future work, this could be implemented in real-time grading operations not only for apples, but also for other similar fruits.