Corn Classification System based on Computer Vision

Abstract

Automated classification of corn is important for corn sorting in intelligent agriculture. This paper presents a reliable corn classification method based on techniques of computer vision and machine learning. To discriminate different damaged types of corns, a line profile segmentation method is firstly used to segment and separate a group of touching corns. Then, twelve color features and five shape features are extracted for each individual corn object. Finally, a maximum likelihood estimator is trained to classify normal and damaged corns. To evaluate the performance of the proposed method, a private dataset consisting of images of normal corn and six kinds of damage corns, including heat-damaged, germ-damaged, cob-rot-damaged, blue eye mold-damaged, insect-damaged, and surface mold-damaged, were collected in this work. The proposed method achieved an accuracy of 96.67% for the classification between normal corns and the first four common damaged corns, and an accuracy of 74.76% was achieved for the classification between normal corns and six kinds of damaged corns. The experimental results demonstrated the effectiveness of the proposed corn classification system.

1. Introduction

Corn is one of the most important foods and the most widely produced feed grain in the world. It also can be processed into a wide range of industrial products. Production of corn in the United States is 984.37 million metric tons produced in 2014 [1]. Due to the effects of corn quality in the price and end usage of the corn, grain-grading standards were developed by the U.S. Grain Standards Act of 1916 [2]. The grain-grading standards are updated and managed by the Federal Grain Inspection Service (FGIS) of the United States Department of Agriculture (USDA). The standards explicitly provided damaged types of corn. Both sellers and buyers now use the standards as a common and worldwide commercial language to decide the type and quality of the corn.

According to the grain-grading standards, corn is classified by moisture, weight, color, shape, odor, and damage [3]. Among these criteria, the moisture, weight, and odor of corn can be evaluated by special instruments such as an electronic analyzer [4]. For the criteria about color, shape, and damage, corn grading is generally observed with the naked eye. It is repetitive and tiring, and, consequently, errors can be made. A plausible way to classify corn is using computer vision to classify corn automatically.

Previous studies have demonstrated that computer vision could be a significant way to analyze grains. Zayas et al. used image processing and pattern recognition to identify whole and broken corn kernels [5]. Ni et al. developed a prototype system to classify whole and broken kernels [6], corn kernels based on their crown shape [7], and grade corn based on their size [8]. Luo et al. used computer vision technology to separate six types of wheat kernels based on their color features [9]. Steenhoek et al. devised a computer vision system to classify corn based on their damage type [10]. Dana and Ivo used image processing to categorize flax cultivar based on seed shape and color [11]. Chen et al combined machine vision and pattern recognition to classify five types of corn based on their shape, color, and geometric features [12]. Arribas et al. presented an automatic leaf image classification system for sunflower crops using neural networks [13]. Gao et al. designed a rapid corn sorting algorithm based on machine vision. The proposed corn classification had a speed of 30 ears/min with a 1280 × 1024 pixel CCD camera [14]. Valiente-Gonzalez et al. devised a computer vision system to automatically evaluate the quality of corn lots by identifying damaged kernels that combined algorithm-based computer vision techniques and principal component analysis (PCA) [15]. Liu et al. proposed an efficient image processing algorithm to detect parameters such as the length, the number of ear rows, and the quantity of kernels in an ear of corn based on a machine vision [16]. Mohammad et al. designed an expert system with ant colony optimization (ACO) to automatically recognize different plant species through their leaf images [17]. Gao et al. designed an automatic detection and classification algorithm for corn product quality and equipment [18]. The algorithm firstly calculated texture features of fresh corn images through wavelet analysis and then measured the separation degree of texture features by the maximum visual entropy function. Finally, according to the texture features and entropy criterion, the fresh corn products were classified. Sun et al. identified and classified damaged corn kernels including undamaged, insect-damaged, and mildew-damaged by using impact acoustic multi-domain patterns [19]. Zhang et al. classified three different degrees of freeze-damage in corn seeds using a VIS/NIR hyperspectral imaging system [20]. Chouhan et al. used computer vision and soft computing methods to identify and classify diseases of the leaf for the plant [21]. Sajad et al. developed a computer vision algorithm that combined color features and a classifier based on ANN with genetic algorithms for detecting existing fruits in aerial images of an apple cultivar and estimating their ripeness stage [22]. These proposed systems mainly focused on algorithms, but the research on classification of damaged corn is less and has paid little attention to the design of corn image capture platforms. In addition, most of them worked well for a small scale of corn, however, since touching corn could not be segmented accurately, their performance may be degraded if the number of corns is increased.

This paper proposed a new corn classification method to classify normal and damaged corn. The main contributions of this work are: (a) a set of images of normal corn and six kinds of different damage corns were collected; (b) color and shape features of corn were fused to train a corn classification model; (c) the performances of corn classification conducted on different test sets were evaluated.

The rest of this paper is organized as follows. The experiment data is described in Section 2, while the proposed classification scheme is present in Section 3. Experimental results are reported and analyzed in Section 4. Finally, this paper is concluded in Section 5.

2. Experimental Data

The corn used in this work were collected at the experimental farm of Southern Illinois University Carbondale (SIUC). Seven kinds of corns were considered in this paper, including normal, heat-damaged, germ-damaged, cob rot-damaged, blue eye mold-damaged, insect-damaged, and surface mold-damaged. As shown in Figure 1, the image of corn was acquired using an optical image collection system that consisted of an imaging table, a color camera, a two-way lighting system, and a computer. The imaging table contained an aluminum frame and a transparent, plastic plate. Corns were arranged on the plastic plate and the plate slides in the aluminum frame. The camera was setup vertically to take top-view images of the corn. The lighting system consisted of upper fluorescent lamps and lower fluorescent lamps. The corn images captured by this platform with a different lighting system are shown in Figure 2. In this work, the corn image captured using upper fluorescent lamps was used in our experiments. The upper fluorescent lamps provided uniform lighting, and the images captured with these lamps were used to extracted color features of the corn. The lower fluorescent lamps placed under the transparent plastic plate provided an overexposure of the image, which was used to extract shape features of the corn. Experimental images were acquired by a DFK 72BUC02 color camera (2592 × 1944) with a M12VM412 lens and saved into a computer (including an Intel Core 5i 3.10GHz, 4GB RAM) connected to the camera.

During image collection, each time there were about 150 corns selected and arranged on the plate. Images collected by this collection system had a resolution ratio of 2592 × 1944 and were saved in BMP format. After collection, normal corns and damaged corns in the image were manually labeled by agronomy experts from Southern Illinois University.

3. Methodology for Corn Classification

The process of corn classification is described in Figure 3 in detail.

3.1. Image Segmentation

To address the problem of touching corn in corn classification, individual corns in the captured image should be accurately segmented first. This paper employed a line profile-based segmentation algorithm (denoted as LPSA) proposed in the work of [23]. Figure 4 describes the segmentation algorithm in detail. The LPSA is usually used to separate touching corn kernels. First, the input corn image was binarized by Otsu’s [24] method to separate corn from the background. Then, LPSA determined the coordinates of the centroid in the object image and created an axis line through the centroid, then it equidistantly created perpendicular lines in the axis line. Pixels that fell on a perpendicular line were summed, and a single value for every perpendicular line was obtained; the resulting vector was called a profile. The axis line rotated, and profiles were generated from other angles. The angle increment was constant. A corn kernel was represented as one nodule in the profile. Two touching corn kernels were presented as two nodules in the profile. The minimum point between the nodules was the touching between the corns. Profiles of a corn object were generated from various angles to find the ultimate angle to draw the split line that separated the touching corn. A result of corn segmentation with LPSA is shown in Figure 5, in which touching corns have been separated effectively.

3.2. Feature Extraction

To distinguish different grades of the corn segmented by LPSA, features about color and shape are extracted in this section, as shown in Figure 6. Particularly, shape features were extracted from the segmented corn area, while color features were extracted from the corresponding corn area in the upper image, in which corns had a uniform illumination. The details of the features are described as follows.

Corn is predominantly white, yellow, and mixed. Most corns are white or yellow. In the market, white corn is more expensive than yellow corn [25]. Therefore, color is an important feature. Corn has two faces, they are: face up and face down. As shown in Figure 7, face up is the side of the corn that contains the germ area, and face down is the side of the corn that does not have a germ area. The germ area is usually white. We used a well-designed method to separate the germ area from corn [26]. As shown in Figure 8, a polygon is used to estimate the germ area. The mean intensity of the germ area calculated from each channel of RGB and HIS color space are then used as the color feature.

For shape features, we extracted five dimensional measurements of segmented corn, i.e., perimeter, area, circularity, rectangularity, and elongation, which were adjusted from the work of Zheng (2008) used for categorizing maize seed [27].

Perimeter was calculated by the method of eight connected chain codes:

$$\mathrm{P}=\mathrm{sqrt}\left(2\right)\times \mathrm{ND}+\mathrm{NX}+\mathrm{NY}$$

(1)

where P represents the perimeter of the corn kernel, ND is the number of odd chain codes, NX is the number of pixels in the horizontal direction, and NY is the number of pixels in the vertical direction. NX + NY is the number of even chain codes.

Area was calculated by counting the number of pixels in the corn image contour:

$$\mathrm{A}={\displaystyle \sum }_{\left(\mathrm{x},\mathrm{y}\right)\in \mathrm{S}}1$$

(2)

where A represents the area of the corn kernel, S is the corn kernel region of image, (x, y) is the coordinate of a pixel inside S, and a pixel value of 1 was assumed in this study.

Circularity was defined as:

$$\mathrm{C}=4\mathsf{\pi }\mathrm{A}/{\mathrm{P}}^2$$

(3)

where C represents the circularity of the corn kernel, P is the perimeter of the corn kernel, and A is the area of the corn kernel.

Rectangularity was defined as:

$$\mathrm{R}=\mathrm{A}/\left(\mathrm{H}\times \mathrm{W}\right)$$

(4)

where R represents the rectangularity of the corn kernel, A is the area of the corn kernel, H is the long axis of the corn kernel, and W is the short axis of the corn kernel.

Elongation was defined as:

$$\mathrm{E}=\min \left(\mathrm{H},\mathrm{W}\right)/\max \left(\mathrm{H},\mathrm{W}\right)$$

(5)

where E represents elongation of the corn kernel, H is the height of the corn kernel, and W is the width of corn kernel.

Examples of five shape features extracted from three samples of normal and damaged corn kernels are listed in

Table 1. Examples of five shape features for three samples of normal and six kinds of damaged corns.

	Name	Perimeter (px)	Area (px)	Circularity	Rectangularity	Elongation
Category
Blue eye mold		306.1493	6239	0.8365	0.7294	0.9681
		273.0782	4556	0.7677	0.6537	0.9647
		338.6346	7320	0.8022	0.7253	0.7500
Cob Rot		335.8061	7691	0.8571	0.76987	0.8108
		366.9605	8193	0.7646	0.6417	0.9825
		318.6346	6684	0.8273	0.7193	0.9109
Germ damage		289.3919	5002	0.7506	0.7061	0.8366
		300.6518	5918	0.82273	0.8189	0.7374
		284.3503	5868	0.9120	0.8162	0.8681
Heat damage		323.5219	7131	0.8562	0.8178	0.7339
		358.9777	8389	0.8181	0.7688	0.7097
		335.7645	6619	0.7378	0.7580	0.6271
Insect damage		358.4924	8148	0.7967	0.7760	0.6720
		359.4335	6171	0.6002	0.6128	0.8962
		337.7645	7300	0.8041	0.7865	0.6555
Surface mold		312.4924	6240	0.8030	0.7212	0.8155
		310.7939	6244	0.8123	0.8030	0.6667
		322.0071	7061	0.8557	0.7930	0.7925
Normal		349.0193	7759	0.8004	0.7070	0.7881
		350.3330	8058	0.8250	0.704	0.8220
		332.5219	7459	0.8477	0.8222	0.7232

. The unit of perimeter and area was pixel (px), and in coordinates of the plane where corn images were captured, the width/length of each pixel was equal to about 0.09 mm.

3.3. Classification

After extracting color and shape features for each corn, we next used a maximum likelihood classifier to classify the grades of corn. Specially, classification of the corn consisted of two steps: training and testing. In the training step, every class of corn and image were captured, and features were extracted for every corn kernel in that class. Mean value and covariance of all corn in the image were obtained for every feature. In the testing step, a maximum likelihood estimator was implemented to classify corn [28]. Images of random classes of corn were captured, segmented, and features were extracted. Li and Yan developed a Bayesian decision rule to minimize misclassifications [29]. The classification formula used in this study is as follows:

$$d_i\left(x\right)=\ln \mathrm{P}\left({\mathsf{\omega }}_{\mathrm{i}}\right)-\frac{\mathrm{n}}{2}\ln \left(2\mathsf{\pi }\right)-\frac{1}{2}\ln \left|{\mathrm{C}}_{\mathrm{i}}\right|-\frac{1}{2}\left[{\left(\mathrm{x}-{\mathrm{m}}_{\mathrm{i}}\right)}^{\mathrm{T}}{\mathrm{C}}_{\mathrm{i}}{}^{-1}\left(\mathrm{x}-{\mathrm{m}}_{\mathrm{i}}\right)\right]$$

(6)

where $\mathrm{P}\left({\mathsf{\omega }}_{\mathrm{i}}\right)$ is the probability of the class to occur, ${\mathrm{C}}_{\mathrm{i}}$ is the covariance matrix of the training feature, ${\mathrm{m}}_{\mathrm{i}}$ is the mean value of the training feature, $\mathrm{x}$ is the ith feature of every corn kernel, and $\mathrm{d}$ is the decision rule.

4. Experiments and Results

4.1. Experiment Settings

In our experiments there were seven kinds of corn, including normal corn and six kinds of damaged corn. For each group, 100 corns were selected to evaluate the proposed method. The ratio between training and testing was generally set to 7 to 3. The sample numbers of training sets and testing sets are shown in

Table 2. Sample numbers of training sets and testing sets in validation experiments.

No.	Category	Numbers of Training Sets	Numbers of Testing Sets
1	Blue eye mold	70	30
2	Cob rot	70	30
3	Germ damage	70	30
4	Heat damage	70	30
5	Insect damage	70	30
6	Surface mold	70	30
7	Normal	70	30

. Considering that heat damage, germ damage, and cob rot damage are three relative common damages of corn, a subset of normal corn and these three kinds of damaged corn was specially selected. The proposed method was firstly conducted on this subset, and then it was conducted on the whole set.

4.2. Results for Three Damaged and Normal Corns

This section demonstrates the classification results for four different corn types (i.e., three types of common damaged corn and normal corn). To evaluate the performance of our method, five different groups of training and testing sets were randomly selected from the whole dataset, and in each group there were 280 samples for training and 120 samples for testing. The proposed method achieved an average classification accuracy of 90.67% with a standard deviation of 4.35%.

Table 3. Confusion matrix for four kinds of classification results. Classes: 1. Normal; 2. Heat-damaged; 3. Germ-damaged; and 4. Cob-rot.

Predicted/Real Class	1	2	3	4	All Data	Classification Error by Class (%)	Classification Accuracy (%)
1	29	0	1	0	30	3.33	96.67
2	0	30	0	0	30	0.00
3	0	0	30	0	30	0.00
4	3	0	0	27	30	10.00

shows the best classification results accompanied by the confusion matrix.

The system correctly classified 29 out of the 30 samples belonging to class 1 (96.67% accuracy), meanwhile, the classification error rate of class 2 and class 3 was 0.00, and this was a perfect state. On the other hand, the highest misclassification rate was in the cob rot class with a 10% error. Analyzing these errors, three samples of class 4 were misclassified in class 1. This meant that samples in class 4 had similar properties with samples in class 1.

Figure 9 depicts the ROC curves of the developed corn classification system for the four defined classes. On the other hand,

Table 4. Performance results using different criteria: sensitivity, accuracy, and specificity, and area under the ROC curve (AUC). Classes: 1. Normal; 2. Heat-damaged; 3. Germ-damaged; and 4. Cob-rot.

Class	Sensitivity (%)	Accuracy (%)	Specificity (%)	AUC
1	96.67	96.67	98.86	0.9884
2	100.00	100.00	98.85	0.9945
3	100.00	99.17	100.00	0.9915
4	90.00	97.50	96.74	0.9833

shows the performance results of the classification method using sensitivity, accuracy, and specificity and the area under the ROC curve (AUC) for each class.

The maximum value of sensitivity (100%) was reached in the heat-damaged and germ-damaged classes. Table 3 proved this fact, since classification error of the two classes was 0. The maximum value of specificity (100%) was given in class 3. Maximum accuracy was given in the heat-damaged class at 100%, and it had the highest sensitivity. In general, all the classes obtained very similar results, so the system would be able to work for corn sorting in intelligent agriculture. For each type, the proposed method achieved an AUC value greater than 0.98. This proved that the proposed classification method was not only accurate but also robust in relation to its parameters. Only class 4 presented a lower AUC value, indicating slight confusion with class 1.

Figure 10 shows a representative image of the classification results from the developed corn classification system. The red, green, white, and blue regions were used to indicate different types of classified corns, including normal corn, cob-rot and heat-damaged, and germ-damaged corn.

Figure 10a is the image before recognition and classification. There were 126 corns including 4 germ-damaged corns and 122 normal corns. Figure 10b is the result from the developed corn classification system. A normal corn was misidentified as cob-rot corn. Its accuracy was 99.2%.

4.3. Results and Discussion for Six Damaged and Normal Corns

On the basis of the above-mentioned classification of four types of corns, the classification experiments of seven types of corns, i.e., six damaged corns and normal corn, were further carried out. Similarly, five different groups of training and testing sets were randomly selected and formed, and in each group there were 490 samples for training and 210 samples for testing. The proposed method achieved an average classification accuracy of 67.48% with a standard deviation of 7.79%.

Table 5. Confusion matrix of seven kinds of classification results. Classes: 1. Normal; 2. Heat-damaged; 3. Germ-damaged; 4. Cob-rot; 5. Blue eye mold; 6. Insect-damaged; and 7. Surface mold.

Predicted/Real Class	1	2	3	4	5	6	7	All Data	Classification Error by Class (%)	Classification Accuracy (%)
1	27	0	0	0	2	1	0	30	10.00	74.76
2	0	27	0	0	0	2	1	30	10.00
3	0	2	26	1	0	0	1	30	13.33
4	0	1	1	27	0	1	0	30	10.00
5	0	2	6	5	16	1	0	30	46.67
6	1	6	4	1	0	16	2	30	46.67
7	0	4	1	1	1	5	18	30	40.00

shows the best classification results for six different types of damaged corns and normal corn accompanied by the confusion matrix.

Notably, the results of the seven classifications using the above-mentioned classification method were unsatisfactory. Table 5 shows that the maximum misclassification rate was in classes 5 and 6 with a 46.67% error, and the misclassification rate of class 7 was 40%. Analyzing these errors, it was found that these errors mainly came from the three newly added categories: blue eye mold (class 5), insect damage (class 6), and surface mold (class 7). Class 5 was mainly misclassified in class 3 and class 4. Class 6 was mainly misclassified in class 2 and class 3. Class 7 was mainly misclassified in class 2 and class 6. This meant that samples in class 5 had very similar properties with samples in class 3 and class 4, and samples in class 6 had very similar properties with samples in class 2 and class 3, and samples in class 7 had very similar properties with samples in class 2 and class 6.

According to this case, some aspects should be improved in further research work. For example, classification methods should be improved using deep-learning, and feature selection should be optimized through adding texture features. In addition, the image collection system can be optimized, where the lighting system can become more uniform to ensure preserving true color of the corn and a more accurate feature calculation. Fresh corn can be harvested from the field to perform the official classification test using computer vision technology. Since these were preliminary results, more damage samples could be studied, including sprout-damaged samples. The repeatability of the system can be tested. For example, two damaged sample groups can be selected with 140 normal corns and 20 damaged corns in each group. This can be repeated to capture 10 images of each group to evaluate the repeatability of the system.

5. Conclusions

A corn classification system based on computer vision was developed to provide an alternative solution to the traditional classification test. Images were acquired using an image collection system, which consisted of an imaging table, two-way lighting system, a camera, and a computer. A line profile segmentation algorithm was adopted to segment groups of touching corns to individual corn images. Twelve color features and five shape features were extracted from the segmented individual corn images. A maximum likelihood estimator was used to classify corns to normal, cob-rot, germ-damaged, heat-damaged, and other damage types. To validate the corn classification system, four and seven groups of corns were tested including normal samples and different types of damaged samples. This classification method achieved better performances in four classifications for corn images. The results showed that the corn classification system provided accurate classification of the corns into four classifications. In general, a corn classification system based on computer vision can be used to offer considerable advantages for the detection of small touching objects.