Research on a Multi-Lens Multispectral Camera for Identifying Haploid Maize Seeds

Abstract

Haploid breeding can shorten the breeding period of new maize varieties and is an important means to increase maize yield. In the breeding program, a large number of haploid seeds need to be screened, and this step is mainly achieved manually, which hinders the industrialization of haploid maize breeding. This article aims to develop a multispectral camera to identify the haploid seeds automatically. The camera was manufactured by replacing narrow-band filters of the ordinary CCD camera, and the RGB, 405 nm, 980 nm and 1050 nm images of haploid or diploid seeds were simultaneously captured (the characteristic wavelengths were determined according to color and high-oil markers of maize). The performance was tested using four maize varieties with the two genetic markers. The results show that the developed multispectral camera significantly improved the recognition accuracy of haploid maize seeds to 92.33%, 97.33%, 97% and 93.33% for the TYD1903, TYD1904, TYD1907 and TYD1908 varieties, respectively. The cameras in the near-infrared region (wavelengths of 980 nm and 1050 nm) achieved better performance for the varieties of high-oil marker, with an increase of 0.84% and 1.5%, respectively. These results demonstrate the strong potential of the multispectral imaging technology in the haploid seed identification of maize.

1. Introduction

Maize (Zea mays L., also commonly known as corn) is one of the important food crops in the world, and its output has a significant influence on global food security [1,2,3]. The world population is gradually growing, while arable land is limited, resulting in an increasing demand for maize yield [4]. The cultivation of new varieties of maize is the key means to boost per-unit yield. There are many breeding methods to obtain new maize varieties [5]. Compared with other breeding methods, the haploid breeding technique has the following advantages: (1) shortening the breeding period; (2) overcoming the incompatibility of distant hybridization; (3) reducing land and labor use; (4) improving breeding efficiency, etc. [6,7]. However, the proportion of haploid seeds is low in haploid breeding, and only about 10% of the required haploid seeds can be gained in a single breeding cycle [8]. It is difficult to identify haploid seeds in mixed maize seeds (haploid and diploid), which restricts the engineering of haploid breeding of maize. Therefore, it is of great significance to develop automatic detection technology of haploid seeds.

Morphological identification (MI), cyto-anotomy identification (CI) and gene marking (GM) are the three main technologies to identify haploid maize seeds [9]. MI refers to judging whether maize is haploid or not by observing the morphological structure of maize at the seedling stage. Although this method is intuitive and effective, the detection cycle is long and the cost is high. CI is complicated and extremely destructive, and haploid seeds are identified by dissecting all seeds. GM is the most widely used and effective haploid identification method at present [10]. It marks the haploid and diploid seeds with different genotypes and phenotypes, and a large number of haploid corn seeds can be selected manually. The advantages of this method are that haploid seeds can be identified at the seed stage, and it is easier to achieve automatic haploid screening. Color marker and high-oil marker are two commonly used gene markers. Color marker uses the R1-nj gene to induce pigment differences in maize seeds, so that diploid embryos have purple marks, while haploid embryos do not [11]. The color difference in seed embryos enables human beings to intuitively identify haploid and diploid corn seeds. However, the expression of R1-nj varies greatly among different varieties of maize, resulting in a high recognition error rate by the human eye when the color difference of the embryo is faint. Haploid maize seeds can also be identified by high-oil marker, and the oil content of induced diploid embryos is significantly higher than that of haploid embryos [12]. However, some varieties exhibit overlapping oil content in haploid and diploid seeds, making it difficult to distinguish. The simultaneous use of color marker and high-oil marker is an effective means to improve the recognition accuracy of haploid seeds [13]. In practice, the haploid corn seeds marked by these two methods are mainly identified by artificial means, which takes a lot of manpower and time, and also cannot meet the needs of the breeding industry. Therefore, there is an urgent need to develop a rapid and non-destructive haploid seed identification and screening method.

Some detection techniques based on gene markers have been used for the automatic identification of haploid maize seeds, including RGB image, nuclear magnetic resonance (NMR), near-infrared spectroscopy (NIR), hyper-spectral imaging (HSI) and multispectral imaging (MSI). RGB image uses the RGB camera to capture the color marker of the seed embryo, and this method has not been satisfactory for haploid seeds with a small color difference [14,15,16,17,18,19]. NMR separates haploids based on oil marker, but NMR is cumbersome in operation, slow in sorting and costly in manufacturing [8,20,21,22,23]. NIR is widely used for the quantitative detection of liquids and powders, such as the contents of protein, oil, sugar and pigment, but the accuracy is easily affected by the size and shape of seeds when identifying haploid maize seeds [24,25,26,27,28,29]. Compared with NIR, HSI technology can obtain the external characteristics of seeds, such as color, size and texture, as well as the internal chemical composition distribution of the seeds with spectral and image information, which counteracts the effects of the size and shape of the seeds on the spectral information [30,31,32,33,34]. However, the data acquisition of HIS is slow and its cost is high. MSI is similar to HSI, and the key difference is that MSI only obtains spectral images under characteristic wavelengths [35,36,37,38]. The acquisition is fast, and the cost is greatly reduced. However, MSI needs to be customized according to the application scenario. At present, there is no report on MSI technology applied to the detection of haploid seeds of maize.

The main objectives of this study are as follows: (1) To design a four-channel multi-lens MSI camera, which can collect the information of color marking and high-oil marking of maize seeds simultaneously; (2) To research the preprocessing algorithm of the MSI image of haploid seeds, to establish the classification model of haploid seeds based on convolutional deep neural network, and to realize the identification of haploid seeds with color marking and high-oil marking; (3) To establish a detection system of haploid seeds of maize and to provide technical support for the following automatic sorting of haploid seeds.

2. Materials and Methods

2.1. The Design of Multispectral Camera for Haploid Maize Identification

The developed MSI camera is shown in Figure 1a, including an RGB camera and cameras with wavelengths of 405 nm, 980 nm and 1050 nm. The RGB camera can capture the shape and color of seeds. The 405 nm camera is in the range of 380 nm~470 nm of the violet light, which can strengthen the image of purple marks on the embryo. The 980 nm and 1050 nm cameras are in the band range of near-infrared, and their wavelengths are the strong absorption peaks of the C-H functional groups characterizing the oil contents, so as to achieve the identification of haploid seeds with high-oil marker. In this way, 4 MSI images of haploid seeds can be captured simultaneously. The manufacturing scheme of the MSI camera is shown in Figure 1b. The optical filter of the ordinary CCD camera (manufactured by Xingkai Security Technology Company (Shenzhen, China), camera diameter 15 mm, resolution 1280 × 720, signal-to-noise ratio ≥48 dB, frames 30 f/s, 5 V powered by the image processor) was removed and replaced with a narrow-band filter in front of the camera. It was successively replaced by the filter with a wavelength at 405 nm, 980 nm and 1050 nm (manufactured by Fuzhe Laser Technology Company (Shenzhen, China), with a diameter of 14–15 mm and a thickness of 1–1.2 mm). When the camera operates, the spectral information of the seed first passes through the lens, then reaches the filter. The filter blocks all spectra except for the characteristic wavelengths, and finally, the CCD sensor perceives the seed image information at the characteristic wavelengths. Consequently, the four images at RGB, 405 nm, 980 nm and 1050 nm are obtained and transmitted to the image processor via USB.

The designed fixing device of camera is shown in Figure 1c, including a camera base and a filter base. The filters were inserted and fixed through the slots on the side of the filter base. The diameter of the circular hole for fixing filter is 12 nm, which is good to install and replace the filter. The detection system of haploid seeds is shown in Figure 1d, which includes seed placement platform, light source, MSI camera, image processor (Raspberry PI 4b, Raspberry PI, Cambridge, UK) and displayer. The light source uses two symmetrically placed tungsten lamps (12 V, 20 W). When working, the tungsten lamp shines the light on maize seed; then, the light is reflected to the MSI camera. The MSI camera transmits the multispectral images to the image processor; then, the processor discriminates haploid and diploid seeds and sends the identification results on the display. The image processor uses the Rasberry Pi 4b, which has four USB ports to connect four cameras and facilitates the sequential capture of images via USB connections.

2.2. Samples and Sample Preparation

Maize samples were divided into two categories: color-marker section and high-oil-marker section. All samples were provided by a breeding company (China National Tree Seed Group Co., Ltd., Beijing, China). In the first section, two varieties (TYD1903 and TYD1904) were obtained using the traditional color inducer CAU5 as the paternal line. The embryo and endosperm of the diploid seeds were both labeled purple, while only the endosperm of the haploid seeds was marked purple. In the second section, the high-oil inducer material CHOI3 was used as the paternal line to produce two other varieties (TYD1907 and TYD1908). These varieties generated haploid and diploid seeds with both oil content and color markers. Specifically, the diploid seeds not only had a purple embryo marker but also a higher oil content compared to the haploid seeds. The total sample size was 400 seeds, with 100 seeds in each category.

2.3. Chemical Composition Determination

The pH-differential method was used for the determination of the anthocyanin content (AC, mg·L^–1). Three seeds from each haploid and diploid group of every variety were randomly chosen from the four maize varieties for AC determination. Cyanidin-3-O-glucoside (CAS No.: 7084-24-4) was employed as the standard sample, with a molar absorptivity (ε) of 26,900 M⁻¹cm⁻¹ and a molar mass (MW) of 432 Da. Then, the AC value was determined using the formula AC=a·b/(ε·l) × MW × 10³, where a represents the difference in absorptivity between two pH solutions, b denotes the dilution rate and l signifies the path length of the colorimetric vessel (cm). During testing, the values of b and l were established as 24 and 1 cm, respectively. The oil content rate (OCR, %) of haploid and diploid seeds was determined by the NMR-based oil testing instrument NM120-015V-I (Shanghai Newman Electronic Technology Co., Ltd., Shanghai, China).

2.4. Multispectral Image Collection

The detection system of haploid seeds (Figure 2) was established, and the MSI camera was pointed at the seed embryo, and its height was adjusted to ensure that the four cameras could collect seed images at the same time. The distance between the MSI camera and the acquisition platform was determined to be 150 mm. The focus of each camera was adjusted to obtain clear images. The black platform was use to acquire MSI images with clean background. The fixed rod of tungsten lamp on left and right side was at 60 degrees with the vertical rod of the holder, and the distance between the halogen tungsten lamp and the vertical rod was 280 mm. The MSI images of maize seeds were collected one by one, and 4 images of each were obtained at the same time. The images were stored by category for the subsequent establishment of identification model of haploid seeds.

2.5. Image Preprocessing

Pytorch was applied to preprocess the obtained images of the maize seeds (100 images each variety). The process includes the following: (1) Transforms.Resize was used to scale the image sizes in proportion to AlexNet’s input size of 224 × 224. (2) Transforms.ColorJitter was used to adjust the brightness, contrast, saturation and tone of the images with value of 0.1, thus obtaining an expanded 400 image sets for each variety. Considering that the position of seeds might affect haploid recognition, the rotation (90 degrees in one step) and translation (50 up, down, left and right) of image were conducted to obtain an expanded 800 image sets with orientation changed. Thus, additional 1200 images were obtained. (3) Transforms.ToTensor and Transforms.Normalize were used to normalize images. This process can reduce the occurrence of gradient explosion and gradient disappearance and accelerate the convergence of the model. KS method was used to divide the data set. For 1300 sets of each category, 700 sets were used as the model training and the other 600 sets were used as the model testing.

2.6. Classification Model of Haploid Seeds

AlexNet was applied to establish the classification models of haploid and diploid seeds. It is a classical convolutional neural network, whose network structure consists of 5 convolutional layers and three fully connected layers (i.e., 2 fully connected hidden layers and 1 fully connected output layer) [39]. Each convolution layer contains convolution kernel, bias term, activation function, and local response normalization (LRN) module. The first, second, and fifth convolutional layers are followed by a maximum pooling layer, and the last three layers are fully connected layers. The final output layer, softmax, converts the network output into probability values that are used to predict the category of the image. The nonlinear activation function, modified linear unit (ReLU), is used in the AlexNet. Compared with traditional activation functions (sigmoid and tanh), ReLU can effectively solve the problem of gradient disappearance while maintaining the fast computation speed, thus making training more efficient. LRN was used to suppress the response of neighboring neurons and avoid overfitting, and this can improve the generalization ability of the network.

$$b_{x,y}^i=\frac{a_{x,y}^i}{{(\mathrm{k}+\mathsf{\alpha }{\sum }_{j=\max \left(0,i-\frac{n}{2}\right)}^{\min \left(N-1, i+\frac{n}{2}\right)}{\left(a_{x,y}^j\right)}^2)}^{\beta }}$$

(1)

where $b_{x,y}^i$ is the normalized value of the ith feature map at position (x, y), $a_{x,y}^i$ is the original value of the ith feature map at position (x, y), n is the neighborhood size, N is the number of feature maps, and k, α and β are hyperparameters with values of 2, ${10}^{-4}$ and 0.75, respectively.

3. Results and Discussion

3.1. Analysis of Seed Multispectral Images

The results depicted in Figure 3 illustrate the outcomes of analyzing both haploid and diploid seeds across color-marked and oil-marked varieties. In general, diploid seeds exhibited higher AC values compared to haploid seeds across all four maize varieties. This distinction can be attributed to the utilization of the R1-nj pigment gene during maize seed induction, resulting in the presence of a purple marker in diploid embryos, whereas haploid embryos lack this marker. Notably, TYD1904 demonstrated the highest AC, with the greatest disparity between haploid and diploid seeds. Conversely, TYD1908 exhibited the lowest AC, while TYD1903 displayed the smallest difference between haploid and diploid AC values. Distinct OCRs were noted between haploid and diploid seeds for TYD1907 and TYD1908, whereas TYD1903 and TYD1904 showed significant overlaps. This suggests that distinguishing between haploid and diploid seeds based on oil content alone is challenging for color-marked varieties. Furthermore, the average OCRs of diploid samples for TYD1907 and TYD1908 were noticeably higher than those of haploid samples.

The images of haploid and diploid seeds of four varieties in different wavelengths are shown in Figure 4. For the RGB images of all varieties, it can be observed that the embryo purple of TYD1904 is the deepest, while that of TYD1908 is the lightest, consistent with the measured anthocyanin content results. The anthocyanin content of TYD1903 and TYD1907 is roughly the same, but in the RGB images, the embryo purple of TYD1907 is more pronounced than that of TYD1903. This is because more anthocyanins of TYD1903 are distributed inside the seeds. For the 405 nm images, it can be seen that the embryo purple was enhanced, especially in the varieties of TYD1904 and TYD1907. Compared to the RGB and 405 nm images, the 980 nm and 1050 nm images contain the smaller purple difference between haploid and diploid seeds, but the differences from TYD1907 and TYD1908 are greater than that from TYD1903 and TYD1904. Overall, there are differences between the images of haploid and diploid seeds, which is the basis for identifying the haploid seeds from diploid seeds.

3.2. The Performance of AlexNet-Based Model

The recognition accuracy of haploid seeds of each variety under a single wavelength is shown in

Table 1. Results of the AlexNet-based model for identifying haploid seeds with single image.

Varieties	RGB (%)	405 nm (%)	980 nm (%)	1050 nm (%)	Average (%)
TYD1903	90.33	88.00	78.00	75.67	83.00
TYD1904	96.67	95.83	79.83	77.00	87.33
TYD1907	95.83	95.00	81.17	79.00	87.75
TYD1908	91.50	90.83	79.50	77.33	84.79
Average (%)	93.58	92.42	79.63	77.25	85.72

. The RGB camera had the highest recognition accuracy for haploid seeds with an average of 93.58%, followed by the 405 nm camera with an average of 92.42%. The performance of the 980 nm camera and the 1050 nm camera was comparatively poorer, ranging from 77.25% to 79.63%. Moreover, the performance of the 1050 nm camera was worse compared to the 980 nm camera. The reason for this is that as the wavelength of the filter increases, the intensity of the transmitted wavelength decreases under the same light source, leading to a weaker image signal at the wavelength of 1050 nm. The RGB and 405 nm cameras had significantly higher recognition accuracy for the varieties with an obvious purple marker (TYD1904 and TYD1907), reaching over 95%. The 980 nm and 1050 nm cameras had higher recognition accuracy for the varieties with a high-oil marker (TYD1907 and TYD1908) compared to the varieties with only a color marker (TYD1903 and TYD1904), reaching accuracies of 87.75% and 84.79%, respectively. This is because the 980 nm and 1050 nm cameras are in the near-infrared region and can identify both the anthocyanin difference and the oil difference between haploid and diploid seeds.

The recognition accuracy of the four varieties under different combinations of cameras are shown in Figure 5 and

Table 2. Results of the AlexNet-based model for identifying haploid seeds with three or four cameras.

Varieties	RGB-405 nm-980 nm (%)	RGB-405 nm-1050 nm (%)	RGB-980 nm-1050 nm (%)	405 nm-980 nm-1050 nm (%)	RGB-405 nm-980 nm-1050 nm (%)
TYD1903	91.50	91.17	90.83	89.00	92.33
TYD1904	97.33	97.17	97.00	96.33	97.33
TYD1907	96.50	96.50	96.67	96.33	97.00
TYD1908	92.83	93.17	93.00	92.50	93.33

. It can be observed that the accuracy of the combination of two cameras ranged from 78.67% to 97%, while the accuracy of the combination of three cameras ranged from 89% to 97.33%. The full combination of four cameras yielded the best performance with an accuracy range of 92.33% to 97.33%. It was evident that the recognition accuracy of the combination of multiple cameras was higher than that of a single camera, and with an increase in the number of cameras, the recognition accuracy increased (Figure 6).

The presented analysis indicates that combinations with the RGB camera performed better than those without the RGB camera. For instance, the combination of the RGB camera and 980 nm camera for the TYD1903 variety achieved 90.5%, which was higher than the 88.33% achieved by the combination of the 405 nm camera and the 980 nm camera. The reason for this result is that the RGB camera captured images with three channels (i.e., R, G and B channels), resulting in higher image clarity. Additionally, the color differences among the four varieties are more pronounced than the differences in oil content.

For varieties with a clear color marker, the accuracy of the RGB camera was already high. Adding other cameras could further improve recognition accuracy, but the improvement margin was limited. For example, the recognition accuracy for TYD1904 using the RGB camera was 96.67%, and it increased to 97.33% when other cameras were added, resulting in a marginal increase of 0.66%. However, for varieties with a light color marker, it is essential to combine other cameras with the RGB camera. For TYD1903, the recognition accuracy with the RGB camera alone was 90.33%, which increased to 92.33% when three other cameras were added, resulting in an increase of 2%.

Moreover, the 980 nm and 1050 nm cameras performed better for varieties with a high oil content. For instance, the accuracy of the combination of the RGB, 980 nm and 1050 nm cameras for TYD1907 and TYD1908 increased by 0.84% and 1.5%, respectively, while for TYD1903 and TYD1904, the accuracy increased only by 0.5% and 0.33%, respectively. This indicates that adding cameras in the near-infrared region can improve the recognition accuracy of haploid seeds, especially for the varieties with a high-oil marker.

3.3. Comprehensive Analysis

Through the above analysis, it is evident that the developed MSI camera has significant performance advantages in identifying haploid maize seeds compared to a single camera. Furthermore, it exhibits clear cost advantages in comparison to existing MSI cameras on the market. As shown in

Table 3. The price list of components used in the MSI camera.

Component Name	Number	Unit price (USD)	Total (USD)
CCD camera	4	14.6	139.1
Filter 405 nm	1	16.7
Filter 980 nm	1	13.9
Filter 1050 nm	1	38.9
Fixing device	1	11.2

, the CCD camera costs USD 14.6, while the filters for 405 nm, 980 nm, and 1050 nm cost USD 16.7, USD 13.9 and USD 38.9, respectively. The total cost of the MSI camera is only USD 139.1, making it relatively affordable compared to market-available MSI cameras (which typically cost more than USD 1000).

Compared to the HSI camera and NIR and NMR spectrometers, MSI has significant advantages in real-time seed detection systems (

Table 4. The comparison with different devices for detecting haploid seeds.

Detection Device	Price	Detection Speed	Accuracy	Remarks
MSI camera	Low	High	Medium	Images are associated with the characteristic wavelengths.
HSI camera	High	Slow	High	Slow data acquisition and processing speed.
NIR spectrometer	Medium	High	Medium	Generally used for detection of homogeneous liquids or powders
NMR spectrometer	High	Medium	Medium	The cost is very high, commonly used in medical fields

). However, existing MSI cameras on the market usually have fixed wavelengths and are often mounted on drones for the acquisition of normalized difference vegetation index (e.g., NDVI), making them difficult to directly apply to the identification of haploid maize seeds. The wavelengths of the images captured by the developed multispectral camera were selected based on the characteristics of haploid maize seeds, and the filters can be easily replaced through the slots on the side of the filter base for expanding detection purposes in the future. In addition, the image augmentation technique was utilized in this study to expand the data set and construct an AlexNet-based model. To mitigate the potential for overfitting, we plan to acquire data sets from a broader variety of maize breeds in future studies. This proactive approach will serve to enhance the robustness of the model, particularly in its ability to accurately identify haploid seeds across a range of maize varieties.

4. Conclusions

This study developed a multi-lens MSI camera for the identification of haploid maize seeds, and the performance was tested using four maize varieties with two different genetic markers. The conclusions of the research can be drawn as follows:

(1)

Compared to the conventional RGB camera, the developed MSI camera significantly improved the recognition accuracy of haploid maize seeds. The accuracy of the MSI camera for identifying haploid seeds of four maize varieties (TYD1903, TYD1904, TYD1907 and TYD1908) was 92.33%, 97.33%, 97% and 93.33%, respectively, while the accuracy of the RGB camera was 90.33%, 96.67%, 95.83% and 91.5%, respectively. These results demonstrate the strong potential of MSI technology in haploid maize seed identification.

(2)

The cameras in the NIR region (wavelengths of 980 nm and 1050 nm) showed better identification performance for the varieties with a high-oil marker. The accuracy of the combination of the RGB, 980 nm and 1050 nm cameras for TYD1907 and TYD1908 increased by 0.84% and 1.5%, respectively, while for TYD1903 and TYD1904, the accuracy increased only by 0.5% and 0.33%, respectively. Therefore, adding cameras in the NIR region can improve the recognition accuracy of haploid seeds, especially for the varieties with a high-oil marker.

(3)

The MSI camera was manufactured based on the ordinary CCD camera, and the overall cost was much lower than for existing MSI cameras on the market. Furthermore, the developed MSI camera is specifically tailored for the identification of haploid maize seeds, making it easier for this MSI camera to be widely applied in haploid maize breeding.