Research on UAV Remote Sensing Method of Mold Detection Suitable for Pericarp of Citri Reticulatae ‘Chachi’ Warehouses

Abstract

Once the Pericarp of Citri Reticulata ‘Chachi’ (PCRC) develops mildew while in storage, the rapid spread of the flora after the occurrence of mold can cause huge losses. As such, inspecting whether the PCRC is moldy is important. In this paper, we propose an alternative inspection method, namely that of utilizing a small UAV with a camera to inspect the PCRC mildew in the top stacks under consideration. Specifically, we first address the light problem in the collected images with different lights via a multi-spectral method, and find that 625–740 nm of lighting has a significant effect on mildew. Second, we utilize the ultrared 1.4R-G method to extract the features of mildew with Otus binarization. We can see that the mold-free area is less than 95% in an image categorized as having mildew. The proposed mildew inspection method achieved 93.3% accuracy. Our method could send the inspection information to a control system, achieving rapid closed-loop automatic control and reducing the mildew-related loss.

1. Introduction

The Pericarp of Citri Reticulatae ‘Chachi’ (PCRC), as a type of homology in medicine and food, is prepared using the aging, dry and mature cultivated varieties of the pericarp of Citri reticulata Blanco [1]. It contains hesperidin, which helps prevent liver diseases, and has anti-cancer, anti-oxidant and anti-inflammatory properties [2,3,4,5]. Flavonoids, which have anti-tumor effects, and volatile oils, which can resist respiratory system as well as other diseases [6], will change in complex ways as the storage years increase [7,8,9]. Some studies have confirmed that the storage characteristics of PCRC should be long-term aging with an annual increase in the storage volume. Once PCRC is impregnated by mold, its active ingredients become significantly reduced and the volatile components will significantly change [10], which creates a higher demand for the controllability of mildew risk in the warehouse. Currently, because the price of PCRC is positively correlated with the storage time, its manufacturers usually have large-scale warehouses. PCRC stacks plastic frames upwards and downwards in a large warehouse, with the top of the plastic frame open and the four sides of the grid sealed (Figure 1). The general stacking height is high to save warehouse space. Conventional manual inspections can only be observed from the side because the side walls of the plastic frame are in the line of sight, making it difficult to effectively retrieve mold information. While the multi-layer stacking structure of the frames is loose, manual inspection operations such as climbing to the top of the observation present increased safety risks with only relatively low efficiency. Traditional inspection methods have limitations and high costs, while the automation devices of the warehouse environment cannot form closed-loop control with people [11]. Therefore, in this study, we propose an observation method that employs an auto-guided unmanned aerial vehicle (UAV) equipped with a camera to inspect the mildew from the top of the plastic frames to perform the noncontact mold inspection of PCRC in the different locations of large warehouses. When the internal air is airless in a large warehouse with a long-term static state, the temperature and humidity at the top of the warehouse space are higher than in the other locations, so the PCRC there is more susceptible to mildew. It is highly practical and feasible to use the flight capability of UAVs to inspect mildew from above the plastic frames.

UAV inspection is an advanced intelligent inspection method. Compared to the traditional manual inspection method, using remote sensing technology with industrial cameras and customized light sources by UAV has the advantages of simple operation, portability, and high efficiency [12,13,14]. Using UAV remote sensing platforms in agricultural monitoring has become a trend in agricultural research [15,16,17]. For example, Walmart has established a research and development team to design commercial drones for inventory inspections and cargo transportation, as well as to monitor out-of-stock conditions and logistics delivery [18]. The Age Steel Company in Dubai, UAE, achieved product tracking in a yard using drones by combining a quadrotor drone platform with radio frequency identification and machine vision, avoiding high-temperature manual work and improving the efficiency [19]. Xiao Bin proposed a UAV with a vision for cargo defect inspection systems, which solves the problem of missing cargo and surface defect identification through the flexible control of UAVs with image-processing methods [20]. Wang Jie proposed that UAVs have the advantage of dexterity and convenience, allowing them to fly freely in warehouses to collect cargo and bookshelf aisle images. Autonomous drone navigation cargo image matching can be achieved using image-processing algorithms, resulting in a complete cargo inventory [21]. The technical solution proposed in this study can be realized using current UAV technology.

Noncontact quality inspection research results and applications in non-destructive rapid inspection, such as for seeds, fruits, and other agricultural products, are gradually increasing [22,23,24]. The current machine vision method, which is based on morphology, color, texture, and other distinguishing image information extraction and recognition methods, provides a theoretical basis for this study. Taking photos of agricultural products with a camera, custom-segmenting them in a computer, and then identifying the region of interest using binary images constitute a reliable method with simple operation and high result accuracy. For example, Bu Xiangyu used color moments and a color co-occurrence matrix to obtain the region of interest features from rapeseed leaves infected with disease spots. This improves the recognition accuracy of rapeseed diseases when combined with the Dempster Shafer evidence theory to fuse feature decisions [25]. Blasco et al. successfully segmented the citrus peel disease spot region using three steps, namely seed selection, employing an iterative method of area growth, and then splitting and merging [26]. Diao Zhihua et al. proposed a recognition method based on improved excess red features, which extracts disease-like areas and binarizes them before using an area thresholding method to segment out pest mite spots in disease-like spots [27]. Zhang Nannan et al. proposed using the HSV model to quickly identify moldy corn particles and mold grades, thereby improving the images of colored corn [28]. As such, the threshold of the V component is used to distinguish normal from moldy, and the H and V component thresholds are used to distinguish mild mold from severe mold.

In summary, among the previous studies of rapid mildew detection, whilst some were based on traditional biochemical experiments but with long cycles and poor real-time performances, others focused on rapid target detection by UAV, which is susceptible to external influence during the collection process, and thus its adaptive light source needs to be strengthened. Although some related algorithms have been used and are effective in the rapid detection of agricultural products, few applications have been seen for the identification of the moldy characteristics of PCRC. Therefore, current research into the detection of moldy PCRC is relatively rare, and more exploration is needed.

In this study, we propose a UAV method using PCRC as the research object, with a camera to inspect the mildew in the warehouse, which can quickly identify the occurrence of PCRC mold. UAV has the advantages of high efficiency, high accuracy, and good safety. The ability to monitor the PCRC mold in real-time, which lays an important foundation for the subsequent realization of unmanned mold monitoring, is particularly useful for good practical application value in the large warehouses of PCRC.

2. Materials and Methods

2.1. Storage Environment and UAV Inspection Methods of PCRC

In this study, we propose the use of UAV as a mobile platform that observes downwards from above at the top of the plastic frame, whilst the camera obtains real-time pictures of PCRC at multiple locations in the warehouse; then, the computer processes the obtained image information to determine whether the PCRC is moldy. If the mold has been detected, the computer will send the signal to start the automatic equipment such as an air conditioner, humidity adjustment system, and air exchange system, which will operate to stop the mold in a timely manner and decrease the growth rate of the mold. Meanwhile, warning information will be sent to the warehouse manager, prompting the manager to make timely remedial measures. The rapid, closed-loop, and automatic control of PCRC warehouse monitoring is more timely, accurate, and effective. The work plan is shown in Figure 2. The important meaning of this closed-loop system is that UAV with a camera can replace traditional manual detection by sampling and judging the mold.

The distance between the top of the plastic frame stack and the roof of the warehouse is 150 cm, as shown in Figure 3a, which is suitable for small UAV flights. The light source is insufficient and the overall environment of the warehouse is dark, which is also influenced by the day-to-day variation in external natural light, as shown in Figure 3b. Therefore, in this study, we added an onboard light source system, which can help obtain a clearer image with stable quality, as well as increase the recognition rate of the moldy characteristics of PCRC, by considering the influence of the light source when the UAV with a camera acquires an image.

To develop a machine vision method to identify the mildew of PCRC in this study, we planned to manually induce the mildew of a sample of PCRC and designed an experimental device to then automatically and continuously acquire images of the mildew of this sample, to ultimately verify the effectiveness of the machine vision method in identifying the mildew.

2.2. Experimental Design of Spectral Characteristic Analysis of PCRC

The spectral reflection curve will change dynamically with the change in the properties of PCRC, so the spectral reflection curve of PCRC can be used to show the degree of mold. In this study, we selected 17 pieces of PCRC with “DaHong” maturity in 2020 from Xinhui District, Jiangmen, Guangdong Province. Because PCRC has the characteristic of three petals and, based on the area and shape of the spectral collection instrument window, each piece of PCRC was again divided into three independent collection areas, yielding 51 sets of samples. The PCRC samples are shown in Figure 4, with the red circular area corresponding to the window of the spectral acquisition instrument. The PCRC samples in the red circular area were classified as non-moldy in area I, lightly moldy in area II, and moderately moldy in area III, based on their actual degree of mold.

As shown in Figure 5, the ASD Fieldspec3 photogrammetric spectrometer (in the spectral range of 350–2500 nm) was used in this experiment to measure the spectral data of PCRC with different levels of moldiness. This instrument can measure and observe the reflectance spectrum curve in real time, as it has the benefits of a high signal-to-noise ratio, high reliability, and high repeatability. Each group of samples was collected three times to ensure stability, perform the data analysis of raw spectra, and screen out the sensitive feature bands with regard to the degree of mildew.

2.3. Analysis Method of Image Acquisition and Identification of Mildew in PCRC Samples

Mildew is a common natural phenomenon, but it grows slowly and requires a certain level of humidity and temperature. After several experimental tests, in the first stage, the mold from the inner side of PCRC appeared as white fluff which require a microscope to identify; as it was not serious, we did not conduct much research at this point. In the second stage, it took the shape of gray-green irregular spots, a stage which we took seriously. Finally, it evolved into black patches, at which time the mold was very serious and the loss would be relatively large. Therefore, this paper focused on the second stage, The moldy spots in the PCRC image are green, while the non-moldy PCRC is non-green. Thus, their color characteristics can be used to identify them. Using UAV to carry vision systems to achieve large-scale and rapid detection in large warehouses is a complex and workload-intensive systematic task. This paper focuses on extracting mold features from the obtained images of PCRC. Therefore, in order to improve the controllability of PCRC mildew samples and limit the interference of other environmental variables, a special environmental test box for PCRC mildew, which is a closed experimental box with a black interior and a controlled temperature of 25 °C, as well as an interior humidity of approximately 85%, was designed in this study. The camera and computer were connected through a USB 2.0 interface to obtain pictures (Figure 6).

In this paper, we propose an experimental pipeline (Figure 7) to complete the differentiation of moldy PCRC and non-moldy PCRC based on the excess red feature, binarization processing, and area threshold method.

3. Results and Discussion

3.1. Characteristic Waveband Analysis and Enhanced Light Source Design for Mildew Spectra of PCRC

Three sets of data were collected and averaged from each PCRC sample collection area. The average reflectance spectral data curves of 153 sets of non-moldy, lightly moldy, and moderately moldy PCRC areas were collected using the existing hyperspectral detection system, as shown in Figure 8a. The specific bands with significant differences were further analyzed by calculating the variance, as Figure 8b shows the variance of the three kinds of spectra.

The maximum variance of 0.011 occurs at 780 nm in the near-infrared band, and the overall red light band (625–740 nm) region is also significant, with variance values greater than 0.007. Given the subsequent need to perform the aging process or several other functions of the machine vision research, these planning studies are mainly in the visible band, so this test still employs the visible light camera. The light source is a combination of a 650 nm red LED light source and an existing light source, which can enhance the visibility of the effect of mold in the images.

According to the results of the pre-test spectral reflectance curve, red light can be added for image acquisition to increase the contrast between the moldy and the non-moldy areas. To achieve a better red light ratio, the same type of white LED and red LED is used to design the enhancement light source, using different electrical power excitations for the red and white light ratios. The white LED and red LED are both ring distributions with the same structure, and both contain 56 beads in the same arrangement; therefore, the red LED light source has a wavelength of 650 nm. The experimental investigation was designed for the ratio selection of the enhancement light source, then the image acquisition was performed on a black carbon fiber plate in a black box, and a JHSM500m vertical 5 million color camera with a focal length of 6 mm, a shooting height of 23 cm, and a resolution of 2592 × 1944 was used. The same PCRC sample was collected and the same moldy area and non-moldy areas were dug (Figure 9f) (whilst the moldy area was measured at the upper left coordinates X = 1338, Y = 632 starting point, the non-moldy area was measured at the upper left coordinate X = 1324, Y = 1214, and both dug a rectangular area of 85 × 80 pixels).

The similarity of the color histograms of the moldy and non-moldy areas in the images of PCRC with different light source ratios was compared. Since the larger the obtained correlation value is, the higher the similarity of the color histograms is, and the smaller the image differences are, this causes a smaller correlation. It can distinguish the moldy and non-moldy areas. Based on the data a, b, and c from

Table 1. Analysis of the difference between non-moldy and moldy areas under multiple light sources.

No.	White Light Electric Power/mW	Red Light Electric Power/mW	Correlation
a	360	0	0.00319
b	360	195	0.00288
c	360	360	0.00244
d	0	360	0.00036
e	195	360	0.00129

, when the white light electric power is constant, a higher ratio of red light electric power means a smaller correlation. The correlation of the d of the pure red light is the smallest, and thus, adding a red LED light source with a wavelength of 650 nm to the collected images can improve the difference between the moldy and non-moldy areas, thereby improving the recognition rate. Therefore, including a red light source in the subsequent UAV camera module set is critical.

Natural light environments differ due to the characteristics of different warehouses and the different times of day in the same warehouses as well as the height of the UAV and its actual distance from the PCRC. Choosing an optimal light source power is a complex problem that must be based on the actual conditions of the application site. No in-depth analysis was performed in this study and the subsequent tests were based on the laboratory simulation of a controlled environment.

3.2. Analysis of Moldy Identification Test on PCRC Samples

Currently, the more commonly used techniques for separating color image processing, which involve the different combinations of color feature factors, such as 2G-R-B, 2R-G-B, and so on, have been effective in other detection tasks [29,30,31]. However, for PCRC mildew, we used the color property to further divide the original color image into two parts for processing; the part showing green color being classified as the moldy region, and the remaining non-green region being classified as the non-moldy region. The excess red features in Equation (1) were selected for the preprocessing segmentation of the image [32].

$$ExR=1.4\times R-G$$

(1)

where R, G, and B represent the red, green, and blue components, respectively.

The main steps of the proposed method for processing the moldy PCRC images in this study are as follows:

(1) Acquiring the original images; (2) Denoising the images and performing contrast adjustment based upon this; (3) Graying out the acquired color images based on the excess red (1.4R-G) feature of the images, and obtaining the binary images by threshold segmentation using the Otsu method [33]; and (4) Superimposing two images. Following the above process, the moldy region was effectively extracted with a significant effect.

Based on Figure 10 and Figure 11, the black pixels excluding the external background are the moldy areas, and the white background is the non-moldy area. The experimental results show that the use of excess red features with the Otsu method can better segment the moldy area, and perform the necessary preparation for subsequent recognition. The misjudgment of the non-moldy PCRC was caused by the uneven shape and uneven light, so we added a special light source and obtained multiple photos from different angles to reduce this error, which will also reduce misclassification. Because PCRC itself also has some autologous scars, the miscalculations cannot be completely eliminated, but can be minimizes as described previously. Therefore, a fault-tolerant threshold value needs to be given.

After removing the moldy areas, the pixel points were iterated to calculate the pixel, and the area of interest was extracted from the entire PCRC to traverse its internal global pixel points. Then, the percentage of the non-moldy area was calculated. Given that the area of the non-moldy region (white pixel points after segmentation) was n, the entire region (internal global white pixel points) is m, and the percentage is r. The calculation equation can be obtained as follows:

$$r=\frac{n}{m}\times 100\%$$

(2)

Given the threshold value to determine whether the sample is moldy, because there is no standard instrumentation in the PCRC industry to quickly discern the degree of moldiness, it is basically executed according to expert experience. In this paper, we started from the color features, whilst the white fluffy features were difficult to be extracted, and the presence of a small amount of similarity in the color autogenous scars of PCRC was misidentified. Based on the data from

Table 2. Identification ratio.

PCRC Quality	Non-Moldy Area n	All Areas m	Ratio r (%)
Non-moldy	1,619,428	1,627,746	99.49%
Non-moldy	1,297,719	1,313,758	98.78%
Non-moldy	1,381,373	1,409,750	97.99%
Moldy	1,600,767	1,693,612	94.52%

, industry experts calibrated the algorithm, and the threshold value was finally selected to be 95%, resulting in the false extraction rate falling within 5%. The 5% includes misclassified pixels or real moldy spots, but the overall proportion is small, so it can meet the requirements of enterprises.

To verify the generality of the algorithm in this study for identifying moldy and non-moldy PCRC regions, fifteen different quality states of PCRC from non-moldy to moldy were collected for the discrimination test of the above model, and the statistical recognition rates are shown in

Table 3. Experimental results after recognition algorithm processing.

No.	Ratio (%)	Identification Results	Actual Results
1	94.74%	Moldy	Moldy
2	95.11%	Non-moldy	Non-moldy
3	95.04%	Non-moldy	Moldy
4	85.03%	Moldy	Moldy
5	90.76%	Moldy	Moldy
6	96.61%	Non-moldy	Non-moldy
7	90.12%	Moldy	Moldy
8	89.87%	Moldy	Moldy
9	96.92%	Non-moldy	Non-moldy
10	90.66%	Moldy	Moldy
11	68.02%	Moldy	Moldy
12	93.55%	Moldy	Moldy
13	91.10%	Moldy	Moldy
14	90.68%	Moldy	Moldy
15	84.09%	Moldy	Moldy

.

Based on the statistics, 14 of the 15 PCRC samples had a 93.3% identification rate, which complies with the discrimination requirements. However, from the statistics, we can see that samples 1, 2, and 3 are all around this threshold, so we plotted samples 1, 2, and 3 to further analyze their differences and perform the whole pipeline works on them (Figure 12). Based on Figure 12, sample 1 and sample 3 have a small amount of mold at the edges, and whilst sample 2 had more autogenous scars, finally resulting in the ratios being located at the critical threshold and sample 3 being misclassified, the identification of the wrong place for the serial number 3, with a mold identification rate, was 95.04%, which was located at the critical threshold. This method has a low recognition rate for the white fuzz in the least serious mold, although there is some uncertainty, and it also demonstrates the effectiveness of the threshold selection; thus, in the actual classification of such PCRC, double verification by an expert is needed.

4. Conclusions and Outlook

Large PCRC warehouses are mainly concentrated in the Jiangmen area of China, for which there has been a lack of systematic research to date into rapid mold detection. The use of UAVs for the detection of mold in PCRC storage is a future development direction of intelligent warehouses. For large PCRC warehouses, different degrees of mold will bring corresponding economic losses. PCRC is so expensive that these economic losses are worth paying attention to. The location of mold is a random event and we cannot predict it. Regularly and rapidly detecting mold across the whole warehouse and reducing the corresponding economic loss due to mold is a meaningful task. Meanwhile, it is also a strong technological necessity for PCRC companies. This paper proposed a rapid detection method using UAV instead of manual detection, which can detect mold rapidly in the whole large PCRC warehouse. This study innovatively addressed this challenge, and the following points were performed in this study, which obtained useful results:

(1) To make the variables (temperature, humidity, light, etc.) controllable, we used a closed black box to build a real-time image acquisition system with controllable variables for PCRC mildew, and allowed for the design and optimization of subsequent moldy region extraction algorithms. However, there are differences within the actual storage environment, for example, the stacked images of the stored PCRC present a challenge to image segmentation and threshold selection, while the motion path planning and vibration of the UAV affect the quality of the captured images, but the recognition algorithm used in this study still has room for adjustment and optimization, and the experiment of combining the results of this study with the UAV will be the focus of subsequent research.

(2) To reduce the influence of light sources on image acquisition, we designed an enhanced light source based on the multispectral test. The statistical test data from cultivating samples with different degrees of mildew PCRC for the multispectral test showed that the difference in the band was most significant at 780 nm, whereas the overall reflectance difference in the band from 625 to 740 nm regions was more significant. The test confirmed that the enhancement of the light source enhanced the moldy recognition effect, as the addition of red light was collected after the difference between the moldy and the non-moldy areas significantly increased. However, the change in natural light during storage is uncertain, and its interaction with the ratio of the enhanced light source should be further investigated in future studies.

(3) By analyzing the color component rationing relationship between moldy and non-moldy regions, the improved excess red feature 1.4R g could effectively extract the moldy regions, the ratio of the non-moldy region to the total pixel area of PCRC was defined as the moldy judgment threshold, and 95% of the ratio was selected in our experiments with the expert opinions. The validation results showed that 14 of 15 samples were correct, with moldy samples accounting for 95.04%. Misclassification, due to the proximity of the defined value, resulted in a test accuracy rate of 93.3%. This method can meet the requirements of the rapid detection of moldy PCRC, and it provides a theoretical basis for subsequent UAVs equipped with cameras to participate in the inspection and achieve rapid automatic identification.