Abstract
Monitoring food processing is mandatory for controlling and ensuring product quality. Most of the used techniques are destructive, arduous, and time-consuming. Non-destructive analyses are convenient for rapid and conservative food quality assessment. Color images of apple slices during the manufacturing of healthy snacks were used for monitoring the drying processing. The implementation of the image-based analysis was straightforward, feasible, and low-cost. The parameters analyzed during imagen acquisition for normalizing were: contrast enhancement, binarization, and morphologic processing, varying the illumination and reference between the positions of the camera and object under analysis. Several apple features related to color, texture, and shape were extracted with computer vision techniques and also analyzed. During image analysis, the entropy was one of the most relevant computed features according to principal component analysis, and it was also relevant in terms of physical interpretation. The average percentage of entropy increase was 19.81% in the green and blue channels, while it was 16.82% in the red channel. Other relevant visual features were the skewness and kurtosis in the RGB channels; and textural information such as contrast, correlation, and variance.
1. Introduction
In rural communities of developing countries, the agricultural activities related to cultivating, harvesting, and handling fruits are usually important for the region’s economy; however, sometimes, these products become waste when they are not consumed or processed. The production of value-added products can represent job opportunities and new income sources for the population by introducing new feasible practices as an alternative to traditional work [1]—one of such new feasible practices is manufacturing products from fresh raw materials. For example, the dehydrated fruits which are a better choice than fast food or unhealthy and sugary snacks. Additionally, the processing required for its production does not require acquiring costly infrastructure.
The monitoring of food processing is valuable for product quality and process standardization-automation [2]. Another issue is that the outcome of food processing depends on intrinsic (raw material composition) and extrinsic (temperature, slice thickness, time) parameters [3]. Therefore, computer vision could play a significant role since vision is the primary sensor in leaving beings and artificial systems during decision-making and process assessment. Besides the potential use of computer vision for food processing, these systems are effective and low-cost. Moreover, the sensor of any computer system is a camera, which is already portable and available on multiple daily devices.
The use of computer vision during monitoring tasks has been successfully applied in different fields (health care, vigilance, and environment). For instance, in the analysis of sequences of medical images to follow the evolution of lesions or [4], and recognition and tracking of smoke within video signals for detection and monitoring of forest fires for environmental protection [5], and machine vision to address the quality assessment of emulsions in the pharmaceutical industry [6]. In addition, computer vision has been exploited in the food industry to track changes in food properties during processing. For example, computer vision (CV) has been previously used as a non-destructive method for monitoring the drying behavior of organic apple cylinders subjected to different antibrowning treatments [7]. Also, drying processing of turmeric slices has been monitored by imaging features by extracting attributes such as color, morphology, texture, browning, and shrinkage [8]. Computer vision has also been used during the salting process of ham to monitor the salt diffusion within lean and fatty tissue areas [9].
Besides human vision, other sensory properties such as the auditory, tactile, olfactory, and gustatory senses have also been artificially implemented to convert, for instance, deterioration reactions into electrical signals processed by a neural network. The design of electronic tongues and noses combines materials’ physical and chemical properties as sensors for monitoring specific characteristic compounds of aroma and taste present in the gaseous, liquid, and solid materials in food processing and control quality. An electronic nose was a successful method to detect and recognize fresh and moldy apples and the different molds used for their inoculation [10]. Quality assessment of delicious royal apple fresh and contaminated using bacterial cultures was determined using an in-line electronic nose system. This study’s principal component analysis and Ward’s analyses showed a correlation for differentiation of fresh, half, and total contaminated apples [11]. Baietto et al. (2015) have reported the use of electronic nose devices (gas sensors) for the identification of fruit volatile organic compounds for identifying the type of fruit, ripeness, and quality [12]. Applying different data-recognition algorithms to signals detected from metal oxide semiconductor (MOS) sensors was a non-invasive and rapid method to identify apple pesticide residues [13]. Effects of four drying methods (air drying, freeze drying, freeze drying plus microwave vacuum, air drying, and explosion puffing drying) on the color, texture, sensory quality, microstructure, bacterial viability, and storage stability of probiotic-enriched apple snacks were assessed. Twenty trained panelists described the texture, flavor, color, taste, and overall acceptability of the samples based on the 10-point hedonic scale [14]. Fruit samples were dried by convective, microwave-vacuum, and a combined method. Apple slices were previously dehydrated with (1) two hours of erythritol, xylitol, and sucrose; (2) thirty minutes of ultrasound. The aim was to characterize the impact of osmotic dehydration, sonication pre-treatment, and drying method on the physicochemical properties of the dried apples [15].
In the current study, apple characteristics during the drying processing were monitored using computer vision techniques as no invasive and fast methods for food control quality. Various existing image techniques were used in this research. How these techniques are combined and integrated depends on the problem to solve. That is why the image-based methodology reported in this work was specifically planned to monitor the drying. The objective of this study was to use simple and easy-to-use devices such as a digital scale and a mobile phone to monitor weight and capture images during the apple slices drying process that could be further analyzed using computer vision techniques as an alternative non-invasive and fast method for food quality control in rural communities.
2. Materials and Methods
2.1. Apple Snacks Preparation
The apples were washed, soaked in ionic silver solution (0.00014%) for disinfection for 10 min, and cut into slices (4–5 m) using an electric slicer (Chefman RJ49, Mahwah, NJ, USA). Afterward, the slices were dehydrated using an air dryer for 14 h at 68 °C. Dried samples were let cool until room temperature, weighed (25 g), and sealed in pouch bags.
2.2. Computer Vision Measurements
During the drying time interval, eight apple slices from different positions (on the dryer trays) were sampled nine-time instances. The change in weight using a digital balance (OHAUS Scout, Parsippany, NJ, USA) and images for further computer vision analysis were taken during drying. The images were captured with an iPhone XR (Cupertino, CA, USA), where the resolution of each image is 3024 4032 pixels and processed in MATLAB 2071a (Mathworks, Natick, MA, USA) (Figure 1). The visual features extracted by computer vision during the dehydration process were color, texture, and shape.
Figure 1.
General overview of the computer vision methodology.
For this, the image is partitioned into multiple regions (Figure 2 image segmentation). A grayscale image is represented as a function on the two-dimensional space , where and specify the row and column, and is the intensity pixel value at that spatial position. A color image is obtained by mixing three-color channels: red , green , and blue , called RGB, and it is the standard method for the generation of color images on screens, such as computer monitors.
Figure 2.
Image segmentation steps for detection of an apple slice at two time-instances: original image (A,F), contrast enhancement (B,G), thresholding (C,H), filtering of the largest labeled components (D,I), slice detection (E,J).
Panels A to E in Figure 2 show the steps for segmentation of one apple slice after 1 h and 48 min of dehydration. Panels F to J (Figure 2) show the segmentation process for the same apple slice after 9 h and 32 min. Contrast enhancement was used to adjust the range of gray intensity so that all the available intensity values (from 0 to 255) could be used, as observed in panels B and G (Figure 2), which show the result of converting a color image into a grayscale image followed by contrast enhancement.
Afterward, an image was segmented by assigning one binary value to each pixel depending on a threshold. If a pixel value is below a threshold, , then the output binary image is assigned 0 at that position; otherwise, it is assigned 1. This segmentation method is called image thresholding, where the threshold was computed by using the image histogram and Otsu’s method. Panels C and H (Figure 2) show the result of image thresholding applied to a grayscale image. The image histogram is a function that specifies the distribution of pixel intensity values according to ; where is the number of pixels with the intensity and is the total number of pixels within the image. Otsu’s method is a statistical algorithm that determines the optimal threshold to classify pixels into foreground or background by examining the image histogram.
After thresholding, a segmented image contained undesired regions such as crumbs and noise. The eight largest regions, corresponding to apple slices, were kept. In contrast, the others were suppressed, as it is shown in panels D and I (Figure 2). Extraction of connected components was used to label each apple slice within the binarized images shown in panels C and H (Figure 2). Finally, each slice was detected, as shown in panels E and J (Figure 2). Features were extracted from each detected slice.
2.3. Feature Extraction
To obtain color features, three histograms were generated. One histogram was obtained from each RGB channel. Five statistical attributes were computed for each channel: the average value , variance , skewness , kurtosis , and entropy . Figure 3 shows the three RGB channels of an apple slice and the corresponding histograms.
Figure 3.
Set of histograms for the three RGB channels of an apple slice.
For texture analysis, the color image was transformed into a grayscale image using the luminosity model, which consists of a weighted average of the RGB channels: . According to the luminosity model, the green channel is the one that contributes the most, which agrees with the fact that green is the dominant color in apples. Textural features convey statistical information about the relative positions of the pixel intensity values within the region of interest. Textural features are obtained from the gray level co-occurrence matrix (GLCM), which specifies the distribution of pairs of pixel intensities according to the distance between these two pixels and the angle of the line segment that joins them. There are four possible angles: 0° (horizontal), 45° (diagonal), 90° (vertical), and 135° (anti-diagonal). The GLCM of a pair of pixel intensities at distance and angle is defined as
where , and is the number of pixel intensities. Seven textural features were computed from the GLCM: (1) the angular second moment , (2) the contrast , (3) the inverse difference moment , (4) the correlation , (5) the co-occurrence matrix variance , (6) the difference average , (7) the entropy [16,17,18,19].
Shape length features were extracted from apple slices. First, the boundary contour of an apple slice was detected to obtain a sequence of straight-line segments. A straight-line segment joins two adjacent boundary pixels, and it is assigned a code number depending on its direction, as in Figure 4, where eight straight line segments are shown along with their code numbers {0, 1, 2, 3, 4, 5, 6, 7}. The sequence of segment codes obtained from the boundary contour is called chain code [20,21]. Next, an eight-entry histogram was generated from the chain code, where each entry is the number of times a code number occurs, within the chain code, divided by the chain length. This histogram contributes eight shape features, also known as direction length descriptors. Shape curvature is another shape descriptor [21], which is the frequency of occurrence of groups of external concave angles (smaller than 180°) and groups of external convex angles (larger than 180°). An external angle was formed between two adjacent straight-line segments, which were part of the boundary contour of an apple slice, scanned in the clockwise direction. Two groups of convex angles and two groups of concave angles are shown in Figure 5.
Figure 4.
Example of the boundary contour of an apple slice, codes and directions of eight straight line segments (codes), shape feature vector.
Figure 5.
The first row contains six concave angles, and the bottom row contains six convex angles. At each row, the first three angles are characterized by the first code being zero. The last three angles are characterized by the first code being one.
The first half of the upper row is a group of three concave angles where the first code is 0. The second half of the upper row is a show of three concave angles where the first code is 1. The first half of the bottom row is a group of three convex angles where the first code is 0 while the second part is a group of three convex angles where the first code is 1. There are eight possibilities for the first code in an angle so that there are eight groups consisting of three concave angles and eight groups consisting of three convex angles. Thus, a sixteen-entry histogram was generated, where each entry is defined the number of times that members, within a group, occur. This histogram contributed with sixteen curvature descriptors.
2.4. Selection of Features
For those cases where the number of features is high, and there is an interest in determining the most relevant features, Principal Component Analysis (PCA) is used. PCA is a linear transformation where a set of extracted features is transformed into a set of new features , known as scores. Scores are defined in a -dimensional space , these scores are not correlated, and their variance determines the relevance of the scores.
The relationship between the original features (; ) and the scores (; ) is given by the set of linear combinations: for . The coefficient is called loading, and it specifies the quantitative contribution of feature to score . The variance/relevance of a score is specified by its corresponding eigenvalue. The higher the eigenvalue , the higher the relevance of the corresponding score . Each entry is defined as the number of times members occur within a group. This histogram contributed sixteen curvature descriptors.
3. Results
Apple Snacks Characterization
Moisture content of raw apples was 83.5%, and after dehydration process the final values were around 16.5%, determined by weight differences before and after drying the samples (Table 1). Figure 6 shows the number assigned to each apple slice.
Table 1.
Measurements of moisture attributes for the first apple slice during the dehydration process at nine instances of time.
Figure 6.
Number assigned to each apple slice. The assignation was based on its position in the oven.
It is observed that the main loss of water occurred after the first four hours, and the water removal decreased sharply after 10 h of drying. Besides weight and moisture, visual features were measured by analyzing color images in MATLAB 2071a. The images were captured with an iPhone XR (Cupertino, CA, USA), where the resolution of each image is 3024 4032 pixels. Initially, the apple slices were visually smooth in color and texture, and rounded in shape. Because of moisture reduction, visual features changed from uniform to irregular. Four visual features were extracted from each apple slice: fifteen color features (five features from each RGB plane), seven texture features, eight shape length features, and sixteen curvature shape features. The 46 visual features were ranked in terms of relevance by using Principal Component Analysis. The twelve most relevant visual features are (1) entropy of the green channel, (2) entropy of the red channel, (3) skewness of the blue channel, (4) skewness of the green channel, (5) skewness of the red channel,(6) kurtosis of the blue channel, (7) entropy of the blue channel, (8) variance of the co-occurrence matrix, (9) contrast, (10) kurtosis of the green channel, (11) angular second moment, (12) correlation of the co-occurrence matrix. The occurrence matrix was used to compute the numerical value of each texture attribute, and it was obtained by considering 1 pixel between two-pixel values under study and an angle of 0° for the straight-line segment that joins them.
The most relevant visual features corresponded to color and texture. Features from these two groups presented advantages for visual interpretation if they were compared with shape length and curvature. Color and texture are invariant to rigid and non-rigid transformations as opposed to shape length and curvature. Rigid transformations include translation, rotation, and scaling. Figure 7 shows the rotation of an apple slice (slice 2 in Figure 6) taking place from = 11.52 h to = 12.95 h and from to = 14.32 h. During the monitoring process, the translation and rotation of apple slices were needed due to the sample’s position in the dryer, the imaging capture was not the same, and the camera was unfixed. Scaling was introduced because of the distance variation between the object and camera when the image was captured. The dehydration process provokes weight loss and non-rigid deformation of each apple slice. The surface area of each apple slice was reduced (shrinkage) due to diminished moisture content. The use of features invariant to rigid and non-rigid transformations is suggested since the proposed methodology should be low-cost, simple, feasible, and effective.
Figure 7.
Dehydration of apple slice two over time.
Entropy was used to measure the image content in terms of visual perception. Table 2 and Figure 8 show the entropy of the green channel for eight apple slices at nine instances of time during the dehydration process. According to PCA, the entropy of the green channel was the most critical feature. Intuitively, entropy measures the random distribution of energy in a system. The temperatures of an object and the warmer surrounding equalize over time as part of the thermal energy from the warm surrounding spreads to the object. The implication that energy tends to be uniformly distributed implies maximization of entropy. As the dehydration proceeded, there was a balance between the likelihood of brownish regions (dehydrated parts) and the likelihood of yellowish regions (higher moisture). As the dehydration proceeded, the entropy trend escalated. According to Figure 7, apple dehydration was characterized by an increase in entropy at the end of the process. Table 3 shows the average variation of the most relevant visual features during 14.32 h of the dehydration of apple slices. The average percentage of variation of the entropy over the dehydration process corresponds to an increase being 19.81% for the green channel, 16.82% for the red channel, and 19.81% for the blue channel.
Table 2.
Entropy of the green channel of eight apple slices at nine instances of time.
Figure 8.
Entropy curves for the green channel of eight apple slices at nine instances of time.
Table 3.
Average percentage of variation for visual features.
Figure 7 and Figure 8 show that the drying curve of slice 1 (curve in blue color in Figure 8) is below the drying curves of other slices. This sample was the third heaviest slice and was in the upper right corner of the air dryer. These curves suggest that the dehydration of the first slice was faster than the other slices. Thus, the drying process was not uniform in the oven for eight apple slices.
Drying curves of other visual features (entropy of the red channel and skewness) are shown in Figure 9 and Figure 10. Skewness measures the asymmetry of the probability distribution of the pixel values about their mean, indicating the magnitude and direction of the deviation from the normal distribution. A negative skew usually means that the distribution tail is on the left side, while a positive skew indicates that the tail is on the right. The skewness in the green and red channels dropped while the skewness in the blue channel raised. It is observed that contrast decreased during the first four hours, then increased significantly during the next four hours, followed by a slightly decreased until the completion of drying, as shown in Figure 10. The local variation (contrast) within the slice content increased as the drying proceeded, representing higher roughness. The entropy of the red and blue channels escalated with time due to the development of brownish regions.
Figure 9.
Drying curves for other visual features.
Figure 10.
Drying curves for additional visual features.
The box plots in Figure 11 show the variation of visual attributes over time for eight apple slices. Each box represents the time variation of a visual attribute within an apple slice. The line in the middle of a box corresponds to the median of the whole range of feature values (quartile 2). The top of the box (quartile 3) represents the median of the range of feature values between quartile 2 and the maximum, and the bottom of the box (quartile 1) represents the median of the range between the minimum and quartile 2. The cross inside a box is the average value.
Figure 11.
Variation of visual features of apple slices during the dehydration process: entropy of the green, red and blue channels; contrast.
4. Conclusions
Apple slices were dehydrated to produce healthy snacks as new labor and economic activity within a rural community. Monitoring of the dehydration process was based not only on measuring weight and moisture but also on measuring visual features such as color, texture, and forms by analyzing images captured with an iPhone. One advantage of using computer vision to analyze the changes in food images visually is that these techniques are non-invasive and feasible to implement without using expensive laboratory equipment. The most important visual features were entropy and skew in the three RGB channels and texture contrast. Entropy was the most critical visual attribute, increasing with the uniformity of the temperature process. This characteristic could be used for monitoring changes in temperature for processing standardization.
Author Contributions
Conceptualization, D.B.-A. and M.R.-R.; methodology, D.B.-A., M.R.-R. and R.R.-R.; formal analysis, D.B.-A., M.R.-R. and R.R.-R.; writing—original draft preparation, D.B.-A., M.R.-R. and R.R.-R.; writing—review and editing, D.B.-A., M.R.-R. and R.R.-R.; funding acquisition, D.B.-A. All authors have read and agreed to the published version of the manuscript.
Funding
The Mexican National Council of Science and Technology (CONACYT) funded this research, under project grant Redes Horizontales del Conocimiento 314456 and European Union’s Horizon 2020 Research and Innovation. Program under grant agreement No. 952594 (ERA Chair project DRIFT-FOOD).
Informed Consent Statement
Not applicable.
Data Availability Statement
Not applicable.
Acknowledgments
We acknowledge Universidad de las Américas Puebla for the infrastructure and equipment provided and the Food Analysis Laboratory, Intema S.A. de C.V. for their contribution in the performed analysis.
Conflicts of Interest
The authors declare no conflict of interest.
References
- Samsatli, S.; Martinez-Hernandez, E.; Ng, K.S. Towards a sustainable bio- economy: Working in harmony with the environment–food–energy–water nexus. Food Bioprod. Process. 2020, 119, 371–372. [Google Scholar] [CrossRef]
- Linko, S.; Linko, P. Developments in Monitoring and Control of Food Processes. Food Bioprod. Process. 1998, 76, 127–137. [Google Scholar] [CrossRef]
- Ochieng, J.; Kirimi, L.; Mathenge, M. Effects of climate variability and change on agricultural production: The case of small-scale farmers in Kenya. NJAS-Wagening. J. Life Sci. 2016, 77, 71–78. [Google Scholar] [CrossRef]
- Rosas-Romero, R.; Tagare, H.D. Segmentation of endocardium in ultrasound images based on sparse representation over learned redundant dictionaries. Eng. Appl. Artif. Intell. 2014, 29, 201–210. [Google Scholar] [CrossRef]
- Rosas-Romero, R. Remote detection of forest fires from video signals with classifiers based on K-SVD learned dictionaries. Eng. Appl. Artif. Intell. 2014, 33, 1–11. [Google Scholar] [CrossRef]
- Unnikrishnan, S.; Donovan, J.; Macpherson, R.; Tormey, D. Machine Learning for Automated Quality Evaluation in Pharmaceutical Manufacturing of Emulsions. J. Pharm. Innov. 2019, 15, 392–403. [Google Scholar] [CrossRef]
- Raponi, F.; Moscetti, R.; Chakravartula, S.S.N.; Fidaleo, M.; Massantini, R. Monitoring the hot-air drying process of organically grown apples (cv. Gala) using computer vision. Biosyst. Eng. 2021, 223, 1–13. [Google Scholar] [CrossRef]
- Sharma, S.; Dhalsamant, K.; Tripathy, P.P. Application of computer vision technique for physical quality monitoring of turmeric slices during direct solar drying. J. Food Meas. Charact. 2018, 13, 545–558. [Google Scholar] [CrossRef]
- Sánchez, A.J.; Albarracin, W.; Grau, R.; Ricolfe, C.; Barat, J.M. Control of ham salting by using image segmentation. Food Control. 2008, 19, 135–142. [Google Scholar] [CrossRef]
- Jia, W.; Liang, G.; Tian, H.; Sun, J.; Wan, C. Electronic Nose-Based Technique for Rapid Detection and Recognition of Moldy Apples. Sensors 2019, 19, 1526. [Google Scholar] [CrossRef] [PubMed]
- Ezhilan, M.; Nesakumar, N.; Babu, K.J.; Srinandan, C.; Rayappan, J.B.B. An Electronic Nose for Royal Delicious Apple Quality Assessment—A Tri-layer Approach. Food Res. Int. 2018, 109, 44–51. [Google Scholar] [CrossRef] [PubMed]
- Baietto, M.; Wilson, A.D. Electronic-Nose Applications for Fruit Identification, Ripeness and Quality Grading. Sensors 2015, 15, 899–931. [Google Scholar] [CrossRef] [PubMed]
- Tang, Y.; Xu, K.; Zhao, B.; Zhang, M.; Gong, C.; Wan, H.; Wang, Y.; Yang, Z. A novel electronic nose for the detection and classification of pesticide residue on apples. RSC Adv. 2021, 11, 20874–20883. [Google Scholar] [CrossRef] [PubMed]
- Li, C.U.I.; Niu, L.Y.; Li, D.J.; Liu, C.Q.; Liu, Y.P.; Liu, C.J.; Song, J.F. Effects of different drying methods on quality, bacterial viability and storage stability of probiotic enriched apple snacks. J. Integr. Agric. 2018, 17, 247–255. [Google Scholar] [CrossRef]
- Cichowska-Bogusz, J.; Figiel, A.; Carbonell-Barrachina, A.A.; Pasławska, M.; Witrowa-Rajchert, D. Physicochemical Properties of Dried Apple Slices: Impact of Osmo-Dehydration, Sonication, and Drying Methods. Molecules 2020, 25, 1078. [Google Scholar] [CrossRef] [PubMed]
- Haralick, R.M.; Shanmugam, K.; Dinstein, I.H. Textural Features for Image Classification. IEEE Trans. Syst. Man Cybern. 1973, 3, 610–621. [Google Scholar] [CrossRef]
- Löfstedt, T.; Brynolfsson, P.; Asklund, T.; Nyholm, T.; Garpebring, A. Gray-level invariant Haralick texture features. PLoS ONE 2019, 14, e0212110. [Google Scholar] [CrossRef] [PubMed]
- Brynolfsson, P.; Nilsson, D.; Torheim, T.; Asklund, T.; Karlsson, C.T.; Trygg, J.; Nyholm, T.; Garpebring, A. Haralick texture features from apparent diffusion coefficient (ADC) MRI images depend on imaging and pre-processing parameters. Sci. Rep. 2017, 7, 4041. [Google Scholar] [CrossRef] [PubMed]
- Yang, M.; Kpalma, K.; Ronsin, J. A Survey of Shape Feature Extraction Techniques. Pattern Recognit. 2008, 15, 43–90. Available online: https://hal.archives-ouvertes.fr/hal-00446037 (accessed on 12 January 2020).
- Safar, M.H. Shape analysis and retrieval of multimedia objects. In Intelligent Virtual World: Technologies and Applications in Distributed Virtual Environment; World Scientific Publishing Company: Singapore, 2004; pp. 21–51. [Google Scholar] [CrossRef]
- Chen, Y.; Li, F.; Huang, T. Curvature features based shape analysis. In Proceedings of the International Conference on Intelligent Computing, Shanghai, China, 15–18 September 2008; pp. 414–421. [Google Scholar] [CrossRef]
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |
© 2022 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).