A Non-Destructive Method for Grape Ripeness Estimation Using Intervals’ Numbers (INs) Techniques

Abstract

Grape harvesting based on estimated in-field maturity indices can reduce the costs of pre-harvest exhaustive sampling and chemical analysis, as well as the costs of post-harvest storage and waste across the production chain due to the non-climacteric nature of grapes, meaning that they are not able to reach desired maturity levels after being removed from the vine. Color imaging is used extensively for intact maturity estimation of fruits. In this study, color imaging is combined with Intervals’ Numbers (INs) technique to associate grape cluster images to maturity-related indices such as the total soluble solids (TSSs), titratable acidity (TA), and pH. A neural network regressor is employed to estimate the three indices for a given input of an IN representation of CIELAB color space. The model is tested on one hundred Tempranillo cultivar images, and the mean-square error (MSE) is calculated for the performance evaluation of the model. Results reveal the potential use of the Ins’ NN regressor for TSS, TA, and pH assessment as a non-destructive, efficient, fast, and cost-effective tool able to be integrated into an autonomous harvesting robot.

1. Introduction

Grapes are non-climacteric fruits, meaning that they do not ripen any further after being removed from the vine. The latter may result in serious quality and quantity loss of the harvested products, which tend to lose their fluids, dehydrate, and become sensitive to microbial decay during post-harvest handling and storage [1] or thrown away if harvested when immature. The accurate prediction of grape maturity would help determine the exact quality of the yield, as well as the exact harvest dates and locations, so as to timely engage resources, i.e., human labor, refrigerators, vehicles, etc., at specific maturity zones. Traditional ripeness estimation is performed via visual assessment, manual sampling and tasting, and chemical analysis. The latter is either subjective and prone to errors or cost-effective and labor-intensive [2]. Therefore, automated grape ripeness estimation based on quick, intact, and on-site methods, is desirable.

To this end, vision-based methods have been used extensively in grape ripeness estimation applications [3], mainly by using external qualities of grapes such as colorimetric and morphological attributes. Color and appearance are closely related to the chemical and sensory properties of fruits. CIELAB color values are commonly used in food-related research [4,5]. The CIELAB color space is an international standard for color measurements, proposed in 1976 by the Commission International d’Eclairage (CIE). CIELAB is considered uniform by the CIE, meaning that the Euclidean distance between two different color points resembles the color difference perceived by the human eye [6].

However, internal qualities for maturity grading can be more comprehensive. Ripening level monitoring is commonly based on sugar content and total acidity determination, using refractometers and titration methods [7]. Chemical attributes such as soluble solids content (SSC), titratable acidity (TA), and pH are widely used objective indicators to determine grape maturity [8]. Therefore, there has been extensive research for a rapid, reliable, less expensive, and non-destructive way to determine SSC, TA, and pH of grapes, to quantify their technological maturity level. Multispectral and remote sensing are used for the estimation of spatial field variability; however, the determination of specific chemical components that change their concentration during the ripening process is not yet accurate [7,9]. Smart chemometric technologies, based on visible, near-infrared, and mid-infrared spectroscopy, revealed their potential for grape quality monitoring, allowing the quick and accurate determination of chemical components [10]. This technology has the benefit of merging the features of both imaging and spectroscopy that, in the reflectance mode, allows the collection of information about the intensity of the light reflected by grapes as a function of their wavelengths [7,10,11].

Based on the above, in this study, a complete characterization of grape ripeness via image analysis and Intervals’ Numbers (INs) technique is described. Ten grape bunches were monitored during ripening. Images were captured for 11 maturity stages until harvest time, accompanied by an equal number of sampling and chemical analyses to measure SSC, TA, and pH values. The CIELAB color space component b* was extracted from all images and represented by an IN. IN representations were associated with chemical attributes, and both were used to train a neural network (NN) regressor, to predict the maturity-related chemical indicators of any given in-field image. This study is the first, as far as the authors’ knowledge, in which a method is proposed for grape ripeness estimation through numerical estimation of the maturity-related chemical indices—SSC, TA, and pH—from intact and on-site grape cluster images by using IN representations of CIELAB color components. Moreover, it is the first time that INs are applied to a regression problem for grape maturity estimation.

Many applications of INs have been reported in the recent literature regarding neural networks, fuzzy inference systems, and machine learning [12,13,14]. In previous studies, NN-based predictive models have been developed, to map a vector of past INs to a future IN, where an IN represented a distribution of an RGB color channel [12,13]. In this research, an NN model was employed to map a vector of INs representing a distribution of a CIELAB channel to a vector of numbers corresponding to grape maturity indices regarding SSC, TA, and pH. Such a model can operate in cascade with another model, which, based on a vector of grape maturity indices, can take a binary decision regarding individual fruit harvesting. The long-term objective was to develop a reliable (cascade) model that can decide the maturity of an individual grape cluster from images instead of chemical analyses, for homogenous harvesting, i.e., selective harvesting of grapes of the same maturity level, and be integrated with an autonomous harvesting robot.

The rest of the paper is structured as follows: Section 2 reviews related research so as to highlight the contribution of the proposed method. In Section 3, materials and methods are presented—namely, the sampling process, chemical analyses, image acquisition, image pre-processing, INs representation, and the proposed IN NN regressor are analyzed in detail. Results are discussed in Section 4, while Section 5 concludes the paper.

2. Related Research

Red–green–blue (RGB) color imaging is the most cost-effective method to extract color channel values. RGB cameras are very accessible to the user due to their low cost and high availability, including smartphone digital cameras, which everyone owns today. However, RGB color channels display a high degree of association resulting in a limited range of colors than the range the eyes can perceive. For this reason, alternative color spaces have been investigated in the literature, so as to provide effective images to image-based methodologies for grapes ripeness estimation, such as CIELAB, hue saturation intensity (HSI), hue saturation value (HSV), etc.

Rodríguez-Pulido et al. [5] used image analysis to classify four different varieties of berry seeds in the ripened or unripened stage. The seeds were photographed inside an illumination box. CIELAB coordinates were used to obtain morphological and appearance parameters of seeds. HSI color space was used for the segmentation process. Analysis of variance was applied to classify the morphological data of all samples, while discriminant analysis was applied to classify the samples based on the features of appearance of the seed, into ripened or unripened. The same research team in [6] proposed a method to evaluate the phenolic maturity stage of grape seeds based on computer vision. A set of phenolic compounds was extracted in order to obtain reference values for the proposed model. Red grape berries were collected in six stages of maturation and chemically analyzed. The seeds were photographed in an illumination box, and the CIELAB coordinates were used to obtain morphological and appearance parameters. Correlation studies were applied to investigate correlations between appearance data and chemical compounds. A forward stepwise multiple regression was finally applied to estimate the maturity stage of grape seeds from features obtained via the image analysis.

Rahman et al. [15] developed a method in which first the grape bunches were segmented from the background and then classified into mature or underdeveloped. HSV color space was used for the segmentation process. Grape berries were identified from their circular shape. Circles were grouped into clusters based on their spatial vicinity, using a k-means clustering algorithm. Texture and color features were computed from the grape cluster images from both RGB and HSV representations. A support vector machine (SVM) classifier was trained to classify the bunches in either one of the two categories. Avila et al. [16] developed maturity color scales of grape seeds. Images of seeds were acquired during phenolic maturity by using a conventional scanner and a camera. The images were transformed to the invariant illumination color model c₁c₂c₃ (#c₁c₂c₃ hex color code). The hexadecimal color code #c₁c₂c₃ expresses each color as a six-digits combination of numbers and letters defined by its mixture of basic RGB colors. Segmentation was performed with the Otsu method, and the representative color of each image was estimated. A support vector regressor (SVR) was employed to create the color scale associated with the degree of maturity of the fruit. Three stages of maturity were considered: mature, immature, and overmature.

Pothen and Nukse [17] proposed an automated approach to evaluate grape maturity based on spatial maps of grape clusters showing current and predicted distribution of color development. First, berries were located by using three visual properties: color, shape, and surface shading. Second, a measurement for color was extracted at each berry location, using the HSV color space, and four maturity grades were determined. Finally, the rate of color change in tandem with the current spatial map was used to predict future maps. A non-invasive method for measuring SSC and pH from images was proposed in [18]. A set of 52 color features were extracted for grape berry samples from multiple color spaces. Partial least squared regression (PLSR) and multiple linear regression (MLR) were used to predict the chemical attributes from the alternative color spaces, revealing RGB color features as the most important for the prediction.

In [19], a method for ripeness estimation based on grape seed images was introduced by Hernandez et al. The authors proposed a Dirichlet mixture (DMM) as a generative model for clustering grape seeds. The DMM model directly used the color histograms of RGB and HSV color spaces to probabilistically assign the grape seed images to different clusters (two and three classes were tested) using a cluster membership indicator. Cavallo et al. [20] developed a method to classify grape bunches into different quality grades. The acquired images were segmented and then used to extract grape bunch features in the CIELAB color space. A random forest classifier was used to predict the quality grade of grapes in the images. Kangune et al. [21] collected a dataset of images of ripening and unripened grape bunches and extracted color features such as RGB and HSV using histograms. A convolutional neural network (CNN) and an SVM were used to classify the images.

Kaburlasos et al. [22] used INs representations of RGB color histograms to predict the future maturity of grapes using past maturity data. More specifically, an IN-based feed-forward neural network (NN) architecture was used, taking two past INs as inputs in order to predict a future IN, related to different maturity stages. The same research team proposed a recursive IN NN scheme in [12], where the inputs of the NN were three—two past INs and the prediction of a third IN, leading to improved prediction performance.

Ramos et al. [23] proposed a CNN model for image classification based on their maturity stage. Maturity classes were established through analysis of TSS, total anthocyanins, and flavonoids. Two CNN architectures were comparatively tested to classify the images in the correct class. More recently, Wie et al. [24] developed a method for maturity prediction in greenhouse grapes. RGB images during ripening were acquired, and color values were determined. Factors such as the diameter of grape berries, compactness, SSC, TA, and SSC/TA were measured and evaluated. A back-propagation NN (BPNN) was employed to predict grape maturity coefficients.

Table 1 includes comparative information regarding the referenced literature on grape ripeness estimation methods of the last decade (2012–2022) and of the proposed method (PM) so as to comparatively highlight its contribution. Non-destructive methods refer to methods that keep the grape bunch intact. On-site application refers to methods that have been tested in the vineyards for real-time ripeness estimation.

Table 1. Comparative table of grape ripeness estimation methods of the literature and the proposed method (PM), based on color imaging.

Ref.	Ripeness Attribute	Color Space	No. of Images	Prediction Model	Method	Evaluation	Applied to	Non-Destructive	On-Site
[5]	Visual assessment	CIELAB HSI	100	Analysis of variance, Discriminant analysis	Classification (ripened–unripened)	57.15–100% Class. Acc.	Seeds	-	-
[6]	21 Phenolic compounds	CIELAB	100	Forward stepwise multiple regressors	Regression	Up to 0.97 R2	Seeds	-	-
[15]	Visual assessment	HSV RGB	31	SVM	Classification (ripened–unripened)	59.38–96.88% Class. Acc.	Bunch	✓	✓
[16]	Visual assessment	c1c2c3	250	SVR	Regression	22.64 MSE	Seeds	-	-
[17]	Visual assessment	HSV	-	Grading scheme	Prediction	0.42–0.56 R2	Bunch	✓	✓
[18]	SSC pH	HIS, NTSC, YCbCr, HSV, CMY	180	MLR PLSR	Regression	SSC: 0.79–0.92 MSE pH: 0.098–0.12 MSE	Berries	-	-
[19]	Visual assessment	RGB HSV	289	DMM	Classification (2 and 3 classes)	125.04 perplexity	Seeds	-	-
[20]	Visual assessment	HSV CIELAB	800	RF	Classification (5, 3 and 2 classes)	60–100% Class. Acc.	Bunch	✓	-
[21]	Visual assessment	RGB HSV	4000	CNN SVM	Classification (ripened–unripened)	69–79% Class. Acc.	Bunch	✓	✓
[22]	Visual assessment	RGB	13	IN NN feed-forward	Prediction	Up to 13.68 average error	Bunch	✓	✓
[12]	Visual assessment	RGB	13	IN NN recursive	Prediction	Up to 6.65 average error	Bunch	✓	✓
[23]	TSS Anthocyanins Flavonoids	RGB	5040	CNN	Classification (3, 4, 6, and 8 classes)	72.66–93.41% Class. Acc.	Berries	-	-
[24]	Diameter, compactness, SSC, TA, SSC/TA	RGB	-	BPNN	Prediction	0.36–0.65 R2	Bunch	✓	✓
PM	SSC TA pH	CIELAB	100	IN NN regressor	Regression	SSC: 6.62 MSE, TA: 8.02 MSE, pH: 0.28 MSE	Bunch	✓	✓
					Classification *	SSC: 71%, TA: 82%, pH: 73% Class. Acc.

* Classification results are indicative and refer to Case 1 (details provided in Section 4). Different cases (selected thresholds) would result in different (higher/lower) classification accuracies.

As can be observed from Table 1, the proposed method is the only one, as far as the authors’ knowledge, to predict numerical values of maturity-related chemical indices SSC, TA, and pH from intact and on-site grape cluster images.

3. Materials and Methods

In what follows, the sampling process, chemical analyses, image acquisition, image pre-processing, INs representation, and the proposed IN NN regressor are analyzed in detail. The image dataset, the corresponding SSC, TA, and pH values, and the source code of the proposed methodology presented in this section are publicly available [25].

3.1. Samples

Tempranillo red grapes that grow on Pavlidis Estate and Winery [26] in Drama, northern Greece, were collected in 2020, at 11 different growing stages, from veraison to ripeness. Sampling was carried out during the morning period, to limit the impact of high temperatures on the physical sampling of grapes and the relevant compositional quality attributes. In the case of recent rain, sampling was postponed. Sampling was carried out by following a specific protocol for 10 consecutive vine trees: approximately 100 berries were collected from both sides of each vine tree.

In order to collect representative samples for each of the 10 vine trees, berries were removed from the top, middle front, middle back, and bottom of each cluster of the vine tree, leaving intact a representative grape cluster in each vine tree in order to photograph its growing evolution. External grape berries from the same vine tree were removed to measure SSC, TA, and pH values, and correlate them with the reference image of the representative cluster of the tree. The representative cluster of each vine tree remained intact throughout the entire procedure, and the camera captured each sampling date and, thus, its ripeness progress.

The samples were immediately transported to the chemical department of the winery and were subjected to analyses. Quick transportation is critical since the time between collection and measurement can cause post-harvest changes in compositional parameters such as total acidity, pH, and color.

Table 2 includes the sampling dates and the weight of the samples for each of the 10 vine trees involved in the experiment. Missing values on the table result due to the representative grape cluster being removed (defected/rotten/malnourished); thus, the corresponding vine tree was excluded from further sampling and analyses. In total, 100 samples were collected.

Table 2. Sampling dates. Weights (gr) of collected samples for 10 vines for 11 sampling dates.

Vine Tree	1	2	3	4	5	6	7	8	9	10
Day 1—27 July 2020	57	44	37	46	50	50	47	45	47	44
Day 2—3 August 2020	59	50	46	54	51	52	53	51	54	44
Day 3—10 August 2020	69	65	60	69	76	55	77	77	73	87
Day 4—17 August 2020	80	55	51	63	72	85	52	74	56	73
Day 5—20 August 2020	62	55	59	61	68	44	55	56	62	55
Day 6—24 August 2020	64	54	50	60	49	49	51	51	53	62
Day 7—27 August 2020	85	47	76	-	65	60	83	65	-	62
Day 8—31 August 2020	79	89	93	-	99	64	93	63	-	65
Day 9—3 September 2020	72	80	64	-	64	82	103	67	-	68
Day 10—7 September 2020	14.1	118	143	-	123	127	158	127	-	135
Day 11—10 September 2020	12.7	132	133	-	109	127	134	111	-	99

3.2. Image Acquisition

A representative grape cluster was selected in each of the 10 vines, and its growing evolution was photographed at each sampling date, resulting in an image dataset of 100 images. Images were captured with a ZED Mini 3D camera, under natural daylight, including disturbances of varying levels of illumination and shadows. All images were captured at the same time of the day so as to ensure similar brightness by the sun hitting the vines from the same side each time. However, on each day, the weather could be either sunny or cloudy, and the foliage could introduce alternating shadows on the cluster images. Ideally, the same consistent illumination in all images would result in better results for the proposed regressor. Nevertheless, the proposed model is trained with images of varying illumination so as to give diversity to the training process and robustness to the final result. Moreover, the method is designated for in-field real-time maturity assessment by an autonomous harvesting robot; the latter implies that realistic scenarios must be tested, so images with varying illumination need to be used. The camera was mounted on a robotic arm of an autonomous harvester robot [27]. The camera captured the images facing the grape clusters vertically at a fixed distance from the vine tree so as to ensure consistency in the result—at a distance of about 50 cm from the grape cluster and a height of about 70 cm from the ground. Figure 1 illustrates indicative images of three different sampling dates—days 1, 5, and 11—for the representative cluster of the 7th vine tree.

Figure 1. Indicative images of the dataset, depicting the ripening process of the representative cluster of the 7th vine tree: (a) sampling day 1; (b) sampling day 5; (c) sampling day 11.

3.3. Chemical Analysis

Grape samples were destemmed and crushed by hand. Reference SSC, TA, and pH values were measured. Analytical methods for determination of total and active acidity were based on OIV’s International Methods of Analysis of Wines and Musts [28]: (1) total acidity: OIV MA-AS313-01 (titration with bromothymol blue), and (2) pH: OIV –MA-AS313-15-PH (pHmeter Titrator ATT1222 Hach). For the evaluation of sugar concentration in grapes, musts were employed via pycnometry (a Baume hydrometer divided into 1/10 Dujardin–Salleron and a calibrated alcohol thermometer scale −20–100 °C Brannan).

3.4. Image Pre-Processing

In the image pre-processing step, grape cluster segmentation was accomplished so as to identify the grape cluster to be further processed. All input RGB images were annotated so that their pixels belong to either grape cluster or background, by using the LabelMe annotation tool [29]. CIELAB color coordinates were obtained from the RGB segmented images.

Color in CIELAB color space is expressed in terms of a psychometric index of lightness, L*, and two cartesian color mixture coordinates a* (from green to red) and b* (from blue to yellow), related to the visual appreciation of color. After experimentation, the distribution of the b* channel was selected in this study, due to the fact that it demonstrated a greater association with color maturity than the other channel histograms. As the chemical indices under study varied along the different grape maturation stages, related to polymerization reactions occurring during this period, the color distribution in histograms also changed. Figure 2 illustrates the distribution of the b* channel in grape samples of the 7th vine tree on days 1, 5, and 11.

Figure 2. Distribution of b* channel in grape samples of the 7th vine tree: (a) sampling day 1; (b) sampling day 5; (c) sampling day 11.

Histograms emphasized the existence of outliers. Isolated bars outside the histogram distribution, as the far bars on the right of Figure 1a–c, were identified as outliers and had to be removed in the pre-processing step. For this reason, a thresholding method was applied. In order to apply thresholding, two conditions had to be met: (1) the thresholding had to be applicable after 1/3 of the total histogram color values, and (2) after that point, when a value was less than the 0.5% of the maximum distribution value, all values after that were ignored.

3.5. INs Representation

Histograms of the b* channel were represented by INs, by using the algorithm distIN [30]. For all 100 images depicting grape ripeness stages, the corresponding INs were extracted. The main advantage of the induced INs is that they can represent all order data statistics by using a finite number of L intervals. The raw histogram data correspond to a probability density function, while an IN corresponds to a probability distribution function. Probability and possibility distributions have been studied comparatively in the past [31]. An Intervals’ Number (IN) is a mathematical object that can be interpreted either probabilistically or possibilistically [32]. Recently, an IN has been interpreted as a cumulative possibility distribution [33].

From a technical point of view, an IN is rigorously defined by a 3-level hierarchy of lattices as follows: Level 0 includes the chain (R, ≤) of real numbers; Level 1 includes the lattice (I, ⊆) of conventional intervals in (R, ≤); Level 2 includes the lattice (F, ≤) of INs, where an IN E is a function E: [0,1]→I that satisfies the following conditions:

$$\begin{array}{c}{h}_1\ge h_2\Rightarrow E_{h_1}\subseteq E_{h_2}\\ \forall \mathrm{X}\subseteq \left[0,1\right]:\underset{h\in \mathrm{X}}{{\displaystyle \cap }}{E}_h=E_{\vee \mathrm{X}}\end{array}$$

(1)

Defining addition and multiplication in (I, ⊆) as [a,b] + [c,d] = [a+c,b+d] and λ[a,b] = [λa,λb], it follows that (F, ≤) is a cone, where addition and multiplication (by a nonnegative number λ ∈ R) are defined as (E + G)_h = E_h + G_h and (λΕ)_h = λΕ_h, respectively, ∀h∈ [0,1]. A metric distance: d_F: F × F → ${\mathrm{R}}_0^{+}$ is defined as

$$d_{\mathrm{F}}\left(E,F\right)={{\displaystyle \int }}_0^1{d}_{\mathrm{I}}\left(E_h,G_h\right)dh$$

(2)

where $d_{\mathrm{I}}:\mathrm{I}\times \mathrm{I}\to {\mathrm{R}}_0^{+}$ is a metric distance in (I, ⊆) given by d_I([a,b],[c,e]) = v(θ(a˄c)) − v(θ(a˅c)) + v(b˅e) − v(b˄e), where $v:\mathrm{R}\to \mathrm{R}$ is a strictly increasing real function; moreover, $\theta :\mathrm{R}\to \mathrm{R}$ is a strictly decreasing real function.

IN-based models process data distributions potentially toward improving effectiveness comparatively to conventional (arithmetic) models. There are two, equivalent IN representations; the membership-function representation and the interval representation. Figure 3 illustrates the interval representation corresponding to the histograms of Figure 2, for L = 32 intervals. The finite number of intervals may result in significant data compression in the case of large input data. Moreover, no feature extraction occurs since an IN can be implicitly employed as a set of features. It should be noted that the proposed method can also introduce an arbitrarily large number of parameters via parametric functions v (.) and θ (.) per constituent lattice.

Figure 3. Interval-representation of INs computed from histograms in Figure 2.

The correlation of INs with maturity can be observed clearly in interval representations presented in Figure 3. In these representations, a further clustering of data can be observed, which occurs to the right in the case of immature grapes (day 1), and as the phenolic maturity progresses, INs become narrower and move gradually to the left. Thus, the generated INs allow for an effective representation of the maturity of grapes in which color is the influential factor in determining the optimal harvesting point.

3.6. IN NN Regressor

The neural network shown in Figure 4 was trained to estimate SSC, TA, and pH values. In this section, details regarding the setup of the NN are listed.

Figure 4. The proposed IN neural network regressor.

The proposed IN NN regressor scheme is a 4-layer $\left(1\times k\times n\times 3\right)$ feed-forward NN. For each of the 100 samples, a training phase was set, using the leave-one-out cross-validation method. The dataset was split into a training set and a testing set, using all but one sample as part of the training set. The model was built using only data from the training set, and it was used to predict the response values of the one sample left out of the training model. The process was repeated 100 times, equal to the number of samples in the dataset.

Keras Python library [34] was used to build and train the proposed NN. Rectified linear unit (ReLU) activation function was used for the transitions of each layer as follows:

$$R\left(z\right)=\{\begin{array}{c}z, z>0\\ 0, z\le 0\end{array}$$

(3)

where k = 10 and n = 20 neurons for the hidden layers 2 and 3 were selected, respectively, by trial and error. For each training phase, 1000 training epochs occurred. The built model was then used for the testing phase, and the results were recorded.

In the experiments, the neural network received the bunch’s image histogram IN (F₁) as input. Three outputs represented the estimation of values SSC, TA, and pH (F₂, F₃, F₄), as depicted in Figure 4.

4. Results and Discussion

Results are summarized in Table 3. The model was evaluated for two different input representations—the raw histogram (1 input of 255 color values) and the induced IN (1 input of 32 IN values), for comparative reasons. The evaluation of the models was performed in terms of the mean-square error (MSE).

Table 3. Comparative results (MSE) for two input representations: raw histogram and induced INs (PM).

Input Representation	TA	pH	SSC
INs (PM)	8.0280	0.2884	6.6259
Histogram	24.8294	3.1216	9.2671

It is obvious from the results in Table 3 that the proposed method reported higher performance than the raw histogram for the estimation of all three indices, by using IN representation as input to the NN regressor. Therefore, INs were proven as effective representations of the distribution of the selected CIELAB channel corresponding to grape maturity indices SSC, TA, and pH. This can be attributed to the finite number of L intervals, equal to 32, compared with the 255 values of the histogram. The IN representation may compress the initial distribution; however, the induced IN included all order statistics of the initial data, resulting in no loss of valuable inherent information. It should be noted that the latter can be extremely advantageous in the case of large input data.

Moreover, most of the methods included in Table 1 involved feature extraction and training of a machine learning model. However, current approaches to feature extraction are ad hoc; in contrast, in the proposed method, no feature extraction occurred since an IN could be implicitly employed as a set of features. This is the reason why the proposed method was directly compared with histograms’ input representation (Table 3): in both cases, no feature extraction was performed.

In order to provide a direct comparison to other ripeness methods reported in the literature, based on color imaging of those included in Table 1, the regression results were translated to classification accuracies. Two classes were defined based on the values of the chemical indices: ripened and unripened, as presented in Table 4. The boundaries of the two classes emerged from [3]; for SSC, TA, and pH, regardless of the grape variety, the limits that collectively indicated a ripened grape were defined (Table 4). Two cases were examined; in the first, the boundary of the two classes was selected in the middle of the proposed maturity range (Case 1), while in the second, the boundary of the two classes was selected to be the lower limit of the proposed maturity range (Case 2).

Table 4. Indicative limits of basic chemical attributes SSC, TA, and pH in ripened wine grapes [3], and identification of two classes (ripened and unripened) based on a selected threshold (middle value and lower value).

Limits for Ripened Grapes	Identification of Two Classes Case 1: Threshold on Middle Value		Identification of Two Classes Case 2: Threshold on Lower Value
	Ripened	Unripened	Ripened	Unripened
3.2 < pH < 3.5	pH > 3.35	pH < 3.35	pH > 3.2	pH < 3.2
11.1 < SSC < 12.8 (Baume)	SSC > 11.95	SSC < 11.95	SSC > 11.1	SSC < 11.1
4 < TA < 7 (g L−1)	TA > 5.5	TA< 5.5	TA > 4	TA< 4

Actual values of the chemical attributes and estimated values were assigned to the two classes. The interpretation of regression results to classification accuracies, for both examined cases, are included in Table 5.

Table 5. Interpretation of regression results of classification accuracies in two classes, based on selected thresholds, for all three chemical attributes.

Classification: Case 1 (Threshold on Middle Value)		Classification: Case 2 (Threshold on Lower Value)
pH:	73%	pH:	69%
SSC:	71%	SSC:	62%
TA:	82%	TA:	75%

Selecting the threshold in the middle of the maturity range resulted in better classification results. The conversion of estimated values into classes is useful for real-time decision making in order for an autonomous harvesting robot to perform selective harvesting [35]. The value of the selected threshold is not fixed, since it is decided by the user depending on the degree of maturity of the grapes he wishes to collect. It is obvious that optimal maturity is not related to a standard threshold value but to the desired value depending on the user and the intended post-harvest use of grapes [3]. For example, in the wine industry, the maturity level of harvested grapes determines the procedure, diffusional, enzymatic, or biochemical processes that would be subsequently applied, while for table grapes, the refractometric index is considered along with the sugar/acid ratio so as to determine grape maturity that reflects consumers’ acceptability [8]. Therefore, a different selection of thresholds would result in different classification accuracies, higher or lower. The latter is considered an additional advantage of the proposed method, since it allows users to determine the desired maturity level of grapes they wish to harvest based on three different chemical indices, revealing a useful flexible tool for on-site grape quality assessment.

Classification results (%) of the proposed method are included, along with regression results, (MSE) in Table 1, for comparative reasons. Even if the results are indicative and refer to a specifically selected threshold (Case 1), they are comparable and better in some cases. Results indicate that it is possible to estimate the maturity of grape clusters by applying a NN regression model to IN data. Moreover, the results of this study proved that CIELAB color space is representative enough to be used for the SSC, TA, and pH prediction of grapes. The proposed method is not a substitute for conventional chemical analysis; however, it is an attractive and objective alternative for the estimation of grape maturity due to its simplicity, versatility, and low computational and economical costs. Furthermore, as can be observed from Table 1, the proposed method is the only one, as far as the authors’ knowledge, to predict numerical values of maturity-related chemical indices SSC, TA, and pH from intact and on-site grape-cluster images. Finally, the proposed method can consider the user’s feedback to translate estimation values in grape quality classes for on-the-spot decision making, toward automated selective harvesting.

Future research involves the application of the proposed model to other grape cultivars to evaluate its robustness. The long-term objective is to develop a reliable model able to determine the maturity of grape clusters from images in real time instead of relying on chemical analyses, and one that is integrated with a harvester robot for selective harvesting.

5. Conclusions

In this study, an IN NN regressor was proposed for the estimation of the ripeness level of grapes from color images using the CIELAB color space. Ripeness level was measured in terms of three well-known maturity-related chemical indices—SSC, TA, and pH. Results indicated the reliable performance of the proposed method in estimating the maturity indices.

The proposed method can be used as a reliable tool to assess the chemical attributes of grape clusters during maturation, saving time and chemical reagents and, thus, allowing wine growers to make fast decisions regarding the exact time and location of harvesting. Moreover, the proposed method can be integrated with an autonomous harvesting robot as an on-the-spot decision-making algorithm for harvesting grape bunches, as producers are interested in collecting bunches of similar degrees of maturity intended for specific uses, e.g., eating, wine production, etc.