Study on Predicting Blueberry Hardness from Images for Adjusting Mechanical Gripper Force

Abstract

Precision and non-damaging harvesting is a key direction for the development of mechanized fruit harvesting technologies. Blueberries, with their soft texture and delicate skin, present significant challenges for achieving precise and non-damaging mechanical harvesting. This paper proposes an intelligent recognition and prediction method based on machine vision. The method uses image recognition technology to extract the physical characteristics of blueberries, such as diameter and thickness, and estimates fruit hardness in real-time through a predictive model. The gripping force of the mechanical claw is dynamically adjusted to ensure non-destructive harvesting. Firstly, a chimpanzee optimization algorithm (ChOA) was used to optimize a prediction model that established a mapping relationship between fruit diameter, thickness, weight, and fruit hardness. The radial basis network optimized by the chimpanzee optimization algorithm (ChOA-RBF) model was compared with a non-optimized model, and the results showed that the ChOA-RBF prediction model has significant advantages in predicting fruit hardness. Next, an orthogonal experiment further verified the model, showing that the prediction error between the model’s values and actual values was less than 5%. Additionally, considering practical applications, a simple and efficient two-parameter method was proposed, removing the weight parameter and predicting fruit hardness using only diameter and thickness. Although the two-parameter method increases the prediction error by 0.36% compared to the three-parameter method, it reduces the number of convergence steps by 71 and shortens the computation time by one-third, significantly improving iteration speed. Finally, further crushing experiments showed that using the two-parameter method for hardness prediction through parameter extraction via visual recognition resulted in a relative error of less than 8%, with an average relative error of 3.91%. The error falls within the acceptable range for the safety factor design. This method provides a novel solution for the non-damaging mechanized harvesting of soft fruits.

1. Introduction

Blueberries, with their delightful taste and rich nutritional value, are highly favored by consumers [1]. According to the 2024 report by the International Blueberry Organization, China’s blueberry cultivation area reached 84,420 hectares, with a production of 563.46 kilotons, ranking first in the world [2]. Traditional manual picking methods have disadvantages such as high costs, low efficiency, fruit loss, and reliance on seasonal labor. These factors increase the cost of blueberry harvesting [3] and severely limit the development of the blueberry industry [4]. In response to these challenges, mechanical automation technology has begun to show promise in the agricultural sector, bringing new hope to the blueberry industry. However, blueberries, being cluster-bearing crops, require selective individual harvesting. Their soft texture and fragile skin make them susceptible to damage, and conventional mechanical grippers cannot meet these requirements, necessitating a flexible and precise design for the mechanical claws.

To achieve a rapid, non-destructive mechanical harvesting of blueberries, both domestic and international researchers have extensively studied blueberry growth, fruit characteristics [5], stem mechanical properties [6], and the connection characteristics between berries and stems [7,8]. Currently, inertial vibration harvesting devices, which use inertial forces to separate fruits from the stems, are the most widely used harvesting equipment abroad. This method is more suitable for high-growing, tree-like blueberry varieties, such as those in North America. However, in China, where blueberries are primarily grown as low shrubs in greenhouses, differences in temperature, lighting, pruning, fertilization, and irrigation lead to staggered, multi-batch maturation. As a result, sorting and selectively harvesting mature fruits while preserving immature ones [9] has become a critical challenge for blueberry harvesting robots.

With the rapid development of artificial intelligence, robotic technologies for harvesting fruits such as apples and pears have matured. From fruit recognition [10] to end-effector design [11] and path planning [12,13], automation has been largely achieved. These fruits, with their harder texture and resistant skin, do not require consideration of the gripping force between the mechanical claw and the fruit. However, the soft texture of blueberries presents a major hurdle, making intelligent recognition, non-destructive harvesting, and selective picking the key challenges in the development of blueberry harvesting robots. Notably, designing a non-destructive method for harvesting blueberries remains unresolved. In blueberry hardness research, Hu et al. correlated the mechanical properties of the fruit using texture profile and puncture analysis [14], while spectral data from randomly selected frogs were used to estimate the mechanical properties of blueberries, providing a reference for predicting their mechanical characteristics. More recently, Oh et al. integrated sensory attributes with tool measurements to identify and predict blueberry hardness [15], enhancing the selection of desired blueberry textures through mechanical parameter estimation. These methods offer valuable insights for predicting blueberry hardness, yet they have not addressed pre-harvest, non-destructive harvesting applications, and relatively little research has been conducted in this area, both domestically and internationally.

In blueberry maturity recognition, Yang et al. proposed a lightweight blueberry fruit detection algorithm for multi-scale targets based on an improved YOLOv5, incorporating a new attention mechanism [16]. Experiments on a self-made blueberry dataset achieved a mean average precision (mAP) of 83.2%. MacEachern et al. applied YOLOv4 for blueberry maturity detection and demonstrated its high accuracy [17]. However, due to the considerable computational load of the YOLOv4 model, its performance significantly decreases when migrated to smaller embedded devices. Zhai et al. proposed an automatic blueberry recognition method based on YOLOv5 and CIE saturation enhancement [18]. By enhancing the image dataset for training, recognition accuracy, recall, and mAP improved by 3.0%, 2.0%, and 2.6%, respectively. However, the similarity in color features between immature blueberries and leaves, along with fruit occlusion by leaves, led to lower real-time detection accuracy. In response, Gai et al. introduced an improved multi-coordinate attention (MPCA) mechanism based on YOLOv8, which enhanced feature extraction during the training process [19], achieving an accuracy of 84.6%, a recall rate of 91.3%, and an mAP of 94.1%. In addition, deep learning models have achieved remarkable results in image correction and object detection. Xu et al. proposed a specular highlight removal method based on the generative adversarial network (GAN). The method uses an attention mechanism to generate highlight intensity masks, which remove highlights while preserving image details, providing an effective preprocessing method for feature extraction and analysis [20]. Zhuang et al. developed a dual-image and dual-local contrast measure (DDLCM) algorithm, which enhances target saliency through local feature enhancement, significantly improving the recognition accuracy of small and weak targets in complex backgrounds [21]. These studies highlight that visual algorithms are effective for blueberry maturity recognition. However, the ability to directly determine the gripping force for harvesting based on image data remains a crucial factor in improving operational efficiency and fruit preservation. Thus, understanding the relationship between fruit color and the gripping force is essential, a concept not yet explored in the aforementioned algorithms.

The intelligent recognition of fruit characteristics such as size and maturity can drive the development of precision agriculture by enabling the evaluation of various crop traits using computational intelligence methods. Gholipoor et al. quantified the influence of plant height [22], canopy width, and the number of fruits per plant on pepper yield using a neural network with a pepper image database and analyzed the contribution of each trait to yield prediction. Nagnath et al. [23] compared different deep learning algorithms for banana maturity and quality detection, concluding that the CNN model is the most suitable for classifying banana fruit maturity and quality. Qiao et al. utilized a backpropagation (BP) neural network to predict the quality of strawberries during cold-chain transportation based on environmental parameters such as temperature [24], humidity, oxygen, and carbon dioxide, providing decision support for management. Lin et al. developed a machine learning model (MLM) to predict the hardness of corn seeds, aiming to improve grinding efficiency [25], reduce seed breakage during transportation, and select high-quality crops. H. Rocha et al. used a radial basis function (RBF) model to predict the yield of hard wheat based on planting area and various climate variables [26]. The model’s input parameters included soil moisture, temperature, organic matter, total carbon, nitrogen, air temperature, solar radiation, and potential evapotranspiration, while the output was field-measured CO₂ emissions. Different regression methods each have their advantages; the RBF network offers strong nonlinear mapping capabilities [27], making it suitable for various predictive models, while BP’s backpropagation algorithm is rigorous and offers strong generalization ability [28]. Random Forest (RF), an ensemble learning technique, is known for its robustness and ease of implementation [29]. Therefore, selecting the appropriate predictive model for different applications is crucial.

Machine learning techniques can establish nonlinear mapping relationships between multiple parameters, showing high flexibility and accuracy in prediction tasks. In recent years, the deep integration of visual recognition and machine learning has promoted the development of intelligent perception. These two technologies complement each other in feature extraction and prediction modeling. Ma et al. combined the canopy reflectance model of row crops with backward propagation neural networks (BPNNs) to estimate fractional vegetation cover (FVC), demonstrating the effectiveness of visual recognition and machine learning algorithms in agricultural parameter estimation [30]. Cai et al. used the YOLACT model to identify ice block images and applied image processing algorithms to estimate fracture features, proving that deep learning combined with image analysis can effectively extract physical features of objects [31]. Ma et al. proposed a pixel dichotomy coupled with a linear kernel-driven model (PDKDM) to estimate FVC in drought areas through random sampling, verifying the feasibility of non-contact biological parameter estimation based on visual feature extraction and modeling [32]. Based on the aforementioned research, challenges such as low detection accuracy in blueberry maturity recognition and the difficulty of non-destructive harvesting due to the fruit’s soft texture remain significant barriers to the efficient automation of blueberry harvesting. Therefore, this paper proposes a novel approach that uses visual recognition to precisely extract the physical characteristics of blueberries and employs regression prediction algorithms to quickly estimate fruit hardness. By integrating fruit hardness with a safety factor to adjust the gripping force of the mechanical claw in real-time, non-destructive harvesting is achieved. This study applies the chimp optimization algorithm (ChOA) to optimize the RBF, BP, and RF models for estimating fruit hardness. For comparison, non-optimized versions of the RBF, BP, and RF models were also evaluated. Using actual measurements of blueberry fruit diameter, thickness, and weight, the study investigates the performance differences in the models and compares the detection accuracy of various neural networks for blueberry maturity. The optimal model for hardness prediction is determined. Based on these findings, blueberries were categorized into three size classes (large, medium, and small) according to their diameter, and an orthogonal experiment was conducted to validate the radial basis network optimized by the chimpanzee optimization algorithm (ChOA-RBF) model. In practical harvesting, fruit weight, which must be either manually assigned or predicted, can present a challenge. To simplify the model structure, improve prediction speed, and eliminate the weight parameter that cannot be directly obtained from images, a more efficient two-parameter method is proposed. This method is experimentally verified and evaluated. The proposed method effectively resolves the conflict between gripping force and skin damage, ensuring fruit integrity during harvesting. It offers a new approach for the non-destructive harvesting of soft fruits and lays the theoretical foundation for dynamically adjusting the mechanical claw’s gripping force in harvesting robots.

2. Materials and Methods

The research process of this study is outlined in Figure 1, which includes four key steps: feature extraction, model construction, model evaluation, and recognition/application. Data analysis and model construction were performed using MATLAB R2023b software.

2.1. Experimental Field

China’s blueberry cultivation is mainly distributed in Shandong, Jilin, Liaoning, and Yunnan provinces. In northern China, blueberries are primarily grown in greenhouses, while in southern regions, they are cultivated in open fields. With the advancement of greenhouse technology and its constant temperature advantages, greenhouses have become an increasingly common cultivation method. Consistent with previous studies [33], we selected greenhouse-grown blueberries in northern China for our study, choosing the rabbit-eye blueberry variety, which is widely grown across both northern and southern China, as the research subject. The experiments and data collection were conducted at Qingdao Wolin Blueberry Agricultural Co., Ltd. (Qingdao, China) (latitude 119°89′, longitude 35°75′). The experimental greenhouse is shown in Figure 2a, with a width of approximately 7.5 m, a length of approximately 105 m, and a center height of 2.2 m. Figure 2b illustrates the greenhouse plants and fruit, with each greenhouse containing 2–3 rows of blueberry plants and corridors 1.6 m wide. Each greenhouse can accommodate 8000–10,000 blueberry plants. As shown, the blueberries are low-growing shrubs with spherical shaped fruit. The diameter (d) of mature fruits ranges from 13 to 24 mm (measured at the equator), and the thickness (t) ranges from 8 to 15 mm (from stem to fruit flower-end). Based on the fruit’s morphological characteristics and size variations, fruit hardness also differs accordingly, as shown in the sample fruit in Figure 2b. During the harvesting process, only fruits of the same maturity were considered for picking, while immature fruits, overripe fruits, and spoiled fruits were excluded. Hence, the study and experiments focus on blueberries with the same maturity level.

2.2. Design Plan

2.2.1. Analysis of Method Rationality

Several post-harvest studies indicate that genotype, maturity, calcium content, and post-harvest management are the primary factors influencing blueberry hardness [34]. Variations in genetic traits, maturity, and post-harvest moisture loss [35] can directly affect the hardness of blueberry fruit. However, during the pre-harvest phase, blueberries within the same area and of the same variety typically share similar farming practices and environmental conditions. As a result, factors such as genotype, maturity, calcium content, and post-harvest management can be considered consistent. In the study of pre-harvest fruit hardness, this research employs a compression testing method to measure blueberry hardness and analyzes the correlation between the hardness and physical characteristics of the same variety. Specifically, the study investigates the relationship between physical features such as diameter, thickness, and weight, and fruit hardness, further validating the proposed method’s rationality. First, statistical methods were used to quantify the relationship between the diameter, thickness, weight, and hardness of blueberries. For 400 blueberry samples from different greenhouses, the diameter and thickness were measured with calipers, while the fruit hardness and weight were recorded using a fruit testing platform. The data were then analyzed, and the Spearman correlation coefficient was used to evaluate the linear relationship between the physical features and fruit hardness. A correlation heatmap was created to illustrate these relationships. Figure 3 displays the correlation heatmaps between diameter and hardness, thickness and hardness, and weight and hardness. The correlation coefficients clearly indicate the degree to which each physical feature influences hardness. Generally, a correlation coefficient |r| ≥ 0.8 suggests a high correlation between two variables, while 0.5 ≤ |r| < 0.8 indicates a moderate correlation, and |r| < 0.3 suggests no significant correlation [36]. The results reveal that there is a positive correlation between the diameter, thickness, and weight of blueberries and their hardness. Notably, the correlation between diameter and thickness with hardness is stronger, while the correlation between weight and hardness is 0.72, the weakest among the three parameters. This indicates that fruit hardness is more closely related to size. This finding provides a reliable basis for subsequent hardness prediction.

Based on these analyses, the proposed method for hardness prediction, which integrates machine vision technology, is scientifically sound. By accurately identifying the diameter, thickness, and weight of the fruit, the hardness can be effectively estimated, enabling the adjustment of the mechanical claw’s gripping force and ensuring that fruit damage is avoided during the harvesting process.

2.2.2. Research Process

In the harvesting process of small berries, the gripping force of the robotic arm is approximately proportional to the fruit’s hardness. That is, the gripping force must be large enough to ensure the fruit is securely held but not too large to avoid damaging the fruit. Therefore, the design of the robotic arm’s gripping force must carefully consider the fruit’s hardness to ensure that excessive pressure is not applied during gripping, thus maintaining the fruit’s integrity.

Due to the thin skin of blueberries, rapid recognition of fruit hardness is a key factor influencing the dynamic adjustment of the robotic claw’s gripping force for the precise, non-destructive harvesting of small berries. This paper provides a detailed analysis of the physical properties of blueberries, using computational intelligence methods to establish a mapping relationship between characteristics such as diameter, thickness, and weight, and the mechanical properties of the fruit. This allows for the determination of the optimal fruit hardness prediction model, which serves as the technical foundation for determining the optimal gripping force of the mechanical claw. As shown in Figure 4, 400 blueberries of different shapes were selected for this study. Using relevant experimental equipment (vernier calipers and fruit testing machines), the diameter, thickness, weight, and maximum pressure that would not damage the fruit were measured. The fruits were then categorized into three size groups based on their diameter: large, medium, and small. Four different computational intelligence methods were employed to model the functional relationships between the physical characteristics of the fruit and its hardness. The detection accuracy and error of the different prediction models were compared to identify the best model. An orthogonal experiment was then conducted to validate the optimal model. Finally, based on the optimal model, the weight parameter cannot be directly obtained through the image recognition system during the harvesting process and must be assigned and predicted. To enable a faster and more convenient estimation of fruit hardness, this study employs a two-parameter method, where the fruit hardness is estimated using only the fruit’s diameter and thickness. This method has been experimentally verified and evaluated, improving both accuracy and efficiency. It provides a new approach for non-destructive harvesting of soft fruits and enables the real-time control of the mechanical claw’s gripping force (F_c).

2.3. Analytical Methods

2.3.1. Prediction Model

As illustrated in Figure 5, the ChOA algorithm, inspired by chimpanzee hunting behaviors [37], includes two main phases: exploration and exploitation. In the exploration phase, chimpanzees drive, block, and chase prey to locate targets. In the exploitation phase, they engage in active attack. ChOA assumes that the best solution, driver, blocker, and chaser can identify the prey’s location, while other chimpanzees update their positions based on the optimal chimpanzee’s location to achieve exploration and exploitation for optimization. This process helps identify optimal model parameters for precise predictions [38].

1.

Driving and Chasing the Prey

$$d\left(t\right)=\left|c\times X_{\mathit{prey} }\left(t\right)-m\times X_{\mathit{chimp}}\left(t\right)\right|$$

(1)

$$X_{\mathit{chimp} }(t+1)=X_{\mathit{prey} }\left(t\right)-a\times d$$

(2)

In Equation (1), $d\left(t\right)$ represents the distance between the prey and the chimpanzee, $t$ is the current iteration count, $a$ and $c$ are coefficient vectors, and $m$ is a chaotic vector generated by chaotic mapping.$X_{\mathit{prey}}$ is the position vector of the prey, and $X_{\mathit{chimp}}$ is the position vector of the chimpanzee. The parameters $a$ and $c$ are defined as follows:

$$a=f\left(2r_1-1\right)$$

(3)

$$m={\mathrm{Chaotic}}_{\mathrm{v}\mathrm{a}\mathrm{l}\mathrm{u}\mathrm{e}}$$

(4)

$$c=2r_2$$

(5)

In Equation (3), $f$ is a linear decay factor that decreases linearly from 2.5 to 0 as the iterations increase. $r_1$ and $r_2$ are random numbers between $[0,1]$. The parameter $a$ is a random variable within the range $[-2f,-2f]$. When a is within $[-1,1]$, the prey stops moving, requiring the chimpanzee to attack to end the hunt. By reducing $f$, the chimpanzee is forced to finish the hunt, with its next position located randomly between its current position and the prey’s position.

2.

Attack

$$\left\{\begin{array}{c}{d}_A=\left|c_1{X}_A-m_{1}X\right|\\ d_B=\left|c_2{X}_B-m_{2}X\right|\\ d_C=\left|c_3{X}_C-m_{3}X\right|\\ d_D=\left|c_4{X}_D-m_{4}X\right|\end{array}\right.$$

(6)

$$\left\{\begin{array}{c}{X}_1=X_A-a_1\times d_A\\ X_2=X_B-a_2\times d_B\\ X_3=X_C-a_3\times d_C\\ X_4=X_D-a_4\times d_D\end{array}\right.$$

(7)

$$X(t+1)=\left(X_1+X_2+X_3+X_4\right)/4$$

(8)

The chimpanzee’s final position is randomly distributed within a circle defined by the positions of attacking, obstructing, chasing, and driving chimpanzees. In other words, the prey’s position is estimated based on the four best individuals, while the other chimpanzees update their positions randomly nearby.

3.

Social Motivation

Once a certain amount of food is obtained, some chimpanzees abandon their duties and fall into a chaotic feeding state. Similarly to the dragonfly optimization algorithm, this chaotic behavior helps the ChOA algorithm avoid becoming stuck in a local optimum during the final iterations. The original study assumes that chimpanzees have an equal probability of selecting either a normal position or a position generated by a chaotic model:

$$X_{\mathit{chimp}}(t+1)=\left\{\begin{array}{cc}{X}_{\mathit{prey}}\left(t\right)-a\times d& \mu \le 0.5\\ {\mathrm{Chaotic}}_{\mathrm{v}\mathrm{a}\mathrm{l}\mathrm{u}\mathrm{e}}& \mu >0.5\end{array}\right.$$

(9)

Here, $\mu$ is a random value within $\left[\mathrm{0,1}\right]$. Notably, if $\mu <0.5$ and $\left|a\right|<1$, the current chimpanzee position is updated using Equation (2). If $\mu <0.5$ and $\left|a\right|>1$, a random chimpanzee’s position is selected for the update. If $\mu >0.5$, the current chimpanzee position is updated using Equation (5). This iterative update method resembles the whale optimization algorithm.

4.

RBF Prediction Model

The RBF neural network is trained using a backpropagation algorithm, adjusting weights and biases iteratively to reduce error. Unlike BP networks, the RBF network requires optimization of parameters such as the center, width, and offset of its basis functions. RBF networks typically train faster than BP networks since they only adjust a limited number of parameters in a localized approach, avoiding BP’s global optimization requirement. BP networks may become stuck in local minima, while the RBF model, due to its basis function properties, effectively achieves global optimization. The RBF model is also more robust with noisy data due to its distance-based local response function, providing stable outputs. RBF excels in handling complex nonlinear relationships, especially in high-dimensional spaces, whereas RF primarily relies on ensemble decision trees, while RBF directly tackles nonlinearity through its radial basis functions.

As shown in Figure 5, the RBF neural network is a three-layer feedforward network with a single hidden layer, structured as depicted in Figure 5. The input layer parameters X_i are called design variables, while the hidden layer functions ${\mathrm{\Phi }}_{\mathrm{i}}$ are nonlinear mappings of these variables, known as basis functions. The output layer is a linear combination of these basis functions. The output function is expressed as follows [39]:

$$y=\sum {\mathrm{\omega }}_{\mathrm{i}}{\mathrm{\Phi }}_{\mathrm{i}}$$

(10)

In this Equation (10), $y$ represents the output response, and ${\mathrm{\omega }}_{\mathrm{i}}$ represents the $\mathrm{i}$-th linear weight. Commonly used basis functions for neural networks include multiquadratic functions, Cauchy functions, and Gaussian functions. In this study, the Gaussian activation function is applied to the neural network. The Gaussian radial basis function is defined as follows:

$${\mathrm{\Phi }}_{\mathrm{i}}=\mathrm{\phi }(\mathrm{r})=\mathrm{e}\mathrm{x}\mathrm{p}\left(-{\displaystyle \frac{r^2}{2{\sigma }^2}}\right)=\mathrm{e}\mathrm{x}\mathrm{p}\left(-{\displaystyle \frac{{‖x-c_i‖}^2}{2{\sigma }^2}}\right)$$

(11)

$‖·‖$ is the Euclidean norm, and $c_i$ represents the $\mathrm{i}$-th fixed center point. When the input data deviates from the center point, the value of the radial basis function rapidly decreases to zero. $\sigma$ represents the rate of reduction and is symmetric radially around the center. The unknown parameters $c_i$ in Equation (11) are determined through training. The RBF neural network structure can be designed using various algorithms, but selecting the RBF network’s centers is critical to the design. This study uses the k-means clustering method to determine center points iteratively, after which the size of all Gaussian RBF kernels can be calculated accordingly.

$$\sigma ={\displaystyle \frac{c_{max}}{\sqrt{2n_c}}}$$

(12)

$c_{max}$ represents the maximum distance between any two center points, and $n_c$ represents the number of center points. The linear weights ${\mathrm{\omega }}_{\mathrm{i}}$ between the hidden layer and the output layer can be obtained using the pseudoinverse matrix method:

$$\mathrm{\omega }=G^{+}\tilde{y}$$

(13)

In this Equation (13), $\tilde{y}$ is the expected response vector, and $G^{+}$ is obtained by solving the pseudoinverse of matrix $G$, defined as follows:

$$\left\{\begin{array}{c}G=\left\{g_{qi}\right\}\\ g_{qi}=exp\left(-{\displaystyle \frac{n_c}{c_{max}^2}}‖x_q-c_i‖\right),q=\mathrm{1,2},L,n;i=\mathrm{1,2},\cdots n_c\end{array}\right.$$

(14)

$x_q$ represents the input parameters.

2.3.2. Algorithm Flow

As shown in Figure 6, in the ChOA algorithm, each potential solution to the optimization problem is represented by a group of “chimpanzees”, with each chimpanzee corresponding to a position in the search space. The ChOA-optimized model was set with a maximum iteration step of 300, population size of 50, default aggressiveness factor of 1.0, and repulsion factor of 2.0. First, the parameters of the ChOA algorithm (such as position) are mapped to the RBF neural network, where the RBF network is trained to compute each chimpanzee’s fitness value. The chimpanzees are then randomly divided into independent groups, with the position of each chimpanzee calculated accordingly. In each iteration, the best-performing chimpanzee from each group is selected, and all chimpanzee positions are updated. This process repeats until a termination condition is met (e.g., minimum error or maximum number of iterations). Once the condition is met, the algorithm outputs the global optimal position of the best chimpanzee. For the RBF neural network optimization, data are first input and randomized, and the RBF network’s topology is determined. Then, the chimpanzees and associated parameters are initialized, and the global extremum is recorded. With each ChOA algorithm iteration, the global optimal position is ultimately mapped to the RBF neural network, forming an optimized model based on the global optimal solution. This model, once trained, can predict the output vector.

2.3.3. Model Construction and Evaluation

Root mean square error (RMSE) is a commonly used metric for measuring prediction accuracy in continuous data models. It reflects the root mean square difference between predicted values and actual values, indicating the average deviation of the predictions from the true values. RMSE is one of the most commonly used performance evaluation metrics in regression prediction tasks. A smaller RMSE indicates lower prediction error and higher prediction accuracy. An RMSE of zero means the model’s predictions were completely accurate. Additionally, when comparing the predictive performance of different models or datasets, RMSE is advantageous because it is in the same units as the original data, allowing for a fair comparison of actual error sizes. Therefore, RMSE is selected as the primary evaluation metric for the prediction model in this study. The mathematical expression of the RMSE loss function is shown in Equation (15), where $n$ is the number of samples, $y_i$ is the true value, and ${\widehat{y}}_i$ is the predicted value. In addition to RMSE, Mean Absolute Error (MAE), Mean Bias Error (MBE), and the coefficient of determination (R²) were also used to comprehensively evaluate the model’s predictive performance.

$$\mathrm{RMSE} =\sqrt{{\displaystyle \frac{1}{n}}\sum _{i=1}^n{\left(y_i-{\widehat{y}}_i\right)}^2}$$

(15)

2.3.4. Modeling the Relationship Between Fruit Hardness and Mechanical Gripper Force

After obtaining the fruit’s hardness using the prediction model, the gripping force of the mechanical claw is adjusted according to the real-time prediction of the fruit’s hardness. A key challenge in this relationship is that the fruit hardness and the gripping force of the mechanical claw are not directly proportional. Using the predicted hardness value to set the gripping force could potentially lead to fruit damage or unstable gripping. Therefore, we introduce a safety coefficient to refine this relationship.

In this model, the fruit’s hardness is predicted in real-time using machine vision, while the gripping force of the mechanical claw is adjusted through the safety coefficient. Specifically, the relationship between the gripping force and fruit hardness is expressed as

$${\mathrm{F}}_{\mathrm{c}}={\displaystyle \frac{\mathrm{F}}{\mathrm{s}}}$$

(16)

where Fc is the gripping force of the mechanical claw, F is the real-time predicted fruit hardness, and s is the safety coefficient, set at 1.3 in this study. The selection of the safety coefficient depends on various factors, including the fruit’s physical properties, the prediction error in hardness, and the need to avoid fruit damage. By appropriately adjusting the safety coefficient, the gripping force can be optimized, ensuring the integrity of the fruit while improving the efficiency and accuracy of the mechanical harvesting process.

2.4. Visual Recognition and Experimental Design

After determining the model, we need to perform recognition and experimentation. The data were collected at the Wolin Blueberry Base in Qingdao, using a Canon camera for shooting. The recognition device was the Canon camera from Canon Corporation in Tokyo, Japan (R6 MARK II) used by the research institute. During the experimental validation process, we used visual recognition equipment to collect blueberry fruit data from different greenhouses and seasons. Under consistent lighting conditions, images of mature fruits were captured from different angles (front view, side view, and bottom view), as shown in Figure 7.

To ensure the richness of the dataset, we applied data augmentation methods such as rotation, scaling, contrast enhancement, and noise addition in random combinations to the collected images. Through data augmentation, we obtained 4210 images, which were used as training images for maturity model recognition and for parameter extraction. In this study, we classified blueberries into three categories: mature, semi-mature, and immature. Mature blueberries are dark purple, large in size, and have a deep overall color. Considering real harvesting situations (taste, sales, transportation, texture), we selected fruits with a maturity score of 0.9–0.95 (fully ripe fruits have a score of 1.0) for image recognition. Fruit diameter, thickness, and other relevant parameters were extracted from the images and used as inputs for real-time fruit hardness prediction. The model’s prediction data were recorded after operation. After the recognition, the fruits were harvested and tested using a test machine, as shown on the right side of Figure 7. The test machine measured the maximum hardness the fruit could withstand through a pressure test, with an accuracy of 0.01 N. The recognition, harvesting, and testing processes for fruits of the same maturity were carried out simultaneously to avoid fruit spoilage due to storage.

3. Results and Discussion

3.1. Model Prediction Results

3.1.1. Results and Analysis

In this study, the 400 experimental data points were randomly shuffled and divided into a training set and a testing set at a 4:1 ratio. During model prediction, the fruit diameter, thickness, and weight from the training set were used as input samples and fed into the prediction model. The output was the predicted fruit hardness. The model’s prediction results are shown in Figure 8, with the X-axis representing the true values and the Y-axis representing the predicted values. The closer the data points are to the center line, the more accurate the prediction. The green range indicates that the results are within a 5% error margin. From Figure 8a, it can be observed that the ChOA-RBF model consistently produces errors within 5% for the predicted samples, significantly outperforming other models. The ChOA-RBF model shows the best overall performance. As model accuracy decreases, the data points with larger errors fall outside the green range, and the number of data points outside the 5% error range indicates lower accuracy, with the BP model showing the lowest accuracy.

After the model training was completed, the testing set data were input into the model to test the accuracy of the predicted fruit hardness. The results, shown in Figure 9, indicate that the ChOA-RBF model’s predicted values are closer to the real measured values, with smaller errors. In contrast, the other five models exhibit poor fitting for individual data points.

The evaluation metric calculation results for the model are shown in

Table 1. Comparison of different evaluation metrics for six prediction models on the training and testing sets.

Models	Evaluation Index
	Training Set				Testing Set
	RMSE	MAE	MBE	R2	RMSE	MAE	MBE	R2
ChOA-RBF	0.5963	0.4124	0.0184	0.8313	0.5490	0.4123	−0.0120	0.7910
ChOA-BP	0.7238	0.5114	0.0487	0.6992	0.7206	0.4931	0.0651	0.6233
ChOA-RF	0.6223	0.4729	0.0201	0.7952	0.5573	0.5211	−0.0146	0.7655
RBF	0.7117	0.4908	−0.0283	0.7175	0.6709	0.5501	−0.0153	0.5845
BP	0.7504	0.5963	−0.0624	0.6693	0.8135	0.6384	0.0565	0.5778
RF	0.7265	0.5287	0.0517	0.6776	0.7637	0.4826	0.0634	0.7134

. It can be seen that the ChOA-RBF prediction model performs the best across all indicators. The root mean square error (RMSE) values for the training and testing sets are 0.5963 and 0.549, respectively, while the measured hardness values range between 8.3 and 12 N, meeting the precision requirements of this study. Additionally, the MAE and MBE values for the ChOA-RBF model are both less than 1, significantly outperforming the other five prediction models. Regarding the coefficient of determination (R²), a value greater than 0.7 is typically considered indicative of a good fit. The R² value for the ChOA-RBF prediction model in the training set is 0.8313, with the deviation between the testing and training sets being less than 10%, indicating that the model has good generalization ability. From the comparison of performance parameters, the model performance ranking is as follows: ChOA-RBF > ChOA-RF > RBF > ChOA-BP > RF > BP. Furthermore, by comparing the performance parameters with and without ChOA optimization, it is clear that the optimization process improves the accuracy of the model.

Finally, an additional 40 sets of measured blueberry physical property data (note: these 40 sets of data include fruits from greenhouses in May and August) were input into four different prediction models to obtain the predicted fruit hardness values. The predicted values were compared with the real measured values, and the relative errors are shown in Figure 10. The ChOA-RBF model exhibited the smallest error. Among the 40 real data points, the maximum error was 8.3%, with the overall performance being the best. Therefore, in the following studies, the ChOA-RBF model is adopted, using blueberry parameters such as diameter, thickness, and weight as inputs to perform real-time prediction of fruit hardness, providing data support for the precise control of the mechanical claw’s gripping force.

3.1.2. Orthogonal Test Validation

To verify the validity of the ChOA-RBF model, an orthogonal experiment was conducted [40]. In the experiment, the blueberry fruit had a diameter range of 13–25 mm, a thickness range of 8–15 mm, and a weight range of 1–5 g. The orthogonal experimental design generated 27 data points, which were input into the ChOA-RBF training model for prediction. The output results are shown in

Table 2. Orthogonal experiment table, comparison between actual values and model prediction results.

No.	Diameter (mm)	Thickness (mm)	Weight (g)	Predicted Data (N)	Actual Data (N)	Absolute Error (N)	Relative Error (%)
1	13.07	8.02	1.21	8.35	8.44	−0.09	−1.02
2	13.19	9.45	1.25	8.89	8.81	0.08	0.89
3	13.31	9.56	1.28	8.37	8.28	0.09	1.03
4	13.54	9.56	1.31	8.76	8.63	0.13	1.47
5	14.52	11.02	1.58	9.00	8.94	0.06	0.68
6	14.19	9.56	1.72	8.67	8.62	0.05	0.62
7	14.77	10.24	1.91	8.81	8.78	0.03	0.33
8	15.25	9.57	1.95	9.18	9.16	0.02	0.27
9	15.28	10.19	1.98	9.78	9.61	0.17	1.76
10	16.46	10.17	2.15	9.52	9.73	−0.21	−2.19
11	16.65	12.01	2.25	10.06	9.84	0.22	2.23
12	16.81	11.73	2.24	10.52	10.19	0.33	3.21
13	17.74	10.74	2.76	10.17	10.08	0.09	0.85
14	18.03	11.15	3.17	10.48	10.51	−0.03	−0.27
15	18.65	12.10	3.24	9.76	9.63	0.13	1.35
16	19.05	13.14	3.83	9.87	9.87	0	0
17	19.97	12.24	3.19	9.73	9.63	0.1	1.03
18	20.07	13.43	3.90	9.53	9.39	0.14	1.47
19	20.36	12.61	3.57	10.35	10.24	0.11	1.03
20	20.38	12.19	3.61	12.07	11.64	0.43	3.68
21	21.10	11.19	3.92	11.07	11.26	−0.19	−1.69
22	21.36	12.78	4.23	11.52	11.83	−0.31	−2.62
23	21.44	12.71	3.98	11.43	11.24	0.19	1.66
24	22.96	14.73	4.34	12.45	12.17	0.28	2.34
25	23.01	14.01	4.21	11.89	11.73	0.16	1.38
26	23.89	14.83	4.88	11.71	11.62	0.09	0.75
27	24.77	14.16	5.46	12.95	12.51	0.44	3.54

. To ensure the consistency of data accuracy, all values are rounded to two decimal places. By comparing the predicted values from the orthogonal experiment ChOA-RBF model with the actual measured values, the maximum relative error was found to be 3.68%, the minimum absolute error was −0.31 N, and the maximum absolute error was 0.44 N. Under the previously defined safety factor, the error range is acceptable. It can be observed that when the absolute error is positive, the model predicts a higher maximum pressure tolerance than the actual measured value. This means the predicted value exceeds the fruit’s rupture pressure, and using this value directly as the gripping force for the mechanical claw would risk fruit rupture. If the error is too large, the fruit could be crushed. When the absolute error is negative, it indicates that the model predicts a lower maximum pressure tolerance than the actual value. In this case, the gripping force of the mechanical claw would be less than the rupture pressure of the fruit, making the harvesting process relatively safe. Although the fruit is less likely to be crushed, if the error is too large, the gripping force might be insufficient, resulting in failed harvesting. Therefore, the safety factor is critical in the relationship between fruit hardness and the mechanical claw’s gripping force. Moreover, to address potential issues from positive or negative errors, the incorporation of flexible materials and curved surface designs in the mechanical claw structure to improve gripping efficiency and safety is recommended. Experimental validation showed that the relative errors between the actual measured results and the orthogonal table predictions were all less than 5%. This outcome indicates that the prediction accuracy of the model is within an acceptable range, confirming the effectiveness and reliability of the ChOA-RBF model in predicting blueberry fruit hardness.

3.2. Two-Parameter Method

3.2.1. Design Principles

In the blueberry harvesting process, a visual recognition device can quickly determine the fruit’s diameter and thickness, but the fruit’s weight cannot be directly obtained through image recognition. This requires either manual input or a comparative prediction to estimate the weight. This step adds an additional computational process, increasing the operational complexity and time cost. Furthermore, as identified in Section 2.2.1, the weight of the fruit is most strongly correlated with its diameter and thickness—the larger the fruit, the heavier it tends to be. However, the correlation between weight and hardness is relatively weak, with a correlation coefficient of 0.72. Similarly, in rows 2, 7, 22, and 26 of the data, it was found that fruits with similar weights had different diameters and thicknesses, yet the hardness of the fruits varied significantly. This further confirms that fruit hardness is closely related to its diameter and thickness, while the correlation with weight is weaker.

Therefore, this study adopts the “two-parameter method”, which uses only the fruit’s diameter and thickness to predict and verify blueberry hardness. This reduction in the number of parameters not only improves work efficiency and simplifies the harvesting robot’s operation process but also maintains the accuracy and reliability of the prediction.

3.2.2. Prediction Results and Validation

The two-parameter method was used to re-establish the mapping relationship between fruit diameter, thickness, and fruit hardness. The 400 data points were input into the ChOA-RBF training model. After training with the same model parameters, the prediction results for the training and testing sets are shown in Figure 11a and Figure 11b, respectively. It can be observed that the data points are mainly concentrated on the lower side of the X = Y-axis, with a few points slightly outside the range. The predictions made using the two-parameter method are more conservative, with predicted fruit hardness values slightly lower than the actual measured values. The error, when compared to the three-parameter method, has decreased, but the error range remains within 6%.

Table 3. Comparison of evaluation metrics for the ChOA-RBF model using the two-parameter and three-parameter methods on the training and testing sets.

Methods	Evaluation Index
	Training Set				Testing Set
	RMSE	MAE	MBE	R2	RMSE	MAE	MBE	R2
Three-parameter	0.5963	0.4124	0.0184	0.8313	0.5490	0.4123	0.0565	0.7910
Two-parameter	0.7013	0.5138	0.0194	0.7076	0.6328	0.5261	0.0571	0.6534

compares the evaluation metrics of the two-parameter method with the three-parameter method. It can be seen that after removing the “weight” parameter, the R² of the two-parameter method decreased by about 0.13, and the RMSE value increased by about 0.1, resulting in a slight decrease in the accuracy of the evaluation metrics. The results show that the MAE and MBE increased by 0.1 and 0.01, respectively, indicating a slight decline in the performance of the two-parameter model compared to the three-parameter model.

To further compare the differences between the two-parameter method and the three-parameter method, we compared the iteration process and actual prediction results of the models. The iteration results are shown in Figure 12a.

Due to the removal of the “weight” parameter, the two-parameter method had a faster convergence rate, reaching convergence in 107 steps with an optimal fitness value of 0.10213. The three-parameter method, on the other hand, reached convergence in 188 steps with a fitness value of 0.10207. Although the three-parameter method had a slight advantage in accuracy, it required more time and computational resources. Additionally, we selected 20 other blueberry samples for prediction and experimental comparison, with results shown in Figure 12b. The results show that the prediction error of the two-parameter method remains within 8%. Although it is higher than that of the three-parameter method, the impact on practical applications is negligible, while the speed improvement of the algorithm significantly outweighs the slight increase in error. Moreover, considering the actual working efficiency, the three-parameter method requires calculations and predictions for weight, whereas the two-parameter method simplifies the computation steps, reducing recognition time and providing faster execution speed. Therefore, despite a slight decrease in prediction accuracy, the two-parameter method remains a feasible and effective solution, offering higher practicality in real-world blueberry harvesting robot applications. While the evaluation metrics in Table 3 show a slight increase in error for the two-parameter method, the actual predictions reveal that the difference between the two methods is not significant.

3.3. Model Recognition and Application

After confirming the ChOA-RBF prediction model and applying the two-parameter method, further experimental validation was conducted. As described in Section 2.4, we selected fruits with a maturity index of 0.95 and extracted parameters (diameter and thickness) from the recognition images for real-time prediction. These fruits were then harvested, and a pressure test was conducted using a test machine. The predicted results were compared with the actual measured values. The results for 40 blueberry samples, collected from different perspectives and different greenhouses, are shown in

Table 4. Comparison between predicted results from the ChOA-RBF model using the two-parameter method and actual measurements from the test machine based on the recognition images.

No.	Diameter (mm)	Thickness (mm)	Actual Result (N)	Predicted Value (N)	Error (N)	Relative Error (%)
1	15.27	10.73	9.40	8.72	0.68	7.23
2	22.68	12.24	11.54	11.13	0.41	3.55
3	15.61	13.16	10.63	10.93	0.30	2.82
4	19.32	14.13	9.31	9.21	0.10	1.07
5	14.26	9.39	8.48	8.62	0.14	1.66
6	18.84	14.91	10.23	10.45	0.22	2.15
7	13.71	9.58	8.51	8.32	0.19	1.98
8	13.15	12.38	9.98	10.50	0.52	5.21
9	15.28	12.96	9.81	10.12	0.31	3.16
10	16.67	12.41	8.80	8.18	0.62	7.05
11	19.38	12.13	10.82	10.58	0.24	2.22
12	18.52	12.69	11.23	10.96	0.27	2.40
13	15.13	10.37	10.01	10.73	0.72	7.19
14	18.14	13.31	10.53	9.92	0.61	5.79
15	19.53	14.44	11.40	10.82	0.58	5.09
16	13.41	9.36	8.25	8.43	0.18	2.18
17	15.31	9.91	9.36	8.74	0.62	6.62
18	16.65	11.92	8.80	9.18	0.38	4.32
19	13.59	10.97	9.01	9.32	0.31	3.44
20	18.25	12.14	10.08	9.87	0.21	1.73
21	13.95	9.69	8.20	8.61	0.41	5.00
22	14.71	11.04	12.25	11.65	0.60	4.90
23	14.25	12.15	10.32	9.91	0.41	3.97
24	18.73	11.9	8.54	8.53	0.01	0.12
25	17.79	12.49	9.90	10.39	0.49	4.95
26	15.31	9.96	8.71	9.24	0.53	6.08
27	15.38	11.87	9.17	9.08	0.09	0.98
28	17.54	11.86	12.20	11.36	0.84	6.89
29	20.65	13.92	11.86	11.45	0.59	4.97
30	13.18	10.33	7.86	8.12	0.26	3.31
31	12.91	10.10	8.82	8.17	0.65	7.37
32	15.83	12.48	9.44	9.91	0.47	4.98
33	17.38	12.77	9.32	9.81	0.49	5.26
34	12.83	9.60	8.20	8.16	0.04	0.49
35	16.74	10.89	10.14	9.87	0.27	2.66
36	15.24	12.15	8.43	8.91	0.48	5.69
37	22.76	14.43	12.26	11.66	0.60	4.89
38	22.59	14.46	11.99	11.57	0.42	3.50
39	13.01	9.86	8.21	8.22	0.01	0.12
40	18.24	13.15	10.58	10.97	0.39	3.69
AVG	16.54	11.86	9.81	9.76	0.39	3.91

. It can be observed that the prediction errors between the predicted values and actual values are all within 1 N, with relative errors under 8%. Compared to the experimental errors conducted earlier, these data variables were obtained through visual recognition, and there is some discrepancy when compared to the actual data measured with a caliper. The final result’s hardness value error is likely caused by the measurement errors in the fruit diameter and thickness extracted through recognition. Although there are a few data points with relatively large errors, the maximum relative error was 7.37%, and the average relative error was 3.91%. This error range is acceptable in practical applications when the safety factor is applied.

It is important to note that the subject of this study is soft-textured blueberries, which are small fruits that are highly prone to damage due to their soft texture, thus increasing the difficulty of harvesting. Therefore, the applicability of this research to larger, harder fruits like pears or apples may be limited. However, this study provides a unique approach to solving the issue of mechanical claw damage during fruit harvesting.

Additionally, this research did not further consider the impact of factors such as leaf obstruction, recognition from different positions, and errors in feature extraction. Other factors like time-domain variations, pests, and different varieties’ effects on fruit hardness were also not addressed. In future optimizations of harvesting robot designs, these factors may need further investigation and consideration to improve the accuracy and reliability of the system.

4. Discussion

With the rise in labor costs and the advancement of agricultural technology, mechanized crop harvesting is bound to become the future trend of agriculture. However, the development of blueberry harvesting robots has been slow due to the fruit’s clustered growth and soft texture. Additionally, blueberries have low fruit hardness and fragile skin, making them easily damaged by conventional mechanical claws. This necessitates the flexible design and precise control of mechanical claws. This study addresses the challenge of non-destructive harvesting for soft fruits by proposing a method that extracts blueberry features through a visual recognition device and predicts fruit hardness. Based on the predicted hardness, the mechanical claw’s gripping force is dynamically adjusted to prevent fruit damage and achieve precise, non-destructive harvesting. We first trained and simulated ChOA optimized RBF, BP, and RF models to compare their differences and detection accuracy. To further verify the simulation’s accuracy, we conducted orthogonal experiments. The results revealed the low correlation of the weight parameter and its complexity in extraction, prompting us to propose the simpler and faster two-parameter model, which estimates fruit hardness using only diameter and thickness.

Subsequent experimental validation and evaluation demonstrated that the two-parameter model improves execution speed while maintaining prediction accuracy compared to the three-parameter model. However, we also observed that the recognition device and prediction accuracy still require further optimization. As visual algorithms advance, recognition efficiency and accuracy can be continuously improved. Moreover, large amounts of data revealed the presence of positive and negative prediction errors. Positive errors indicate that the gripping force may crush the fruit, while negative errors suggest insufficient gripping force, potentially causing fruit slippage. We addressed this issue by introducing a safety factor in our study. In practical applications, the use of flexible materials and curved mechanical claw designs could further enhance harvesting efficiency and accuracy. These aspects will be considered in our future design and development.

In this study, we focused on rabbit-eye blueberries, a representative variety widely grown in northern greenhouses and southern open fields. In future extensions, we plan to incorporate different varieties and planting environments as parameters to optimize our model. This will enable the model to automatically adjust to various harvesting conditions across different varieties and regions. The proposed method could also be applied to other soft fruits with fragile skins, such as waxberries and peaches. By simulating human decision-making, the mechanical claw dynamically adjusts gripping force based on external features and image details, preventing damage. This image-based fruit hardness prediction method provides a new approach to non-destructive harvesting and offers a theoretical foundation for the dynamic adjustment of gripping force in harvesting robots.

5. Conclusions

The automatic harvesting of blueberries faces challenges due to the soft texture and fragility of the fruit. Therefore, this paper proposes a method that extracts blueberry features through a visual recognition device and predicts fruit hardness. Based on the predicted blueberry hardness, the gripping force of the mechanical claw is preset to prevent fruit damage and achieve precise, non-destructive harvesting. This method effectively solves the problem of damage between the harvesting device and the fruit. The main research conclusions of this paper are summarized as follows:

1.

A New Approach for Predicting Fruit Hardness: A novel method is proposed to directly extract the fruit’s pressure resistance from the blueberry image, identifying the correlation between fruit color and hardness. By predicting the hardness, the gripping force of the mechanical claw can be adjusted. Calculations and experiments demonstrate that this design is feasible and can improve harvesting efficiency while preventing fruit damage.

2.

Mapping Physical Properties to Hardness Using the ChOA-RBF Model: The ChOA-RBF model is applied to establish the mapping relationship between physical properties and hardness, compared with RBF, BP, and RF models from different angles. Evaluation metrics and error plots indicate that the ChOA-RBF model has significant advantages in fruit hardness prediction. Orthogonal experiments confirm that the maximum error of this model does not exceed 5%.

3.

Two-Parameter Method as a Practical Solution: Considering the limitations of the visual recognition device and practical operational applications, a two-parameter method is proposed as an alternative. Though the R² value and some parameters slightly decrease, the actual prediction error remains under 8%. Furthermore, the method results in faster execution speed and higher harvesting efficiency in practical applications.

4.

By comparing the model recognition results with the actual test results, it was found that there is some discrepancy, which is caused by slight errors in the parameters extracted by the recognition device. However, the final result shows that the maximum relative error is less than 8%, with an average relative error of 3.91%, which is acceptable within the error range considering the safety factor of 1.3.