Automatic Estimation of Apple Orchard Blooming Levels Using the Improved YOLOv5

Abstract

The estimation of orchard blooming levels and the determination of peak blooming dates are very important because they determine the timing of orchard flower thinning and are essential for apple yield and quality. In this paper, we propose an orchard blooming level estimation method for global-level and block-level blooming level estimation of orchards. The method consists of a deep learning-based apple flower detector, a blooming level estimator, and a peak blooming day finding estimator. The YOLOv5s model is used as the apple flower detector, which is improved by adding a coordinate attention layer and a small object detection layer and by replacing the model neck with a bidirectional feature pyramid network (BiFPN) structure to improve the performance of the apple flower detector at different growth stages. The robustness of the apple flower detector under different light conditions and the generalization across years was tested using apple flower data collected in 2021–2022. The trained apple flower detector achieved a mean average precision of 77.5%. The blooming level estimator estimated the orchard blooming level based on the proportion of flowers detected at different growth stages. Statistical results show that the blooming level estimator follows the trend of orchard blooming levels. The peak blooming day finding estimator successfully positioned the peak blooming time and provided information for the flower thinning timing decision. The method described in this paper is able to provide orchardists with accurate information on apple flower growth status and is highly automated.

1. Introduction

Flower thinning is beneficial to improving apple yield and fruit quality, and choosing a reasonable timing of flower thinning helps to reduce crop load levels, improve fruit tree disease resistance, reduce yield fluctuations, and improve bud quality [1,2,3]. Mechanical [4] and chemical [5] flower thinning has been widely used to improve the automation of the flower thinning process and reduce labor intensity, but the judgment of flower thinning timing has not achieved automation. Because apple flower growth levels are influenced by weather and temperature and natural environmental factors, the timing of flower thinning remains an unpredictable part of apple production with large variations within years and from year to year [6]. Generally, orchard blooming levels are monitored by growers counting a limited number of fruit trees in the orchard within a selected location, a method based on visual estimation that is not representative and time-consuming [7]. Computer vision and deep learning techniques can effectively gather information on orchard blooming levels to help farmers’ decision-making and reduce reliance on experience.

In the refinement of intelligent orchard management, apple flower/fruit identification and growth stage estimation based on image information have received considerable attention from many scholars. Meanwhile, computer vision techniques have been widely used in crop pest and disease identification [8], fruit detection [9], and automatic detection of flowering stages [10]. In early work on apple flower detection, traditional machine vision methods based on color and threshold were widely used. The authors of [11] analyzed flower density variation in orchards based on image processing methods that adjust color thresholds to determine the correlation between flower density and orchard yield. The authors of [12] developed a method for adjusting the hue, saturation, and brightness of the image in order to estimate the number of flower clusters of an individual fruit tree in a high-density apple orchard. However, the limitations of the above-mentioned color and threshold-based methods are the dependence on lighting conditions, poor robustness of the algorithm, and the manual-based adjustment of threshold parameters, which makes this method inapplicable in actual production practice [12]. Obviously, the use of manual feature extraction and color thresholding-based methods is challenging when it comes to distinguishing apple flowers with different phenological periods and shaped contours in complex natural environments. The application of UAV remote sensing images to analyze apple tree flowering intensity has recently received attention, and the authors of [13] explored the potential of UAV RGB high-resolution imagery to measure flowering intensity. Image segmentation techniques were used to segment the white pixels, which corresponded to apple blossoms in orthophotos and single photographs. The results showed a maximum correlation of 0.54 between white pixels and flower clusters for each tree. Those results show the complexity of using UAV images. The authors of [14] mapped the cluster density of apple flowering by processing unmanned aerial vehicle (UAV) images using binary classification and K-nearest neighbor algorithms, respectively. The results show that flower cluster detection is limited by various factors, such as weather, canopy volume, flowering phenology, intercropping, soil color, and the number of bands to be analyzed. The authors of [13] suggested that there are still other methods to be researched before discarding the use of UAV RGB images for estimating flowering intensity.

In agriculture, uncontrollable natural environmental conditions and complex backgrounds cause difficulties in the application of traditional machine vision methods. With the development of deep learning, convolutional neural networks (CNN) are widely used for classification [15], detection [16], and segmentation [17] of agricultural products for their efficient automatic feature extraction capability and high stability. In terms of estimating the phenological stages of flowers and fruits using deep learning techniques, the authors of [18] proposed an improved YOLOv3 model for detecting apples at three growth stages: young, swollen, and ripe in orchards with fluctuating illumination and complex backgrounds. The authors of [19] proposed a lightweight CNN named Fusion-YOLO (F-YOLO) and achieved the detection of tea chrysanthemum at three growth stages, budding, early flowering, and full-blooming, with the highest detection average accuracy of 89.53%. From the above work, it could be seen that the deep learning technique shows high accuracy and robustness in the detection of agricultural products at different phenological stages.

Therefore, deep learning algorithms are applied to the classification [20], detection [21], and segmentation [22,23,24] of apple flowers. The authors of [25] proposed a CNN+SVM-based model for segmenting apple flowers in images, which outperforms methods based on color analysis and verifies that the hierarchical features extracted by CNN can effectively combine color and morphological information. The authors of [26] implemented the Mask R-CNN algorithm to perform instance segmentation of apple flowers. Different image enhancement techniques were implemented, and it was verified that the implementation of image augmentation was essential to reduce the validation loss and improve the detection accuracy of the segmentation algorithm. The network achieved an average precision (AP) of 0.86 in the test dataset. The authors of [27] proposed the FCNs-Edge model for generating apple flower density mapping, and FCNs-Edge implemented a novel end-to-end apple flower segmentation algorithm, which was able to obtain pixel-level F1 scores of up to 85.6%. The authors of [28] used generative modules and various image preprocessing methods for data enhancement of the apple flower dataset. The pruning inference proposed by the authors can automatically deactivate part of the network structure according to different conditions and reduce the network parameters. The model can achieve 90.01%, 98.79%, and 97.43% in precision, recall, and mAP, respectively, in detecting the apple flowers. The authors of [29] proposed a weakly supervised flower/fruit counting network based on deep learning named CountNet, implementing flower and fruit counting of apples. The aforementioned work focuses on the segmentation of apple flowers at pixel level and object level detection at individual growth stages in fruit trees; however, this paper is more concerned with the estimation of blooming levels in orchards.

In previous work on apple orchard blooming level estimation, the authors of [30] proposed a system capable of estimating blooming intensity and peak blooming date from tree images. The system has three stages: a visual flower detector based on a deep convolutional neural network, followed by a blooming level estimator, and a peak blooming day finding algorithm. The trained detector was able to detect flowers on trees with an average precision score of 0.68. However, there is still much room for improvement in the performance of the apple blossom detector, and the ‘on-sight estimator’ for blooming intensity estimation, which uses the average blossom size as one of the explanatory variables, introduces instability into the estimator because the flower size in the image is affected by the angle and distance of the shot. The authors of [31] suggested that further segmentation of blossom areas could contribute to the extraction of detailed growth information of apple flowers. Therefore, the authors proposed an instance segmentation model that improved Mask Scoring R-CNN (MASU R-CNN) with a U-Net backbone for detecting and segmenting apple flowers in bud, semi-open, and fully open at three different growth statuses. Although instance segmentation of apple flowers for different growth states provides richer information on apple flower morphology, according to the discussion in [30], the estimation of blooming levels in global apple orchards is based on statistics of apple flowers at different growth stages. The authors of [7] proposed a CNN-based DeepPhenology model to implement an 8-stage apple flower phenology distribution estimation, which is able to effectively map apple flower distribution on an image-level, row-level, and block-level without labeling individual flower clusters. The results show an average Kullback–Leibler (KL) divergence value of 0.23 over all the validation sets. Since the input to the model is limited to a quadrangle, information on the flowering status of certain parts of the whole tree is lost.

Based on the above research and discussion, an algorithm that can capture information on the blooming level of apple flowers of the whole fruit tree at a larger scale and automatically and stably estimate the global or block-level blooming level of an orchard is needed for the production practice of orchard flowering management. Therefore, this paper proposes an automatic orchard blooming level estimation algorithm, which consists of three parts: an apple flower detector, a blooming level evaluator, and a peak bloom day finder. First of all, we improve the YOLOv5s model and use the improved model as an apple flower detector. Based on this, we tested the robustness of the apple flower detector under different light conditions and examined the interannual knowledge transfer ability of the apple flower detector using the apple flower datasets for the years 2021 and 2022. The blooming level estimator achieved block-level blooming level estimation in orchards, and the peak blooming date was pinpointed by the peak blooming day finding estimator. In the end, the detection results of the apple flower detection model before and after the improvement were analyzed, and the performance of the detection model proposed in this paper was compared with that of the baseline model.

2. Materials and Methods

2.1. Definition of Blooming Level

To achieve a block-level or global estimate of orchard blooming levels and to illustrate the inputs for the apple flower detection network, it is necessary to define blooming levels according to the morphology of apple flowers in different phenological periods. Generally, orchardists categorize the growth state of apple flowers into three stages: ‘Bud’, ‘Full Flowering’, and ‘Withered Flower’. However, this division does not provide a fine estimation of apple flower phenological distribution.

The buds of apple flowers are mixed buds that sprout in spring and produce a new section of tips. In the Yellow River basin, apple tree flowering is mostly concentrated in April, and the flowering period lasts for about 2–3 weeks. Based on the above growth characteristics of apple flowers and the way of dividing the phenological stages proposed by [7], we have adopted a definition of six stages of blooming levels, and the growth states of apple flowers at each blooming level are shown in Figure 1. Initially, the buds of apple flowers are small, dense, and green in color, which are called the ‘Spitting buds’ (Figure 1a). As the pedicels elongate and the buds gradually swell, the buds take on a white or pinkish globular shape. However, at this stage, each bud remains close together and the buds are generally spherical in shape, which is called ‘Tight Clusters’ (Figure 1b). In the next stage, the buds gradually spread out and swell, and each cluster of apple flowers will have 5–6 white or pink balloon-like buds, denoted the ‘Balloon Blossom’ stage (Figure 1c). Apple flowers in the ‘Balloon Blossom’ stage will soon enter the next stage, called the ‘King Bloom’ stage (Figure 1d). During the ‘King Bloom’ stage, the central flower in a cluster is fully bloomed, while the other flowers in the cluster remain in balloon-like buds and grow around the central flower. The next stage is the gradual opening of the bulb around the queen flower, which is called the ‘Full Bloom’ stage (Figure 1e). When there are faded petals in a cluster of apple flowers, this stage is called the ‘Petal Fall’ stage (Figure 1f). As opposed to [7], the flowering potential of the buds is unknown, considering the effect of the previous year’s thinning of fruits and flowers. Heavy fruiting can also partially or completely inhibit the formation of flower buds [32]. Meanwhile, the distinction between flower buds and leaf buds is commonly not obvious and hard to identify [20]. Therefore, the detection of buds at the stage before ‘SB’ was discarded in this study.

2.2. Image Collection and Dataset Building

2.2.1. Image Collection

The data were collected at Tianping Lake Experimental Demonstration Base of Shandong Fruit Tree Research Institute (117°2′ E, 36°13′ N), as shown in Figure 2a. The varieties of fruit trees in the orchard are Gala, Fuji, and Wanglin, which are planted in a modern dwarf stock dense planting pattern with a row spacing of approximately 3.5 m and a tree spacing of approximately 1.2 m.

To accurately record and observe the morphological changes of apple flowers throughout their growth cycle, we built an apple flower image acquisition device. The working conditions of this device and the structure of this device are shown in Figure 3. The apple flower image acquisition device mainly includes the track, mobile platform, for data collection, a Dell G3 computer (Intel i5-8300 CPU USA, 12G memory, NVIDIA GeForce GTX 1050Ti GPU USA), and a ZED (Stereolabs) camera. The device was set up in an observation site in an orchard, and by moving the platform, images of 20 adjacent fruit trees in the observation site can be acquired. The computer, ZED camera, and other filming equipment were mounted on a mobile image collection platform, which could be moved along the guide track. The ZED camera is mounted at 1.68m from the ground and 0.8 m from the apple trees at an angle of 0°–30° from the horizontal ground. We recorded video using the ZED Explorer and saved it in SVO format, the camera initialization parameters for video recording were: 720P video mode, 60 frames per second, a monocular resolution of 1280 × 720 pixels, and a depth range of 0.5–20 m. We used the “ZED Depth Viewer” software to capture images with significant differences in the video to ensure that richer data were covered. In this study, we also captured apple flower images outside the observation site using an iPhone7Plus mobile phone, and the shooting working state was consistent with the data acquisition device, and the resulting image resolution was 3024 × 3024 pixels.

In this study, a two-year dataset was collected in 2021–2022, with data collected at the same locations. We collected 1000 images in April 2021, and most of these images were collected at the ‘Full Bloom’ stage as well as the ‘Petal Fall’ stage in the orchard. In April 2022, we collected image data of apple flowers from budding to wilting.

Table 1. Summary of the dataset.

Years	Date	Blooming Level	Total Images	Weather
2021	11 April 2021	FB	203	Light Rain
	12 April 2021	FB	452	Cloud
	13 April 2021	FB/PF	382	Clear
	14 April 2021	PF	462	Cloud
2022	1 April 2022	SB	337	Clear
	2 April 2022	SB/TC	613	Clear
	3 April 2022	TC	221	Clear
	4 April 2022	TC/BB	493	Clear
	5 April 2022	BB/KB	297	Clear
	6 April 2022	KB/FB	321	Clear
	7 April 2022	FB	502	Clear
	8 April 2022	FB	236	Clear
	9 April 2022	FB	212	Clear
	10 April 2022	FB/PF	500	Cloud
	11 April 2022	PF	452	Cloud

Notes: See Figure 1 for an explanation of the blooming level abbreviations.

summarizes the collection time, orchard blooming level, number of images collected, and weather information for the dataset.

2.2.2. Dataset

Of the data collected in 2022, 1800 images were filtered and used for model training and validation, and the dataset is called ‘Apple Flower A’. We use the 2021 data for generalizability tests of the model for apple flower detection in different years, and this dataset is called ‘Apple Flower B’.

The large amount of image data helps to improve the performance of the apple flower detection model and enhance the feature extraction capability. Data augmentation can effectively expand the sample size of small-scale datasets while improving network generalization and avoiding overfitting. In this paper, the Apple Flower A dataset was divided into a training set and a validation set in a ratio of 8:2. The images in the training set data were adjusted using vertical flip, horizontal flip, random scaling, cutout, and brightness transformation. Finally, the training set image samples were expanded to 4000 images with no overlap between the training and validation sets. The augmented results are shown in Figure 4. The flower clusters were labeled using the LabelImg tool, in which the apple flowers in different growth stages and shading situations were surrounded by a minimum external rectangular box.

Figure 5 shows the apple flower image data collected in the natural environment. As can be seen in Figure 5, the background of apple flower images is complex, apple flowers in the ‘Spitting Buds’ and ‘Tight Clusters’ stages occupy few pixels in the images and are unclear features (Figure 5a). Flower clusters are heavily obscured by leaves and branches, and the uncertain lighting conditions in the orchard cause uneven image brightness in the dataset (Figure 5b).

2.3. Algorithm Process

Based on the apple tree blooming intensity and peak blooming date estimation systems proposed by [30], the apple flower growth state supervision algorithm in this study is divided into three parts: the apple flower detector, the blooming level estimator, and the peak blooming day finding estimator.

The results of the apple flower detector detection will be used as the input to the blooming level estimator and the peak blooming day finding estimator. For the apple flower detector, we improve the YOLOv5s model and use the improved model as the apple flower detector.

The blooming level estimator calculates the percentage of each blooming level stage to discriminate the apple flower growth status based on the results of the apple flower detector.

Finding the date of peak blooming is important because this date determines the timing of flower thinning and apple yield is affected by it. In current apple flower thinning practice, the peak blooming date is determined as the day when 80% of the trees in the entire orchard reach ‘King Bloom’ and ‘Full Bloom’ stage. The peak blooming date finder will make decisions for orchard thinning timing.

2.4. YOLOv5 Model

The performance of the apple flower detector directly affects the results of the blooming level estimator and orchard peak blooming day finder. Currently, the single-stage object detection algorithm YOLO [33,34,35,36] has been widely used in fruit and vegetable detection [37] and weed detection [38], which can balance speed and detection accuracy compared to the two-stage object detection algorithm. Therefore, in this study, the YOLOv5s model [39] was selected as the apple flower detection model in the v6.0 release of the YOLOv5 model.

The structure of the YOLOv5s model is shown in Figure 6, which is mainly composed of the backbone, neck, and head. The backbone part uses CSPdarknet53 as the feature extraction network, the neck part uses the FPN+PAN structure, which achieves the fusion of backbone network features by up-sampling and down-sampling operations, YOLO head implements the decoding of the input features and the prediction of the object.

2.5. Apple Flower Detection Model

Although YOLOv5 already has excellent object detection capability, it is still challenging to detect apple flowers in the unstructured environment of orchards and under complex background and lighting conditions. Improving the detection performance of the apple flower detector for each growth stage can provide a reliable basis for the blooming level estimator and the peak blooming day finding estimator. Therefore, based on the characteristics of the data set in this paper, we improved the YOLOv5s model.

2.5.1. Coordinate Attention Layer

In the early stage of apple flower growth, the smaller buds morphologically resemble the ends of branches of fruit trees, and the green buds are surrounded by green leaves, which makes it difficult to distinguish the buds from the background. The attention mechanism can make the model attend more to the object of interest in the image during the training process and improve the efficiency of feature extraction. Consequently, we expect to improve the network’s ability to mine budding features through the attention mechanism.

The coordinated attention (CA) mechanism [40] adds location information to the channel attention to allow the network to perform attention operations over a larger area, and to facilitate the attention module to capture spatial long-range dependencies with precise location information, this attention mechanism decomposes the global pooling into a pair of one-dimensional feature encoding operations to form direction-aware and position-sensitive attention feature maps to be complementarily applied to input features, the structure of which is shown in Figure 7.

The CA attention module is a flexible plug-and-play module that can improve network accuracy with a small overhead. The high-level features of the backbone network are rich in semantic information. To make the attention mechanism analyze the effective semantic information in the network, the CA attention module is inserted into the end of the backbone network in this paper.

2.5.2. Small Object Detection Head

Three scale feature layers exist in the YOLOv5s model. For the input image of size 640 × 640, feature maps of sizes 80 × 80, 40 × 40, and 20 × 20 are generated after 8×, 16× and 32× subsampled, respectively, and these feature maps are sent to the head section after feature fusion at the neck. The information of small object flowers in the image exists in the shallow features of the network. Obviously, too large a subsampled rate will destroy the flower features at the ‘Spitting Buds’ and ‘Tight Clusters’ stages, causing the model to detect less effectively.

A larger feature map can reduce the receptive field of the feature map in the input image, reduce the distortion of shallow feature information in the network, and improve the small object detection performance. In this paper, a small object detection head is added to the YOLOv5s network, this four-head structure can ease the negative influence caused by the violent object scale variance [41]. The Conv module is linked in layer 18 of the network to integrate the semantic information brought by the deep structure and continues to upsample and expand the feature map. At layer 21, the up-sampled feature map of 160 × 160 size is fused with the layer 2 feature map in the backbone network in a Concat fusion operation to obtain more shallow feature information.

2.5.3. Improved Feature Fusion Structure

The feature fusion structure of the YOLOv5s neck is a PAN+FPN structure [36,42], which enables the fusion of feature information with stronger semantic relationships in the deeper layers of the network with the rich location information in the shallow layers. When detecting apple flowers that are in a complex background, we hope to fuse the valid information from the backbone part of the network at the neck of the network to reduce the interference of background information. However, the PAN+FPN structure does not emphasize the apple flower targets of interest in the image during the feature fusion process after upsampling and subsampling. The bidirectional feature pyramid network (BiFPN) structure [43] can compensate for the shortcomings of the above feature fusion structures, and this paper combines the BiFPN structure to optimize the neck of the YOLOv5s network.

The BiFPN structure is shown in Figure 8. Compared with PAN+FPN, this structure removes nodes with only one input edge and adds an extra channel between the original input and output nodes at the same layer for more feature fusion. The traditional Concat fusion operation cannot pinpoint the features of interest and increases the number of parameters of the network. BiFPN introduces a simple and efficient weighted feature fusion mechanism, which introduces trainable weights to adjust the degree of contribution of different feature maps. BiFPN uses a fast normalization fusion strategy to normalize the weights to a value ranging from 0 to 1, improving the training efficiency. The weighted fusion method is shown in Equation (1).

$$O={\displaystyle {\sum }_i\frac{w_i}{\epsilon +{\displaystyle {\sum }_j{w}_j}}{I}_i}$$

(1)

where $I_i$ is the input feature map, $w_i$ and $w_j$ are the learnable weights of the input feature map, which are guaranteed to be greater than 0 by the ReLU activation function in each training, and $\epsilon =0.0001$ is added to the denominator to prevent numerical instability.

As shown in Figure 8 and Figure 9, after the addition of the small object detection layer of YOLOv5s, the feature fusion network takes the four different-scale feature maps output from layers 2, 4, 6, and 9 of the backbone network as the input to the BiFPN structure, respectively. With the improved neck, a path is added to layers P3 and P4 with cross-scale connections and weighted feature fusion at the feature fusion nodes. Equations (2) and (3) describe the two feature fusion processes in the P4 layer.

$$P_4^{td}=Conv\left(\frac{w_1{P}_4^{in}+w_{2}Resize(P_5^{in})}{w_1+w_2+\epsilon }\right)$$

(2)

$$P_4^{out}=Conv\left(\frac{w_1^{\prime }{P}_4^{in}+w_2^{\prime }{P}_4^{td}+w_3^{\prime }Resize(P_3^{out})}{w_1^{\prime }+w_2^{\prime }+w_3^{\prime }+\epsilon }\right)$$

(3)

where $Conv$ is the convolution operation; $Resize$ is used for upsampling or subsampled operations for resolution matching.

3. Results

3.1. Model Training

The hardware platform for this paper is configured as follows: NVIDIA GeForce RTX 3070 GPU with 8 GB of video memory and 24 GB of memory. All models are run on the Ubuntu 20.04 operating system, PyTorch is the deep learning framework, Python is the programming language, and CUDA is used to accelerate the GPU.

The 2022 apple flower dataset was used for the training of the apple flower detector.

Table 2. Model training parameters.

Training Parameters	Values
Input Image Size	640 × 640
Batch Size	4
Iterations	160,000
Momentum	0.937
Weight Decay Rate	0.0005
Initial Learning Rate	0.01

shows the detailed hyperparameter configuration during the model training. The input image is adjusted to 640 × 640 pixels and the batch size is set to 4. Experiments show that the loss of the model converges and smooths out after 160,000 iterations. The momentum, initial learning rate, and weight decay rate have been experimentally optimized.

3.2. Performance Evaluation of Flower Detectors

Detection of apple flowers in the natural environment requires consideration of the performance of the detection network for selecting the optimal detection model. The mean average precision (mAP) and average precision (AP) are used for model evaluation in this paper. mAP and AP are related to precision (P) and recall (R), and their relationships are shown in Equations (4)–(7).

$$P=\frac{TP}{TP+FP}\times 100\%$$

(4)

$$R=\frac{TP}{TP+FN}\times 100\%$$

(5)

$$AP={\displaystyle {\int }_0^{1}P(R)dR}\times 100\%$$

(6)

$$mAP=\frac{1}{M}{\displaystyle \sum _{k=1}^{M}AP(k)\times 100\%}$$

(7)

where $TP$, $FP$, and $FN$ indicate the number of true positives, false positives, and false negatives in the model detection results; $M$ is the total number of categories. Since the blooming level was divided into six stages, $M=6$ in this study.

3.3. Apple Flower Detection in Natural Environment

The light conditions of orchards in the natural environment are complex, and in the actual operation, the light conditions of orchards are influenced by the time of day, area, and weather, and uneven light can affect the apple flower detection model. Therefore, this study tested the performance of the optimized model on a dataset of apple flowers at various growth stages in different lighting environments. Figure 10 shows the detection results for six orchards with different blooming levels in a more illuminated environment. As can be seen from Figure 10, the model in this paper performs well in determining apple flowers at the ‘Full Bloom’ and ‘Petal Fall’ stages, and flowers disturbed by strong light are still successfully detected by the model even in images with strong light and drastic changes in light intensity. The flowers at the ‘Spitting Buds’ and ‘Tight Clusters’ stages are smaller and also have missed detection when they are obscured by branches and leaves, but most of them are still detected by the model.

At nightfall, the low light intensity in the orchard increased the difficulty of feature extraction. As can be seen in Figure 11, for apple flowers in the post-bud stage, the weaker light intensity had little effect on the detection results of the model, and the model still maintained high detection accuracy. However, detecting apple flowers at the ‘Spitting Buds’ and ‘Tight Clusters’ stages is challenging for the model. In Figure 11a,b, the number of missed detections by the model for apple flowers at the ‘Spitting Buds’ and ‘Tight Clusters’ stages increased with decreasing light intensity, and even false identifications occurred.

However, in general, the model proposed in this paper can provide accurate detection of apple flowers at all blooming levels with high robustness under different light, flower density, and occlusion conditions. The optimized model as an apple flower detector can meet the requirements of block-level or global blooming level estimation and peak blooming date finder for orchards.

3.4. Inter-Annual Knowledge Transfer

Different natural environmental conditions in different years can cause changes in the growth state of apple flowers, resulting in shifts in the distribution of image information in different years compared to the image data collected in 2022 in this study. Therefore, the ability to use a model trained in a specific year and to perform inference on the apple flower dataset from different years is important for the generalization of the model, which avoids retraining the model each year. This interannual knowledge transfer capability of the model is demonstrated by the generalizability of the model across years. In this paper, the apple flower detector trained on 2022 data was used to validate its effectiveness, using AP and mAP metrics to measure the model’s ability to generalize to different years of data.

We selected 200 images from each of the apple flower datasets in 2021 and 2022, respectively, for evaluation, and

Table 3. Results of evaluating the generalizability of flower detectors across years.

Years	mAP (%)	AP (%)
		FB	PF
2021	86.4	87.2	85.6
2022	86.6	87.5	85.8

shows the ability of the apple flower detector to generalize across years between 2021 and 2022. It is noteworthy that most of the images in the Apple Flower B dataset were collected after the ‘King Bloom’ stage. As can be seen from Table 3, the mAP values of the model in this paper are 86.4% and 86.6% in the detection results of 2021 and 2022, respectively, and the difference between the two is small. In 2021, the model obtained AP values of 87.2% and 85.6% in the ‘Full Bloom’ and ‘Petal Fall’ stages, respectively, which were almost the same as the test results of 87.5% and 85.8% in 2022. The test results showed that the apple flower detector was insensitive to apple flower data from different years and the model had good inter-annual knowledge transfer capability. However, the ability of the model to detect apple flowers at other blooming levels across years is unclear since apple flower data before the ‘FB’ stage are lacking in the 2021 dataset. Therefore, richer inter-annual data are needed to evaluate the performance of the model.

3.5. Blooming Level Estimation

In the practice of blooming level estimation, it is often more practical to estimate the global or block-level blooming level of an orchard than to detect flowering in individual fruit trees.

In this paper, apple flower blooming levels are considered independent events. The blooming level estimator counts the percentage of each blooming level stage based on the output of the apple flower detector at a given point in time. The percentage threshold for blooming level determination is set at 60% based on the experience of fruit farmers. Obviously, this blooming level estimation method is a higher challenge for the apple flower detector. To verify the effectiveness of the blooming level estimator, the data within the observation sites were collected and counted, and the estimated results were compared with the ground truth values of the percentage of each blooming level obtained by manual counting and statistics. Figure 12 shows the results of the apple flower detector compared with the ground truth for six time periods. From Figure 12, it can be seen that the results of the apple flower detector can accurately follow the trend of apple flower growth level for eleven days from 1 April to 11 April, which indicates that the blooming level estimation method is effective. It is noteworthy that the results from 2 April to 4 April differed from the ground truth values by 4–8%, which is a large difference compared to the results of the other blooming level stages. The reason for this is that apple flowers at the ‘Spitting Buds’ and ‘Tight Clusters’ stages during this period are less distinct and have fewer pixels, making them harder to detect by the detector. As the apple flowers grow, their features and differences with the environmental background gradually become obvious, so the difference between the detection results of this model and the real values decreases and converges.

3.6. Blooming Peak Day

Tracking the change in the number of apple flowers during the ‘King Bloom’ and ‘Full Bloom’ stages facilitates peak blooming date finding. The peak blooming curve graph (Figure 13) can be used as a peak blooming day finding estimator, and we plotted the curve using the output of the apple flower detector, with the detected images all coming from the observation site. It can be seen from Figure 13 that each blooming level curve will peak sequentially over time, and the interval between each peak is about one day, which indicates that the apple flower growth status changes greatly in a short period of time. ‘KB’ is a landmark stage of apple flower growth that has an important influence on flower pollination [44]. The timing of flower thinning is generally decided by the ratio of ‘King Bloom’ clusters and ‘Full Bloom’ clusters, therefore the distinction between ‘KB’ and ‘FB’ growth stages and merging them in the peak blooming day finding estimator helps farmers to have a more accurate picture of orchard blooming levels and make decisions. The ‘KB + FB’ curve in Figure 13 is the sum of the ‘King Bloom’ and ‘Full Bloom’ curves, and the horizontal coordinate point corresponding to the first intersection of this curve with the 80% peak blooming threshold line is the global peak blooming date in the orchard. The results show that the apple flowers in the observation site reached peak flowering around 6 April, which is highly consistent with the judgment of orchard management experts. Therefore, the peak blooming day finding estimator can be used to find the peak blooming date and provide a reliable basis for the determination of the day of flower thinning. Through the analysis in Section 3.4, the good inter-annual detection capability of the apple flower detector provides reliable support for the peak blooming day finding estimator, which made it possible to apply the peak date finder in the following year.

4. Discussion

4.1. Comparison of Improved Model with YOLOv5s

To further evaluate the apple flower detection model proposed in this paper, the Apple Flower A dataset was used for training the apple flower detection model and the YOLOv5s model, and the performance of both algorithms was evaluated using the same test set. The AP values and mean average precision of the test results for each blooming level stage are shown in

Table 4. Comparative experimental results of the improved model and the YOLOv5s model.

Model	mAP (%)	AP (%)
		SB	TC	BB	KB	FB	PF
YOLOv5s	75.8	40.0	74.9	84.6	82.2	87.2	85.7
Ours	77.5	44.8	76.7	85.8	84.1	87.5	85.8

. From the results in Table 4, it can be seen that the model proposed in this paper obtained 77.5% of mAP compared to the YOLOv5s model with an improvement of 1.7%. The contribution of the mAP enhancement comes from the improved detection capability of the model in this paper for small objects and objects with insignificant features. Compared with the YOLOv5s model, the AP of flowers at the ‘Spitting Buds’ and ‘Tight Clusters’ stages was improved by 4.8% and 1.8%, respectively.

The detection results in Figure 14 show that many apple flowers at the ‘Spitting Buds’ and ‘Tight Clusters’ stages were not detected by the YOLOv5s model, but the model in this paper significantly reduced the phenomenon of missed detection of the flower bud. In Figure 15, the clusters in the ‘Petal Fall’ stage are messy, and apple flowers in the transition from ‘Full Bloom’ to the ‘Petal Fall’ stage are not easily distinguished from each other, which makes it difficult to determine the blooming level. The detection results of YOLOv5s show more false identifications and overlapping object boxes, but the model in this paper accurately determines the blooming level of the messy flower clusters, reducing the false detection of clusters and the duplicate identification of objects. From the above results, it can be seen that the apple flower detection model in this paper shows good classification and small-target bud detection performance in detecting apple flowers with different blooming levels.

4.2. Comparison of Proposed Model with Baseline Model

Comparison with the baseline models can reveal the patterns in the process of apple flower detection by different models and verify the effectiveness of the models in this paper. We selected three baseline object detection models, Faster R-CNN [45], YOLOv3 [35], and SSD [46], for experiments, respectively. These models were trained and tested on the Apple Flower A dataset. The experimental results of the baseline models and the models in this paper are shown in

Table 5. The results of this paper’s apple flower detection model are compared with the baseline model.

Model	mAP (%)	AP (%)
		SB	TC	BB	KB	FB	PF
Faster R-CNN	73.4	37.0	71.4	83.1	79.6	84.6	84.5
SSD	73.9	39.5	70.5	83.8	80.2	85.1	84.2
YOLOv3	74.6	40.4	72.9	83.6	81.3	85.0	84.7
Ours	77.5	44.8	76.7	85.8	84.1	87.5	85.8

. The mAP of each baseline network fluctuated around 74%, and the YOLOv3 model among the baseline models achieved the highest mAP value of 74.6%, while the mAP of this paper’s model was 77.5%, which was 2.9% higher than that of the YOLOv3 model.

The APs of the present model for apple flower detection at the ‘Spitting Buds’ and ‘Tight Clusters’ stages were 44.8% and 76.7%. Compared with the baseline models of YOLOv3, which performed best for the detection of the above two blooming levels, the AP was improved by 4.4% and 3.8%, respectively. Compared with the evaluation results of other blooming levels, all tested models commonly got lower AP in the ‘SB’ stage for the reason that the network lost more unobvious small bud features in the subsampled process. Meanwhile, the too-small apple flower targets caused difficulties in manual labeling, and the missed labeling was regarded as a negative sample, which affected the training of the network. Moreover, the differences in apple flower morphology at the ‘SB’ stage caused by different varieties and the slowly evolving bud features from the ‘SB’ to ‘CB’ stage affected the classification ability of the network. Therefore, the apple flower detector based on the deep learning object detection model has much improved space for the detection of small target buds, and the negative influence of losing the object after resizing the image can be mitigated by sliding the detection window in large-sized images. At the blooming level with obvious apple flower features, the AP of this study model was higher by 1–4% compared with other baseline models. Based on the above theory and practice, it is demonstrated that the proposed apple flower detection model has advantages in apple flower detection at different blooming levels.

5. Conclusions

This study proposes an apple flower blooming level estimation method to provide guidance on the timing of flower thinning, which consists of three modules: an apple flower detector, a blooming level estimator, and a peak blooming day finding estimator. The apple flower blooming level was divided into six stages, and the distribution of apple flowers in different growth forms was used to determine the block-level or global blooming level of the orchard. Based on the natural environment and apple flower growth features, the improved YOLOv5s model was used as the flower detector in this paper. Through the training of the flower detector, the detector achieved the detection of apple flowers at each growth stage in the complex natural background, with different light intensities and different years of data. The experimental results show that the YOLOv5s model, which is added with a coordinated attention machine module and a small object detection layer, can effectively improve the detection performance of the detector for smaller buds in the images. The BiFPN structure effectively improves the complex feature extraction ability and reduces the false detection rate of apple flowers in the messy falling flower stage.

The blooming level estimator is based on statistics of the output of the apple flower detector, and the blooming level is considered an independent event. The flower detector obtained lower accuracy during the budding period of apple flowers, but the results of the blooming level estimator only fluctuated by 4–8%. The results show that the blooming level estimator reduces the sensitivity to miss and error detection by the apple flower detector. The peak blooming day finding estimator achieved the finding of peak blooming dates at the orchard block level, and the obtained results were recognized by orchard managers.

In this paper, a model for apple flower blooming level estimation is preliminarily discussed. We believe that this method can be easily generalized to the estimation of blooming levels in other species of fruit, such as peach and pear flowers. Furthermore, improving and studying the field estimation model of apple fruit set based on variety effects after flowering thinning is instructive for later fruit thinning and yield prediction. The performance of the global blooming level estimation in the orchard is to be tested, and the next step is to record the variation of apple flower growth status in different locations of the orchard. The development of an autonomous mobile robot with embedded data collection equipment makes it possible to capture global orchard images. Future work will map the distribution of blooming levels in the orchard to enable the selection of flower thinning timing in different areas of the orchard, which will help to provide more detailed management and decision information on the nutrient load balance of each fruit tree.