Grapevine Plant Image Dataset for Pruning

Abstract

Grapevine pruning is conducted during winter, and it is a very important and expensive task for wine producers managing their vineyard. During grapevine pruning every year, the past year’s canes should be removed and should provide the possibility for new canes to grow and produce grapes. It is a difficult procedure, and it is not yet fully automated. However, some attempts have been made by the research community. Based on the literature, grapevine pruning automation is approximated with the help of computer vision and image processing methods. Despite the attempts that have been made to automate grapevine pruning, the task remains hard for the abovementioned domains. The reason for this is that several challenges such as cane overlapping or complex backgrounds appear. Additionally, there is no public image dataset for this problem which makes it difficult for the research community to approach it. Motivated by the above facts, an image dataset is proposed for grapevine canes’ segmentation for a pruning task. An experimental analysis is also conducted in the proposed dataset, achieving a 67% IoU and 78% F1 score in grapevine cane semantic segmentation with the U-net model.

Dataset:https://github.com/humain-lab/Buds-Dataset.

Dataset License: Creative Commons Attribution 4.0 International.

1. Summary

Grapevine pruning poses a difficult problem if a specific automation solution is required. From the agricultural domain, it is known that a grapevine is separated into different parts. Based on this knowledge, several attempts by computer vision and the image processing research community have been made to solve the problem through image data processing, with the support of stereoscopic cameras [1,2] and 3D laser scanners [3,4]. Most of these are non-invasive approaches, except [4] when a robotic system is used for grapevine pruning, based on image processing and computer vision methods. Other proposed publications are based on RGB images without the addition of distance estimation or other real-world attributes. These methods, with the support of real-world attributes from the scene, aim to decompose the grapevine from given images [5,6,7,8] to achieve a pruning cutting points estimation. The grapevine structure consists of three basic parts, (a) the trunk, which is the larger woody part of the plant, (b) the cordons, which are thinner than trunks and of which there are usually two in each plant, and (c) the canes starting on the top of cordons, which are the thinner woody parts of the plant and the tallest, as presented in Figure 1.

Some basic visual characteristics of the plant are the fact that the trunk is always vertical, the cordons are horizontal, and the canes are in a random direction with a vertical formation. In the winter, during grapevine pruning, the shoots, spurs, suckers, and water sprouts do not exist. As mentioned above, the basic concept of pruning is to remove the last year’s canes from the cordons and let the new canes grow to give new grapes [10]. This leads to the conclusion that canes are the object of interest in this specific problem. Grapevine pruning via computer vision and image processing has been proven to be a very challenging problem, since canes are increasing in length, in random directions, and very near each other. This structure creates a high overlap between objects of interest, and, with the addition of a complex background and noise from the environment, image analysis is very difficult, especially for an invasive approach where a robotic system must take action. Another challenge in this problem is the complex background, where a grapevine image will contain not only the foreground plant but also the background, especially during winter, when the grapevine has no leaves at all. Finally, each winery applies different pruning strategies, such as pruning some canes completely and just shortening the rest. Different pruning strategies add some constraints to the problem, which makes it more complex than simple. A feasible solution is the usage of stereoscopic cameras, where, based on an RGB image and depth map (real distance values), the foreground and background could be separated in a given image [2]. The above issues justify why most studies in the literature focus on non-invasive approaches [1,2,3,5,6,7,8], where background and foreground algorithms, object detection [7], and segmentation [6] deep learning algorithms have been used without pruning points’ estimation. In invasive approaches [4], a closed environment surrounds the plant to be pruned with the support of robotic arms and computer vision methods based on 3D data processing. Despite the pruning, canes should also be removed, and in a fully automated scenario, the pruning strategy might change from the classical approach, since robotic manipulators with the support of cameras cannot mimic humans completely. Achieving this through image analysis means that all canes should be segmented in order to extract the whole canes’ bodies for cutting point estimations on them. Semantic segmentation is a very popular method in robotic systems that interact in a wild-free environment. With this approach to grapevine pruning, automation methods can achieve cutting point estimations or feature extraction mechanisms, since the whole area of interest is segmented. Pruning could be characterized as a problem for which it is very hard to find robust solutions, since vineyard plants do not have more than a 3 m distance from each other and have a very complex structure. Based on the above issues, the objectives of this study are:

• To provide to the computer vision and image processing research community a dataset that will motivate further research on the automation of the very difficult work of pruning in vineyards, for example by accurately estimating cutting points.
• To propose a dataset of images collected from a grapevine during the pruning period, with the complexity and difficulties of the free environment of a vineyard and corresponding to the real conditions of pruning.
• To justify the value of the proposed dataset with the application of a semantic segmentation model to segment and deconstruct the basic pieces of grapevine plants.

2. Data Description

The proposed dataset contains 100 image samples from grapevine plants during the winter and vine pruning season. Each image is encoded in PNG with a resolution of 1920 × 1080 and RGB color space. For each image sample, a hand-annotated mask is produced with the same resolution, and it contains marked pixel areas that point to a specific target class. Each marked pixel area has its unique color attribute and corresponds to each target class, whereas the rest of the image is colored in black and points to the background class. In Figure 2, a data sample is presented with the RGB image on the left (Figure 2a) and the marked pixel areas for each target class on the right (Figure 2b). In Figure 3, a mixed image between the RGB image (Figure 2a) in grayscale and its image mask (Figure 2b) is presented.

The data samples’ structure is connected to the semantic segmentation methods, which is a classification approach for each pixel in a given image. The proposed dataset solves a 4-class problem (trunk, cordon, cane, background), where each datum consists of two images, as presented in Figure 2 and Figure 4. In Figure 2a or Figure 4a in particular, the challenges presented in the previous section are visually justified. It is clear that the background creates a deformation between the target area of interest and the remaining image content. A basic approach with semantic segmentation solutions would be a CNN architecture as a backbone for feature extraction in a pair of images and a fully connected layer on top to apply the classification at the pixel level of the given images. In Figure 5, a mixed image between the RGB image (Figure 4a) in grayscale and its image mask (Figure 4b) is also presented.

A huge challenge, for example, in these approaches is handling false-positive estimations of the model, where areas classified in a target class but belonging to the background class are segmented. This is one of the challenges that motivated computer vision researchers to propose and implement a huge number of CNN architectures and image processing methods in order to achieve robust solutions to semantic segmentation approaches. An important factor that determines the appearance of false positives in segmentation model estimations is the complexity of the image content, which includes the targeted area of interest. The aforementioned issue plays a significant role in invasive robotic systems because these perform based on these estimations. For this reason, only one model is not enough for a robust solution, and, consequently, supplementary methods are vital to solid estimations. A possible solution might be a fusion of foreground-background algorithms and semantic segmentation models under one learning process, where the foreground-background algorithms would provide supplementary support to the segmentation models in order to emphasize the foreground features and produce better-segmented areas. Returning to the proposed dataset, the number of data samples could be characterized as small, but it contains high values of object instances in each image. In

Table 1. The number of object instances for each target class.

Trunks	Cordons	Canes
128	241	1316

, the number of object instances from each class is presented for the whole dataset.

Despite the small number of data samples, the number of object instances is high enough to support image processing or a computer vision and deep learning method, such as semantic segmentation with the support of augmentation methods, in order to produce a huge amount of images for training deep learning model architectures. Additionally, many images contain more than one grapevine plant. The imbalance between the object instances that appear in the dataset can be a problem for any methodology, since the objects of each class have a uniqueness in texture, color, and geometric features, along with the characteristic location of each object inside the image content. The selected classes emerged based on the age and mass of each plant part, where the trunk was first, followed by the cordon and the cane.

3. Methods

For the construction of the dataset, images were collected from a vineyard in the Greek city of Drama. Drama is an area in which many wineries are intensively active, and its selection has a special value. In

Table 2. Vineyard structure information of Pavlidis winery.

Location	Lat: 41.20101042956404,
	Long: 23.95247417327117
Posts’ height	3–3.5 m
Posts’ width	7.62 cm
Wires’ diameter	3 mm
Irrigation pipes’ diameter	16 mm
Plants’ distance (same row)	1.2–2 m
Plants’ distance (row–row)	2.2 m
Corridor width	2.2 m

, data concerning the location, the structure, and the density of grapevines in the vineyard are presented.

Since the dataset is proposed for pruning, the objects of interest are the canes, which provide enough to be processed by a deep learning model. Creating a dataset is important for providing a variety of instances to avoid overfitting. However, the work of pruning has some particularities. For example, pruning is mainly carried out in the morning with daylight. Additionally, the robot should see the plants vertically and not at an angle. That is why all images are captured vertically and with daylight. To obtain diversity in the dataset, images were taken from different distances in order for the model to learn to be scale-invariant. Scale invariance in this problem is very important, allowing one to discriminate areas that might belong to other corridors. Additionally, different distances are taken in order to accommodate robotic system scenarios where a camera is mounted on the end effector of a robotic manipulator along with the cutting tool. In this scenario, the object of interest might change in scale in camera images based on robotic arm movements.

3.1. Obtaining the Dataset

The device that was used to collect the image data was a ZED Mini 3D camera with a ZED SDK installed and the appropriate software application for image capturing and file exportation.

Table 3. Specifications of the ZED Mini 3D camera.

ZED Mini 3D
Resolution	1920 × 1080
FOV	90° horizontal, 60° vertical, 100° diagonal
Aperture	f/2.0
Sensor format	16:9
Sensor size	1/3″

provides details about the camera’s settings. The images were hand-annotated with free polygon shapes. Each shape was characterized with a class name, and for each image, a JSON file was produced. Next, with each image and corresponding JSON file, which contained the annotated areas with the appropriate class names, the image masks were produced. The RGB images were kept the same without any further processing, maintaining the image content as it was collected.

The ZED Mini camera exposes all camera settings to the user, such as brightness, contrast, hue, saturation, gamma, sharpness, white balance, exposure, and gain. Each parameter can be adjusted manually, and changes are applied to both sensors without supporting individual adjustment. The camera settings are presented in

Table 4. ZED Mini camera settings.

ZED Mini 3D
Brightness	4/8
Contrast	4/8
Hue	0/11
Saturation	4/8
White Balance	auto
Gain	auto
Exposure	auto

.

3.2. Dataset Validation

To evaluate the dataset (

Table 5. Amount of data before and after data augmentation.

Augmentation	Train Data	Test Data
No	90	10
Yes	360	40

), we implemented a U-Net [11] model in TensorFlow [12] with a segmentation-models library [13]. We used the U-Net model with ResNet34 [14] (

Table 6. Details of used models.

Model	Convolutional Layers	Parameters (Millions)
ResNet34	34	21,797
U-Net	23	7765

) as the backbone, which was pre-trained in ImageNet. Additionally, data augmentation techniques were applied in order to increase the data. Specifically, we applied horizontal flip and 15 and −15 degrees rotation because they do not change the structure of a vineyard plant. The train and validation splits were 90% and 10%, respectively. The training process was done in Google Colab in order to deploy a 12 GB RAM GPU. The implementation achieved 67% IoU and 78% F1 Score (Equations (1) and (2)), with parameters that are presented in

Table 7. Parameters used for training.

Learning Rate	0.0001
Batch Size	8
Optimizer	Adam
Backbone	ResNet34
Epochs	50

. Figure 6 and Figure 7 present two results from the trained U-Net.

$$\mathrm{IoU}=\frac{Area of Intersection of two masks}{Area of Union of two masks}$$

(1)

$$\mathrm{F}1\mathrm{Score}=\frac{True Positives}{True Positives+\frac{1}{2}\left(False Positives+False Negatives\right)}$$

(2)

With the above results, the applicability of the image dataset is justified. The predicted masks could be used for further processing to estimate cutting points at the start of the canes on top of cordons. Despite pruning, the above image dataset and application could be used for further plant analyses such as grape maturity growing.

4. User Notes

The way that this dataset was created is intended to provide data for model development in a vineyard pruning task. Vineyard pruning based on robotic systems is a very difficult task. A proposed procedure for a vineyard pruning task is roughly as follows. A mobile robot stops in front of a vine in a preselected location. The vision system mounted on the robotic manipulator detects the desired area. The acquired images are processed by means of the trained models, and the specific cutting points are calculated. The system, based on the optimum practices, selects the cutting point. An initial calculation of the target location permits the execution of the path planning algorithm, which will create an initial path for the manipulator. The manipulator starts to move towards the target, following this path. The vision system for each cycle acquires an image and, after processing, provides the coordinates of the target with a better accuracy (visual servoing). When the tool arrives at the desired location, a cutting procedure is implemented. To extend this scenario, a second collaborative mobile robot follows the previous one, and, with a second robotic manipulator and a properly designed gripper at the end-effector of it, it removes the cut branch. When the cutting procedure is completed for that specific vine, the mobile robots move to the next vine.