Design and Simulation of Two Robotic Systems for Automatic Artichoke Harvesting

Abstract

The target of this research project was a feasibility study for the development of a robot for automatic or semi-automatic artichoke harvesting. During this project, different solutions for the mechanical parts of the machine, its control system and the harvesting tools were investigated. Moreover, in cooperation with the department DISPA of University of Catania, different field structures with different kinds of artichoke cultivars were studied and tested. The results of this research could improve artichoke production for preserves industries. As a first step, an investigation on existing machines has been done. From this research, it has been shown that very few machines exist for this purpose. Based also on previous experiences, some proposals for different robotic systems have been done, while the mobile platform itself was developed within another research project. At the current stage, several different configurations of machines and harvesting end-effectors have been designed and simulated using a 3D CAD environment interfaced with Matlab^®. Moreover, as support for one of the proposed machines, an artificial vision algorithm has been developed in order to locate the artichokes on the plant, with respect to the robot, using images taken with a standard webcam.

1. Introduction

In recent years agriculture has received the focus of different research groups that operate in the robotics and automation field. Moreover, electronics, informatics and automation solutions have been proposed as commercial products by most of agriculture machine and tools suppliers. Nevertheless, in-field fully autonomous operations yet have some barriers to overcome in order to become relatively common and cheap as well as reliable, in comparison to the great number of applications in the manufacturing industry.

The main reasons for this delay is due to the fact that agricultural tasks are carried out in an outdoor environment, often really unstructured, dynamic and most of the time heavily conditioned by water, mud, wind, light, dust, chemicals and so on. Moreover, the actual cost of a robotic system is usually too high with respect to the other production costs and with respect to the standard season operators’ salary. Despite these premises, different solutions exist for some specific agricultural operations. In [1,2] a robotic system for vineyard and greenhouses operation, like transportation and spraying, is reported. Several other examples of robotic harvesting systems have been developed and reported in [3,4,5]. However not so many commercial systems exist and most of these are simply research laboratory prototypes. In [6], an interesting review on the fruit harvesting problem and the possible automatic solutions is reported. Among the harvesting robots that have been developed, it is possible to cite the Orange Picking Robots [7] developed by the University of Catania, where a detailed study on the vision processing analysis was carried out and several testing of the picking mechanism were developed. In [8,9], two in-depth reviews about artificial vision applications to the agricultural problem and useful algorithm are reported, while in [10] a possible solution to the fruit detection problem has been implemented and tested. Another interesting research activity has been carried out by Wageningen University that performed several studies and field tests for a cucumber harvesting robot [11]. As regard harvesting, many applications concentrated in greenhouse environment, where it is simpler to move the harvesting system and to locate vegetables, as in the case of strawberry harvesting [12]. Only a few works concentrate on vegetable harvesting in outdoor fields, as in the case of radicchio harvesting [13]. Other interesting applications of robotics in agriculture include precision spraying and precision fertilization [14], inspection and treatment of plants [15], selective herbicide application [16], just to name a few.

As far as we know, there is only one example of a machine for artichoke harvesting. The aim of the BIOCARD EU FP6 STREP project [17,18] was to demonstrate the economical and technical feasibility of a global process to improve annual lifecycle Cynara cardunculus exploitation for energy applications. Among the different activities of that project, the development of specific machinery able to harvest the capitula and separate directly in the field seeds from cynara biomass was addressed. Concerning automatic artichoke harvesting for fresh vegetable market, our research work represents a preliminary step toward the introduction of robotics into this specific field. In particular the main peculiarities of our approach are from one side the design and development of an automated robotic system for harvesting and on the other side the fact that the research was carried out working concurrently with the growers and with the experts in horticulture, to select suitable plants that could make the process of harvesting easier.

2. Investigation about Artichokes Field Design and Current Harvest Methodology

The purpose of this research project was a feasibility study for the development of an automatic or semi-automatic machine for artichokes harvesting. This kind of machine will work on specific cultivations for industrial use of the artichokes, like for preserves production. These kinds of cultivations are more indicated for maximum productivity with minimum interest to artichoke appearances and could be selected in a suitable way, in order to be compatible for the automatic harvesting while maintaining organoleptics properties. This task is easier with respect to many other crops because most artichoke cultivars have an annual or bi-annual life cycle. Moreover, suitable cultivars that have specific peculiarities for industrial use and that could simplify automatic harvesting have been tested.

In our study case, with the help of the department DISPA of the University of Catania, two cultivars have been selected for their properties: Harmony F1 hybrid and Madrigal F1 hybrid. Moreover, the field structure and density have been designed in a suitable way, in order to allow a small vehicle to move along the rows. The first field structure was realized with a single row of plants, with different densities between 0.4 m and 0.7 m. The row distance has been kept at 1.4 m for each density. The second field structure, shown in Figure 1, is realized with doubled interlaced rows, with different densities between 0.55 m and 1 m. The distance between the rows is kept at 1.4 m.

Figure 1. Some field structure used during test: doubled interlaced rows.

Figure 1. Some field structure used during test: doubled interlaced rows.

Until now, artichokes are manually harvested and collected inside wooden or plastic-made boxes for retail business. The only difference with harvesting for industrial business is that artichokes are collected directly inside trucks or small tractors, like that in Figure 2. Manual harvest is often made by using some support machines, mainly for all the collected artichokes. From some statistical analysis in the Sicily area, the harvest cost is about 20% of the total production cost. In order for automatic harvesting to be convenient, a suitable machine with good harvesting rate must be developed. A typical harvesting rate of 10 workers is about 30,000 artichokes/day, with a typical 8 h long working time, which is 6.25 artichokes each minute per worker, so any kind of machine must at least have a similar harvesting rate.

Figure 2. A small tractor used for manual artichokes harvesting.

Figure 2. A small tractor used for manual artichokes harvesting.

3. Machines Design and Simulations

On the basis of several field observations and measurements and following discussions with local experts, two different harvesting machines have been designed and simulated. The two machines are based on different harvesting methodologies and each one has its own advantages and drawbacks. Each of the two uses a mobile platform that relies on two rubber tracks. It is possible to use electrical motors or thermal engine ones. A suitable small electrical tracked vehicle was developed and tested by the authors along another research project and is reported in [1,2]. This is narrow enough to be able to move between the plant rows. The two machines have been simulated in a 3D CAD environment, that can be interfaced to Matlab^® packages. In order to implement a more realistic simulation, first of all a graphical representation of an artichoke with trunks, leaves and immature flowers (Figure 3) was prepared. Each artichoke head has a known geometrical position in the simulated field (Figure 4) and this property has been used during the simulations in order to cut or pick-up the artichoke itself with the simulated machine.

Figure 3. The simulated artichoke plant.

Figure 3. The simulated artichoke plant.

Figure 4. The simulated artichokes field.

Figure 4. The simulated artichokes field.

The first proposed machine uses a dual manipulator configuration. One manipulator has three degrees of freedom (DOF), while the other has only two DOF. Once the machine has reached a suitable position to use the cutting tool on the first manipulator, the system can adjust the position of the second manipulator in order to pick another artichoke without moving the mobile base. Each manipulator has an end-effector capable to cut the artichoke and to drop it directly inside a specific box or on a conveyor belt (Figure 5). Each DOF could be pneumatically or hydraulically actuated. The main peculiarity of this machine is that it can harvest artichokes with high precision with the help of an artificial vision system described in next paragraph. However, due to the mechanical delays, the system can not be fast enough to satisfy the required minimum harvesting rate. The plant will not be damaged too much by the harvesting procedure, but, due to some possible faults in the artificial vision system, some artichokes will not be seen and consequently not harvested.

Figure 5. The robot for artichokes selective harvesting: (a) and (b) two different views of the first proposal and (c) a detailed view of the gripping tool.

Figure 5. The robot for artichokes selective harvesting: (a) and (b) two different views of the first proposal and (c) a detailed view of the gripping tool.

The control architecture should be composed by four main blocks: the vision system, the mobile platform position control system, the manipulators control system and the main supervising and coordinating control system. Once the supervising and coordinating block is signaled from the vision system that an artichoke is found in the image in a suitable workspace, it stops the mobile base at the given position. Then sends the artichoke coordinates, relative to the base, given by the vision system to the first manipulator. At the same time, if another artichoke is found on the operative space of the second manipulator, also this is commanded to move and harvest. Then the procedure is repeated until another artichoke is found in the image or the end of the corridor is reached.

The second type of machine uses a very large cutting floor with moving blades (Figure 6). This kind of machine will cut the plant at a suitable height. The height value will be estimated time by time before starting the harvesting session and regulated accordingly. This machine could be fast enough, but it could damage plants and, if the height of the cutting floor is not well regulated, the artichokes could be also damaged. In particular also some foliage will be cut and collected during the harvesting of the artichokes. Most of the separation can be performed by using some selecting tools onboard the machine. The remaining foliage can be easily removed during the subsequent factory processing phases. The damage caused on the plants is not relevant because the life cycle of these cultivars is annual or bi-annual. For all the proposed machines, the 3D environments have been interfaced with Matlab^® for simulating their control system.

Figure 6. (a) The robot for artichokes mass-harvesting. (b) Machine details of the cutting floor.

Figure 6. (a) The robot for artichokes mass-harvesting. (b) Machine details of the cutting floor.

4. Artificial Vision Algorithm

The first proposed machine uses two standard cartesian manipulators, one with two and the second with three linear joints to pick artichokes with a suitable end-effector (cutter). In order to perform a correct positioning of the cutter with respect to the artichokes, an artificial vision algorithm based on images acquired with a standard webcam mounted on-board the robot, was adopted. All images are taken as top view of the field at a mean distance of about 0.6 m from the top of the plant. For development purposes, the algorithm has been implemented in a Matlab^® environment using the Image Processing Toolbox; it has been tested with static images from the camera. In Figure 7, a typical image taken with the camera is shown. This image is then converted in black and white (Figure 8) and scaled down in resolution, in order to implement a very fast recognition algorithm.

Figure 7. Camera picture of an on-field artichoke.

Figure 7. Camera picture of an on-field artichoke.

Figure 8. Black and white representation.

Figure 8. Black and white representation.

4.1. Details of the Image Processing Algorithm

Different methodologies have been used to obtain better results. Among the others, erosion-diffusion techniques, correlation with sample images, Artificial Neural Networks, co-occurrence matrix, cluster symmetry, wavelets and so on. Due to the fact that artichokes have quite the same color of the image background (mainly leafs but also terrain with weeds), image colors are not very useful to identify them. On the other side, searching for high-symmetry clusters in the image could help. In case the picture contains portion of terrain, these areas are considered as noise. Using a combination of the previous methodologies as a sequence of image filters, it is possible to obtain a high true positive rate in each image. Best results have been obtained with a preliminary black and white conversion, followed by an erosion-diffusion step. After that, the algorithm searches for high-symmetry regions. To compute such regions, the Specular Image method was applied. In the main image, a moving sub-image denoted as “kernel” is established around a specific pixel on the image and then the left and right parts of this kernel are compared each other as well as the upper and lower part. In steps of 10 pixel, a new kernel is defined around the next 10^th pixel and so on for all the main image pixels. In order to maintain a suitable speed for the algorithm the image of was scaled down in resolution to 512 × 384 pixels. Since for all the images taken in the used experimental conditions, artichoke commonly cover an area of about 60 × 60 pixels, a kernel dimension of 105 × 105 pixels was chosen as a good compromise. When the two parts of the kernel have about 3000 similar pixels the image is considered has having a high symmetry.

In all those regions, patterns are then classified using GLCM (Gray-Level Co-occurrence Matrix) [19] and Wavelets methodology [20] in order to filter out most of the false positive targets. These two analyses aim to obtain some kind of digital signature of the artichoke in its different shape, light conditions and so on. From different trials, is has been shown that leafs and other kind of noises on the images have different signatures in the sense of GLCM and Wavelets analysis. Each of the two methods are first performed on different samples images containing typical artichoke, leafs and terrain in real conditions. These samples images have typical dimensions of 51 × 51 pixels. After the two methods on sample image with suitable pre-filters have been applied in order to find out thresholds for all the different parameters, auto and cross correlation operations are performed between samples images and related parameters have been computed and compared.

As regards the GLCM analysis, the Contrast output parameter is considered, as it was found as the only parameter able to discriminate an artichoke from leaves and terrain. Then, the Entropy and Standard Deviation of the same sub-image are computed. From a series of experimental trials on real images, it was found that the sub-image can be considered a true positive, when the parameters move in the ranges descripted in Table 1.

Table 1. Ranges for the used parameters of the Gray-Level Co-occurrence Matrix (GLCM) algorithm.

Parameter	Minimum value	Maximum value
Contrast	0.0755	0.16
Entropy	0.2745	0.33
Standard Deviation	0.2045	0.24

Table 1. Ranges for the used parameters of the Gray-Level Co-occurrence Matrix (GLCM) algorithm.

Parameter	Minimum value	Maximum value
Contrast	0.0755	0.16
Entropy	0.2745	0.33
Standard Deviation	0.2045	0.24

Regarding the second analysis, a two dimensional wavelet function decomposition is then used. The best results have been obtained using a wavelet function “db8” of the Daubechies family with three levels of decomposition. After this operation, always applied over sample images, the approximation parameter have been rejected while all the three detail parameters have been collected in the “signature” matrix. A specific kernel image with “high symmetry” is considered a true target when the correlation between the two signature matrixes is greater than 0.87.

4.2. Results of the Image Processing Algorithm

In Figure 9 the original image with some information overlapped about the real artichoke position with respect to the robot, is shown. Green squares are related to true positive artichokes coordinate points, while red squares indicate false positive. The robot will then reach only the true positive points. During different tests, the vision algorithm has shown good capabilities to detect true and false positive (85% to detect true positive and 76% to detect false positive) on a set of about 20 typical artichoke images taken in a real field in sunny conditions.

Figure 9. Localization data computed by the artificial vision algorithm, on the original image.

Figure 9. Localization data computed by the artificial vision algorithm, on the original image.

5. Results

In order to get an estimation of the performance of the two solutions, ad-hoc simulations were done because each of the two machines uses a different approach.

As regards the mass-harvesting machine, the main problem is the different height of artichokes on the plant, even if all of them are at the same maturity level. The real case is even worst, because of the presence of artichokes not ready to be harvested (too small or not mature). For this kind of simulation, several measurements were done on real fields to get a statistical distribution of the positions and heights of the artichokes. In Figure 10 the mean of the heights of different order artichokes (main stem/bud, first order and second order stems/buds are considered [21]) that are at the same time on the plants, for nine different repetitions, is shown. Each repetition was made in different days, normally with one week interval. In this experiment, the cultivar Madrigal F1 in simple rows was considered. These measurements were also used to implement the Matlab^® simulations for the control system of the two machines, while operating in the 3D CAD environment. In these simulations the position distribution of the artichokes and the recognition rate of the vision algorithm (for the selective harvesting robot) were also taken into account. The planar position was reproduced from the measurements on a real field, while the height of the buds was modeled using a Gaussian distribution with the mean and variance get from the field. In this way it was possible to estimate the harvesting time for each robot, as reported in Table 2.

Figure 10. Mean height in centimeters for artichokes of the cultivar Madrigal F1 in simple rows for nine different manual harvesting sessions.

Figure 10. Mean height in centimeters for artichokes of the cultivar Madrigal F1 in simple rows for nine different manual harvesting sessions.

Table 2. Harvesting performance of the two proposed solutions derived from the Matlab^® simulations.

	Selective Harvesting robot	Mass Harvesting robot
Speed	2 m/s (Arm)–0.2 m/s (Vehicle)	0.2 m/s (Vehicle)
Productivity	0.13 Artichokes/s	1.2 Artichokes/s
Productivity per hour	480 Artichokes/h	4,320 Artichokes/h
TOTAL Productivity (Working day)	3,840 Artichokes/8 h	34,560 Artichokes/8 h
Weight harvested (Unit weight of typical artichoke 200 g)	96 kg/h	864 kg/h

Table 2. Harvesting performance of the two proposed solutions derived from the Matlab^® simulations.

	Selective Harvesting robot	Mass Harvesting robot
Speed	2 m/s (Arm)–0.2 m/s (Vehicle)	0.2 m/s (Vehicle)
Productivity	0.13 Artichokes/s	1.2 Artichokes/s
Productivity per hour	480 Artichokes/h	4,320 Artichokes/h
TOTAL Productivity (Working day)	3,840 Artichokes/8 h	34,560 Artichokes/8 h
Weight harvested (Unit weight of typical artichoke 200 g)	96 kg/h	864 kg/h

In order to obtain an estimation of the efficiency of the two machines, in terms of achieved useful yield, a simulation of the proposed harvesting methodologies has been also done in the fields by manually harvesting artichokes using the two proposed methodologies. Operators were trained about the two protocols. With the mass-harvesting protocol, operators harvested all the artichokes that are above a certain height, established before the session. The harvesting days were chosen by skilled operators in order to maximize the presence on the whole field of artichokes to be harvested. Artichokes that are at the cutting level or above the cutting level, but not ready to be harvested, were tagged as “damaged”. The same experimental flow applies to the selective-harvesting protocol.

After the harvesting phase, collected artichokes were counted and cumulative data for different harvesting sessions are shown in the graphs in Figure 11 and Figure 12.

It can be observed that the mass harvesting robot has a good productivity, comparable to the given requirements. However, the percentage of damaged artichokes is too high to be considered suitable for the intended purposes. The selective harvesting has a small rate of damaged artichokes, but is too much slow, complex and expensive to be really implemented in the field.

Figure 11. Total harvested and damaged artichokes for the selective harvesting robot.

Figure 11. Total harvested and damaged artichokes for the selective harvesting robot.

Figure 12. Total harvested and damaged artichokes for the mass harvesting robot.

Figure 12. Total harvested and damaged artichokes for the mass harvesting robot.

6. Conclusions

The problem of artichoke harvesting has been investigated by means of literature search, in-field tests and manually simulating the robotic harvesting. Exploiting the fact that artichoke plants are mainly annual or bi-annual, it is possible to easily design the field in order to be compatible with mechanical harvesting operations, by means of a small autonomous vehicle with suitable manipulators or tools. Also artichoke cultivars have been selected to maximize mass production and to have plant field more compatible with mechanical harvesting.

With the help of a 3D CAD environment, two different approaches to the artichoke harvesting problem have been proposed. By using this methodology, some advantages and drawbacks of the proposed solutions have been fully analyzed, taking into account field observation on real plants and local expertise. Moreover, as a support for one of the proposed solution, an artificial vision algorithm has been developed and tested in order to recognize artichokes on the plant to allow the robot to pick them. From the obtained results it follows that further work is needed before building and testing a real prototype, since the obtained performance for those systems is still not satisfactorily. In any case the proposed approaches give us a suitable tool and a strong basis for the development, simulation and test of new solutions. This work represents a preliminary study about possible solutions for automatic artichoke harvesting. In the next phases of the research, further collecting criteria and performance requirements will be certainly assessed.