Safety of Automated Agricultural Machineries: A Systematic Literature Review

Abstract

Automated agricultural machinery has advanced significantly in the previous ten years; however, the ability of such robots to operate safely will be critical to their commercialization. This study provides a holistic evaluation of the work carried out so far in the field of automated agricultural machines’ safety, as well as a framework for future research considerations. Previous automated agricultural machines’ safety-related studies are analyzed and grouped into three categories: (1) environmental perception, (2) risk assessment as well as risk mitigation, and (3) human factors as well as ergonomics. The key findings are as follows: (1) The usage of single perception, multiple perception sensors, developing datasets of agricultural environments, different algorithms, and external solutions to improve sensor performance were all explored as options to improve automated agricultural machines’ safety. (2) Current risk assessment methods cannot be efficient when dealing with new technology, such as automated agricultural machines, due to a lack of pre-existing knowledge. Full compliance with the guidelines provided by the current International Organization for Standardization (ISO 18497) cannot ensure automated agricultural machines’ safety. A regulatory framework and being able to test the functionalities of automated agricultural machines within a reliable software environment are efficient ways to mitigate risks. (3) Knowing foreseeable human activity is critical to ensure safe human–robot interaction.

1. Introduction

The world human population will reach 8.5 billion in 2030, 9.7 billion in 2050, and 10.9 billion in 2100 [1]. In addition, about 9% of the global population (770 million in 2021) is undernourished [2]. Meeting the food demands of an increasing population, in addition to tackling the issue of undernourishment, will require new innovative strategies in agriculture. Moreover, farm workers are migrating from rural to urban areas due to poverty, limited access to social protection, inequality, and environmental degradation [3]. To meet the increased demand for agricultural products, with limited inputs, modern farmers will turn to automation, using technologies such as AI, machine learning, sensors, and other digital hardware as well as software to boost productivity and meet rising food demands in a sustainable way that is less dependent on inputs and the labor force.

There has been considerable development of AI-based automation in the areas of pruning, spraying fertilizer, autonomous weed removal, and harvesting. Ramin et al. [4] reviewed 152 research papers that focused on the research and development of autonomous weed control, field scouting, and harvesting equipment. Similarly, Lytridis et al. [5] reviewed 77 papers that focused exclusively on agricultural cooperative robots that were split into five different topics, including (a) human–robot cooperation, (b) cooperative multiple unmanned aerial vehicles (UAVs), (c) cooperative multiple unmanned ground vehicles (UGVs), (d) hybrid teams of UAVs as well as UGVs (UAVs/UGVs), and (e) cooperative manipulation by multi-arm systems.

Despite tremendous development in agriculture automation, automated agricultural machines have not yet reached a commercial scale [4] owing to several obstacles, including machine system safety, data privacy, initial investment costs, associated expenses, and a lack of knowledge of their potential benefits [6,7]. One of the major obstacles to the full deployment of automated agricultural machines is safety. Automated agricultural machines must be designed to work safely, their safety record must be confirmed, and, more importantly, they must be viewed as safe by end users. Therefore, investigating and ensuring the safety of automated agricultural machines will be critical for their adoption.

The main objective of this work is to survey, organize, review, and summarize the different bodies of published research related to the safety of automated agricultural machines, including discussing potential future work.

2. Materials and Methods

A systematic literature research method that utilized the identification, screening, and inclusion procedures outlined by the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement (https://prisma-statement.org/ accessed on 3 October 2022) was used in this study [8]. A systematic literature review was utilized in this study as it is method-driven, replicable, and comprehensive. In addition, this method aims to reduce bias through exhaustive literature searches, providing a scientific as well as transparent process, and by providing an audit record of the reviewers’ procedures [9].

2.1. Identification

The systematic literature review was conducted by using the Scopus database, as it was identified as being the most relevant for publications in the areas of engineering and manufacturing [10]. The search was conducted in October 2022. Titles, abstracts, and keywords were used as the search fields in the Scopus electronic database. The query used in the search was “object detection agriculture” OR “human–robot interaction in agriculture” OR “robot perception agriculture” OR “robotic safety agriculture” OR “human factor ergonomic agriculture” OR “autonomous agricultural vehicles safety” OR “risk assessment agriculture robotics”. These keywords were selected based on a preliminary search that found them effective in finding an initial list of known relevant articles. Furthermore, the automated search options in the Scopus electronic database were used to include the following criteria: (1) years (2012–2022), (2) subject areas (agricultural and biological sciences, engineering, and computer science), (3) keywords (agriculture, agriculture robots, object detection, deep learning, robotics, computer vision, object recognition, precision agriculture, and remote sensing), and (4) language (English). In total, 1209 matching articles were identified and exported into an Excel spreadsheet for further analyses.

2.2. Screening

The authors examined all of the 1209 articles by reading the titles and abstracts. The criteria for inclusion in the next stage were as follows:

• Article is about robots and/or autonomous systems;
• Article is about agriculture applications;
• Article is about safety.

All of the 1209 articles were reviewed at this stage, and only 40 articles were found to meet all of the inclusion criteria.

2.3. Eligibility

The authors examined all of the 40 articles by reading the full texts. The criteria for inclusion in the next stage were as follows:

• The full text of the article is available;
• Article is about automated and autonomous systems;
• Article is about agriculture applications;
• Article is about safety;
• Articles is not a review paper;
• Article is not a book.

Review papers and books were not included in this systematic review, as the goal of this paper was to review current primary research findings. Including review articles and books might skew the data on the amount and type of research conducted. A total of 40 articles were processed in this stage; 9 articles were excluded as they did not meet the criteria, leading to 31 articles being left for the next stage. Figure 1 shows a flow diagram of the review’s methodology.

2.4. Additional Review

To identify additional articles related to the safety of automated agriculture machines that might have been missed by the keywords, each of the 31 articles were found in Scopus and saved as an article list. The automated search options in the Scopus electronic database were then used to obtain all of the articles that were either cited or referenced by these 31 articles. In total, 1444 articles were identified and exported into an Excel spreadsheet for further analyses. Moreover, the Excel data filter option was used to only include articles from 2012 to 2022. This narrowed the total down to 1114 articles. The same screening process method conducted in Section 2.2. was applied to the 1114 articles, which narrowed the total down to 86 articles. The remaining articles were read and included if they met the criteria stated in Section 2.3. A total of 51 articles was obtained; however, it was the case that 20 out of the 51 articles were already in the initial 31 articles found. This occurred as a result of several articles citing other works that were already identified in the initial examination. Thus, these articles were excluded, and the remaining 31 articles were considered. Overall, a total of 62 articles were found and reviewed. Figure 2 shows a flow diagram of the additional review’s methodology.

3. Results

The reviewed papers were organized into three categories that improve the safety of automated agricultural machines: (1) environmental perception, (2) risk assessment as well as risk mitigation, and (3) human factors as well as ergonomics, as illustrated in Figure 3. A total of 48 articles are related to the perception of automated agricultural machines, 9 articles focused on risk assessment as well as risk mitigation, and 5 articles involved human factors as well as ergonomics. Overall, there were 43 journal articles, 17 conference articles, 1 standard, and 1 thesis.

The geographical region of the first author was considered to be the geographical location of the publication. Over half of these publications came from just four countries: Denmark, with 10 papers, and the USA, Italy, and China, with 8 each. The next tier was Germany, Australia, Japan, and Canada, with three–four publications each. Korea, Greece, the United Kingdom, Brazil, Taiwan, Norway, Portugal, Finland, Israel, Spain, and Poland all had one–two publications each (Figure 4). On average, five–six publications were published per year, with a minimum of two and a maximum of nine articles. There were no clear trends in recent publications on articles related to the safety of autonomous systems in agriculture (Figure 5). This trend contrasts with trends for articles published on AI, or autonomous systems in OSH which showed a rapid increase in the last 10 years [11].

3.1. Environmental Perception

Robotic technology has been extensively developed for agricultural tasks in recent years to produce intelligent vehicles that can boost productivity and competitiveness. Safety is a significant concern with regard to autonomous vehicles, as unforeseen obstructions and highly deformable terrain in working areas must be robustly identified and avoided to prevent accidents. Failure to recognize and avoid obstacles as well as dangerous terrain in a timely manner can cause serious problems, including harm or death to humans as well as animals and/or damage to robots in cases of incidents. As a result, accurate and reliable environmental perception is vital [12].

The perception of agricultural environments is achieved using perception sensors and machine learning or artificial intelligence. Perception sensors, positioning and guidance systems, and associated algorithms are used to identify and classify significant objects, and a safety-related control system is used to control machines without human intervention. (1) Perception sensors are used to collect and record images (including all types of obstacles and jeopardizing terrains) in agricultural environments. Examples of perception sensors used with automated agricultural machines include stereo vision cameras, LiDAR, radars, thermal cameras, RGB cameras, and laser scanners.

Table 1. Advantages and disadvantages of different sensor modalities for outdoor perception [12,13].

Sensor Modality	Functional Methods	Advantages	Disadvantages
Stereo vision camera	Stereo vision cameras capture wavelengths in the visible range. The images they produce recreate almost exactly what our eyes see.	Natural interpretation for humans	Risk of occlusions
		Relatively high resolution	Sensitive to lighting conditions
		Relatively high sampling rate	Poor performance in low-visibility conditions (rain, fog, smoke, etc.)
LiDAR	LiDAR is an optical remote sensing technology that measures the distance or other properties of a target by illuminating the target with light.	High accuracy, high density	High costs related to accuracy and range
		Narrow beam spread Fast operation	Some risk of occlusion No color or texture information
Radar	Radars measure the distance between an emitter and an object by calculating the ToF of an emitted signal and received echo.	Long range (up to 100–150 m)	Three-dimensional map limited to pencil beam radar
		Panoramic perception (360°)	Difficulty in signal processing and interpretation
		Multiple targets	No detection of small objects
Ultrasonic sensor	An ultrasonic sensor uses sonic waves, in the range of 20 kHz to 40 kHz, to measure the distance to an object.	Reliable in any lighting environment	Poor detection range and resolution
		Less impact on the environment	Sensitive to smoothness and angle to obstacles
ToF camera (time of flight camera)	ToF cameras measure distances with modulated light based on the ToF principle.	Provides direct 3D measurements Invariant to illumination and temperature No moving parts	Low resolution, limited field of view (FoV) Relatively short detection range (depends on illumination power)
Thermal camera	A thermal camera is a device that forms a heat image and is used to detect heat radiation that is emitted by all living things.	Invariant to illumination	Relatively low resolution
		Robust against dust and rain Detects humans and animals	Some risk of occlusion Difficulty in calibration

provides a description of some of these perception sensors (Table 1). (2) Machine learning algorithms analyze the data collected. Algorithms are trained with the datasets for the identification and classification of significant objects. Examples of machine learning algorithms used with automated agricultural machines are convolutional neural networks and hidden Markov models. The goal of training the algorithms is to allow them to robustly identify and classify images in various agricultural environment settings. (3) Finally, when the algorithms are well-trained, and the hazards are identified during robots’ operations, logic is sent to the robot to either stop, change direction to avoid an obstacle, or reduce its speed [11,13], as shown in Figure 6.

Robust obstacle detection in agricultural settings is challenging due to the complex and unstructured nature of such environments. Furthermore, training algorithms on all possible obstacles is complex because many hazards have camouflaged appearances or structures and are diverse. There are four main categories of obstacles encountered in agriculture, including positive, negative, and moving obstacles, as well as deformable terrain. A positive obstacle is any object higher than ground level, whereas a negative obstacle is lower than ground level. Examples of positive obstacles include trees, metallic poles, and buildings. Examples of negative obstacles include holes and ditches. In the case of a positive obstacle, an automated agricultural machine risks a collision; conversely, a negative obstacle can cause a crash. A dynamic obstacle is something that suddenly appears in front of a robot, generally represented by human operators, other moving machinery, animals, or even children. Moreover, obstacles can also vary widely based on the type of crop, fruit, and vegetable, as well as the curving of the landscape. Figure 7 depicts examples of positive, negative, and moving obstacles commonly seen in agricultural settings [12].

Furthermore, because an autonomous vehicle’s stability can be jeopardized by highly deformable terrain, deformable terrains remain another potential hazard faced by automated agricultural machines. As a result, effective terrain identification and classification, as well as subsequent appropriate automated machine reactions, are crucial aspects of improving safety. Figure 8 displays rough terrains that are frequently found in agricultural settings.

3.1.1. Single Perception Sensor

Given the challenging environment in which automated agricultural machines operate, numerous works have been developed regarding robust obstacle detection and terrain classification systems to improve their safety. For example, Freitas et al. [14] introduced the use of 3D laser scanners to detect obstacles in agricultural settings. An autonomous orchard vehicle was equipped with a 3D laser scanner and tested in a field. As the vehicle moves forward, the laser measures different lines on the ground, allowing the creation of 3D point clouds representing the terrain surface as well as the obstacles detected. Within the same year, the authors of Kohanbash et al. [15] proposed a safety architecture with which to guide the development of safe automated agricultural machines, with an initial focus on autonomous agricultural vehicles. They developed a new hierarchy of agricultural robotic safety based on three elements: (1) sensing/computing/actuation technologies, (2) the user interface, and (3) establishing standards based on the level of automation implemented in machines. A John Deere tractor was retrofitted to make it autonomous by following their hierarchy and was equipped with a laser scanner in addition to wheel and steering encoders. Based on their hierarchy, the retrofitted tractor was able to detect and stop in front of obstacles, but could not drive around them. Additionally, Bellone et al. [16] looked at autonomous vehicles safely navigating through challenging terrain and detecting positive, negative, and dynamic obstacles by using stereo vision techniques. All types of obstacles and terrain imperfections were correctly detected by the system; however, the results were less precise because the stereo vision techniques could not produce point clouds of equal density and quality. In line with this issue, the authors of Reina et al. [17] proposed the use of machine learning to improve radar classification in outdoor settings. A self-trained radar classifier was developed, where the ground model automatically learned during a bootstrapping stage and was continuously updated based on the most recent ground labels to forecast ground instances in subsequent scans.

Dvorak et al. [18] looked at the potential of an ultrasonic sensor to detect things that are often seen in an outside farming environment. The technology correctly identified a water jug and oriented strand board but underperformed in recognizing animals, a mannequin (human model), and a fence. Moreover, Yang et al. [19] presented human detection via an automated tractor using stereo vision in an agricultural environment. The method detected a human in the range of 4 to 11 m with an error distance of about 0.5 m in both stationary and motion conditions during daytime. Furthermore, Ross et al. [20] used a novelty detector to inform stereo vision matching to improve obstacle detection systems for autonomous robots in agricultural field scenarios. This approach has no lighting dependence and requires no pretraining because the novelty is based solely on current image data. While the system correctly detected all of the obstacles during the day, there were some problems when detecting the obstacles at night. More recently, the authors of Fleischmann et al. [21] created a stereo-vision-camera-based obstacle-detecting system. The entire categorization process was based on dividing the created point cloud into cells and analyzing the cells further. Within the same year, a sophisticated sensor camera was introduced by Campos et al. [22], who explored the safety of automated agricultural machines through robust obstacle detection with a unique CCD high-resolution camera operating in the visible RGB spectral range. Automatic video analyses were performed to detect static and dynamic obstacles in agricultural environments via spatial–temporal analyses. A key feature of this method is that it does not require any training process. The authors Yan et al. [23] used a stereo vision camera combined with fuzzy logic and a neural network to successfully detect vehicles, humans, trees, plastic bars, and other common obstacles in an agricultural environment.

Researchers also employed a single 2D laser scanner sensor to detect obstacles in an agricultural environment [24]. The sensor was placed in front of an automated agricultural tractor and was designed to find a man-made round pole in a farm setting. The system was able to accurately identify the obstacles and stop the vehicle; however, in the field’s corners, where the sensor was sensing beyond the field plot’s margins, the system produced false positives. In order to recognize an environment and identify as well as avoid obstacles in agricultural settings, the researchers of Inoue et al. [25] used an object detection system based on a stereo camera and convolutional neural network. The study tested the positions of obstacles and landmarks as well as the position of detected objects in relation to the robot rather than the system’s ability to detect and avoid obstacles. The outcomes, however, revealed that the robot’s final position inaccuracy was 5.83 m. Rotating the robot caused an error to accrue, resulting in a self-position value that was not accurate. Consequently, a single sensor seems to be unable to reliably detect all obstacles in agricultural settings, which is not surprising considering the variety and complexity of common field obstacles; however, since each perception sensor has its strengths and shortcomings, perhaps combining multiple sensors to improve obstacle detection in agricultural settings may be a better method [26]. Table 1 shows the functionalities, as well as some advantages and disadvantages of different sensor modalities for outdoor perception.

3.1.2. Multiple Perception Sensors

Considering the fact that one sensing technology cannot be effective in all situations encountered in agricultural environments, a set of sensor technologies was evaluated in [27]. The authors used a stereo vision camera, LADAR (Laser Detection And Ranging), thermal camera, and radar. The stereo vision camera effectively identified the obstacles. The thermal camera revealed high temperatures on living beings, automobiles, and water. The radar was able to create a panoramic image of the surrounding area; however, the LADAR was unable to recognize moving obstacles in time when the tractor was moving. Recently, the authors of [28] used three imaging sensors (RGB, thermal, and stereo cameras) and two active sensors (LiDAR and radar). The RGB camera was able to recognize upright pedestrians, but its performance degraded for more complicated postures. Detecting items that extend significantly above the ground was easy with the LiDAR sensor and stereo camera. The thermal camera demonstrated excellent skills by detecting things of different temperatures, such as humans and living obstacles. More recently, Christiansen et al. [29] conducted the same research as above but with fewer sensors. The authors used an RGB camera, stereo camera, thermal camera, and LiDAR. Additionally, different obstacles were considered. The obstacles included standing/lying adult and child mannequins, two real humans, and an ISO 18497 barrel. The authors obtained similar results to those stated previously. The authors of Ross et al. [30] explored the usage of multiple stereo vision cameras to detect obstacles in agricultural environments during both the daytime and nighttime. The system successfully detected all obstacles during the daytime, but humans and tires were not detected during the nighttime.

Nissimov et al. [31] suggested that a Kinect sensor could be combined with an infrared laser emitter, infrared camera, and RGB camera to develop an obstacle detection system for greenhouse environments; however, problems with smooth and shiny surfaces, misalignment between RGB and depth images, a time delay (30 s) for a stable depth measurement after a quick rotation, synchronization, and mismatch between the RGB and depth images’ fields of view and points of view were all mentioned as potential sources of error. In Ball et al. [32], a vision-based obstacle detection and navigation system for a crop field robot was created. To follow the crop rows, the robot used a combination of GPS, inertial sensors, and a stereovision camera sensor to navigate the terrain. An extra set of stereoscopic wide-angle cameras was utilized to observe and detect obstacles in the area 10 m ahead of the robot. More recently, Reina et al. [11] looked into combining stereovision, LiDAR, radar, and thermography to improve agricultural robot environmental awareness. Stereovision and LiDAR were coupled first, followed by radar and stereovision, and lastly stereovision and thermography. Additionally, to detect and avoid obstacles in an agricultural area, Franzius et al. [33] investigated a low-cost compact module made up of color cameras and an ARM-based processor board installed on an autonomous lawn mower. The results showed that the technology maintains good mowing performance while reducing collision incidents. Moreover, a new method for multimodal obstacle detection by fusing RGB cameras and LiDAR sensing with a conditional random field (CRF) was explored [34]. The proposed method was evaluated on a dairy paddock and different orchards with a perception research robot in Australia. The results have shown a better performance of the proposed method compared to when a single perception sensor was used.

Using four 120-degree viewing angle cameras in conjunction with the YOLO-v3 object identification system, the authors of Jung et al. [35] recently investigated the potential to improve obstacle detection skills. For testing, the cameras were put on a tractor. The system had an accuracy of 88.43% in identifying obstacles. In a more contemporary work, Skoczeń et al. [36] explored an obstacle detection system for agricultural mobile robot applications that used four RGB-D cameras and two LiDAR sensors. The system was able to correctly detect obstacles with a high accuracy of 95.2%. The key aspect about this system is that it generates maps that contain information on the working area, unknown places, and obstacle position. Moreover, multiple sensors have also been investigated for robust outdoor terrain classification. Reina et al. [37] conducted an outdoor terrain classification with LiDAR and two stereovision cameras. Their system successfully detected the obstacles and classified the outdoor terrain. Another study on statistical ground categorization using a combination of radar, monocular vision, four 2D laser scanners, a thermal infrared camera, and RTK was conducted by [38]. With an average classification accuracy of roughly 80%, the system was successful in detecting drivable surfaces. This feature exemplifies the potential value of combining radar and vision, in which the radar provides range information and the vision provides color-based sub-ground separation for an augmented map of the environment.

Reina et al. [39] explored the estimation of terrain classification supervised learning method to recognize terrain surface for a vehicle operating in rural settings with a stereo vision camera and electrical current as well as voltage sensors. Their system successfully predicted (89.1%) the terrain classifications. Additionally, Kragh et al. [40] investigated obstacle detection and terrain classification in agricultural fields by using a single 3D LiDAR as well as several visual and pose sensors. The system successfully classified the terrain and detected objects, such as humans, animals, cars, and buildings, with an accuracy of 91.6% and 81.1%, respectively. Finally, Christiansen et al. [41] developed an anomaly detection method for obstacle perception by using a stereovision camera composed of two Flea 3 GigE color cameras. The sensors benefit from integrating deep convolutional neural networks (CNNs) with background subtraction or anomaly detection to overcome the problem of recognizing distant and obstructed objects that are not recognized as obstacles by traditional CNNs. The studies reveal that the proposed technique can detect a human at longer ranges, up to 90 m, while maintaining real-time operation in agricultural settings.

Figure 9 displays the total number of each sensor/device used in all of the articles to improve the safety of automated agricultural machines. The most used sensor/device was stereo vision cameras, which were used 23 times. Another commonly used sensor was the 3D LiDAR (16 times), as well as thermal cameras, radars, and 3D laser scanners (nine times each). RGB cameras were used eight times, 120-degree cameras were used four times, and monocular vision cameras and ultrasonic sensors were used two times. All other sensors/devices (i.e., trinocular stereo vision cameras, HD webcams, ToF cameras, 2D laser scanners, 3D cameras, web cameras, 360-degree cameras, omnidirectional stereo vision cameras, and high-resolution CCD cameras) were only used once.

3.1.3. Datasets and Algorithms

With the introduction of numerous high-quality datasets of urban and highway driving, person detection from automobiles has advanced quickly recently; however, no large-scale benchmark is available for the same problem in off-road or agricultural contexts. To enhance research in an agricultural setting, Pezzementi et al. [42] proposed the National Robotics Engineering Center (NREC) Agricultural Person-Detection Dataset. Two perception platforms (a tractor and a pickup truck) were used to collect the data. The dataset consists of 19,000 sampled person-free photos and 76,000 labeled person images. The dataset highlights several key challenges of the domain, including varying environments, substantial occlusion by vegetation, people in motion and nonstandard poses, and people seen from several different distances. Furthermore, Kragh et al. [43] presented a multimodal dataset for recognizing agricultural obstacles. A platform of sensors/cameras, including a stereo camera, thermal camera, web camera, 360 camera, LiDAR, and radar, was mounted on a tractor. The tractor was used to mow the grass on a 3.3 hectare field featuring several static and moving obstacles. A drone was utilized to capture the positions of all of the obstacles, which were then manually labeled and synchronized with all sensor data. The whole dataset is available at https://vision.eng.au.dk/fieldsafe/ (accessed on 3 November 2022). Moreover, the lack of public datasets that address the recognition of 3D pedestrians in various agricultural situations has also held back relevant research in this area. The authors of this study created datasets for 3D pedestrian recognition in agriculture called “FieldSafePedestrian”, which contain field photographs taken during both the day and at night [44]. The data can be downloaded at https://github.com/tjiiv-cprg/3D-Pedestrian-Detection-in-Farmland (accessed on 1 November 2022). All of these datasets can be used to support agricultural robot object detection research.

The type of algorithm utilized might be a way to make obstacle detection systems more effective. Doerr et al. [45] claim that effective and robust computational methods are essential for detecting image features via image processing or dealing with sensor data fusion to provide the basic information required by agricultural vehicle autonomous guidance systems. Therefore, the choice and implementation of methods and signal-processing algorithms are critical. Many algorithms have been proposed in the literature for the detection of foreign objects or obstacles to the operation of autonomous vehicles. Yin et al. [46] evaluated the effectiveness of different feature recognition algorithms (average height, density, connectivity, and discontinuity methods) for a specific LiDAR sensor in obstacle detection. Additionally, a 3D camera was mounted on a field robot that same year to detect obstructions in an agricultural setting; however, impediments were distinguished from the noise and background data were gathered with the 3D camera by using noise and background reduction techniques. After segmenting and extracting information about obstacles, a clustering algorithm was utilized to determine when the field robot should slow down or halt during autonomous runs in the field. Recently, an information-processing architecture for multimodal obstacle as well as environment detection and recognition approach for process evaluation in agricultural areas was created by the authors of [47]. A sensor platform, inverse sensor model, fusion and mapping, and process assessment comprise the proposed architecture’s four components. Furthermore, for the safety of agricultural automated machines, especially those moving at a reasonably high speed, the capacity to identify obstacles more quickly and respond appropriately is crucial. Sadgrove et al. [48] developed a quick obstacle detection technique to be used in complex agricultural settings, called the multiple-expert color feature extreme learning machine (MEC-ELM). The MEC-ELM can locate and categorize items quickly, with 84% precision and 91% recall in weed detection in 0.5 s per frame, thanks to the color implementation employed with the SAT.

More recently, to decrease the rate of missed detection and erroneous identification of multiple farming obstacles, resulting in improved real-time detection performance, an enhanced YOLOv5 algorithm based on the k-means clustering algorithm and CIoU loss function was developed [49]. The upgraded YOLOv5 algorithm outperformed the recurrent convolutional neural network (R-CNN) in terms of small-target-obstacle identification. Moreover, it is challenging to detect obstacles precisely and effectively in orchard environments since they are complex and unstructured. Therefore, to recognize common obstacles in orchards, such as people, cement columns, and utility poles, Li et al. [50] enhanced a lightweight object detection approach based on the YOLOv3 object recognition algorithm. In the model, a Gaussian model was added to enhance the detection effect, while the MobileNetV2 network was employed to decrease the running time when extracting picture features. In terms of accuracy and speed, the system surpasses previous models, such as the single-shot detector (SSD) and R-CNN. Furthermore, agricultural robots currently utilize vision-based algorithms to detect barriers based on their classification. As a result, it is complex to locate new classes of things, which are frequent in agricultural settings given the variety of objects. Convolutional autoencoders are used to recognize any items deviating from the regular pattern in the strategy suggested by Mujkic et al. [51]. In terms of object detection, the system has demonstrated superior performance to the most recent vision-based algorithms.

Contemporary research suggests using a deep learning algorithm to locate electricity lines in paddy fields to assure the safe operation of an agricultural drone sprayer. The tiny-YOLOv3 model was used to train the dataset of electricity lines, which was collected in an agricultural setting. Powerlines that had bounding box labels were used to create the training dataset. The results of electricity line detection in real-time frames per second (FPS) are, on average, 12.5 FPS. The weakness of this method is that when the rice is not grown sufficiently, the suggested algorithm may mistake ridges for electricity lines [52]. To improve the UAV’s perception of its surroundings and its capacity for autonomous obstacle recognition as well as avoidance in an agricultural setting, the authors of Wang et al. [53] also integrated a stereo vision camera with a deep-learning-based object detection system. The convolutional neural network (CNN) model with the YOLOv3 object detection algorithm was trained and evaluated using a dataset. The findings indicate that the CNN model has an average obstacle detection accuracy of 75.4%. Moreover, another study was performed to improve the classic A* algorithms used in the obstacle detection system of plant protection UAVs using dynamic heuristic functions, search point optimization, and inflection point optimization based on the fusion of data from monocular cameras and millimeter wave radars. The improved algorithm increased the capability of the obstacle avoidance system by decreasing the number of grid searches, turning points, and data processing time [54].

3.1.4. Sensing Strategies

The perception sensor fitted on the automated agriculture machine may become less accurate due to vibration. Accordingly, a study by Periu et al. [55] found that, by reducing vibrations, a designed mounting and stabilizing system can improve the accuracy of a LiDAR sensor; however, the mean error distance between the real and detected locations of the obstacles rose as speed increased. Furthermore, for agricultural robotic equipment to operate safely, sensor robustness is essential. To determine, with confidence, if a specific sensor is reliable or not, adequate testing must be performed; however, because of the expense and high level of unpredictability in the agricultural environment, conducting sufficient tests is not always practical. To compare and evaluate the autonomous human detection of sensor systems for the functional safety of autonomous agricultural robots, the authors of Meltebrink et al. [56] developed a dynamic test stand method that has real environment detection areas (REDAs) for each sensor system. This test method allows for continuous, long-term testing of commercially accessible sensor systems throughout the year on a dynamic test stand.

Xue et al. [57] demonstrated the potential of a velocity control approach method for automated agricultural machines to prevent collisions. The control technique includes two steps: (1) a velocity generator for collision avoidance with a cloud model and (2) collision prediction in dynamic situations with an upgraded obstacle space–time grid map. The outcomes of the field tests demonstrate the strong viability and efficiency of the suggested velocity control approach. Finally, Santos et al. [58] suggested a novel open source solution called AgRobPP-CA to autonomously perform obstacle avoidance during robot navigation. AgRobPP is a ROS-based open source path planning framework (robot operating system). AgRobPP considers the robot’s center of mass and continuously checks the present robot’s trajectory for predictable collisions or potentially hazardous inclined zones. AgRobPP operates in real-time to avoid nearby obstacles, allowing deviations, avoiding unforeseen obstacles, and avoiding steep slope zones that can cause the robot to roll over, as has been the case for several tractors.

In conclusion, in this first sub-section of the review, existing research exploring the usage of automated agricultural machines’ perception systems to improve their safety was presented. The studies have focused on four main research areas: (1) The use of a single perception sensor to detect and avoid obstacles. Given the complexity of agricultural settings, it is challenging for a single perception sensor to effectively identify all types of obstacles. (2) The use of multiple perception sensors. Studies have shown that they result in considerable improvements in obstacle detection and avoidance in comparison to single perception sensors. (3) The availability of datasets of agricultural environments (including all types of obstacles) to help automated agricultural machines’ perception systems be sufficiently trained and subsequently perform well in detecting obstacles. Studies have started creating and making such datasets available. Likewise, studies have also focused on improving the types of algorithms used to manipulate data. (4) Finally, external methods intended to improve automated agricultural machines’ perception systems, such as by reducing vibrations and velocity controls.

3.2. Risk Assessment and Risk Mitigation

Another way to improve the safety of automated agricultural machines is through risk assessment and risk mitigation. There are several types of risk assessment and safety analysis methods, including fault tree analysis (FTA), failure mode and effects analysis (FMEA), human factors analysis and classification system (HFACS), and hazard analysis and risk assessment (HARA) [59].

While FMEA is a commonly used and recognized tool in engineering, it has various limitations, including subjectivity, time, cost, and information management. It also strongly relies on prior knowledge, such as experience with design engineering and risk assessment procedures, potential applications or misuses of machines, previous “accidents” and other historical occurrences affecting the equipment, and common design concerns [60]. The reliance on pre-existing knowledge provides a substantial barrier to the efficacy and usability of FMEA when dealing with new and unique technologies, such as agricultural automated machines. In line with this issue, these studies [59,61] explored if historical data involving non-automated machinery could be used to validate numerical values assigned during an FMEA assessment of automated agricultural machines. The authors concluded that using existing historical data to inform the values assigned to an FMEA of automated agricultural machines would not be appropriate. Additionally, the authors of Sandner [61] investigated the possibility of using historical fatality data of non-automated machinery to improve the efficacy of hazard analysis and risk assessment (HARA) in conducting a risk assessment of automated agricultural technology. They determined that using an operating scenario generator based on historical data to examine automated agricultural machines improves the efficacy of HARA.

The International Organization for Standardization (ISO) has been used worldwide for several decades to manage risk/safety. In general, types A, B, and C make up the majority of safety standards in the machinery industry. Type A standards are basic safety standards covering basic concepts, design principles, and general aspects that can be applied to all machinery. Type B standards are generic safety standards that cover safety aspects or one type of safeguard that can be used across a wide range of machinery. Type C standards are machine safety standards dealing with detailed safety requirements for a particular machine or group of machines. In the work presented in Shutske et al. [62], the authors studied which standards and risk assessment methods are commonly referenced by professionals working with automated agricultural machines in the industry. In their investigation, eight standards were taken into account, including ISO 12100:2010 [63], ISO 2519-1:2018 [64], ISO 2519-2:2018 [65], ISO 2519-3:2018 [66], ISO 2519-4:2018 [67], ISO 18497 [68], ANSI S318.18 [69], and ANSI S354.7 [70]. Overall, it was the case that most of them used ISO 12100:2010 when working with automated agricultural machinery. This is not surprising, since ISO 12100:2010 is a type A standard that covers a general overview of machinery safety. In addition, most individuals use a lower number of risk analysis methods and techniques with highly automated machinery than they would when analyzing non-automated machinery.

The application of risk assessment is pertinent for the application of type C standards to convert the provided design principles into specific detailed machine requirements. Type C standards contain precise safety criteria for a single machine or group of machines [12]. One type C standard for a highly automated agricultural machine is ISO 18497. To achieve safe operation, this document outlines design guidelines for highly automated agricultural equipment operations. It covers all significant risks, dangerous circumstances, and occurrences, including those that call for a particular response and are mechanical, electrical, or mobility-related. The ISO 18497 standard contains guidelines for ensuring the safety of agricultural automated machinery. The suggestions are divided into two groups: (1) safety requirements as well as protective or risk reduction measures, and (2) the verification and validation of safety requirements as well as protective or risk reduction measures. Moreover, all safety requirements can be divided into four safety systems: perception, safeguarding, control, and supervisory systems. Examples of safety requirements include measures for machine enabling operations, operational processes, machine operational status, machine operational speeds, perception systems, the verification of minimal system perception as well as safety performance, and remote halting.

However, several factors were missed in the standard. The requirements, for example, only apply to “field operations”, not to scenarios or settings in which human operators, service staff, or others might carry out repairs, travel on roadways, or mount/dismount equipment. Injuries sustained during these operations have been recorded and confirmed in previous studies [71,72]. Furthermore, there are no operations in farmyards or barns, nor on public roads. Additionally, there is no rationale for accommodating maintenance programs, whether executed in the field or in a traditional maintenance shop. These limitations were also mentioned in a more recent study [73]. Nevertheless, this standard is currently under revision to provide substantial improvements.

Under the general requirement of operational procedures, it is written that “It shall not be possible to enable highly automated operation without the perception system confirming that the hazard zone is obstacle-free”. The perception system refers to the perception sensors (LiDAR, radar, stereovision, etc.) explored above under the section on obstacle detection. The perception sensor represents the eyes of the automated agricultural machine. The obstacle is classified by the standard as a barrel-shaped object with a height of 80 cm and a width of 38 cm that is supposed to simulate a human seated position (Figure 10). Steen et al. [74] questioned the validity of the standard for ensuring the safety of highly automated agricultural machinery from an obstacle detection perspective. To detect the barrel as stated by the standard, the authors first trained a deep conventional network. Second, the algorithm detectability of the ISO obstacle on row crops and grass mowing was assessed. The results indicated that the program could detect the ISO obstacle with nearly 100% precision on row crops and 90.8% precision on grass mowing, while not detecting people and other very distinct obstacles such as animals. The key outcome of this research is that the obstacle presented in the standard is not fully adequate to ensure the safe operation of agricultural automated machinery.

In addition, obstacles are not the only hazards present during field operations. Uneven terrain, such as slopes, and avalanches could lead to autonomous vehicle rollovers. Unfortunately, these critical aspects have not been addressed within the standard.

Another study, conducted by Basu et al. [75], suggested a conceptual regulatory framework to reduce the risks associated with small automated agricultural machines from the perspective of a practical and self-contained engineering guide while simultaneously fostering innovation. The following are the main features of the legal system: (1) Multiple parties may be held jointly and severally liable for the use or operation of an automated agricultural machine. (2) Every law that establishes obligations contains equivalent defenses for dodging obligations or minimizing damages. (3) A few defenses specifically address the peculiarities of an automated system. (4) Unless the law specifically prohibits it, contracts can be used to define the obligations and rights of individual parties and shield them from liability. (5) The utility of the activity and its social as well as economic value must be considered when a court decides to award damages for loss and injury.

From a practical standpoint, because the machines must be taken out of the field to be checked, testing every piece of software and hardware that goes into automated agricultural machines is difficult, risky, and time-consuming. In order to address these issues, Kelber et al. [76] developed a method called hardware-in-the-loop (HIL) that makes it possible to verify and validate automated agricultural machinery’s software in a reliable, safe, and adequate manner. Two physical devices are required for hardware-in-the-loop validation: a machine simulator that executes a mathematical model of the automated agricultural machine in real-time and an HIL rack that houses all of the machine hardware as well as software. Automated agricultural machines experienced hardware-in-the-loop validation prior to field testing, and the results showed good consistency between the two test techniques. The repeatability, test coverage assurance, economic viability, testing time reduction, risk minimization, and anomalous behavior identification are the main benefits of hardware-in-the-loop approaches.

In conclusion, in this second sub-section of the review, existing research exploring risk assessment and hazard analysis research efforts to improve the safety of automated agricultural machines was presented. Overall, since the current risk assessment and hazard analysis methods rely on pre-existing knowledge of the equipment, they cannot be effective when dealing with new and revolutionary technologies where pre-existing data are not available, such as automated agricultural machines. Moreover, the guidelines provided by the international organization for standardization (ISO 18497) and existing research exploring these guidelines to improve the safety of automated agricultural machines were presented. While the recommendations provided by the standard are valuable, they only target field operations overlooking other operations, including repairs, travel on roadways, mount/dismount equipment, farmyards, barns, and guidelines to deal with hazardous deformable terrain. Overall, full compliance with the current requirements provided in ISO 18497 cannot guarantee the safety of automated agricultural machines. Finally, a regulatory framework and being able to test the functionalities of automated agricultural machines within a reliable software environment are efficient ways to mitigate risks.

3.3. Human Factors and Ergonomics

Understanding the human factor perspective during human–robot interaction remains a potential method with which to improve the safety of automated agricultural machines. To evaluate and improve the human factor design of a previously developed grafting robot, ergonomic analysis software was used to create computer-simulated human models [77]. The models were then integrated with a real working environment during grafting operations under various working conditions. The studies revealed that the robot needs to be redesigned to include leg space, adjust the distance between the rootstock and scion, and the operator seat location. These changes will significantly reduce the lower back and upper limb stresses of the potential operator [77]. Recently, Bashiri et al. [78] used a tractor driving simulator to assess the operator’s situational awareness when working with agricultural semiautonomous vehicles. Overall, the authors found out that as the automation level increases, the level of operator situation awareness decreases. Thus, considering the level of human situation awareness and robot’s level of automation could improve safety and work efficiency during human–agricultural robot collaborative and cooperative operations.

More recently, researchers conducted field studies in both open and enclosed conditions with strawberry harvesters, evaluating their work alongside that of a Thorvald robot. The research characterized interactions between the robot performing an in-field transportation task and human fruit pickers. From a safety and ergonomic perspective, the information gathered in this study can be used to improve the design of agricultural automated machinery, allowing safer human–robot collaboration [79]. In addition, Benos et al. [80] looked at the safety and ergonomics of human–robot interaction in agricultural operations. The authors highlighted various hazards that could jeopardize human safety and measures for reducing the risk of injury and methods for safe collaboration. Moreover, given that a crucial element in accomplishing safe human–robot interaction is human awareness, the authors of Anagnostis et al. [81] believe that providing the activity “signatures” of the workers has the potential to increase human awareness during human–robot interaction, thus contributing toward establishing safe human–robot interaction. Therefore, the authors investigated human activity variations during human–robot collaboration in an agricultural setting by using a variety of wearable sensors. The dataset was made publicly available (https://ibo.certh.gr/open-datasets/ (accessed on 7 November 2022)) for future usages of ergonomic analyses and machine learning models for human–robot interaction in an agricultural environment.

In conclusion, in this third sub-section of the review, existing research exploring human–robot interaction from an ergonomic perspective to improve the safety of automated agricultural machines was presented. Overall, (1) testing the interaction between an automated agricultural machine and a human within a simulated environment helps identify safety concerns associated with the design of the machine in advance, and subsequently provides improvement before the machine reaches the users. (2) Knowing the foreseeable human activity related to a particular task when interacting with an automated agricultural machine is essential to improve the design of the machine and subsequently ensure safe human–robot interaction.

4. Discussion

The main objective of this study was to survey, organize, review, and summarize eleven years of work addressing the safety of automated agricultural machineries and set up the direction for potential future research opportunities. In this paper, previous efforts were reviewed and organized into three categories: (1) perception, (2) risk assessment as well as risk mitigation, and (3) human factors as well as ergonomics.

• Environmental perception: The studies have focused on four primary methods: (1) The use of a single perception sensor to detect and avoid obstacles. (2) The use of multiple sensors in obstacle detection and avoidance. (3) The availability of datasets of agricultural environments (including all types of obstacles) to help automated agricultural machines’ perception systems be sufficiently trained and subsequently perform well in detecting obstacles. (4) To improve the perception systems of automated agricultural machines, external ways to improve sensor performance were investigated. Finally, given that it was observed that most of these prior research efforts focused exclusively on obstacle detection while overlooking the reaction of the automated machine afterward, future research should expand on how automated agricultural machines should robustly avoid obstacles upon detection and continue performing their tasks without downtime.
• Risk assessment, hazards analysis, and standards: Overall, since the current risk assessment and hazard analysis methods rely on pre-existing knowledge of equipment, they cannot be effective when dealing with new and revolutionary technologies where pre-existing data are not available, such as automated agricultural machines. A promising future research direction is the development of real-time situational safety risk assessment methods for automated agricultural machines. While the recommendations provided by the ISO 18497 standard are valuable, they only apply to field operations overlooking other operations, including repairs, travel on roadways, mount/dismount equipment, farmyards, and barns. In addition, the standard is missing guidelines with which to deal with hazardous deformable terrain. Overall, full compliance with the current requirements provided in ISO 18497 cannot guarantee the safety of automated agricultural machines. For future directions, the requirements provided with the ISO 18497 standard should be (1) improved to make sure that complying with the guidelines effectively ensures the safety of machines and (2) expanded to cover all other potential areas where automated agricultural machines are likely to be used. ISO 18497 is currently in review, and a future version is expected to be released. Regulatory frameworks and being able to test the functionalities of automated agricultural machines within a reliable software environment are efficient ways to mitigate risks.
• Human factors and ergonomics: Overall, (1) testing the interaction between an automated agricultural machine and a human within a simulated environment helps identify safety concerns associated with the design of the machine in advance, and subsequently provides improvement before the machine reaches users. (2) Knowing the foreseeable human activity related to a particular task when interacting with an automated agricultural machine is essential to improve the design of the machine and subsequently ensure safe human–robot interaction. For future directions, given the wide variety of human sizes, the dimensions of a wide variety of human sizes should be considered in future simulated environment experiments.

5. Conclusions

Doubtlessly, automated agricultural machines will become inseparable parts of modern farms; however, without robust safety systems the full deployment of automated agricultural machines would be delayed or even jeopardized. As a result, diverse significant research efforts have been carried out towards this end. This research provides a holistic, synthesized overview of all previous research efforts in a comprehensive manner and sets up future research directions. The outcome is that the perception of automated agricultural machines, risk assessment as well as risk mitigation techniques, and human factors as well as ergonomics have all been explored to make automated agricultural machines safer. Overall, ergonomists, safety engineers, physicians, manufacturers, Internet of things developers, governmental officials, and international organizations must work together to make automated agricultural machines safer.