An Agricultural Event Prediction Framework towards Anticipatory Scheduling of Robot Fleets: General Concepts and Case Studies

Abstract

Harvesting in soft-fruit farms is labor intensive, time consuming and is severely affected by scarcity of skilled labors. Among several activities during soft-fruit harvesting, human pickers take 20–30% of overall operation time into the logistics activities. Such an unproductive time, for example, can be reduced by optimally deploying a fleet of agricultural robots and schedule them by anticipating the human activity behaviour (state) during harvesting. In this paper, we propose a framework for spatio-temporal prediction of human pickers’ activities while they are picking fruits in agriculture fields. Here we exploit temporal patterns of picking operation and 2D discrete points, called topological nodes, as spatial constraints imposed by the agricultural environment. Both information are used in the prediction framework in combination with a variant of the Hidden Markov Model (HMM) algorithm to create two modules. The proposed methodology is validated with two test cases. In Test Case 1, the first module selects an optimal temporal model called as picking_state_progression model that uses temporal features of a picker state (event) to statistically evaluate an adequate number of intra-states also called sub-states. In Test Case 2, the second module uses the outcome from the optimal temporal model in the subsequent spatial model called node_transition model and performs “spatio-temporal predictions” of the picker’s movement while the picker is in a particular state. The Discrete Event Simulation (DES) framework, a proven agricultural multi-robot logistics model, is used to simulate the different picking operation scenarios with and without our proposed prediction framework and the results are then statistically compared to each other. Our prediction framework can reduce the so-called unproductive logistics time in a fully manual harvesting process by about 80 percent in the overall picking operation. This research also indicates that the different rates of picking operations involve different numbers of sub-states, and these sub-states are associated with different trends considered in spatio-temporal predictions.

1. Introduction

Precision agriculture concept is based on information-intensive agriculture, prescription farming and identification of suitable tools to minimize cost through enhanced productivity while simultaneously conserving the resources [1]. Nowadays, cost-effective intelligent systems based on automation and robotics technologies are needed for improved performance in agricultural fields. Such cutting-edge solutions must also be inherently safe and reliable for human and environment [2,3]. Harvesting operations in agricultural farms are not only about safely picking up the crop yields but also about optimal resource management and operational logistics planning. These concerns are more critical in soft-fruit farms, as the short-shelf life of the fruits, that requires transportation of picked fruits to a cold storage facility as soon as possible. During picking operations, human pickers spend around 20–30% of the overall working time just in logistics activities such as transportation and unloading, where they walk with trays of picked fruits and empties by moving back and forth from the crop to the storage location [4,5]. Shortage of skilled farm workers is another major difficulty faced by many commercial soft-fruit growers around the world, resulting in fruits left unpicked, especially during worst-case scenarios like economic crisis and pandemic [6].

Though farming systems employ large-sized machines like tractors, nearly 10% of the field area is used for operating these machines for logistics operations [7]. Hence, the usage of small size robots, or even a fleet of them, has an advantage over conventional farm vehicles for better utilization of the agricultural land area. Moreover, modular autonomous robots such as the Thorvald platform can be easily reconfigured for different in-field tasks and do not demand for any special arrangements [8].

An initial Wizard-of-Oz (WoZ) study in [9] was conducted with the Thorvald robot in a real strawberry farm in the presence of human pickers (Figure 1), to check the advantages that a robot brings to workers during harvesting. The authors evaluated how an autonomous robot for in-field transportation system would assist fruit pickers and what benefits this solution could provide to the production process. The in-field transportation is still an open problem for the production of various fruit crops (e.g., strawberries, raspberries, apples), which demand the deployment of sustainable management approaches for robot fleets in real field applications [10]. Optimal management of multi-robot systems is the key idea to reduce operating time of harvesting and, thus, deploying these robots based on spatio-temporal predictions of harvesting events would improve efficiency of the human pickers and eventually benefit the farmers.

The conceptual idea of using a fleet of coordinated vehicles for agricultural tasks is not a novelty. Many research groups are developing fleet management systems for dealing with task management, motion planning, traffic control, data analysis, and fleet monitoring (e.g., battery levels, mission status, robots locations and so on). In [11] the authors discuss some hardware architectures for robot fleets to improve reliability, lower development costs, and allow software integration from different developers. Three different hardware topologies for fleets of robots were introduced to reduce the computing system to a minimum: central-external controller, central-internal controller and immerse controller. In [12], the authors address the design, development, testing and assessment of a new generation of heterogeneous robot fleets (ground and aerial), equipped with innovative sensors, enhanced end-effectors, and improved decision-making algorithms for effective weed and pest management in the crop fields. A novel augmented reality system is proposed in [13] to help farmers supervise the operation of autonomous agricultural machinery towards efficient fleet management and safe operation based on situational awareness. A cloud-based in-field fleet coordination system for multiple sequential operations on large-scale farms is introduced in [14], wherein the authors developed a collaborative self-driving operating system for fleet management using a strip-state updating algorithm for successive field operations under an optimal strategy for conditioning the waiting time between sequential operations. Following this trend, fleets of autonomous robots designed to monitor plants, eliminate weeds and harvest crops with human awareness can definitely become commercially available in the coming years.

1.1. Problem Formulation

The main motivation for this work comes from the RASberry project, a.k.a. Robotics and Autonomous Systems for Berry Production, that aims to develop an autonomous fleet of robots operating in human-assisting environments for agricultural farms of strawberry production [7]. The tasks available in the field are shared among the fleet of robots which are governed by a central coordination system to ensure an optimized task distribution across the entire fleet. To optimize the number of robots in a fleet and their workflow, the fleet coordination component must learn how to anticipate human worker needs and optimize the spatial distribution of robots available for a picker. According to [4], autonomous robots were provided “on-demand” of pickers for in-field logistics. They have used an online mobile-based application to request, cancel or to get the status of a particular transportation robot. We suspect that the “on-demand” policy in in-field logistics is prone to human errors with inability to use digital devices, or waste in time for pickers while waiting for robot to reach to them. To overcome this, a key idea for an improvement in in-field logistics is shown in Figure 2, that emphasizes sending a robot to the picker’s location by “anticipating-demand” instead of “on-demand” while they are in picking operation. This solution aims to get rid of the portable digital device and is also capable of reducing the pickers’ waiting time for a robot.

1.2. Related Works

Deploying fleet of robots is a cutting-edge solution, which is predominant in warehouse industries [15] but with still few applications in agriculture [16]. Much of the existing works in agri-robotics mostly address problems of path planning, and goal assignment of multiple robots and farm vehicles [17,18], with a few use cases on human-robot assistance.

Agriculture is not a money driven industry, which means it also asks for productivity gain on each cost invested in the implementation of robots on the field [2]. Indeed, different regions have distinct field characteristics, and infrastructure. Under such a dynamic environment, a faster analysis tool would help in exploration of different processes and operation in in-field logistics before actually deploying the robots. In this context, the Discrete Event Simulation (DES) framework is adopted from [4], to model the picking scenarios in presence of soft-fruit human pickers and fleet of robots and validating our prediction framework.

Spatio-temporal predictions in agricultural fields are discussed mainly for crop management and yield estimation [19,20,21]. Still, there is not much concern regarding the modeling of the human picker’s motion in constrained agricultural environments. In a recent work [22], the authors have developed a prediction approach for localising and tracking human pickers inside polytunnels using a novel Topological Particle Filter (TPF) algorithm. However, this solution is still limited to “on-demand” calling a robot for transport assistance. Our proposed prediction framework is working towards the automatic calling of the robot by “anticipating-demand”.

According to [23], machine learning-based methods were applied for human motion prediction integrated with video data or used for other forecasting applications. However, such methods have a few common pitfalls such as hyper-parameter tuning, depth and complexity of the models. Recently, there is a growing interest in dynamic object motion prediction based on Bayesian approaches for efficient human-robot collaboration [24,25], combined with optimal motion planning and collision-free trajectory generation for mobile robots [26,27].

Hidden Markov Models (HMMs) [28], a particular case of Bayesian network, are attractive for the modeling of the state progression [29]. However, conventional HMMs has the limitation of dealing with irregularly sampled continuous-time data such as in our case of picking operation performed by human pickers. Based on proven research on similar type of data in medical applications of diseases progression [30,31,32], human and vehicle motion prediction [33], and human-motion recognition [34], we propose the use of a Continuous-time HMM (CtHMM) for the spatio-temporal prediction framework for different picking scenarios.

1.3. Our Proposal and Contribution

The proposed spatio-temporal prediction framework consists of two prediction models: one is called picking_state_progression model and other is called node_transition model. The former predicts the temporal dynamics of harvesting activities (tray_full_time) and the latter predicts the spatial dynamics for individual pickers (tray_full_node). The validation of the prediction framework is carried out by using the DES framework [4] fed with data from real harvesting operations. Then, two test cases are considered: in Test Case 1, we have the picking_state_progression model working for a given average picking_time to observe how efficiently a temporal progression of a state can be predicted, without loosing too much of information over a period of time. In Test Case 2, the temporal prediction outcome obtained from the picking_state_progression model and the mean_picking_rate are given as input in the node_transition model, to predict picker’s location in a 2D discretized topological map of the agricultural field. The relation between the picking_time and mean_picking_rate is given as follows: the picking_time for a “tray-full event” is equal to the topological node-to-node distance divided by the mean_picking_rate multiplied by the farm yield.

In this work, our main contributions are as follows: (i) develop a spatio-temporal prediction framework of the picker activity status by exploiting the constrained 2D discrete map information of the environment called topological map and using statistical learning methods. This model should investigate the possible progression of a state and accordingly making anytime prediction of the location of the pickers; (ii) based on simulations with real-field data through case studies, we have also analyzed the feasibility of our prediction models in reducing unproductive time such as, transportation time or waiting time of human pickers during picking operation along the crop fields.

This paper is organized as follows: in Section 2, we present the system overview, followed by a brief discussion on the topological map and a state abstraction for the picking operation. Section 3 presents the proposed spatio-temporal prediction framework, the states of the individual CtHMMs and the predictions obtained from the CtHMMs; in Section 4, we introduce the experimental evaluation and the analysis of the simulation results for two case studies; Section 5 finally presents the concluding remarks and perspectives for future works.

2. System Overview

In this section, an overview of the whole field picking operation for the RASberry use case and some of the necessary tools to achieve this goal are reported, which are needed to better understand the evaluation of the results.

2.1. The Topological Map

In the DES framework, the agricultural field is discretized into a 2D topological map [35], which is a graph of discrete points located in a 2D space called as topological nodes. A general representation of the topological map is shown in Figure 3, where a topological node is denoted as a waypoint node. According to [36], it provides a unified system that can tackle different challenges in terms of continuous navigation of autonomous robot in agriculture field. It can be described as a tuple $\mathcal{T}=〈N,E,A,nav〉$, where N is a set of physical location points $\{n_1,n_2,\cdots ,n_{\kappa }\}$, $\kappa$ is the total number of locations, and $E\subseteq N\times N$ shows the set of possible edges connecting the nodes element i-th row and j-th column of E, defined as:

$$E_{ij}=\left\{\begin{array}{c}1,\mathrm{if}{n}_i\text{connects to}{n}_j,\hfill \\ 0,\mathrm{elsewhere}.\hfill \end{array}\right.$$

(1)

The navigation from one topological node to another is possible only if an edge exists between them. Here, A is the set of actions performed by robots to navigate by mapping of each edge to navigation action, $nav$. The picking process is discretized by looking at the changes in the picker activities while moving from one topological node to another.

2.2. A State Abstraction for Picking Operation

To better understand the distinct states of human pickers in a real-time fruit picking operation, below we discuss manual and robot-assisted picking operation procedures. The picking operation protocols are almost similar for different fruit picking environment such as strawberry polytunnels or apple orchards. The operation is described as follows: at the outset of picking operation, a group of human pickers are assigned to a certain row of an agriculture farm. They will start picking fruits in tray by moving forward or backward (i.e., ascending or descending order respectively, of topological map node IDs) in a given crop row. When a tray is full (called a “tray-full event”), in manual operation Figure 4, they themselves will transport and unload the tray by moving to the local storage. However, in robot-assisted operation Figure 5, they will push a button interface of an in-house app called CAR (Call-A-Robot) system on a mobile device and wait for the robot to come to nearby location [22]. Then, the pickers get the service in transportation and unloading of the fruit tray(s). In both cases, after finishing unloading the tray, they resume picking again from the last position they were picking, or in a new row if the previous row is finished. This cycle of events repeats until the harvesting tasks is completed for all crop rows in the field. For the sake of simplicity, we have assumed that the robot speed and the transportation speed of the human picker are the same.

The bottleneck problem in both manual and robot-assisted picking operations is that the pickers have to spend significant time on unproductive activities, either transporting trays or waiting for the robot to arrive respectively. Reducing the time of unproductive activities forms the basis for the evaluation of our proposed anticipatory scheduling framework.

Categorizing the Picker’s States

According to [4], we have categorised pickers activities into six states from the real-time data observations in RASberry use-case picking operation as Idle, Transportation, Picking, Unloading, Loading, and Waiting, as shown in Figure 6a. Among all states, Picking (or picking state) is found to be the dominant one which depicts the time duration of a “tray-full event” (or tray_full_time) and all other states are highly dependent on it. Thus, in this work, we are concerned with modeling picking_state_progression for prediction. An example of time duration of each picker’s state is shown in Figure 6b.

3. A Topological Map-Based Spatio-Temporal Prediction Framework

This section presents the proposed spatio-temporal prediction framework for the RASberry use case, discussing the details of the individual CtHMMs and the system architecture processes for “tray-full events” predictions used in our test case studies.

3.1. Spatio-Temporal Prediction Modules

As shown in Figure 7, our prediction framework consists of two modules each with a particular CtHMM. The first module comprises the picking_state_progression model, which outputs the expected time of filling up of a tray as well as predicts temporal progression with respect to percent filling of a tray, for a given average picking_time as input. Here, we assume that every state (e.g., the picking state) is associated with a probability of transition to another state, which is a function of state duration and resets to zero once a transition to another state occurs. Therefore, without loss of generality, we consider that the picking_state_progression as an exponential function defined by the mean of resident time in a state. Being a dominant state, it has high chances of self-transition for input picking_time. Then, the picking_time is discretized into several segments called sub-states. The sub-states imply that the state of picker will exhibit the same behavior for a certain period. Then, each sub-state signifies the percentage of the tray that is full at a given time in the future. These achieved sub-states correspond to “hidden states” of the picker. Later, we will determine the number of such sub-states required in the spatial prediction model using a statistical model evaluation method.

The second module comprises the node_transition model, which predicts topological node location for input temporal progression of a tray and mean_picking_rate. The node_transition model is a spatial prediction model created by exploiting the discrete nodes of the topological map of a given agricultural field into a CtHMM and using it to predict the picker location at any time in the future. In the next section, we will explain in detail both individual CtHMMs.

3.2. Design of Individual CtHMMs

Each CtHMM model represents the sequence of observed feature vectors corresponding to each hidden state, considering an emission probability. A graphical representation of a general model state transition is shown in Figure 8. A CtHMM can be represented as a tuple $\lambda =〈SS,A,\pi ,O,B〉$ for state transition at each unit of time t [37]. These parameters for our models are defined in what follows: SS is the set of hidden states $SS=\{s{s}_1,s{s}_2,\cdots ,s{s}_n\}$ and n is the number of states in the hidden state sequence. For the picking_state_progression model, $s{s}_i\in SS$ is the i-th sub-state within the picking operation starting from an empty tray and ending with a “tray-full event”, which is equivalent to a fixed progression of filling a tray. In the node_transition model, these hidden states are the topological nodes in the environment (node_ID), representing the spatial position of the picker. The initial state or prior probability vector $\pi \in {\mathbb{R}}^{1\times n}$ is given by:

$${\pi }_i=\mathit{P}\left(s{s}_i\right),i=1,\cdots ,n,$$

(2)

which determines in which state $s{s}_i$ the picking process begins. $A\in {\mathbb{R}}^{n\times n}$ is the state transition matrix where its entries $a_{ij}$ provide the transition probability from state $s{s}_i$ to state $s{s}_j$ as:

$$a_{ij}=P\left(s{s}_j\right|s{s}_i),s{s}_i,s{s}_j\in SS,$$

(3)

where the state transition coefficients satisfy

$$a_{ij}\ge 0,i,j=1,\cdots ,n,\mathrm{and}{\Sigma }_{j=1}^n\left(a_{ij}\right)=1,\forall s{s}_i\in SS.$$

(4)

Here, O is the set of observable states $O=\{o_1,o_2,\cdots ,o_T\}$ or observations, where T is the number of samples in the observed response sequence. For the picking_state_progression model, $o_k\in O$ is the k-th observation (emission) within the picking operation, which is equivalent to the percentage of picking progress and denoted by x. In the node_transition model, these observable states are the topological node location denoted by node_ID. For every hidden state, $B\in {\mathbb{R}}^{n\times (n+1)}$ is an output observation distribution whose entries $b_{ik}$ provide the emission probability as:

$$b_{ik}=P\left(o_k\right|s{s}_i),o_k\in O,s{s}_i\in SS.$$

(5)

3.3. Predictions from the CtHMMs

The overall processes involved in the prediction from either of the CtHMM are shown in Figure 9. In this case study, data from the harvesting operation of a commercial strawberry production field were collected from devices such as cameras, Garmin GNSS receiver, and CAR system [22]. The data from all the sensors were processed to extract the timestamps of each picker activity. For example, we have visually inspected and fetched the time duration of a tray full event by tracking each picker in the video manually. Similarly, we have processed GNSS timestamps as well as CAR system timestamps to classify different picker activities. More details about the CAR system can be found in [22]. After that, temporal features were extracted and mapped to different picker’s state (see Figure 6a). Next, the average picking_time for a tray and the mean_picking_rate are fed into the CtHMM for picking_state_progression and node_transition models respectively. The parameters of the CtHMM are calculated using the aforementioned Equations (2)–(5). From our proposed architecture, the objective is to predict “when” and “where” a picker would fill a complete tray with respect to the topological node location, given the observations of the picker’s current picking progress.

Since the sub-state driven models are nearly nested, it is expected that the more complex model with higher number of hidden states would have larger posterior likelihood, and at least a good fit, as the simple model. However, the main questions that arise when we apply CtHMM are: (i) how to enquire the certainty of the state posterior distribution of the observation sequences from different sub-state models and accordingly choose an optimal model with suitable number of hidden states? And (ii) how to get the `best-fit’ hidden state sequence $SS=\{s{s}_1,s{s}_2,\cdots ,s{s}_n\}$ of the given observation sequence $O=\{o_1,o_2,\cdots ,o_T\}$ and parameters of the sub-state model $\lambda$?

The first problem is solved using the Kullback–Leibler divergence (KLD) method as being suitable for model comparison in the Bayesian framework, that intuitively measures how much a given arbitrary distribution is away from the true distribution [38]. Basically, it computes the information loss in the fitted model relative to that in the reference model and is denoted as:

$$KL\left(p\right|\left|q\right)=\left\{\begin{array}{c}{\int }_{-\infty }^{\infty }p\left(x\right)·\ln (p\left(x\right)/q\left(x\right))dx\mathrm{otherwise},\hfill \\ \infty \mathrm{if}p\ne 0,q=0.\hfill \end{array}\right.$$

(6)

where $p\left(x\right)$ is represents the “true” distribution of data (observation) and $q\left(x\right)$ represents the approximation (model) of $p\left(x\right)$. In this paper, KLD is compared against uniform distribution $q\left(x\right)$, which is defined as a 1-D array of n samples with an equal probability of $1/n$. Apparently, the model with minimum KLD index is considered as the optimal choice. However, in this paper, the threshold KLD index is set close to 1 for a model with a given sub-state size and picking operation time interval, to be considered a good fit. In other words, a simpler model with less number of sub-states is compromised with a complex model with more number of sub-states only if the KLD value is found within range 1 ± 0.05. The second problem is solved by using the Viterbi algorithm which is based on the maximum likelihood estimation (MLE) [39]. Next, we will present two case studies and their corresponding experimental analysis.

4. Experimental Evaluation

In this section, we present the experimental evaluation to demonstrate the effectiveness and feasibility of the proposed prediction framework. We have carried out multiple DES trials with prediction models running on the Ubuntu 16.04 LTS operating system based-workstation with Intel^® Core^TM i7-7500U CPU and 16 GB RAM. Our DES framework follows the picking protocols from the RASberry use case for picking operation, as discussed in Section 2.2. The DES framework codes were written in Python, enabling easy integration with the Robot Operating System (ROS) framework [40]. The DES trials with/without prediction framework have adopted and initialized the field variables, according to the DES framework explained in [4], as follows: normalized farm yield as $0.0650\pm 0.0013$ crates per discrete node distance, picker’s normalized picking_rate and transportation_rate as $0.0375\pm 0.0007$ ms⁻¹ and $1.00\pm 0.02$ ms⁻¹ respectively. In our experiments, we are using 6 rows of an open rectangular field of strawberry plants, each row 120 m long and separated by 1.5 m with equidistant topological node-to-node distance of 5 m. The resulting topology (see Figure 3) of the site corresponds to a comb pattern with the main transportation route containing the local store connecting starting points of all rows. On average, the simulations reported in this work took approximately 13–15 min to simulate all the events of roughly $2:30$ h clock time.

Our CtHMM-based prediction framework predicts the next “tray-full event” for each of the pickers not only based on field conditions, but also based on their individual conditions (e.g., tray start node, time and picking directions, pickers having a break, and so on). These predictions are made periodically to incorporate any changes in the conditions of individual pickers. For example, after each picker completes a “tray-full event” in the field, the mean_picking_rate and average picking_time for that tray are calculated from their tray_start_node, tray_start_time, tray_full_node and tray_full_time, and are used to have a better mean rate to be used for future predictions. The actual tray_full_node and tray_full_time are obtained from the CAR system which is used by the picker to call for a robot in regular operations and as a monitoring system when robots are allocated based on the predictions. The experimental tests were carried out and the analysis of the results are presented in the following subsections.

4.1. Test Case 1: To Choose an Optimal Picking_State_Progression Model

We will first explain in details the parameters chosen for the picking_state_progression model and then, in the later part, we show the analysis of the experiment outcome. In this experiment. we want to get the optimal number of hidden states (sub-states) for a given picking_time in order to achieve an optimal model.

Accordingly, the picking_time is divided into n sub-states where each sub-state means 100/n % of a “tray-full event” occurrence. Once picker is in picking state, he/she would be in such a state till a tray is finished, which means the sub-state transition always happens in forward direction. Since the tray filling is naturally progressive, the given sub-state transition probability in the model, denoted by the state transition matrix A, is cyclic, having an extremely low probability in the reverse direction.

The output observation distribution B is set for the prediction model such that each sub-state has $70\%$ of probability to emit itself as an observation, another $10\%$ for each neighbouring sub-state and $0.1\%$ for all observations, to ensure numerical stability. We also include $10\%$ change for occurrence of “unknown” noisy observation, which each node is equally likely to emit in prediction. The initial state probability $\pi$ is set uniformly as 1/n, for each hidden state (sub-state).

To validate our picking_state_progression model, several runs on DES framework were executed considering different number of sub-states. First, we start with a random sub-state size of 5, and the corresponding graph results for each test run with sub-state size incremented up to 30 are plotted in Figure 10. Then, the goodness of the fit to the data is checked using the Viterbi algorithm. The log-likelihood distribution is then inferred using KLD analysis. For the sake of simplicity and practical aspects, we have only considered few situations to demonstrate the KLD analysis. According to Figure 10 and from the results shown in

Table 1. Changes in the KLD value versus the number of sub-states.

Number of Sub-States	10	15	20	25	30
KLD Value	0.46	0.60	0.71	0.92	1.006

, for a picker’s mean_picking_rate equal to $0.0375$ ms⁻¹ and the prediction time interval 9, the KLD value for the prediction model with number of sub-states 30 was found to be closer to the true distribution than in other cases.

4.2. Test Case 2: Spatio-Temporal Predictions of the Picker’s Motion

Based on the results obtained from Test Case 1, a picking_state_progression model with 30 substates is combined with node_transition model to make the complete “tray-full event” prediction in Test Case 2. Here, it is assumed that the picker is following the picking protocol (see Section 2.2). The CtHMM parameters (A, $\pi$, B) used in this model are calculated as the same way as in Test Case 1.

To check the accuracy of the predictions, the DES framework was run with both CtHMMs and analyzed against the actual data obtained from the DES running based on the manual picking operation, with two pickers. The prediction results were evaluated with respect to the occurrence of a “tray-full event” for a single picker and the overall picking operation with multiple pickers, to understand the feasibility of using an anticipatory scheduling strategy. In this work, our focus is on reducing the time consumption in logistics operations based on the human picker’s activities predictions, although the DES trials are executed with robots equal to the number of pickers in each farm row, as shown in Figure 3. However, the explicit evaluation of an optimal number of robots required to assist the human pickers in the field, the task allocation, and the concerns regarding the scheduling of robots are out of the scope of this work for now.

Here, for the sake of better understanding, we will adopt the acronyms mentioned in detail in

Table 2. List of Acronyms and Abbreviations used in the text with their definition.

Acronym	Definition	Description
TFT	Tray Full Time	Time taken to fill a tray, denoted by tray_full_time.
PTE	Prediction Time Error	Difference between actual and predicted $tray\text{\_}full\text{\_}time$ (TFT), denoted by prediction_time_error.
TTT	Transportation Time per Tray	Travel time taken by picker to unload a full fruit tray to the local storage, denote by transportation_time.
ANCT	Average Nodes Covered per “tray-full event”	Nodes distance on an average a picker covers for filling up a tray.
NTT	Node Transition Time	Time taken by a picker while picking, to move from one node to its subsequent node.
ATT	Average “tray-full event” Time	For overall process, an average time taken by a picker to fill a tray.
APCT	Average Process Completion Time	For overall process, an average time taken to pick a complete farm.
STT	Save on Transportation Time	Based on the usage of the prediction model, a reduction in transportation time.
FoO	Frequency of Occurrence	Likeliness of an event to occur.
PAE	Percentage Average Error	For overall process, the absolute difference between actual and predicted values in percent.

: $tray\_full\_time$ (TFT); $prediction\_time\_error$ (PTE); $transportation\_time$ per tray (TTT); number of average nodes covered per “tray-full event” (ANCT); Node transition time (NTT); average “tray-full event” time (ATT); average process completion time (APCT); save on transportation time (STT); frequency of occurrence (FoO); percentage average error (PAE).

4.2.1. Analysis for a Single Picker

As mentioned previously, this analysis is needed for a specific CtHMM model with mean_picking_rate, and it would keep providing the same results for the same picking start node and time. Otherwise, different CtHMM models can be created for different pickers based on their picking dynamics or changes in field dynamics (say, changes in node-to-node space), resulting in changes in “tray-full event” prediction (and consequently PTE) as well. As shown in

Table 3. Analysis of the “tray-full event” for a picker (Id 00).

	Start				Prediction				Actual				PTE per Tray
						End	End			End	End		prediction_time_error
ray No.	row_ID	node_ID	Time (s)	Direction	row_ID	node_ID	Time (s)	Direction	row_ID	node_ID	Time (s)	Direction	Time (s)
1	00	00	1.9	F	00	16	2096.7	F	00	16	2114.9	F	18.9
2	00	16	2393.6	F	02	08	4630.3	F	02	07	4475.5	F	−154.5
3	02	07	4664.7	F	02	23	6764.5	F	02	22	6641.3	F	−123.2
4	02	22	6978.1	F	02	11	8949.0	R	02	11	8935.6	R	−13.4
5	02	11	9163.8	R	04	04	11,138.5	F	04	04	11,150.2	F	11.7
6	04	04	11,311.9	F	04	19	13,281.5	F	04	20	13,399.4	F	117.9
7	04	20	13,718.8	F	04	13	15,680.1	R	04	13	15,686.2	R	6.1

, it is observed that a picker with an Id 00 starts picking at 1.9 s in forward direction from row_ID 00 and node_ID 00. Then, Tray 1 is anticipated to finish in 2096.7 s, but actually it takes 2114.9 s with same number of covered nodes, 16, and so on. Similarly to the real appointed nodes row_IDs, the picker is predicted to cover 3 rows (row_ID 00, 02, 04) and to finish 7 full trays in total completion time of 15,680.1 s whereas in actual it takes 15,686.2 s. The predicted forward (F) and reverse (R) directions for picking trays are found to be same as the actual data. However, in few cases of the predicted “tray-full event” (e.g., Tray No. 2, 3, 6), the node_IDs are found to be deviated by $\pm 1$ node from the actual tray full node location of a picker. If the predicted node is ahead of actual node, for example Tray No. 2 and 3, the PTE per tray is always found negative. Such a value came up as a result of an extra prediction time used to cover one extra node. Otherwise, when the predicted node is equal to (or less than) the actual node, PTE gives mostly positive value, except for a few cases (e.g., Tray 4). It is worth mentioning that reducing the distance between adjacent topological nodes from 5 m (say, 2.5 m) can also reduce this negative PTE.

In our case, as a convention, when applying an anticipatory scheduling strategy with robots, a negative value of PTE is called as picker_waiting_time for the robot (e.g., Tray No. 2, 3, 4), whereas a positive value is called as robot_waiting_time (e.g., Tray No. 1, 5, 6, 7) for the picker. It is worth noticing that, we are concerned in this work only for saving the picker_waiting_time and not the robot_waiting_time. However, for deploying an optimized anticipatory scheduling of the robot fleets for human pickers both waiting times need to be considered in a next work.

The predicted results for human-robot interaction based on an anticipatory scheduling are evaluated, as shown in

Table 4. Class categorization of “tray-full event” Predictions based on PTE values.

Class	Pred.	Act.	PTE	Who’s	Empirical	FoO
	Node	Node	per Tray (s)	Waiting	% STT (Approx.)
1	same	same	$0<$ PTE ⋘ TTT	Robot	100	High
2	behind	ahead	$0<$ PTE ≤ TTT	Robot	$(80,100$)	High
3	ahead	behind	−2TTT ⋘−PTE <−TTT	Picker	Not Applicable	Medium
4	same	same	−TTT ⋘ PTE $<0$	Picker	$(85,100$)	Low

. These results show us how much probable STT can be achieved, according to the predicted node_ID for a “tray-full event” and PTE. Then, four main prediction classes were observed. In Class 1, a robot can be sent to the actual current node of a picker and wait till he/she finishes a tray. Then, the picker can unload to the robot without even travelling much distance with a filled tray. In Class 2, the picker might need to travel just about one node (i.e., about 5 m) to unload, and can save on transportation time by avoid travelling all the way to the local storage. In Class 3 and Class 4, the picker will be waiting for a short period of time, till a robot reaches to the picker’s current node location. This waiting time would be extremely small (e.g., PTE of Tray 4) in comparison to the average TTT.

Considering the transportation speed $1.0\pm 0.02$ ms⁻¹ for a picker, as discussed earlier, the TTT is calculated as approximately $121.57\pm 13.39$ s to unload 7 trays. Hence, for all classes, % STT per tray is obtained by using the following relation:

$$\mathrm{STT}(\%)=\frac{\mathrm{TTT}-\mathrm{PTE}}{\mathrm{TTT}}\times 100.$$

(7)

In Class 3, a robot is predicted to be sent to a node located ahead the actual picker location (e.g., Tray 2, 3). Then, a robot can not obviously pass the picker, but it needs to stop at the picker’s current tray full node. The PTE value in such classes consists of an “offset” of an extra predicted Node Transition Time (NTT). Here, NTT is defined as “how much time a picker takes to move from one node to its subsequent node while picking”, being calculated as:

$$\mathrm{NTT}=\frac{\mathrm{TFT}}{\mathrm{ANCT}}.$$

(8)

When removing the NTT from PTE it will provide us the PTE value (picker_waiting_time or robot_waiting_time) for the predicted node which should be the same as the picker’s actual “tray-full event” node. The actual NTT and predicted NTT are normally distributed as $146.57\pm 5.70$ s and $129.22\pm 7.28$ s respectively. Thus, for a particular situation like Tray 2, PTE is found as $-25.2$ s instead of $-154.5$ s. However, for such classes in general, picker_waiting_time is normally distributed as $17.35\pm 2.23$ s. Correspondingly, considering the situation where a predicted node is either the same as or ahead of picker’s actual tray full node, % STT per tray is calculated by using Equation (7), as approximately $85.72\pm 0.37$%.

4.2.2. Analysis for an Overall Picking Operation

According to the prediction analysis shown in

Table 5. Overall Picking Operation Analysis Matrix with Values Expressed as $\mathrm{mean}\pm \mathrm{std}$.

	Prediction	Actual	PAE
APCT (s)	15,787.85$\pm 142.48$	15,726.02$\pm 56.28$	$0.38\pm 0.77$
ANCT	$16.07\pm 0.82$	$15.78\pm 0.80$	$1.8\pm 0.10$
ATT (s)	$2255.26\pm 28.78$	$2246.57\pm 11.37$	$0.38\pm 1.09$

, for the overall picking process of 7 tray full events with two pickers, the predicted APCT were 15,787.85$\pm 142.48$ s. On average, this value is $0.38\%$ more than the actual APCT, which is 15,726.02$\pm 56.28$ s. The increase in the predicted APCT was expected as a result of the randomly distributed picking and transportation rates. Based on the “tray-full event” analysis of the whole process, the predicted ANCT by an average picker to fill a tray is approximately $16.07\pm 0.82$ with an ATT normally distributed as 2255.26 $\pm 26.78$ s. On the contrary, in actual, ATT is approximately $2246.57\pm 11.37$ s and ANCT is approximately $15.78\pm 0.80$. Hence, based on the PAE value of the overall process, the predicted picking process is approximately $0.38\pm 1.09\%$ slower than the actual picking process. In addition, the predicted process shows an “offset” node coverage of approximately $1.8\pm 0.10\%$, which means $0.28\pm 0.03$ number of nodes more than the actual number of nodes to fill a tray.

Based on the predicted and actual values of ATT, the PTE value is calculated as picker_waiting_time of $8$ s with a little deviation of $24.64$ s. Hence, this ensures that the PTE value is falling into an acceptable category of Class 1, 2, or 4, where either picker has to wait at the “tray-full event” node for a while or travel a few meters of distance, in order to unload a tray. Accordingly, based on the estimated values of TTT and PTE for the overall process, the % STT is calculated by using Equation (7), as approximately $79.74\pm 9.70$. Additionally, considering the safety of the picker while interacting with robot under an anticipatory scheduling strategy, the error in node prediction is found within the acceptable range.

It is worth mentioning that the current pilot study is conducted to check the feasibility of the proposed prediction framework for fruit picking operations. It is also intended to create a baseline prediction model against manual picking operation. Here, running the DES framework for manual operation implies that the robots are included and sent to the picker’s location as/when they call for a robot. Conversely, running the DES framework with CtHMMs implies that the robots are included and sent to the picker’s location in advance by anticipating the requirement of the robot during the picking operation. Future research work will focus on implementing autonomous robot-assisted picking operations, with and without an anticipatory scheduling framework. Furthermore, the DES framework can be extended to include a fleet of robots with a high number of pickers in the operation. Please see [4] for allocating logistics tasks to a fleet of robots. It is noteworthy that our prediction framework can be easily extended to a large number of human pickers. We can also group pickers with similar picking rates together and use a single model for making predictions for each of them.

5. Conclusions and Perspectives

In this work, we have presented a preliminary study for developing the anticipatory scheduling of robots in the agricultural field to assist human pickers in various operations. The utilization of the DES framework allows the systematic analysis of different picking scenarios based on predictions. Based on our simulation results for an individual picker, at each “tray-full event”, the prediction methodology has depicted four classes, with respect to the predicted and actual node location in the topological map. Three classes (1, 2 and 4) have proved to be favorable for the anticipatory scheduling when the expected node is located ahead or at the same node as the actual node of the picker.

Spatial and temporal predictions using two CtHMMs have been shown to be quite effective with small error in operating time of approximately $0.38\%$ greater than in the actual case for the overall picking process. An average picker is predicted to cover just $0.28\pm 0.03$ number of extra nodes to fill a tray picking in similar direction as that of the actual case. Another advantage of this study is that scheduling a robot based on our prediction model may reduce the transportation time approximately by $79.74\pm 9.70\%$, on overall picking operation. In our case studies on real-world farms, it was observed that the pickers hardly ever take a break until the harvesting operation is completed, and this is not considered in the evaluations. Even if a picker takes a break from the harvesting operation, due to the periodic predictions being made with our spatial-temporal prediction framework, predictions for the individual pickers who are taking a break can be removed from this particular round of predictions. We believe that reducing transportation time can improve the utilization of human pickers in agricultural fields and help reduce their physical strain.

The main outcomes of this work are quite promising and allow us to glimpse future applications of such an event prediction framework for human-robot collaboration in manufacturing and autonomous driving in urban environments. In the future, the focus would be to implement our prediction model in different agricultural fields to analyze the effectiveness of the anticipatory scheduling strategy in real-time and real-world scenarios. Further extending this work, we intend to improve our prediction framework, including other picker states such as Idle, Transportation, Loading, and Unloading.