Optimizing Autonomous Wheel Loader Performance—An End-to-End Approach

Abstract

Wheel loaders in mines and construction sites repeatedly load soil from a pile to load receivers. Automating this task presents a challenging planning problem since each loading’s performance depends on the pile state, which depends on previous loadings. We investigate an end-to-end optimization approach considering future loading outcomes and transportation costs between the pile and load receivers. To predict the evolution of the pile state and the loading performance, we use world models that leverage deep neural networks trained on numerous simulated loading cycles. A look-ahead tree search optimizes the sequence of loading actions by evaluating the performance of thousands of action candidates, which expand into subsequent action candidates under the predicted pile states recursively. Test results demonstrate that, over a horizon of 15 sequential loadings, the look-ahead tree search is 6% more efficient than a greedy strategy, which always selects the action that maximizes the current single loading performance, and 14% more efficient than using a fixed loading controller optimized for the nominal case.

1. Introduction

The task of most construction and mining machines, such as wheel loaders, is to perform long sequences of earthmoving operations in dynamic environments. Optimizing this task involves selecting loading and transportation actions that maximize the gain over time, according to some performance measure. This is challenging for several reasons. First, the performance of a single loading cycle depends highly on the local shape of the pile and the soil properties [1]. The loading actions need to be carefully adapted to these conditions. Second, each loading alters the pile shape. From the perspective of future loadings, such a change can be either for the better or worse. There might be actions that are of low performance in the short term but are necessary to achieve higher gains in the future [2]. Therefore, a greedy strategy of always choosing the loading action that maximizes the performance for a single loading might be sub-optimal in the long run. Such a strategy might gradually deteriorate the pile state or lead to excessive transportation between the dig location and the load receiver. This motivates considering future states when selecting the actions.

We refer to the problem of finding the action sequence that maximizes the total performance of many wheel loading cycles, given the initial pile state, as the wheel loader end-to-end optimization problem. To the best of our knowledge, this problem has not been scientifically explored previously. In fact, until recently, it was computationally intractable. In our previous work [3], we developed data-driven models capable of predicting the loading performance and the subsequent state of the pile within a few milliseconds. Model inputs are the previous pile state and selected action parameters. This makes it possible to rapidly evaluate the net performance of numerous alternative action sequences in advance. These models are referred to as a world model, as they predict the new state the environment will transition into, given the previous state and a selected action, and the observations associated with the state transition [4].

This paper investigates methods for solving the wheel loader end-to-end optimization problem and analyzes the obtained solutions. The research questions include what characterizes the optimal action sequence, how sensitive the results are to the initial pile state and planning horizon, what the computational demands are, and the feasibility of practical use of the method. The idea is that the methods could be implemented in site management systems with high-precision monitoring of equipment and the 3D topography, sending coordinated work plans to the individual machines.

We use a look-ahead tree search approach illustrated in Figure 1. Future action candidates for the foreseen future states are identified recursively, offline. The action candidates are evaluated, using data-driven models developed in previous research [3], to compute their respective performance and expand into further action candidates under the new states. The method should apply to any kind of world model with the same prediction capabilities. This approach is, reportedly, similar to how experienced operators intuitively plan their work, but we expect that a computerized model and search algorithms can make more accurate predictions and plan over longer horizons. To understand the importance of the look-ahead search, we examined the effect of using different planning horizons and objectives.

The wheel loader is assumed to have a high-level planner and a low-level control system, which work as the hierarchical architecture described in refs. [5,6]. The high-level planner selects a subtask action, such as bucket filling, dumping, or V-shaped transportation, according to the current state of the machine and the work site. A low-level controller performs the subtasks given a motion plan and control parameters. For bucket filling, the machine is equipped with a type of admittance controller [7]. Our controller takes four action parameters that determine how reactive the boom and bucket actuation is in response to the momentaneous dig force [3]. The performance of a loading cycle is measured in terms of mass, time, and work, with contributions from each subtask.

We test the effect of the look-ahead tree search by considering a wheel loader tasked with performing 15 sequential loading cycles. The resulting performance and computational cost are compared with that obtained with a greedy search and a nominal strategy.

2. Related Work

The end-to-end optimization problem of sequential loading was discussed in [2], considered intractable, and substituted with a simpler problem of optimizing each loading cycle by searching in a lower-dimensional action space implicitly constrained by the pile state. An average of 90% efficiency (bucket payload relative to its capacity) was achieved over 20 cycles. The study was limited to 2D, used a heuristic simulation model for soil displacements and settling, and did not consider time or energy consumption. Attempts were made to improve the performance further by using heuristics to avoid action plans that produce challenging terrain configurations, but the optimization was computationally prohibitive.

What is a good dig location in the pile, given its shape in terms of a heightmap, has been studied from the perspective of both bucket filling and the transportation between the dig and the dump location. In [8], a coarse-level planner finds the points along the pile contour closest to the load receiver, and a refined planner was used to avoid curved regions of the pile surface to minimize bucket-side loading and drop-in filling efficiency. A sequential coarse-to-fine planner was studied in [9] and compared to a greedy planner. The coarse planner first produces a sequence of dig regions ordered radially around the pile centroid. The fine planner then selects the dig points inside each region that have the best local pile shape. Simulations (cellular automata) of 50 consecutive loadings showed that the coarse-to-fine strategy maintains a good pile shape and performs at 80–90% while a greedy strategy (always selecting the dig location with optimal pile shape) initially performs equal or better for ten loadings but drops down to about 60% after 25 loadings.

Models for adapting the loading control to the pile shape have been studied with different approaches. In [10], reinforcement learning was used to train a multiagent system for autonomous control of an underground load–haul–dump (LHD) vehicle using a depth camera, lidar, and force and kinematics sensors. One agent was used to select a favorable dig position, given the observed pile shape. A second agent had the task of steering the vehicle toward the target position and controlling the forward drive as well as the lift and tilt cylinders to perform bucket filling. The agents were trained in a simulated environment to learn control policies designed for high productivity and energy efficiency over multiple loading cycles while also avoiding collisions and wheel slip. The system achieved bucket filling with 75% of maximum capacity on average and used 4% less energy by actively selecting favorable dig locations. The vehicle was confined to operate in a narrow mine drift with little variability in viable dig position or how to navigate to and from the dump location. It is untested how well this approach applies to controlling a wheel loader above ground with much larger action and observation space. The study [11] explored multiobjective optimization of an LHD control strategy using surrogate models based on data from sequences of simulated loadings in a pile of fragmented rock. Each simulated sequence used a parameterized dig trajectory planner that adapts to the local pile state. Based on many simulations with different parameters, the learned model can identify the loading strategies that ensure a high average performance and avoid those that systematically deteriorate the pile state.

The optimization of the V-shaped transportation path between the dig and dump location has been addressed in several studies, focusing on finding the shortest path with curvature consistent with the vehicle’s minimum turning radius [12,13], taking into account the vehicle dynamics and construction working site constraints [14], non-uniform trajectories under forward and reverse driving conditions [15], or minimizing fuel consumption [16]. Time- and energy-efficient tracking of a planned path has been explored in several studies using optimal control [17,18].

The optimal bucket filling trajectory was researched in [19] using the discrete element method (DEM) and used in [20] to find the optimal short loading cycle using dynamic programming. For the sake of computational efficiency, the simulations were limited to quasi-2D piles sloped with the material’s angle of repose. The effect of the pile shape on the optimal trajectory or the resulting 3D pile state has not been explored using DEM simulation, apart from the work in [11].

3. Problem Formulation

We focus on the short loading cycle, in which the wheel loader repeatedly loads and dumps soil from a pile to a load receiver. Each cycle can be divided into a sequence of subtasks: V-turn-1, loading, V-turn-2, and dumping. These subtasks are illustrated in Figure 1. We assume the wheel loader is equipped with a low-level control system for each subtask. A high-level system (an operator or agent) activates the low-level systems by selecting a set of subtask action parameters: ${\mathit{a}}^{\mathrm{V}1}$, ${\mathit{a}}^{\mathrm{load}}$, ${\mathit{a}}^{\mathrm{V}2}$, and ${\mathit{a}}^{\mathrm{dump}}$, respectively.

At the beginning of each cycle $n=1,2,\dots ,N$, the wheel loader is located by the load receiver at a position ${\mathit{x}}_n^{\mathrm{dump}}$. The V-turn-1 subtask is to drive the wheel loader to a selected loading position ${\mathit{x}}_n^{\mathrm{dig}}$ somewhere along the edge of the nearby pile. A low-level controller plans and executes a V-turn motion that starts at ${\mathit{x}}_n^{\mathrm{dump}}$ and ends at ${\mathit{x}}_n^{\mathrm{dig}}$, avoiding collisions with the environment, and with endpoints heading normal to the load receiver and the pile, respectively. As subtask action parameters, we use the target loading position, ${\mathit{a}}_n^{\mathrm{V}1}={\mathit{x}}_n^{\mathrm{dig}}$.

The next subtask is the actual loading from the pile. It starts with the machine approaching the selected loading position and ends with the vehicle reversing from the pile with a filled bucket. The task is carried out by an automatic bucket filling controller with some action parameters, ${\mathit{a}}_n^{\mathrm{load}}$, that may be adapted to the current pile shape and material properties. The loading transforms the pile from a state ${\mathit{H}}_n$ into a new state ${\mathit{H}}_{n+1}$ and results in some amount of soil mass $M_n$ in the bucket. We identify the state with the geometric shape of the pile.

After bucket filling, the V-turn-2 subtask is performed. It ends with the wheel loader approaching the receiver located at a position ${\mathit{x}}_n^{\mathrm{dump}}$, which is the corresponding action parameter ${\mathit{a}}_n^{\mathrm{V}2}$. The final subtask is to empty the bucket’s contents onto the load receiver with an action ${\mathit{a}}_n^{\mathrm{dump}}$. When the receiver is filled, this subtask might require some adjustment action to prevent spillage. A completed loading cycle thus involves a selection of action parameters that we collect in a vector ${\mathit{a}}_n=[{\mathit{a}}_n^{\mathrm{V}1},{\mathit{a}}_n^{\mathrm{load}},{\mathit{a}}_n^{\mathrm{V}2},{\mathit{a}}_n^{\mathrm{dump}}]$.

A loading cycle involves spending some amount of mechanical work $W_n$ and time $T_n$ to displace a certain mass $M_n$ from the pile to the receiver. Each of the subtasks contributes to the cycle time and work, while it is only the loading subtask that produces a mass measurement. We attribute each loading cycle a performance, which we measure by the performance vector:

$${\mathcal{P}}_n={\left[{\displaystyle \frac{M_0}{M_n}},{\displaystyle \frac{T_n}{T_0}},{\displaystyle \frac{W_n}{W_0}}\right]}^{\mathsf{T}},$$

(1)

where $M_0$, $T_0$, and $W_0$ are some characteristic values used for normalization. Note that the performance vector is a function of the pile state, the location of the load receiver, and the selected action, i.e., ${\mathcal{P}}_n({\mathit{H}}_n,{\mathit{a}}_n)$. We pose the end-to-end optimization problem of N sequential loading cycles as the problem of finding the sequence of actions ${\left\{{\mathit{a}}_n\right\}}_{n=1}^N$ that, given the initial pile state ${\mathit{H}}_1$ and the dump location ${\mathit{x}}^{\mathrm{dump}}$, satisfy

$$\underset{\{{\mathit{a}}_1,\dots ,{\mathit{a}}_N\}}{argmin}{\displaystyle \sum _{n=1}^N}{\mathit{w}}^{\mathsf{T}}{\mathcal{P}}_n,$$

(2)

where $\mathit{w}$ is a vector of positive weights for controlling the trade-off between maximizing loaded mass and minimizing time and work.

The posed optimization problem is computationally challenging to solve. An action sequence ${\left\{{\mathit{a}}_n\right\}}_{n=1}^N$ renders a sequence of pile states ${\left\{{\mathit{H}}_n\right\}}_{n=1}^N$ and performance measurements ${\left\{{\mathcal{P}}_n\right\}}_{n=1}^N$. These cannot be computed independently due to the strong dependency on the evolving pile state. The computational complexity for exhaustive search scales exponentially as ${\left(t_{\mathit{a}}{D}_{\mathit{a}}\right)}^N$, where $D_{\mathit{a}}$ is the number of action candidates that must be evaluated for each pile state and $t_{\mathit{a}}$ is the computational time for doing so.

Assumptions and Delimitations

A number of delimitations and simplifying assumptions are made to bring down the difficulty of the problem. We focus on loading a single type of non-cohesive and homogeneous soil, namely gravel. The surrounding ground is assumed to be flat. We assume no soil is spilled from the bucket during the V-turn-2 subtask or when dumped on the receiver. Spillage would affect subsequent performance, either by loss in control precision or traction or by the need to clear the ground.

The receiver is located at a fixed location and orientation relative to the pile. We assume it is immediately replaced by another receiver at the same location when full. The wheel loader always approaches the receiver’s center position, orthogonally, and simply empties the bucket without considering the shape of the body. The contribution to the cycle performance is then a constant value. We thus ignore the selection of the dumping action parameter ${\mathit{a}}^{\mathrm{dump}}$ and the V-turn-2 parameter ${\mathit{a}}^{\mathrm{V}2}$ from the problem but account for the contribution of the actions to the net performance.

Automatic bucket filling starts with the loader heading toward a dig location along the edge of the pile at a nominal target speed. To limit the dimensionality of the problem, the  heading is always normal to the pile contour, and the edge is discretized into a finite number of candidates of dig locations separated by a step size smaller than the width of a bucket.

We assume there is always room for a collision-free V-turn between the receiver and the pile. As the dump location is known, the V-turn planner requires only the dig location and heading as input. The V-turns are performed with some performance ${\mathcal{P}}_n^{\mathrm{V}1}$ and ${\mathcal{P}}_n^{\mathrm{V}2}$, and we assume that we can predict them (Section 4.2) with sufficient accuracy given the path and the bucket load mass.

For the loading, we assume access to a world model in the following form:

$$\begin{array}{cc}\hfill {\mathit{H}}_{n+1}& =\mathbf{\Phi }({\mathit{H}}_n,{\mathit{a}}_n),\hfill \end{array}$$

(3)

$$\begin{array}{cc}\hfill {\mathcal{P}}_n^{\mathrm{load}}& =\mathbf{\Psi }({\mathit{H}}_n,{\mathit{a}}_n),\hfill \end{array}$$

(4)

where $\mathbf{\Phi }$ is a pile state predictor model and $\mathbf{\Psi }$ is a performance predictor model, which take as input the previous pile state ${\mathit{H}}_n$ and the reduced action control parameter ${\mathit{a}}_n=[{\mathit{x}}_n^{\mathrm{dig}},{\mathit{a}}_n^{\mathrm{load}}]$.

With these simplifications, the problem to solve is

$$\underset{{\{{\mathit{x}}_n^{\mathrm{dig}},{\mathit{a}}_n^{\mathrm{load}}\}}_{n=1}^N}{argmin}{\displaystyle \sum _{n=1}^N}{\mathit{w}}^{\mathsf{T}}{\mathcal{P}}_n.$$

(5)

with the predicted performance ${\mathcal{P}}_n={\mathcal{P}}_n^{\mathrm{load}}+{\mathcal{P}}_n^{\mathrm{V}1}+{\mathcal{P}}_n^{\mathrm{V}2}$ that must be computed sequentially along with the predicted pile state evolution ${\left\{{\mathit{H}}_n\right\}}_{n=1}^N$ starting from the initial state ${\mathit{H}}_1$.

4. Method

This section describes our models for predicting the outcome of a single loading cycle and the search method to find the optimal sequence of wheel loader actions.

4.1. Loading Prediction

We use the data-driven world model developed in [3] for a Komatsu WA320-7 wheel loader doing automatic bucket filling in piles of gravel using an admittance controller. The models predict the loading outcome:

$$\begin{array}{cc}\hfill {\mathit{H}}^{\prime }& =\mathbf{\Phi }(\mathit{H},{\mathit{x}}^{\mathrm{dig}},{\mathit{a}}^{\mathrm{load}}),\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill {\mathcal{P}}^{\mathrm{load}}& =\mathbf{\Psi }(\mathit{H},{\mathit{x}}^{\mathrm{dig}},{\mathit{a}}^{\mathrm{load}}),\hfill \end{array}$$

(7)

in terms of the resulting heightmap ${\mathit{H}}^{\prime }$ and loading performance, ${\mathcal{P}}^{\mathrm{load}}={\left[M_0/M_{\mathrm{load}},T_{\mathrm{load}}/T_0,W_{\mathrm{load}}/W_0\right]}^{\mathsf{T}}$, given the current heightmap $\mathit{H}$, dig location ${\mathit{x}}^{\mathrm{dig}}$, and action parameters ${\mathit{a}}^{\mathrm{load}}$ for the automatic loading controller.

The dependency on the pile shape is limited to the local pile shape, $\mathit{h}$, in the area of the dig location and with the selected heading. Therefore, the function $\mathbf{\Phi }$ first does a cutout operation of the global heightmap to obtain the local pile surface. This is then fed as input, along with the loading action parameters, to a deep neural network $\mathit{\varphi }$. A replace function then substitutes the predicted local heightmap ${\mathit{h}}^{\prime }=\mathit{\varphi }(\mathit{h},{\mathit{a}}^{\mathrm{load}})$ into $\mathit{H}$ to obtain the predicted global heightmap ${\mathit{H}}^{\prime }$. Similarly, the loading performance is predicted using a deep neural network, ${\mathcal{P}}^{\mathrm{load}}=\psi (\mathit{h},{\mathit{a}}^{\mathrm{load}})$, after first cutting out the local heightmap at the dig location and selected heading. The model has three convolutional layers to encode $\mathit{h}$ before being fed with ${\mathit{a}}^{\mathrm{load}}$ to a multi-layer perceptron. The swish activation is used for all layers. The pile state and performance predictor models are illustrated in Figure 2 and Figure 3, respectively.

The models are trained and validated on a dataset from more than 10,000 random loading actions on various gravel pile shapes. The trained models, with roughly ${10}^7$ parameters, achieved around 95% accuracy in predicting loading performance and 97% accuracy in predicting the resulting pile state.

As in [3], we use different sizes and resolutions of the local heightmap for the pile state and performance predictor models. For the former model, we discretize a quadratic heightmap with 5.2 m sides in a $52\times 52$ grid. For the latter model, we use a $36\times 36$ grid and a side length of 3.6 m.

Optimal Loading Action Parameters

The wheel loader is equipped with the function of automatic bucket filling using admittance control. The controller is parameterized with four parameters ${\mathit{a}}^{\mathrm{load}}\in {[0,1]}^4$ that control how reactive the boom lift and bucket tilt are to the perceived digging resistance, which in turn is an unknown function of the local pile shape. For each pile shape and soil strength, there exists some loading control parameters ${\mathit{a}}^{\mathrm{load}*}$ that optimize the loading performance:

$${\mathit{a}}^{\mathrm{load}*}=\underset{{\mathit{a}}^{\mathrm{load}}}{argmin}{\mathit{w}}^{\mathsf{T}}{\mathcal{P}}^{\mathrm{load}}.$$

(8)

The optimal control parameters can be computed using the gradient descent method with the iterative step:

$${\mathit{a}}^{\mathrm{load}}:={\mathit{a}}^{\mathrm{load}}-\eta \nabla \left({\mathit{w}}^{\mathsf{T}}{\mathcal{P}}^{\mathrm{load}}\right),$$

(9)

where $\eta$ is the step length and $\nabla =\partial /\partial {\mathit{a}}^{\mathrm{load}}$. As an initial guess, we use ${\mathit{a}}_n^{\mathrm{load}}={[1,0,0.0,0.5,0.5]}^{\mathsf{T}}$, which corresponds to the machine thrusting deep into the pile, filling the bucket with little or no lifting of it [3]. This type of action has been suggested for uniformly sloped piles of dry homogeneous soil [21], and we expect the optimization process to converge faster with this initialization. The maximum number of iterations is set to 30 with early stopping (patience 3 for tolerance ${10}^{-4}$). The gradient with respect to ${\mathit{a}}^{\mathrm{load}}$ is calculated by using pytorch.autograd and exploiting that $\psi$ has two input branches, separating $\mathit{h}$ and ${\mathit{a}}^{\mathrm{load}}$ so that the encoding part only needs to be evaluated once during the optimization process.

4.2. V-Turn Model

A loading cycle involves moving the wheel loader back and forth between the pile and the load receiver in a V-shaped pattern connecting the dump location ${\mathit{x}}^{\mathrm{dump}}$ and dig location ${\mathit{x}}^{\mathrm{dig}}$. We estimate the transportation time and work by numerical integration of an assumed equation of motion for the wheel loader, including its motor strength, variable mass, and rolling resistance. For generating the V-turn paths, we use cubic B-splines [22]. In narrow spaces, Dubins curves may be the preferred choice to ensure that the vehicle’s minimum turning radius is respected [23]. Each V-turn path is formed by the combination of two spline curves. V-turn-1 is composed of the splines connecting ${\mathit{x}}^{\mathrm{dump}}\to {\mathit{x}}^{\mathrm{V}1}$ and ${\mathit{x}}^{\mathrm{V}1}\to {\mathit{x}}^{\mathrm{dig}}$ with the condition that the ingoing and outgoing directions should match in the switch back point ${\mathit{x}}^{\mathrm{V}1}$. Similarly, V-turn-2 is composed of two splines connecting ${\mathit{x}}^{\mathrm{dig}}\to {\mathit{x}}^{\mathrm{V}2}$ and ${\mathit{x}}^{\mathrm{V}2}\to {\mathit{x}}^{\mathrm{dump}}$. See Figure 4 for an illustration.

To compute the splines, we follow the method and notations of [24]. The path to be found is represented by five control points, ${\left\{{\mathit{p}}_i\right\}}_{i=0}^5$. These are obtained by solving the following equation system:

$$\left[\begin{array}{cccccc}1& 0& 0& 0& 0& 0\\ N_0^1\left(s_0\right)& N_1^1\left(s_0\right)& 0& 0& 0& 0\\ N_0^2\left(s_0\right)& N_1^2\left(s_0\right)& N_2^2\left(s_0\right)& 0& 0& 0\\ 0& 0& 0& N_3^2\left(s_1\right)& N_4^2\left(s_1\right)& N_5^2\left(s_1\right)\\ 0& 0& 0& 0& N_4^1\left(s_1\right)& N_5^1\left(s_1\right)\\ 0& 0& 0& 0& 0& 1\end{array}\right]\left[\begin{array}{c}{\mathit{p}}_0\\ {\mathit{p}}_1\\ {\mathit{p}}_2\\ {\mathit{p}}_3\\ {\mathit{p}}_4\\ {\mathit{p}}_5\end{array}\right]=\left[\begin{array}{c}{\mathit{q}}_0\\ \alpha {\mathit{d}}_0\\ \mathbf{0}\\ {\mathit{q}}_1\\ \beta {\mathit{d}}_1\\ \mathbf{0}\end{array}\right],$$

(10)

where $N_i^{\left(j\right)}\left(s\right)$ is the jth derivative of the ith spline basis function associated with ${\mathit{p}}_i$ and curve parameter $s\in [0,1]$. On the right-hand side, ${\mathit{q}}_0$ and ${\mathit{q}}_1$ are the spline endpoints with first derivatives $\alpha {\mathit{d}}_0$ and $\beta {\mathit{d}}_1$, respectively, where $\alpha$ and $\beta$ are parameters for controlling the magnitudes of the derivatives.

For a given set of endpoints, ${\mathit{x}}^{\mathrm{dump}}$ and ${\mathit{x}}^{\mathrm{dig}}$ (and heading at these points), we compute the connecting splines, via ${\mathit{x}}^{\mathrm{V}1}$ and ${\mathit{x}}^{\mathrm{V}2}$, that minimize the path length and curvature according to the following weighted objective function [24]:

$${\gamma }_1\sum {\left({\kappa }_i\right)}^2\Delta s_i+{\gamma }_2\sum {\left({\displaystyle \frac{d{\kappa }_i}{ds}}\right)}^2\Delta s_i+{\gamma }_3\sum \Delta s_i.$$

(11)

with the weights set to $\mathit{\gamma }=[10.0,10.0,1.0]$ to avoid sharp curves. We use Powell’s method to limit the search space such that the switch back points must reside in a box with side length $l_2=10$ m positioned at $l_1=5$ m from the endpoints, see Figure 4 for an illustration. These constraints are also chosen to shorten the reversing distance from ${\mathit{x}}^{\mathrm{dump}}$ and ${\mathit{x}}^{\mathrm{dig}}$ to the switch back points. This explains why the V-turn-1 and V-turn-2 paths are not identical although generated using the same ${\mathit{x}}^{\mathrm{dump}}$ and ${\mathit{x}}^{\mathrm{dig}}$. The magnitude of the derivatives at ${\mathit{x}}^{\mathrm{dump}}$, ${\mathit{x}}^{\mathrm{dig}}$, ${\mathit{x}}^{\mathrm{V}1}$, and ${\mathit{x}}^{\mathrm{V}2}$ are fixed at 10, 30, 5, and 5, respectively, for the paths to be straight around the data points. The implementation for computing the paths uses scipy.optimize.

For each given V-turn path, the time and work for driving the vehicle is computed. The first step is to compute the velocity profile $v\left(t\right)$ and traction force $f\left(t\right)$. The velocity is obtained by numerical integrations of the equation of motion:

$$M_{\mathrm{tot}}{\displaystyle \frac{dv}{dt}}+C_{\mathrm{r}}v=f,$$

(12)

where the total variable mass $M_{\mathrm{tot}}=M_{\mathrm{vehicle}}+M_{\mathrm{load}}$ is composed of the vehicle mass $M_{\mathrm{vehicle}}$, $M_{\mathrm{load}}$ is the loaded mass (zero for V-turn-1), and $C_{\mathrm{r}}={\mu }_{\mathrm{r}}{M}_{\mathrm{tot}}g$ is the rolling resistance with gravity acceleration g and rolling resistance coefficient ${\mu }_{\mathrm{r}}$. The machine is accelerated by a traction force $f\left(t\right)$ to reach a preset target velocity during the transportation phase. We control $f\left(t\right)$ via the throttle as described in [25]. When approaching an endpoint, the machine is decelerated by a constant brake force $f=f_{\mathrm{brake}}$. The V-turn duration is computed by the time difference between the endpoints and the work as $\int \max [0,f(t\left)\right]v\left(t\right)dt$ along the path. Note that only positive work is accounted for in the work computation, i.e., no energy can be accumulated during the decelerating phase. Also, we assume the vehicle follows the prescribed path precisely, with no required work for steering and with the bucket and boom angles held fixed during transport. We use a time step of 10 ms, $M_{\mathrm{vehicle}}=15,2$ tonne, ${\mu }_{\mathrm{roll}}=0.01$, and the input throttle value can increase with a rate of $2.0$ units/s. The target speed is 8.0 km/h for both reversing and forwarding except for the nearest 5 m before ${\mathit{x}}^{\mathrm{dig}}$ where it increases to 11.4 km/h. Example V-turns and trajectories with the resulting speed and traction force are shown in Figure 5.

To save computational time during end-to-end optimization, we pre-compute many optimal V-turns using a grid of dig locations and different headings. The result is stored in a look-up table. During optimization, the estimated time and work for the V-turns are interpolated from the look-up table.

4.3. Dumping

We assume the machine always empties the bucket at a fixed location at the receiver, without considering the shape of the loaded mass. Therefore, the emptying time is fixed at 5 s, and no work is associated with the dumping action.

4.4. Look-Ahead Tree Search

To find a near-optimal solution, we develop a look-ahead tree search algorithm that combines the world model and V-turn model for long-horizon predictions. The planning horizon is denoted by integer N and the search depth by $d\le N$. At each planning step, $n=1,2,\dots ,N$, a finite set of I candidate dig locations ${\left\{{\mathit{x}}_i^{\mathrm{dig}}\right\}}_{i=1}^I$ and corresponding loading actions ${\left\{{\mathit{a}}_i^{\mathrm{load}}\right\}}_{i=1}^I$ are considered. We are to select the dig location and loading action that are optimal over the search depth d. At the most shallow search depth, $d=1$, this corresponds to picking the candidate that minimizes ${\mathit{w}}^{\mathsf{T}}{\mathcal{P}}_n({\mathit{H}}_n,{\mathit{x}}_n^{\mathrm{dig}},{\mathit{a}}_n^{\mathrm{load}})$. At the first planning step, $n=1$, we know the pile state ${\mathit{H}}_1$ and can make this pick. For the future planning steps, we must first expand the pile into its next state from the previous, ${\mathit{H}}_{n+1}={\mathit{H}}_n^{\prime }\equiv \mathbf{\Phi }({\mathit{H}}_n,{\mathit{x}}_n^{\mathrm{dig}},{\mathit{a}}_n^{\mathrm{load}})$. At larger search depths, $d\ge 2$, we compute the predicted net performance over the search horizon d using the following approximation of the evaluation function:

$$\begin{array}{cc}\hfill Q_n^d& ={\displaystyle \sum _{k=n}^{n-1+d}}{\mathit{w}}^{\mathsf{T}}{\mathcal{P}}_k({\mathit{H}}_k,{\mathit{x}}_k^{\mathrm{dig}},{\mathit{a}}_k^{\mathrm{load}})\hfill \\ \hfill & \lesssim {\mathit{w}}^{\mathsf{T}}{\mathcal{P}}_n({\mathit{H}}_n,{\mathit{x}}_n^{\mathrm{dig}},{\mathit{a}}_n^{\mathrm{load}})+{\displaystyle \sum _{k=n+1}^{n-1+d}}{\mathit{w}}^{\mathsf{T}}{\mathcal{P}}_k({\overline{\mathit{H}}}_k,{\overline{\mathit{x}}}_k^{\mathrm{dig}},{\overline{\mathit{a}}}_k^{\mathrm{load}}),\hfill \end{array}$$

(13)

where ${\overline{\mathit{x}}}_k$ and ${\overline{\mathit{a}}}_k$ are greedy choices given the pile state ${\overline{\mathit{H}}}_k$; that is, those that minimize ${\mathit{w}}^{\mathsf{T}}{\mathcal{P}}_k({\overline{\mathit{H}}}_k,{\overline{\mathit{x}}}_k,{\overline{\mathit{a}}}_k)$. The pile ${\overline{\mathit{H}}}_k$ is, in turn, the result of an expansion from the pile state at the previous search depth, ${\overline{\mathit{H}}}_{k-1}$, using a greedy choice at that level too (if there is a maximum number loading cycles, N, then the summation in Equation (13) is limited by this in case $n+d>N$. Also, if $d=1$, there is no second or higher term on the right-hand side). This approximation is to avoid doing an exhaustive search over all action candidates. Evaluating Equation (13) requires that pile state at the top level pile ${\mathit{H}}_n$ is expanded into all the candidate piles ${\left\{{\mathit{H}}_i^{\prime }\right\}}_{i=1}^I$ using ${\left\{{\mathit{x}}_i^{\mathrm{dig}}\right\}}_{i=1}^I$ and ${\left\{{\mathit{a}}_i^{\mathrm{load}}\right\}}_{i=1}^I$. The algorithm is displayed in Algorithm 1 and pictorially illustrated in Figure 1.

Algorithm 1: Look-ahead tree search with depth d and planning horizon N

4.4.1. Greedy Strategy

We refer to the choice of $d=1$ as the entirely greedy strategy. The dig location is always selected to optimize the one-step evaluation function $Q_n^{d=1}=Q_{\mathrm{greedy}}\equiv {\mathit{w}}^{\mathsf{T}}{\mathcal{P}}_{\mathit{n}}$ with the loading action optimized for the local pile state at the dig location. In this case, the pile state ${\mathit{H}}_n$ needs only to be expanded once and not into multiple pile state candidates.

4.4.2. Maximum Loading Strategy

Setting $Q_n^{d=1}=Q_{\mathrm{loading}}\equiv {\mathit{w}}^{\mathsf{T}}{{\mathcal{P}}_{\mathit{n}}}^{\mathrm{load}}$ results in dig locations and loading actions that are optimized for short-term loading performance while ignoring the V-turn cost. We refer to this as the maximum loading strategy.

4.4.3. Nominal Strategy

For comparison, we also test the strategy of always selecting the dig location that has the lowest transportation cost, irrespective of the loading performance. This is accomplished with $Q_n^{d=1}=Q_{\mathrm{nominal}}\equiv {\mathit{w}}^{\mathsf{T}}\left({{\mathcal{P}}_{\mathit{n}}}^{\mathrm{V}1}+{{\mathcal{P}}_{\mathit{n}}}^{\mathrm{V}2}\right)$. We refer to this as the nominal strategy as it is a standard method [26] and the natural choice when there is no access to a world model.

4.5. Computational Time

The computational time is what limits us from computing the optimal solution by exhaustive search and instead approximate the evaluation function by Equation (13) at higher search depths. We register the computational time for analyzing a single loading cycle. It includes predicting the new pile state ${\mathit{H}}^{\prime }$ and total performance $\mathcal{P}$, which involves updating ${\mathit{a}}^{\mathrm{load}}$ according to Equation (9) (dominant part) and computing and integrating the V-turns given a selected dig location ${\mathit{x}}^{\mathrm{dig}}$ as described in Section 4.2.

Table 1. Profiling a single loading cycle prediction.

Function		Time [ms]	Dominant Computing Cost
$\mathbf{\Phi }$	cutout	9.0	heightfield rotation
	$\varphi$	2.5
	replace	12.0	heightfield rotation
$\mathbf{\Psi }$	${\psi }^{\mathrm{load}}\to {\mathit{a}}^{\mathrm{load}}$	∼45.0	grad-descent (1.5 [ms] × iteration)
	${\psi }^{\mathrm{V}1}$	2.5	generating 2 × cubic spline
	${\psi }^{\mathrm{V}2}$	2.5	generating 2 × cubic spline
Total		∼73.5

shows the average computational time from profiling the loading predictions made on a desktop computer with an Intel(R) Core(TM) i7-8700K, 3.70 GHz, 32 GB RAM on a Windows 64-bit system and NVIDIA GeForce RTX 2070 SUPER.

The computational time for solving the end-to-end optimization problem depends on the given time for search, the number of child nodes per node (I), and the tree search depth (d). For reference, a greedy search takes about 29.4 s for $d=40$, $I=10$ with 73.5 ms per prediction as it requires $I\times d$ predictions.

5. Results

We test the look-ahead search algorithm considering a wheel loader performing a task of $N=15$ sequential loading cycles. First, we analyze the greedy strategy (prediction horizon $d=1$) and compare it using the load optimizing and the nominal strategies. Next, we analyze the look-ahead tree search algorithm with different search depths to examine the effect of the prediction horizon. The weights were fixed at $\mathit{w}=[2,1,1]$ such that production (loaded mass) and cost (loading time and mechanical work) contribute equally to the overall loading performance.

The initial pile surface ${\mathit{H}}_1$ features a trapezoidal prism, 1.8 m tall with a ${30}^{\circ }$ slope at the front, with Perlin noise [27] added to the surface to avoid ending up in the same local minima. A representative initial pile surface is illustrated in Figure 6.

The location and orientation of the receiver are fixed at ${\mathit{x}}^{\mathrm{dump}}=[x,y,\theta ]=[-12.0\mathrm{m},-3.0\mathrm{m},-30.0°]$. The dig locations are constrained within $-5.0\mathrm{m}\le x\le 8.0\mathrm{m}$ and $0.0\mathrm{m}\le y\le 6.0\mathrm{m}$. The listup function generates candidate dig locations ${\tilde{\mathit{x}}}^{\mathrm{dig}}$ with a spacing of 1 m.

5.1. Greedy Strategy

The greedy strategy with no horizon, $d=1$, is tested using the three evaluation functions described in Section 4.4.1, Section 4.4.2 and Section 4.4.3: $Q_{\mathrm{greedy}}={\mathit{w}}^{\mathsf{T}}{\mathcal{P}}_n$, $Q_{\mathrm{loading}}={\mathit{w}}^{\mathsf{T}}{\mathcal{P}}_n^{\mathrm{load}}$, and $Q_{\mathrm{nominal}}={\mathit{w}}^{\mathsf{T}}\left({\mathcal{P}}_n^{\mathrm{V}1}+{{\mathcal{P}}_{\mathit{n}}}^{\mathrm{V}2}\right)$. The pile states after five loading cycles are shown in Figure 7. The greedy strategy results in loadings that aim for areas in the search region with more soil easily accessible from the dump truck. The maximum loading strategy aims to the right where there is more soil but further from the dump truck. The nominal strategy loads to the left, in proximity to the dump truck, as can be expected when focusing entirely on minimizing the transportation cost.

The evolution of the loaded mass, time, and work over 15 loading cycles is shown in Figure 8, and the net result is listed in

Table 2. The total performance by the three variants of the greedy search strategy in units M [tonne], T [s], and W [MJ].

		Load		V-Turns		Total
Strategy	$\mathit{M}$	$\mathit{T}$	$\mathit{W}$	$\mathit{T}$	$\mathit{W}$	$\mathit{T}$	$\mathit{W}$
Greedy	$\mathbf{64}.\mathbf{4}$	197	6.2	421	9.3	$\mathbf{694}$	$\mathbf{15}.\mathbf{5}$
Max loading	62.6	$\mathbf{189}$	$\mathbf{6}.\mathbf{0}$	453	10.4	717	16.4
Nominal	59.8	260	9.4	$\mathbf{362}$	$\mathbf{7}.\mathbf{5}$	697	16.9

. The greedy strategy performs the best on average and in total. The nominal strategy’s performance drops over time, suggesting that the pile state is deteriorating.

The greedy strategy is 8% more productive and energy efficient than the nominal strategy, using nearly the same time to complete 15 cycles. The maximum loading strategy is 1% more productive than the greedy strategy in loading tasks but spends around 8% more time and energy during V-turns. That is an expected result, as transportation time and energy do not contribute to the evaluation function of the maximum loading strategy. The analysis demonstrates that the evaluation function works effectively.

5.2. Long-Horizon Planning with Look-Ahead Tree Search

The second test uses the tree search algorithm with finite prediction horizon d and evaluation function (13). Ten initial heightfields were prepared with different Perlin noise applied to the 1.8 m high trapezoidal prism. For each of these ten piles, $N=15$ sequential loading cycles were evaluated with the prediction horizon ranging from $d=1$ to $d=N$. The resulting loading outcome and performance evaluation function are shown in Figure 9. The results were averaged over the ten repetitions starting with the different initial piles. The performance converges around $d=4$. Beyond this point, increasing the search horizon further does not enhance the overall performance.

The performance improves by about 5.6% on average when comparing the results between $d=1$ and $d=4$. The total loaded mass is marginally reduced from about 64.7 to 64.0 tonnes (1.1%), while the total loading time and the work improves from about 665 to 632 s (5.0%) and from 14.9 to 13.9 MJ (6.7%), respectively. Note that we also tested with evaluation horizons of $N=10$ and $N=5$. The results at $N=10$ showed a similar improvement ratio, but this trend was not found for $N=5$, probably because of the shorter task length. As we have already found that the greedy strategy ($d=1$) is 6% more performant than the nominal strategy, we conclude that the look-ahead tree search leads to a 14% increase in performance relative to the nominal strategy.

The computational cost of the search increases with the prediction horizon d. On average, the search took 250 and 10,843 predictions with $d=1$ and 4, respectively. The computational time was about 18 and 792 s, respectively, given that a prediction takes 73 ms on average. The computational cost for deciding an immediate optimal action is shown in Figure 10.

The search with $d=15$ required 3181 predictions to decide the first action, which amounts to 232 s, given a prediction takes 73 ms on average. The searches with $d=4$ and $d=1$ required 593 and 13 predictions to decide the first action, amounting to 43 s and 0.9 s, respectively. Note that the number of predictions depends on the prediction horizon, the resolution of listup, and the length along the pile edge within the constraint region. The loading action sequence can make the pile state complex, which can increase the number of predictions.

The optimal loading locations are visualized in Figure 11 for prediction horizons $d=1$ and 4.

The first dig locations are rather similar, but the consecutive sequences are distinct from each other. The greedy search in Figure 11a starts at $x=0$, continues to the right, and then switches to the left. The tree search method in Figure 11b starts a little closer to the dump truck and stays consistently to the left, closer to the dump truck, working its way into the pile with approximately three buckets’ width. The conclusion is that the tree search method transforms the pile in a way that maintains future loading with good outcomes while keeping proximity to the dump truck.

6. Discussion

With the look-ahead search method, it is possible to find action sequences that maximize the total performance over long horizons by increasing the pile state quality for future loadings. This method adapts to the truck’s location and can do so even if it would vary provided that the location is known in advance. In our experiments, the optimal action sequence can be (before any serious code optimization) computed in 43 s with a look-ahead search with depth $d=4$, which is close to the average loading cycle time. Future work should consider other soil properties, such as cohesive or inhomogeneous soil. The resulting pile state is likely to depend more strongly on the selected loading actions than it does for gravel, for which the pile always tends to settle with a local slope near the natural angle of repose. To support more realistic conditions, one should extend the framework to support non-flat ground and spillage. Capturing this with a world model would be more demanding and might benefit from a longer horizon than $d=4$. That would motivate substantial improvements in the computational efficiency of the look-ahead search. In this study, we employed a set of specific weight factors that equally balance productivity (mass per unit time) and energy efficiency (mass per unit energy). It would be interesting, in future work, to explore the impact of different weight factors on the result, e.g., what digging behavior emerges when energy consumption is heavily penalized.

7. Conclusions

We describe and test a look-ahead tree search method to find near-optimal loading action sequences. This method uses a world model to predict future pile states and the performance for each loading cycle. It can dynamically adjust to varying conditions and constraints of the work site. Our results show significant performance improvements, showcasing the benefits of incorporating long-term predictions into decision making. The look-ahead tree search results in a 6% higher performance in sequential loading than a greedy strategy and 14% higher than the nominal strategy. Moreover, the tree search using long-horizon predictions leads to different decisions for immediate actions. This raises the question for future work on how sensitive the optimized action sequence is to variations in soil properties.