# Deep Learning-Based Prediction of Throttle Value and State for Wheel Loaders

^{*}

## Abstract

**:**

## 1. Introduction

## 2. Background

#### 2.1. Problem Statement

#### 2.2. LSTM Network

## 3. Methodology

#### 3.1. The Overview of the Proposed Deep Learning-Based Framwork

#### 3.2. Data Collection and Pre-Processing

#### 3.3. Construction of LSTM

#### 3.4. Construction of BPNN

## 4. Results and Discussion

#### 4.1. Performance Analysis of Deep Learning Model for Different Materials

#### 4.2. Comparison with Different Deep Learning Models

#### 4.3. Performance Analysis of LSTM Networks for Different Sampling Frequency

## 5. Conclusions

## Author Contributions

## Funding

## Institutional Review Board Statement

## Informed Consent Statement

## Conflicts of Interest

## References

- Davila Delgado, J.M.; Oyedele, L.; Ajayi, A.; Akanbi, L.; Akinade, O.; Bilal, M.; Owolabi, H. Robotics and automated systems in construction: Understanding industry-specific challenges for adoption. J. Build. Eng.
**2019**, 26, 100868. [Google Scholar] [CrossRef] - Frank, B.; Kleinert, J.; Filla, R. Optimal control of wheel loader actuators in gravel applications. Autom. Construct.
**2018**, 91, 1–14. [Google Scholar] [CrossRef] - Hemami, A.; Hassani, F. An overview of autonomous loading of bulk material. Int. Symp. Autom. Rob. Constr.
**2009**, 405–411. [Google Scholar] [CrossRef] [Green Version] - Dadhich, S.; Bodin, U.; Sandin, F.; Andersson, U. From Tele-Remote Operation to Semi-Automated Wheel-Loader. Int. J. Electr. Electron. Eng. Telecommun.
**2018**, 178–182. [Google Scholar] [CrossRef] - Hemami, A. Fundamental Analysis of Automatic Excavation. J. Aerosp. Eng.
**1995**, 8, 175–179. [Google Scholar] [CrossRef] - Bobbie, F.; Lennart, S.; Reno, F.; Anders, F. On Increasing Fuel Efficiency by Operator Assistant Systems in a Wheel Loader. In Proceedings of the International Conference on Advanced Vehicle Technologies and Integration (VTI 2012), Changchun, China, 16–19 July 2012; pp. 155–161. [Google Scholar]
- Roberts, J.M.; Duff, E.S.; Corke, P.I. Reactive navigation and opportunistic localization for autonomous underground mining vehicles. Inf. Sci.
**2002**, 145, 127–146. [Google Scholar] [CrossRef] - Dadhich, S.; Bodin, U.; Andersson, U. Key challenges in automation of earth-moving machines. Autom. Construct.
**2016**, 68, 212–222. [Google Scholar] [CrossRef] [Green Version] - Marshall, J.A.; Murphy, P.F.; Daneshmend, L.K. Toward Autonomous Excavation of Fragmented Rock: Full-Scale Experiments. IEEE Trans. Autom. Sci. Eng.
**2008**, 5, 562–566. [Google Scholar] [CrossRef] - Sotiropoulos, F.E.; Asada, H.H. A Model-Free Extremum-Seeking Approach to Autonomous Excavator Control Based on Output Power Maximization. IEEE Robot. Autom. Lett.
**2019**, 4, 1005–1012. [Google Scholar] [CrossRef] - Dobson, A.A.; Marshall, J.A.; Larsson, J. Admittance Control for Robotic Loading: Design and Experiments with a 1-Tonne Loader and a 14-Tonne Load-Haul-Dump Machine. J. Field Robot.
**2017**, 34, 123–150. [Google Scholar] [CrossRef] [Green Version] - Dadhich, S.; Sandin, F.; Bodin, U.; Andersson, U.; Martinsson, T. Field test of neural-network based automatic bucket-filling algorithm for wheel-loaders. Autom. Construct.
**2019**, 97, 1–12. [Google Scholar] [CrossRef] - Sun, W.; Iwataki, S.; Komatsu, R.; Fujii, H.; Yamashita, A.; Asama, H. Simultaneous tele-visualization of construction machine and environment using body mounted cameras. In Proceedings of the 2016 IEEE International Conference on Robotics and Biomimetics (ROBIO), Qingdao, China, 3–7 December 2016; pp. 382–387. [Google Scholar] [CrossRef]
- Yamada, H.; Muto, T. Development of a Hydraulic Tele-Operated Construction Robot using Virtual Reality. Int. J. Fluid Power
**2003**, 4, 35–42. [Google Scholar] [CrossRef] - Fernando, C.L.; Saraiji, M.Y.; Seishu, Y.; Kuriu, N.; Minamizawa, K.; Tachi, S. Effectiveness of Spatial Coherent Remote Drive Experience with a Telexistence Backhoe for Construction Sites. In Proceedings of the ICAT-EGVE, Kyoto, Japan, 28–30 October 2015; pp. 69–75. [Google Scholar]
- Feng, H.; Yin, C.; Ma, W.; Yu, H.; Cao, D. Parameters identification and trajectory control for a hydraulic system. ISA Trans.
**2019**, 92, 228–240. [Google Scholar] [CrossRef] - Yoo, S.; Park, C.G.; You, S.H.; Lim, B. A Dynamics-Based Optimal Trajectory Generation for Controlling an Automated Excavator. Proc. Inst. Mech. Eng. Part C-J. Eng. Mech. Eng. Sci.
**2010**, 224, 2109–2119. [Google Scholar] [CrossRef] - Zou, Z.; Chen, J.; Pang, X. Task space-based dynamic trajectory planning for digging process of a hydraulic excavator with the integration of soil–bucket interaction. Proc. Inst. Mech. Eng. Part K-J. Multi-Body Dyn.
**2018**, 233, 598–616. [Google Scholar] [CrossRef] - Bauza, M.; Hogan, F.R.; Rodriguez, A. A Data-Efficient Approach to Precise and Controlled Pushing. In Proceedings of the Conference on Robot Learning, Zürich, Switzerland, 29–31 October 2018; pp. 336–345. [Google Scholar]
- Byravan, A.; Fox, D. SE3-nets: Learning rigid body motion using deep neural networks. In Proceedings of the 2017 IEEE International Conference on Robotics and Automation (ICRA), Singapore, 29 May–3 June 2017; pp. 173–180. [Google Scholar] [CrossRef] [Green Version]
- Sotiropoulos, F.E.; Asada, H.H. Autonomous Excavation of Rocks Using a Gaussian Process Model and Unscented Kalman Filter. IEEE Robot. Autom. Lett.
**2020**, 5, 2491–2497. [Google Scholar] [CrossRef] - Sandzimier, R.J.; Asada, H.H. A Data-Driven Approach to Prediction and Optimal Bucket-Filling Control for Autonomous Excavators. IEEE Robot. Autom. Lett.
**2020**, 5, 2682–2689. [Google Scholar] [CrossRef] - Koo, B.; La, S.; Cho, N.W.; Yu, Y. Using support vector machines to classify building elements for checking the semantic integrity of building information models. Autom. Construct.
**2019**, 98, 183–194. [Google Scholar] [CrossRef] - Hasan, M.A.M.; Ahmad, S.; Molla, M.K.I. Protein subcellular localization prediction using multiple kernel learning based support vector machine. Mol. Biosyst.
**2017**, 13, 785–795. [Google Scholar] [CrossRef] - Shi, Y.; Xia, Y.; Zhang, Y.; Yao, Z. Intelligent identification for working-cycle stages of excavator based on main pump pressure. Autom. Construct.
**2020**, 109. [Google Scholar] [CrossRef] - Li, Y.; Xu, M.; Wei, Y.; Huang, W. A new rolling bearing fault diagnosis method based on multiscale permutation entropy and improved support vector machine based binary tree. Measurement
**2016**, 77, 80–94. [Google Scholar] [CrossRef] - Arabi, S.; Haghighat, A.; Sharma, A. A deep-learning-based computer vision solution for construction vehicle detection. Comput.-Aided Civil. Infrastruct. Eng.
**2020**, 35, 753–767. [Google Scholar] [CrossRef] - Park, J.; Lee, B.; Kang, S.; Kim, P.Y.; Kim, H.J. Online Learning Control of Hydraulic Excavators Based on Echo-State Networks. IEEE Trans. Autom. Sci. Eng.
**2017**, 14, 249–259. [Google Scholar] [CrossRef] - Kim, J.; Chi, S. Action recognition of earthmoving excavators based on sequential pattern analysis of visual features and operation cycles. Autom. Construct.
**2019**, 104, 255–264. [Google Scholar] [CrossRef] - Yao, Z.; Huang, Q.; Ji, Z.; Li, X.; Bi, Q. Deep learning-based prediction of piled-up status and payload distribution of bulk material. Autom. Construct.
**2021**, 121. [Google Scholar] [CrossRef] - Luo, H.; Wang, M.; Wong, P.K.Y.; Tang, J.; Cheng, J.C.P. Construction machine pose prediction considering historical motions and activity attributes using gated recurrent unit (GRU). Autom. Construct.
**2021**, 121. [Google Scholar] [CrossRef] - Shi, J.; Sun, D.; Hu, M.; Liu, S.; Kan, Y.; Chen, R.; Ma, K. Prediction of brake pedal aperture for automatic wheel loader based on deep learning. Autom. Construct.
**2020**, 119. [Google Scholar] [CrossRef] - Xing, Y.; Lv, C.; Mo, X.; Hu, Z.; Huang, C.; Hang, P. Toward Safe and Smart Mobility: Energy-Aware Deep Learning for Driving Behavior Analysis and Prediction of Connected Vehicles. IEEE Trans. Intell. Transp. Syst.
**2021**, 22, 4267–4280. [Google Scholar] [CrossRef] - Dai, S.; Li, L.; Li, Z. Modeling Vehicle Interactions via Modified LSTM Models for Trajectory Prediction. IEEE Access
**2019**, 7, 38287–38296. [Google Scholar] [CrossRef] - Fang, W.; Ding, L.; Zhong, B.; Love, P.E.D.; Luo, H. Automated detection of workers and heavy equipment on construction sites: A convolutional neural network approach. Adv. Eng. Inform.
**2018**, 37, 139–149. [Google Scholar] [CrossRef] - Kim, J.; Chi, S.; Seo, J. Interaction analysis for vision-based activity identification of earthmoving excavators and dump trucks. Autom. Construct.
**2018**, 87, 297–308. [Google Scholar] [CrossRef] - Zhang, S.; Yao, L.; Sun, A.; Tay, Y. Deep Learning Based Recommender System: A Survey and New Perspectives. ACM Comput. Surv.
**2019**, 52, 1–38. [Google Scholar] [CrossRef] [Green Version] - Hochreiter, S.; Schmidhuber, J. Long short-term memory. Neural Comput.
**1997**, 9, 1735–1780. [Google Scholar] [CrossRef] - Kingma, D.P.; Ba, J.L. Adam: A Method for Stochastic Optimization. arXiv
**2018**, arXiv:1412.6980. [Google Scholar] - Nair, V.; Hinton, G.E. Rectified linear units improve Restricted Boltzmann machines. In Proceedings of the ICML, Haifa, Israel, 21–24 June 2010; pp. 807–814. [Google Scholar]

**Figure 6.**Schematic diagram of working condition division: (

**a**) velocity of wheel loader; (

**b**) lift cylinder pressure; (

**c**) tilt cylinder pressure.

**Figure 8.**Comparison of RMSE from different materials: (

**a**) prediction of throttle value (

**b**) prediction of state. mean and median values are shown with ‘–’ and ‘—’ respectively.

**Figure 9.**Driving data of experienced drivers and the predicted value from different materials: (

**a**) small coarse gravel (

**b**) large coarse gravel.

**Figure 10.**RMSE comparison of different LSTM networks using small coarse gravel: (

**a**) prediction of throttle value (

**b**) prediction of state.

**Figure 11.**RMSE comparison of different LSTM networks using large coarse gravel: (

**a**) prediction of throttle value (

**b**) prediction of state.

**Figure 12.**RMSE comparison of BPNNs and LSTM networks using small coarse gravel: (

**a**) prediction of throttle value (

**b**) prediction of state.

**Figure 13.**RMSE comparison of BPNNs and LSTM networks using large coarse gravel: (

**a**) prediction of throttle value (

**b**) prediction of state.

**Figure 14.**Relationship between sampling frequency and prediction performance using large coarse gravel: (

**a**) prediction of throttle value (

**b**) prediction of state.

**Figure 15.**Relationship between sampling frequency and prediction performance using small coarse gravel: (

**a**) throttle prediction (

**b**) state prediction.

Phase | Path |
---|---|

Forward with no load | V1 |

Bucket filling | V2 |

Backward with full load | V3 |

Forward and hoisting | V4 |

Dumping | V5 |

Backward with no load | V6 |

Phase | Multiple LSTM of Using LCG | Multiple LSTM of Using SCG | Single LSTM of Using LCG | Single LSTM of Using SCG | BPNN of Using LCG | BPNN of Using SCG |
---|---|---|---|---|---|---|

V1 | 0.64 | 0.54 | 0.71 | 0.57 | 0.75 | 0.73 |

V2 | 1.07 | 0.89 | 1.67 | 1.09 | 1.36 | 1.14 |

V3 | 0.83 | 0.78 | 0.86 | 0.92 | 1.18 | 0.91 |

V4 | 1.00 | 0.91 | 1.02 | 1.00 | 1.28 | 1.13 |

V5 | 0.95 | 0.87 | 1.52 | 1.01 | 1.26 | 1.10 |

V6 | 0.96 | 0.90 | 1.10 | 1.06 | 1.16 | 1.01 |

Phase | Multiple LSTM of Using LCG | Multiple LSTM of Using SCG | Single LSTM of Using LCG | Single LSTM of Using SCG | BPNN of Using LCG | BPNN of Using SCG |
---|---|---|---|---|---|---|

V1 | 2.36 | 1.53 | 2.57 | 1.68 | 4.53 | 1.95 |

V2 | 3.44 | 1.34 | 4.56 | 1.75 | 4.49 | 1.85 |

V3 | 2.36 | 1.44 | 2.66 | 1.67 | 3.89 | 2.64 |

V4 | 3.03 | 1.98 | 3.36 | 2.09 | 4.17 | 2.32 |

V5 | 3.08 | 1.18 | 4.30 | 1.43 | 4.01 | 1.89 |

V6 | 2.28 | 1.81 | 2.44 | 1.96 | 3.08 | 2.48 |

Sampling Frequency | RMSE of Throttle Value Using LCG | RMSE of State Using LCG | RMSE of Throttle Value Using SCG | RMSE of State Using SCG |
---|---|---|---|---|

10 HZ | 4.90 | 17.95 | 4.00 | 9.72 |

20 HZ | 2.93 | 11.93 | 2.07 | 5.43 |

50 HZ | 1.51 | 3.53 | 1.44 | 3.00 |

100 HZ | 1.17 | 3.00 | 1.09 | 2.64 |

200 HZ | 0.93 | 2.75 | 0.82 | 1.65 |

Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations. |

© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Huang, J.; Cheng, X.; Shen, Y.; Kong, D.; Wang, J.
Deep Learning-Based Prediction of Throttle Value and State for Wheel Loaders. *Energies* **2021**, *14*, 7202.
https://doi.org/10.3390/en14217202

**AMA Style**

Huang J, Cheng X, Shen Y, Kong D, Wang J.
Deep Learning-Based Prediction of Throttle Value and State for Wheel Loaders. *Energies*. 2021; 14(21):7202.
https://doi.org/10.3390/en14217202

**Chicago/Turabian Style**

Huang, Jianfei, Xinchun Cheng, Yuying Shen, Dewen Kong, and Jixin Wang.
2021. "Deep Learning-Based Prediction of Throttle Value and State for Wheel Loaders" *Energies* 14, no. 21: 7202.
https://doi.org/10.3390/en14217202