A Roadheader Positioning Method Based on Multi-Sensor Fusion
Abstract
:1. Introduction
2. Related Work
3. Framework of Proposed Method
- Geocentric Coordinate System: This system has its origin at the Earth’s center. The x-axis is directed towards the vernal equinox, while the y-axis aligns with the Earth’s rotational axis, pointing to the North Pole. The z-axis, in conjunction with the other axes, creates a right-handed coordinate system. Notably, the outputs from both the gyroscope and accelerometer rely on this system.
- Earth Coordinate System: Centered at the Earth’s core, the x and y axes of this system lie on the Earth’s equatorial plane. Specifically, the x-axis indicates the juncture of the prime meridian and the Equator, whereas the y-axis extends towards the North Pole in alignment with the Earth’s rotational motion.
- Navigation Coordinate System: With the roadheader’s center as its origin, the x-axis of this system is directed towards the geographic east. The y-axis aligns with the geographic north. Complementing the other two axes, the z-axis helps to constitute a right-handed system. This system serves as the referential framework for showcasing the integrated positioning system’s final results.
- Carrier Coordinate System: This system also uses the roadheader’s center as its starting point. Here, the x-axis looks to the right side, and the y-axis faces the roadheader’s front. Meanwhile, the z-axis rises upwards, and together, these axes form a right-handed coordinate system.
- Camera Coordinate System: Rooted at the left camera’s optical center, the x-axis of this system runs rightward along the camera’s baseline. Conversely, the y-axis descends, while the z-axis, combined with the other two, establishes a right-handed coordinate system. This orientation follows the camera’s optical path.
4. Position and Attitude Perception System Based on Improved Stereo Visual Odometry
4.1. Improved Stereo Visual Odometry Front End: Image Pre-Processing with Image Enhancement Module
4.1.1. Extraction of the Reflection Component of the Mine Image
4.1.2. Mine-Image Dehaze
4.1.3. Image Enhancement Experiment
4.2. Stereo Visual Odometry Back End: Position and Attitude Perception for a Roadheader Based on Motion Estimation
5. Position and Attitude Perception for a Roadheader Based on the SINS
5.1. Update of Roadheader Attitude
5.2. Update of Roadheader Velocity and Position
5.3. Error Equation of the SINS
5.4. Design of Kalman Filter
- Predict the state quantity :
- Predict the state covariance matrix
- Update Kalman filter gain
- Update the state variable quantity :
- Update the state covariance matrix:
6. Experiment and Discussion
6.1. Dataset
- Visio struggles with accurately estimating attitude, but it is relatively consistent on overall translation.
- With IMU, the position estimation from triple integration of accelerometer and gyroscope measurements drifts rapidly over time. Also, the IMU results have lower orientation error than Vision; this implies that the majority of IMU drifts are the result of integrating noisier accelerometer measurements, leading to increasingly inaccurate velocity estimates.
- Comparing to ORBSLAM2 [56], our method obtains lower error on all sequences, which can be attributed to the image-enhancement module and the ability of the Kalman filter to fuse the data of IMU and Visio.
6.2. Integrated Positioning Experiment
6.3. Discussions
6.4. Limitation
7. Conclusions
- (1)
- With the goal of achieving the precise positioning and attitude perception of the roadheader, we propose an integrated positioning method based on the fusion of the strapdown inertial navigation system (SINS) and stereo visual odometry. The fundamental concept of this method is to utilize the strapdown inertial navigation system (SINS) as the reference system and incorporate stereo visual odometry as the supplementary system. The data from both systems are combined using the Kalman filter.
- (2)
- To eliminate the influence of low-quality images on the accuracy of stereo visual odometry, we designed an image-enhancement module to preprocess the images. We tested our image-enhancement module on both public and self-built datasets and conducted comparative experiments with other image-enhancement methods. The results show that our image-enhancement module can effectively improve image quality, increase the number of features extracted from images, and improve the accuracy of feature matching.
- (3)
- We tested the proposed integrated positioning method on the KITTI dataset and the EuRoC dataset and compared proposed methods with three other methods. In addition, we designed and conducted integrated positioning experiments on a simulated roadway, with the roadheader as the experimental object. The experimental results demonstrated that the maximum errors for roll, yaw, and pitch were 0.1129°, 1.3589°, and 0.9759°, respectively. Additionally, the maximum errors for displacement along the x, y, and z axes were 0.0360 m, 0.1172 m, and 0.0150 m, respectively.
Author Contributions
Funding
Conflicts of Interest
References
- Deshmukh, S.; Raina, A.K.; Murthy, V.M.S.R.; Trivedi, R.; Vajre, R. Roadheader–A comprehensive review. Tunn. Undergr. Space Technol. 2020, 95, 103148. [Google Scholar] [CrossRef]
- Corke, P.; Roberts, J.; Cunningham, J.; Hainsworth, D. Mining robotics. In Springer Handbook of Robotics; Springer: Berlin/Heidelberg, Germany, 2008; pp. 1127–1150. [Google Scholar] [CrossRef]
- Jiang, X.X.; Li, C.X. Statistical analysis on coal mine accidents in China from 2013 to 2017 and discussion on the countermeasures. Coal Eng. 2019, 51, 101–105. [Google Scholar] [CrossRef]
- Liu, Y.; Li, Y.Q. Research on the automatic laser navigation system of the tunnel boring machine. Seventh Int. Symp. Precis. Eng. Meas. Instrum. 2011, 8321, 484–489. [Google Scholar] [CrossRef]
- Cui, Y.; Liu, S.; Liu, Q. Navigation and positioning technology in underground coal mines and tunnels: A review. J. South. Afr. Inst. Min. Metall. 2021, 121, 295–303. [Google Scholar] [CrossRef]
- Xie, H.P.; Wu, L.X.; Zheng, D.Z. Prediction on the energy consumption and coal demand of china in 2025. J. China Coal Soc. 2019, 44, 1949–1960. [Google Scholar] [CrossRef]
- Liu, M.; Gao, Y.; Li, G.; Guang, X.; Li, S. An improved alignment method for the strapdown inertial navigation system (SINS). Sensors 2016, 16, 621. [Google Scholar] [CrossRef]
- Tian, W.Q.; Tian, Y.; Jia, Q.; Zhang, K. Research status and development trend of cantilever Roadheader navigation technology. Coal Sci. Technol. 2022, 50, 0253–2336. [Google Scholar] [CrossRef]
- Qin, T.; Li, P.L.; Shen, S.J. Vins-mono: A robust and versatile monocular visual-inertial state estimator. IEEE Trans. Robot. 2018, 34, 1004–1020. [Google Scholar] [CrossRef]
- Mur, A.; Raul, J.M.M.M.; Juan, D.T. ORB-SLAM: A versatile and accurate monocular SLAM system. IEEE Trans. Robot. 2015, 31, 1147–1163. [Google Scholar] [CrossRef]
- Campos, C.; Elvira, R.; Rodríguez, J.J.G.; Montiel, J.M.; Tardós, J.D. Orb-slam3: An accurate open-source library for visual, visual–inertial, and multimap slam. IEEE Trans. Robot. 2021, 37, 1874–1890. [Google Scholar] [CrossRef]
- Fu, S.; Li, Y.; Zhang, M.; Zong, K.; Cheng, L.; Wu, M. Ultra-wideband pose detection system for boom-type Roadheader based on Caffery transform and Taylor series expansion. Meas. Sci. Technol. 2017, 29, 015101. [Google Scholar] [CrossRef]
- Fu, S.; Li, Y.; Zong, K.; Liu, C.; Liu, D.; Wu, M. Ultra-wideband pose detection method based on TDOA positioning model for boom-type Roadheader. AEU Int. J. Electron. Commun. 2019, 99, 70–80. [Google Scholar] [CrossRef]
- Roman, M.; Breido, J.; Drijd, N. Development of Position System of a Roadheader on a Base of Active IR-sensor. Procedia Eng. 2015, 100, 617–621. [Google Scholar] [CrossRef]
- Du, Y.; Tong, M.; Liu, T.; Dong, H. Visual measurement system for Roadheaders pose detection in mines. Opt. Eng. 2016, 55, 104107. [Google Scholar] [CrossRef]
- Tao, Y.; Huang, Y.; Chen, J.; Li, P.; Wu, M. Study on measurement error of angle of deviation and offset distance of Roadheader by single-station, multipoint and time-shared measurement system based on iGPS. In Proceedings of the CSAA/IET International Conference on Aircraft Utility Systems (AUS 2018), Guiyang, China, 19–22 June 2018. [Google Scholar] [CrossRef]
- Tian, Y. Inertial navigation positioning method of Roadheader based on zero-velocity update. Ind. Mine Autom. 2019, 45, 70–73. [Google Scholar] [CrossRef]
- Shen, Y.; Wang, P.; Zheng, W.; Ji, X.; Jiang, H.; Wu, M. Error compensation of strapdown inertial navigation system for the boom-type Roadheader under complex vibration. Axioms 2021, 10, 224. [Google Scholar] [CrossRef]
- Wang, H.R. Roadheader combined positioning method based on strapdown inertial navigation and differential odometer. Ind. Mine Autom. 2022, 48, 148–156. [Google Scholar] [CrossRef]
- Yang, J.; Wang, C.; Zhang, Q.; Chang, B.; Wang, F.; Wang, X.; Wu, M. Modeling of Laneway Environment and Locating Method of Roadheader Based on Self-Coupling and Hector SLAM. In Proceedings of the 5th International Conference on Electromechanical Control Technology and Transportation, Nanchang, China, 15 May 2020. [Google Scholar] [CrossRef]
- Wang, L.X. Pose Measurement Technology of Roadheader Body based on Fusion of Visual and SINS. J. Phys. Conf. Ser. 2022, 2363, 012014. [Google Scholar] [CrossRef]
- Zhang, W.; Zhai, G.; Yue, Z.; Pan, T.; Cheng, R. Research on visual positioning of a Roadheader and construction of an environment map. Appl. Sci. 2021, 11, 4968. [Google Scholar] [CrossRef]
- Li, C.; Liu, J.; Zhu, J.; Zhang, W.; Bi, L. Mine image enhancement using adaptive bilateral gamma adjustment and double plateaus histogram equalization. Multimed. Tools Appl. 2022, 81, 12643–12660. [Google Scholar] [CrossRef]
- Zhang, W. Research on image enhancement algorithm for the monitoring system in coal mine hoist. Meas. Control 2023, 00202940231173767. [Google Scholar] [CrossRef]
- Nan, Z.; Yun, G. An image enhancement method in coal mine underground based on deep retinex network and fusion strategy. In Proceedings of the 6th International Conference on Image, Qingdao, China, 21–23 April 2021. [Google Scholar] [CrossRef]
- Zhuang, P.; Ding, X. Divide-and-conquer framework for image restoration and enhancement. Eng. Appl. Artif. Intell. 2019, 85, 830–844. [Google Scholar] [CrossRef]
- Hosseini, S.A.; Abbas, A.S.; Reza, A. Prediction of bedload transport rate using a block combined network structure. Hydrol. Sci. J. 2022, 67, 117–128. [Google Scholar] [CrossRef]
- Zhuang, P. Image enhancement using divide-and-conquer strategy. J. Vis. Commun. Image Represent. 2017, 45, 137–146. [Google Scholar] [CrossRef]
- Ghosh, S.K.; Biswajit, B.; Anupam, G. A novel approach of retinal image enhancement using PSO system and measure of fuzziness. Procedia Comput. Sci. 2020, 167, 1300–1311. [Google Scholar] [CrossRef]
- Shahri, A.A.; Spross, J.; Johansson, F.; Larsson, S. Landslide susceptibility hazard map in southwest Sweden using artificial neural network. Catena 2019, 183, 104225. [Google Scholar] [CrossRef]
- Ibrahim, H.; Nicholas, S.P.K. Brightness preserving dynamic histogram equalization for image contrast enhancement. IEEE Trans. Consum. Electron. 2007, 53, 1752–1758. [Google Scholar] [CrossRef]
- Peng, L.I.; Yong, H.; Kunlun, Y.A.O. Multi-algorithm fusion of RGB and HSV color spaces for image enhancement. In Proceedings of the 37th Chinese Control Conference (CCC), Wuhan, China, 28–27 July 2018. [Google Scholar] [CrossRef]
- Petro, A.B.; Catalina, S.; Jean, M.M. Multiscale retinex. Image Process. Line 2014, 4, 71–88. [Google Scholar] [CrossRef]
- Land, E.H.; John, J.; Cann, M. Lightness and retinex theory. Josa 1971, 61, 1–11. [Google Scholar] [CrossRef]
- Al, H.; Mohammad, A.; Zohair, A.A. Retinex-Based Multiphase Algorithm for Low-Light Image Enhancement. Trait. Du Signal 2020, 37, 733–743. [Google Scholar] [CrossRef]
- Jobson, D.J.; Zia-ur, R.; Glenn, A.W. Properties and performance of a center/surround retinex. IEEE Trans. Image Process. 1997, 6, 451–462. [Google Scholar] [CrossRef] [PubMed]
- Cho, Y.; Ayoung, K. Visibility enhancement for underwater visual SLAM based on underwater light scattering model. In Proceedings of the IEEE International Conference on Robotics and Automation (ICRA), Singapore, 29 May–3 June 2017. [Google Scholar] [CrossRef]
- He, K.; Jian, S.; Xiaoou, T. Single image haze removal using dark channel prior. IEEE Trans. Pattern Anal. Mach. Intell. 2010, 33, 2341–2353. [Google Scholar] [CrossRef] [PubMed]
- Lee, C.; Chul, L.; Chang, S.K. Contrast enhancement based on layered difference representation. In Proceedings of the 19th IEEE International Conference on Image Processing, Orlando, FL, USA, 30 September–3 October 2012. [Google Scholar] [CrossRef]
- Guo, X.; Yu, L.; Haibin, L. LIME: Low-light image enhancement via illumination map estimation. IEEE Trans. Image Process. 2016, 26, 982–993. [Google Scholar] [CrossRef] [PubMed]
- Wang, S.; Zheng, J.; Hu, H.M.; Li, B. Naturalness preserved enhancement algorithm for non-uniform illumination images. IEEE Trans. Image Process. 2013, 22, 3538–3548. [Google Scholar] [CrossRef] [PubMed]
- Vonikakis, V.; Odysseas, B.; Ioannis, A. Multi-exposure image fusion based on illumination estimation. Proc. IASTED SIPA 2011, 135–142. [Google Scholar] [CrossRef]
- Mittal, A.; Soundararajan, R.; Bovik, A.C. Making a “completely blind” image quality analyzer. IEEE Signal Process. Lett. 2012, 20, 209–212. [Google Scholar] [CrossRef]
- Gu, K.; Lin, W.; Zhai, G.; Yang, X.; Zhang, W.; Chen, C.W. No-reference quality metric of contrast-distorted images based on information maximization. IEEE Trans. Cybern. 2016, 47, 4559–4565. [Google Scholar] [CrossRef]
- Wang, R.; Zhang, Q.; Fu, C.W.; Shen, X.; Zheng, W.S.; Jia, J. Underexposed photo enhancement using deep illumination estimation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Long Beach, CA, USA, 15–20 June 2019. [Google Scholar] [CrossRef]
- Gong, Y.; Liao, P.; Zhang, X.; Zhang, L.; Chen, G.; Zhu, K.; Tan, X.; Lv, Z. Enlighten-GAN for Super Resolution Reconstruction in Mid-Resolution Remote Sensing Images. Remote Sens. 2021, 13, 1104. [Google Scholar] [CrossRef]
- Lv, F.; Lu, F.; Wu, J.; Lim, C. MBLLEN: Low-Light Image/Video Enhancement Using CNNs. BMVC 2018, 220, 4. Available online: https://api.semanticscholar.org/CorpusID:52285038 (accessed on 6 September 2023).
- Wei, C.; Wang, W.; Yang, W.; Liu, J. Deep retinex decomposition for low-light enhancement. arXiv 2018, arXiv:1808.04560. [Google Scholar] [CrossRef]
- Calonder, M.; Lepetit, V.; Strecha, C.; Fua, P. Brief: Binary robust independent elementary features. In Proceedings of the Computer Vision–ECCV 11th European Conference on Computer Vision, Heraklion, Greece, 5–11 September 2010. [Google Scholar] [CrossRef]
- Zhenhuan, W.; Chen, X.; Zeng, Q. Comparison of strapdown inertial navigation algorithm based on rotation vector and dual quaternion. Chin. J. Aeronaut. 2013, 26, 442–448. [Google Scholar] [CrossRef]
- Tian, M.; Liang, Z.; Liao, Z.; Yu, R.; Guo, H.; Wang, L. A Polar Robust Kalman Filter Algorithm for DVL-Aided SINSs Based on the Ellipsoidal Earth Model. Sensors 2022, 22, 7879. [Google Scholar] [CrossRef] [PubMed]
- Cui, Y.; Liu, S.; Yao, J.; Gu, C. Integrated positioning system of unmanned automatic vehicle in coal mines. IEEE Trans. Instrum. Meas. 2021, 70, 1–13. [Google Scholar] [CrossRef]
- Huang, Y.; Yonggang, Z. A new process uncertainty robust Student’st based Kalman filter for SINS/GPS integration. IEEE Access 2017, 5, 14391–14404. [Google Scholar] [CrossRef]
- Geiger, A.; Philip, L.; Raquel, U. Are we ready for autonomous driving? the kitti vision benchmark suite. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, Providence, RI, USA, 16 June 2012. [Google Scholar] [CrossRef]
- Burri, M.; Nikolic, J.; Gohl, P.; Schneider, T.; Rehder, J.; Omari, S.; Achtelik, M.W.; Siegwart, R. The EuRoC micro aerial vehicle datasets. Int. J. Robot. Res. 2016, 35, 1157–1163. [Google Scholar] [CrossRef]
- Mur-Artal, R.; Tardós, J.D. Orb-slam2: An open-source slam system for monocular, stereo, and rgb-d cameras. IEEE Trans. Robot. 2017, 33, 1255–1262. [Google Scholar] [CrossRef]
- Liu, Y.; Hongwei, W.; Lei, T. TBM-MSE: A Multi-engine State Estimation Based on Inertial Enhancement for Tunnel Boring Machines in Perceptually Degraded Roadways. IEEE Access 2023, 11, 55978–55989. [Google Scholar] [CrossRef]
- Huang, X.; Liu, Q.; Shi, K.; Pan, Y.; Liu, J. Application and prospect of hard rock TBM for deep roadway construction in coal mines. Tunn. Undergr. Space Technol. 2018, 73, 105–126. [Google Scholar] [CrossRef]
- Yan, Z.; Wang, H.; Geng, Y. Coal-rock interface image recognition method based on improved DeeplabV3+ and transfer learning. Coal Sci. Technol. 2023, 41, 328–338. [Google Scholar] [CrossRef]
Methods | DICM | LIME | NPEA | VV |
---|---|---|---|---|
LIESD | 3.82 | 4.38 | 3.54 | 3.09 |
KinD | 3.33 | 4.77 | 3.51 | 3.37 |
Mbllen | 3.72 | 4.51 | 3.94 | 4.18 |
Enlight-GAN | 3.57 | 3.72 | 4.11 | 2.58 |
Ours | 3.47 | 3.64 | 3.73 | 3.28 |
Methods | DICM | LIME | NPEA | VV |
---|---|---|---|---|
Retinex_net | 5.20 | 4.83 | 5.20 | 5.30 |
KinD | 5.15 | 4.95 | 5.01 | 5.44 |
Mbllen | 5.31 | 5.13 | 5.19 | 5.38 |
Enlight-GAN | 5.13 | 4.88 | 5.15 | 5.45 |
Ours | 5.47 | 5.41 | 5.13 | 5.49 |
Seq. | IMU | Visio | ORB-SLAM2 | Proposed | ||||
---|---|---|---|---|---|---|---|---|
t | r | t | r | t | r | t | r | |
00 | 5.82 | 1.45 | 5.43 | 1.75 | 3.49 | 1.47 | 2.75 | 1.32 |
01 | 9.73 | 3.29 | 8.22 | 3.17 | 7.52 | 2.92 | 6.17 | 2.84 |
02 | 10.72 | 3.26 | 7.02 | 5.11 | 6.69 | 4.73 | 5.52 | 2.57 |
03 | 5.18 | 1.25 | 4.19 | 1.63 | 3.07 | 1.27 | 2.51 | 1.08 |
Seq. | IMU | Visio | ORB-SLAM2 | Proposed | ||||
---|---|---|---|---|---|---|---|---|
t | r | t | r | t | r | t | r | |
MH_01 | 0.45 | 1.53 | 0.49 | 1.65 | 0.34 | 1.62 | 0.26 | 1.37 |
MH_02 | 0.63 | 1.59 | 0.61 | 1.70 | 0.35 | 1.71 | 0.30 | 1.09 |
MH_03 | 0.59 | 1.54 | 0.22 | 1.65 | 0.24 | 1.01 | 0.17 | 0.97 |
MH_04 | 0.51 | 1.38 | 0.72 | 1.53 | 0.38 | 1.27 | 0.21 | 1.12 |
Parameters | CH110 | FSON II |
---|---|---|
Gyroscope zero bias/ | ≤3.5 | ≤0.005 |
Alignment accuracy/(°) | 0.1 | 0.01 |
Heading accuracy/(°) | ≤0.4 | ≤0.02 |
Positioning Method | Assessment Indicators | x/m | y/m | z/m | Roll/° | Yaw/° | Pitch/° |
---|---|---|---|---|---|---|---|
Proposed method | Maximum error | 0.0360 | 0.1172 | 0.0150 | 0.1129 | 1.0616 | 0.9759 |
Average error | −0.0012 | −0.0615 | 0.0073 | −0.0343 | −0.477 | −0.5367 | |
Standard deviation | 0.0150 | 0.0328 | 0.0035 | 0.0358 | 0.2751 | 0.2413 | |
SINS | Maximum error | 0.1148 | 0.3942 | 0.0581 | 0.2203 | 1.3589 | 1.0110 |
Average error | −0.0036 | −0.2061 | 0.0314 | −0.0631 | −0.1227 | −0.5782 | |
Standard deviation | 0.0492 | 0.1143 | 0.0133 | 0.0928 | 0.6188 | 0.2401 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Wang, H.; Li, Z.; Wang, H.; Cao, W.; Zhang, F.; Wang, Y. A Roadheader Positioning Method Based on Multi-Sensor Fusion. Electronics 2023, 12, 4556. https://doi.org/10.3390/electronics12224556
Wang H, Li Z, Wang H, Cao W, Zhang F, Wang Y. A Roadheader Positioning Method Based on Multi-Sensor Fusion. Electronics. 2023; 12(22):4556. https://doi.org/10.3390/electronics12224556
Chicago/Turabian StyleWang, Haoran, Zhenglong Li, Hongwei Wang, Wenyan Cao, Fujing Zhang, and Yuheng Wang. 2023. "A Roadheader Positioning Method Based on Multi-Sensor Fusion" Electronics 12, no. 22: 4556. https://doi.org/10.3390/electronics12224556
APA StyleWang, H., Li, Z., Wang, H., Cao, W., Zhang, F., & Wang, Y. (2023). A Roadheader Positioning Method Based on Multi-Sensor Fusion. Electronics, 12(22), 4556. https://doi.org/10.3390/electronics12224556