# Vibration and Image Texture Data Fusion-Based Terrain Classification Using WKNN for Tracked Robots

^{1}

^{2}

^{*}

## Abstract

**:**

## 1. Introduction

- (1)
- Describing the characteristics of different driving terrains by extracting time-domain and frequency-domain features from the vibration signals of tracked robots These extracted features are then processed using the PCA algorithm to reduce dimensionality, transforming the original vibration signals into more representative feature vectors. Additionally, the PCA-processed vibration signals features are fused with the terrain image texture features to obtain more detailed and precise information to describe the current terrain features of the tracked robot.
- (2)
- To enhance the significance of closer samples in determining terrain classification, the traditional KNN algorithm is modified by introducing a weighting operation, resulting in the improved WKNN algorithm. Combined with the previously extracted feature matrix that combines vibration and image information, the tracked robot achieves terrain classification with high accuracy and robustness.

## 2. Terrain Classification Algorithm

#### 2.1. Vibration Signal Feature Extraction

#### 2.1.1. Time-Domain Extraction

#### 2.1.2. Frequency-Domain Extraction

#### 2.1.3. PCA Algorithm

_{i}is the i-th eigenvalue, x

_{imin}is the minimum value of the i-th eigenvalue, x

_{imax}is the maximum value of the i-th eigenvalue, and X is the i-th eigenvalue after normalization.

- (1)
- Firstly, it is first necessary to standardize the data in Z-groups so that the units and ranges of different features are the same, as shown in Equation (2).

- (2)
- Based on the given data matrix, the covariance matrix C is constructed as follows:

- (3)
- Based on this, the eigenvalues of the covariance matrix C and the corresponding eigenvectors can be obtained.
- (4)
- The selection criteria for the principal components are as follows:

- (5)
- Based on the selected principal components, the projection matrix P is constructed as follows:

_{i}is the feature vector corresponding to the i-th principal component.

- (6)
- Calculating the reduced dimensional data.

#### 2.2. Image Texture Feature Extraction

_{E}, contrast S

_{con}, correlation S

_{cor}, entropy S

_{En}, and difference moment D, are selected as feature vectors in this paper, which are calculated as follows [24,25]:

_{i}and μ

_{j}are the mean values of rows and columns of P, respectively; and σ

_{i}and σ

_{j}are the standard deviations of rows and columns of P, respectively.

#### 2.3. WKNN Classification Algorithm

- (1)
- Calculating the distance between the samples to be classified and the training samples.

_{1}, x

_{2}, …, x

_{n}). If the training samples are y = (y

_{1}, y

_{2}, …, y

_{n}), then the distance d(x, y) between them can be calculated using the following equation:

- (2)
- Selecting the closest k training samples;

- (3)
- Counting the categories of k training samples and obtaining result from a vote.

- (4)
- Classifying the samples to be classified into a category with the most votes.

## 3. Experimental Results

#### 3.1. Experimental Method

#### 3.2. Terrain Classification Based on Vibration Signals

#### 3.3. Terrain Classification by Fusing Vibration and Image Information

## 4. Conclusions

## Author Contributions

## Funding

## Data Availability Statement

## Conflicts of Interest

## References

- Lu, E.; Ma, Z.; Li, Y.M.; Xu, L.Z.; Tang, Z. Adaptive backstepping control of tracked robot running trajectory based on real-time slip parameter estimation. Int. J. Agric. Biol. Eng.
**2020**, 13, 178–187. [Google Scholar] [CrossRef] - Lu, E.; Zhao, Z.; Yin, J.J.; Luo, C.M.; Tian, Z.M. Trajectory Learning and Reproduction for Tracked Robot Based on Bagging-GMM/HSMM. J. Electr. Eng. Technol.
**2023**, 1–13. [Google Scholar] [CrossRef] - Barfoot, T.D. State Estimation for Robotics, 1st ed.; Cambridge University Press: Cambridge, UK, 2017. [Google Scholar]
- Ward, C.C.; Iagnemma, K. Speed-independent vibration-based terrain classification for passenger vehicles. Veh. Syst. Dyn.
**2009**, 47, 1095–1113. [Google Scholar] [CrossRef] - Shi, W.L.; Li, Z.; Lv, W.J.; Wu, Y.P.; Chang, J.; Li, X.C. Laplacian support vector machine for vibration-based robotic terrain classification. Electronics
**2020**, 9, 513. [Google Scholar] [CrossRef] [Green Version] - Xue, K.; Li, Q.; Xu, H.; Wang, T.L. Vibration-based terrain classification for robots using K-nearest neighbors algorithm. Zhendong Ceshi Yu Zhenduan/J. Vib. Meas. Diagn.
**2013**, 33, 88–92. [Google Scholar] - Komma, P.; Weiss, C.; Zell, A. Adaptive bayesian filtering for vibration-based terrain classification. In Proceedings of the 2009 IEEE International Conference on Robotics and Automation, Kobe, Japan, 17 May 2009. [Google Scholar]
- Brooks, C.A.; Iagnemma, K. Self-supervised terrain classification for planetary surface exploration rovers. J. Field Robot.
**2012**, 29, 445–468. [Google Scholar] [CrossRef] - DuPont, E.M.; Roberts, R.G.; Selekwa, M.F.; Moore, C.A.; Collins, E.G. Online terrain classification for mobile robots. In Proceedings of the ASME International Mechanical Engineering Congress and Exposition, Orlando, FL, USA, 5 November 2005. [Google Scholar]
- Bai, C.C.; Guo, J.F.; Zheng, H.X. Three-dimensional vibration-based terrain classification for mobile robots. IEEE Access
**2019**, 7, 63485–63492. [Google Scholar] [CrossRef] - Du, X.B.; Zhu, H. Research on terrain classification of tracked robot based on time-frequency characteristics and PCA-SVM. J. Henan Polytech. Univ. Nat. Sci.
**2019**, 38, 84–90. [Google Scholar] - Woods, M.; Guivant, J.; Katupitiya, J. Terrain classification using depth texture features. In Proceedings of the Australian Conference of Robotics and Automation, University of New South Wales, Sydney, Australia, 2 December 2013. [Google Scholar]
- Wu, H.; Zhang, W.C.; Li, B.; Sun, Y.C.; Duan, D.; Chen, P. Visual terrain classification methods for mobile robots using hybrid coding architecture. In Proceedings of the 2019 IEEE 4th International Conference on Image, Vision and Computing (ICIVC), Xiamen, China, 5 July 2019. [Google Scholar]
- Hu, T.; Li, W.H.; Qin, X.X.; Wang, P.; Yu, W.S. Terrain classification of polarimetric synthetic aperture radar images based on deep learning and conditional random field model. J. Radars
**2019**, 8, 471–478. [Google Scholar] - Filitchkin, P.; Byl, K. Feature-based terrain classification for littledog. In Proceedings of the 2012 IEEE/RSJ International Conference on Intelligent Robots and Systems, Algarve, Portugal, 7 October 2012. [Google Scholar]
- Kurup, A.; Kysar, S.; Bos, J. SVM-based sensor fusion for improved terrain classification. In Autonomous Systems: Sensors, Processing, and Security for Vehicles and Infrastructure; SPIE: Bellingham, WA, USA, 2020; Volume 11415, p. 114150G. [Google Scholar]
- Wang, W.; Zhang, B.T.; Wu, K.Q.; Chepinskiy, S.A.; Zhilenkov, A.A.; Chernyi, S.; Krasnov, A.Y. A visual terrain classification method for mobile robots’ navigation based on convolutional neural network and support vector machine. Trans. Inst. Meas. Control
**2022**, 44, 744–753. [Google Scholar] [CrossRef] - Hanson, N.; Shaham, M.; Erdoğmuş, D.; Padir, T. Vast: Visual and spectral terrain classification in unstructured multi-class environments. In Proceedings of the 2022 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Kyoto, Japan, 27 October 2022. [Google Scholar]
- Helmi, H.; Forouzantabar, A. Rolling bearing fault detection of electric motor using time domain and frequency domain features extraction and ANFIS. IET Electr. Power Appl.
**2019**, 13, 662–669. [Google Scholar] [CrossRef] - Zeng, R.Y. Terrain Identification and Autonomous Control for Tracked Mobile Robot. Ph.D. Thesis, University of Science and Technology Beijing, Beijing, China, 2021. [Google Scholar]
- Wu, X.; Chen, K.; Xie, C.Y.; Liu, S.B.; Mei, L.; Wang, W.S. Research on early warning method of feed pump vibration based on PCA-KNN. Chem. Eng. Mach.
**2019**, 49, 137–142. [Google Scholar] - Zhang, C.Y.; Wang, L.L.; Lv, Z.B.; Tian, X.; Wang, D.H.; Wang, G.Z. Mechanical State Identification Method Based on Vibration Parameter Image Combined with Gray Level Co-occurrence Matrix for Reactors. Noise Vib. Control
**2023**, 43, 154. [Google Scholar] - Meshkini, K.; Ghassemian, H. Texture classification using Shearlet transform and GLCM. In Proceedings of the 2017 Iranian Conference on Electrical Engineering (ICEE), Tehran, Iran, 4 May 2017. [Google Scholar]
- Haralick, R.M.; Shanmugam, K.; Dinstein, I.H. Textural features for image classification. IEEE Trans. Syst. Man Cybern.
**1973**, 6, 610–621. [Google Scholar] [CrossRef] [Green Version] - Mulyono, I.; Lukita, T.; Sari, C.; Setiadi, D.; Rachmawanto, E.; Susanto, A.; Putra, M.; Santoso, D. Parijoto Fruits Classification using K-Nearest Neighbor Based on Gray Level Co-Occurrence Matrix Texture Extraction. J. Phys. Conf. Ser.
**2019**, 1501, 012017. [Google Scholar] [CrossRef] - Cover, T.; Hart, P. Nearest neighbor pattern classification. IEEE Trans. Inf. Theory
**1967**, 13, 21–27. [Google Scholar] [CrossRef] [Green Version] - Suhariningsih, S.; Bastomi, M.Y.; Purwanti, E.; Hariyani, D.A.; Permatasari, P.A.D.; Astuti, S.D. X-ray image based on Gray Level Cooccurrence Matrices (GLCM) K-nearest neighbor (KNN) to detect tuberculosis. In Proceedings of the AIP Conference Proceedings, Surabaya, Indonesia, 14 October 2021. [Google Scholar]

**Figure 1.**Experimental scheme for terrain classification of tracked robots: (

**a**) tracked robot, (

**b**) data acquisition scheme.

**Figure 3.**Texture features of terrain images: (

**a**) energy features; (

**b**) contrast features; (

**c**) correlation features; (

**d**) entropy features; (

**e**) difference moment features.

Features | Description | Features | Description |
---|---|---|---|

Mean amplitude | ${T}_{1}=\frac{{\displaystyle {\sum}_{n=1}^{N}\left|x\left(n\right)\right|}}{N}$ | Kurtosis | ${T}_{11}=\frac{{\displaystyle {\sum}_{n=1}^{N}{\left(x\left(n\right)-\overline{x}\left(n\right)\right)}^{4}}}{\left(N-1\right){T}_{8}^{4}}$ |

Square root amplitude | ${T}_{2}={\left(\frac{{\displaystyle {\sum}_{n=1}^{N}\sqrt{\left|x\left(n\right)\right|}}}{N}\right)}^{2}$ | Variance | ${T}_{12}=\frac{{\displaystyle {\sum}_{n=1}^{N}{\left(x\left(n\right)-\overline{x}\left(n\right)\right)}^{2}}}{N-1}$ |

Maximum value | ${T}_{3}=\mathrm{max}\left(x\left(n\right)\right)$ | Standard deviation | ${T}_{13}=\sqrt{\frac{{\displaystyle {\sum}_{n=1}^{N}{\left(x\left(n\right)-\overline{x}\left(n\right)\right)}^{2}}}{N-1}}$ |

Minimum value | ${T}_{4}=\mathrm{m}\mathrm{i}\mathrm{n}\left(x\left(n\right)\right)$ | Skewness | ${T}_{14}=\frac{{\displaystyle {\sum}_{n=1}^{N}{\left(x\left(n\right)-\overline{x}\left(n\right)\right)}^{3}}}{\left(N-1\right){T}_{13}^{3}}$ |

Peak | ${T}_{5}={T}_{3}-{T}_{4}$ | Waveform factors | ${T}_{15}=\frac{{T}_{8}}{\left(1/N\right){\displaystyle {\sum}_{n=1}^{N}\left|x\left(n\right)\right|}}$ |

Peak value | ${T}_{6}=\mathrm{m}\mathrm{a}\mathrm{x}\left(\left|{T}_{3}\right|,\left|{T}_{4}\right|\right)$ | Pulse factor | ${T}_{16}=\frac{{T}_{5}}{\left(1/N\right){\displaystyle {\sum}_{n=1}^{N}\left|x\left(n\right)\right|}}$ |

Square mean root | ${T}_{7}={\left(\frac{{\displaystyle {\sum}_{n=1}^{N}\sqrt{\left|x\left(n\right)\right|}}}{N}\right)}^{2}$ | Residual gap factor | ${T}_{17}=\frac{{T}_{3}}{{T}_{2}}$ |

Root mean square | ${T}_{8}=\sqrt{\frac{{\displaystyle {\sum}_{n=1}^{N}{\left(x\left(n\right)\right)}^{2}}}{N}}$ | Skewness factor | ${T}_{18}=\frac{{T}_{6}}{{T}_{2}^{2}}$ |

Crest factor | ${T}_{9}=\frac{{T}_{5}}{{T}_{8}}$ | Peak state factor | ${T}_{19}=\frac{{T}_{11}}{{T}_{2}^{4}}$ |

Clarence or margin factor | ${T}_{10}=\frac{{T}_{6}}{{T}_{8}}$ | Yield factor | ${T}_{20}=\frac{{T}_{5}}{{T}_{2}}$ |

Features | Description | Features | Description |
---|---|---|---|

Mean | ${F}_{1}=\frac{{\displaystyle {\sum}_{u=1}^{U}s\left(u\right)}}{U}$ | Peak | ${F}_{5}=\mathrm{m}\mathrm{a}\mathrm{x}\left(s\left(u\right)\right)-\mathrm{m}\mathrm{i}\mathrm{n}\left(s\left(u\right)\right)$ |

Frequency center | ${F}_{2}=\frac{{\displaystyle \sum {}_{u=1}^{U}{f}_{k}s\left(u\right)}}{{\displaystyle \sum {}_{u=1}^{U}s\left(u\right)}}$ | Root mean square | ${F}_{6}=\sqrt{\frac{{\displaystyle \sum {}_{u=1}^{U}{f}_{u}^{2}s\left(u\right)}}{{\displaystyle \sum {}_{u=1}^{U}s\left(u\right)}}}$ |

Variance of mean frequency | ${F}_{3}=\frac{{\displaystyle \sum {}_{u=1}^{U}{\left(s\left(u\right)-{F}_{1}\right)}^{2}}}{U-1}$ | Mean square frequency | ${F}_{7}=\frac{{\displaystyle \sum {}_{u=1}^{U}{f}_{u}^{2}s\left(u\right)}}{{\displaystyle \sum {}_{u=1}^{U}s\left(u\right)}}$ |

Median value | ${F}_{4}=\mathrm{median}\left(s\left(u\right)\right)$ | Root mean frequency square | ${F}_{8}=\sqrt{\frac{{\displaystyle \sum {}_{u=1}^{U}{\left(s\left(u\right)-{F}_{1}\right)}^{2}}}{U-1}}$ |

Model | HWT605 |
---|---|

Communication method | RS485, RS232 |

Voltage | 5 V~36 V |

Current | <40 mA |

Size | 55 mm × 36.8 mm × 24 mm |

Measurement range | ±6 g |

Model | RER-USB4KHBR01 |
---|---|

Resolution and frame rate | 3840 × 2160 30FPS/1902 × 1080 30FPS 3840 × 2160 1FPS/2592 × 1944 1FPS |

Voltage | 5 V |

Current | 200 mA |

Size | 38 mm × 38 mm × 38 mm |

Lens specifications | Size: 1/2.5, Focal length: 1.95 mm DFOV > 120° |

Interface type | USB2.0 HIGH SPEED |

Maximum effective pixels | 3840(H) × 2160(V) |

Output image format | MJPEG/YUY2(YUYV) |

**Table 5.**KNN terrain classification results based on vibration features at v = 0.3 m/s. (ARR = 57.14%).

Train Terrain | Test Terrain | ||||
---|---|---|---|---|---|

Asphalt Road | Cement Road | Marble Road | Grassland | Cobblestone Road | |

Asphalt Road | 71.43% | 28.57% | |||

Cement Road | 14.29% | ||||

Marble Road | 28.57% | 100% | 85.71% | ||

Grassland | |||||

Cobblestone Road | 85.71% | 14.29% | 71.43% |

**Table 6.**KNN terrain classification results based on vibration features at v = 0.7 m/s. (ARR = 60.00%).

Train Terrain | Test Terrain | ||||
---|---|---|---|---|---|

Asphalt Road | Cement Road | Marble Road | Grassland | Cobblestone Road | |

Asphalt Road | 28.57% | 14.29% | |||

Cement Road | 85.71% | ||||

Marble Road | 71.43% | 14.29% | 100% | ||

Grassland | 100% | 14.29% | |||

Cobblestone Road | % | 71.43% |

**Table 7.**WKNN terrain classification results based on vibration using PCA at v = 0.3 m/s (AAR = 62.86%).

Train Terrain | Test Terrain | ||||
---|---|---|---|---|---|

Asphalt Road | Cement Road | Marble Road | Grassland | Cobblestone Road | |

Asphalt Road | 14.29% | ||||

Cement Road | 57.14% | 28.57% | |||

Marble Road | 100% | 100% | |||

Grassland | 85.71% | ||||

Cobblestone Road | 28.57% | 14.29% | 71.42% |

**Table 8.**WKNN terrain classification results based on vibration using PCA at v = 0.7 m/s. (AAR = 80.00%).

Train Terrain | Test Terrain | ||||
---|---|---|---|---|---|

Asphalt Road | Cement Road | Marble Road | Grassland | Cobblestone Road | |

Asphalt Road | 100% | 42.86% | |||

Cement Road | 85.71% | ||||

Marble Road | 14.29% | 57.14% | |||

Grassland | 85.71% | 28.57% | |||

Cobblestone Road | 14.29% | 71.43% |

**Table 9.**KNN terrain classification results based on image texture features at v = 0.3 m/s. (AAR = 77.14%).

Train Terrain | Test Terrain | ||||
---|---|---|---|---|---|

Asphalt Road | Cement Road | Marble Road | Grassland | Cobblestone Road | |

Asphalt Road | 85.71% | 57.14% | |||

Cement Road | 42.85% | ||||

Marble Road | 71.43% | 14.29% | |||

Grassland | 100% | ||||

Cobblestone Road | 14.29% | 28.57% | 85.71% |

**Table 10.**KNN terrain classification results based on image texture features at v = 0.7 m/s. (AAR = 62.86%).

Train Terrain | Test Terrain | ||||
---|---|---|---|---|---|

Asphalt Road | Cement Road | Marble Road | Grassland | Cobblestone Road | |

Asphalt Road | 71.43% | 14.29% | 14.29% | 14.29% | |

Cement Road | 57.19% | 14.29% | |||

Marble Road | 14.29% | 85.71% | 14.29% | ||

Grassland | 57.19% | 14.29% | |||

Cobblestone Road | 28.57% | 14.29% | 14.29% | 14.29% | 57.19% |

**Table 11.**WKNN terrain classification results based on fusion PCA-processed vibration and image texture features at v = 0.3 m/s. (AAR = 97.14%).

Train Terrain | Test Terrain | ||||
---|---|---|---|---|---|

Asphalt Road | Cement Road | Marble Road | Grassland | Cobblestone Road | |

Asphalt Road | 100% | ||||

Cement Road | 100% | ||||

Marble Road | 100% | ||||

Grassland | 85.71% | ||||

Cobblestone Road | 14.29% | 100% |

**Table 12.**WKNN terrain classification results based on fusion PCA-processed vibration and image texture features at v = 0.7 m/s. (AAR = 91.43%).

Train Terrain | Test Terrain | ||||
---|---|---|---|---|---|

Asphalt Road | Cement Road | Marble Road | Grassland | Cobblestone Road | |

Asphalt Road | 100% | ||||

Cement Road | 85.71% | ||||

Marble Road | 14.29% | 100% | |||

Grassland | 85.71% | 14.29% | |||

Cobblestone Road | 14.29% | 85.71% |

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |

© 2023 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Wang, H.; Lu, E.; Zhao, X.; Xue, J.
Vibration and Image Texture Data Fusion-Based Terrain Classification Using WKNN for Tracked Robots. *World Electr. Veh. J.* **2023**, *14*, 214.
https://doi.org/10.3390/wevj14080214

**AMA Style**

Wang H, Lu E, Zhao X, Xue J.
Vibration and Image Texture Data Fusion-Based Terrain Classification Using WKNN for Tracked Robots. *World Electric Vehicle Journal*. 2023; 14(8):214.
https://doi.org/10.3390/wevj14080214

**Chicago/Turabian Style**

Wang, Hui, En Lu, Xin Zhao, and Jialin Xue.
2023. "Vibration and Image Texture Data Fusion-Based Terrain Classification Using WKNN for Tracked Robots" *World Electric Vehicle Journal* 14, no. 8: 214.
https://doi.org/10.3390/wevj14080214