Effective Motion Sensors and Deep Learning Techniques for Unmanned Ground Vehicle (UGV)-Based Automated Pavement Layer Change Detection in Road Construction

Abstract

As-built progress of the constructed pavement should be monitored effectively to provide prompt project control. However, current pavement construction progress monitoring practices (e.g., data collection, processing, and analysis) are typically manual, time-consuming, tedious, and error-prone. To address this, this study proposes sensors mounted using a UGV-based methodology to develop a pavement layer change classifier measuring pavement construction progress automatically. Initially, data were collected using the UGV equipped with a laser ToF (time-of-flight) distance sensor, accelerometer, gyroscope, and GPS sensor in a controlled environment by constructing various scenarios of pavement layer change. Subsequently, four Long Short-Term Memory network variants (LSTMs) (LSTM, BiLSTM, CNN-LSTM, and ConvLSTM) were implemented on collected sensor data combinations for developing pavement layer change classifiers. The authors conducted the experiment to select the best sensor combinations for feature detection of the layer change classifier model. Subsequently, individual performance measures of each class with learning curves and confusion matrices were generated using sensor combination data to find out the best algorithm among all implemented algorithms. The experimental result demonstrates the (a_z + g_x + D) sensor combination as the best feature detector with high-performance measures (accuracy, precision, recall, and F1 score). The result also confirms the ConvLSTM as the best algorithm with the highest overall accuracy of 97.88% with (a_z + g_x + D) sensor combination data. The high-performance measures with the proposed approach confirm the feasibility of detecting pavement layer changes in real pavement construction projects. This proposed approach can potentially improve the efficiency of road construction progress measurement. This research study is a stepping stone for automated road construction progress monitoring.

1. Introduction

Most pavement construction projects often suffer from time and cost overruns due to associated risks in the construction stage [1,2]. Efficient construction progress monitoring (CPM) is vital to enable the as-built progress status of the project by comparing as-built progress with as-planned progress throughout the project for project control [3,4]. The as-built progress measurement is measured for the construction of different road layers (e.g., subgrade, subbase, base, and surface) and then combined along the road project length to evaluate overall pavement construction progress. [5,6]. During this process, each constructed (as-built) road layer with corresponding spatial information (starting and finishing points) is identified and utilized for as-built measurement.

Traditionally, this process is manually carried out, which is characterized by text-based data collection and manual checklists. Project managers typically spend 30–50% of their time gathering and analyzing these as-built collected data [7]. As such, manual CPM is criticized for being manual, tedious, error-prone, and time-consuming [8]. Subsequently, recent technological developments introduced several automated data collection technologies to capture as-built data of pavement construction with less manual intervention. Examples include technologies such as the Global Navigation Satellite System (GNSS), total station, sensor (LiDAR and RGB camera)-equipped unmanned aerial vehicle (UAV) [2,3,9,10,11]. However, the collected as-built data are manually analyzed in the form of point clouds, aerial images, and surveyed data (for example, total station and GNSS) [3,9,12,13]. The interpretation of these data may be inconsistent, subjective, and time-consuming [12,14]. Despite the efficiency gains, as-built component detection remains complex, manual, or semiautomated, which causes delays in the project controlling action [2,12].

Road pavement construction is characterized by its repetitive nature, consisting of activities that are sequentially repeated along the road length, such as constructing various road pavement layers along the road length, including the surface layer, base course layer, subbase layer, and subgrade layer [4,12]. Hence, automated road layer detection is quite crucial information for automated road construction progress monitoring [12]. Despite the importance of automated layer detection, only two research studies have attempted to detect as-built road layers automatically for automated road construction progress measurement. Vick and Brilakis (2018) and Lo et al. (2022) investigated automated road layer detection from point cloud data using a UAV photogrammetry-based 3D reconstruction approach. Both studies strongly support the road layer detection approach to measuring road progress monitoring. However, the proposed 3D construction-based approach is, unable to detect the thin road layer (i.e., asphalt), susceptible to vertical occlusions, and not applicable to segment-wise construction [13,15]. It is also noted that the feasibility of UAV is deterred due to the requirement of training, skill, safety concerns, and complex regulatory approval processes, especially in no-fly zones (control zones, areas of military operations, and restricted areas) [10,16]. Drone operation is forbidden in more than 20 nations, such as Bhutan, Algeria, Kuwait, Bahrain, Egypt, and Cuba [17,18,19,20]. The UGV has evolved as a solution to resolve the issues faced by UAVs and can be operated in a diverse construction environment with fewer safety concerns. Similarly, sensors (motion and distance) can be a better alternative than a vision-based 3D reconstruction approach to detect the features from motion patterns using deep learning techniques for road layer change detection [10].

The as-built progress measurement of road construction requires the information of as-built pavement layer(s) with corresponding spatial information with respect to time, considering the repetitive nature of the road. Hence, road layer detection is quite crucial information for as-built road construction progress measurement. During the construction, various road layers can be detected from their interrelationship with the layer change information along the road length. The scope of the proposed study is only to develop a pavement layer change classifier. Therefore, this paper proposes a novel solution to implement sensors mounted unmanned ground vehicle (UGV) to automate layer change detection by developing a layer change classifier using deep learning techniques for as-built road layer detection. In order to achieve this, the following research objectives were accomplished:

(1)

To develop the layer change classifier to detect the pavement layer change automatically using sensors mounted UGV prototype and four main Long Short-Term Memory techniques (LSTMs).

(2)

To determine the best combinations of sensors to consider as optimum feature detectors for developing the pavement layer change classifiers.

(3)

To examine the LSTM feasibility to detect the road layer change and find the best-performing LSTM network for developing a layer change classifier.

This study proposes an automated approach to detect road layer changes using sensors (accelerometer, gyroscope, ToF sensor, and GPS sensors) mounted UGV and deep learning techniques for road layer detection. Initially, the dataset was generated in a controlled environment by simulating (by physically creating the road) the road layer changes with different combinations of road layers, corresponding thicknesses, and adjoining slopes. Subsequently, four pavement layer change classifiers were developed to detect the road layer change from collected sensor data using LSTM, BiLSTM, CNN LSTM, and ConvLSTM networks. The effectiveness of sensor combinations was assessed by comparing their performance with classifiers. The optimum LSTM was also evaluated by comparing the performance of classifier models to detect road layer change.

2. Literature Review

2.1. Automated as-Built Data Collection Technologies

Researchers have widely utilized sensors such as motion sensors, distance sensors, time-lapse cameras, depth cameras, and LiDAR, mounted on mobile (i.e., UAV and equipment) and fixed platforms to collect as-built data for constructed component detection [9,10,21,22]. Rao et al. (2022) and Omar and Nehdi (2016) stated that mobile platforms are proven more favorable for mounting sensors, especially in horizontal infrastructure, because of unique benefits such as speedy data acquisition, more coverage of construction sites, and no need for the installation of many sensors along the length [10,23]. Despite having many benefits and great popularity, aerial data collected with UAVs are challenged in the construction industry by restrictions due to safety issues, licensing, and training requirements sanctioned by the government, local bodies and aviation authorities [10,16,24]. Many countries consist of complex and time-consuming approval processes for drone operations, such as New Zealand and India [25,26]. By resolving these issues, UGV has started to receive the attention of researchers due to greater flexibility [16]. Recent studies, such as Kim et al. (2019), Asadi et al. (2020), and Park et al. (2019), strongly confirmed the viability of the UGV as a platform to equip sensors for automated data collection in construction [16,27,28]. On the other hand, Ryu et al. (2019) and Kim and Cho (2020) reported that motion sensor data typically require less computational power to process and analyze compared to imaging sensors generating the image, point cloud, and video data [29,30]. Additionally, occlusion problems (i.e., line of sight) can be avoided with motion sensor data, which is a bigger issue with vision-based approaches [30]. Consequently, the motion-sensing approach is an increasingly embraced choice that bolsters the impetus to equip motion sensors for pattern detection based on progress monitoring in construction [31].

2.2. As-Built Object Detection in the CPM Domain

Conventional methods adopt manual visual inspection to assess the as-built construction for CPM, which may be time-consuming, laborious, dangerous, and inaccurate [32]. Automated object recognition has been widely researched to recognize the status of projects, activity recognition, as-built quantity estimation, and productivity analysis in the CPM domain [11,33,34,35]. Many CPM-related studies have investigated building information modelling (BIM) and point cloud-based automated as-built component detection of vertical buildings, e.g., Golparvar-Fard et al. (2011), Turkan et al. (2012), Bosche et al. (2015), and Braun and Borrmann (2019) [36,37,38]. Techniques such as the global descriptor, semi-global-matching (SGM) algorithm, and iterative closest point (ICP) algorithm have been used to detect the as-built components from the point cloud data. However, object detection from the point cloud suffers challenges due to unstructured 3D point cloud data, inconsistent point density, noisy data, cluttered scenes, and computing intensiveness. Patel et al. (2021) identified that most CPM-related studies focused on vertical structures, whereas limited studies investigated road construction progress monitoring [3]. The previous road CPM studies concentrated on earthwork progress [39], productivity [40], and resource activity monitoring [41]. However, these research studies did not concentrate on pavement construction progress measurement, and they also were not able to provide direct physical progress. Subsequently, Vick and Brilakis (2018) researched road layer detection using the UAV-enabled 3D reconstruction process and space-partitioning method [12]. The proposed approach recognizes the road layer by comparing road layer surface elevation from 3D point cloud (as-built data) and BIM 4D (as-planned data). Despite worthy contributions, the study was criticized due to the lack of ability to recognize the thin road layers and having complete reliance on BIM to detect as-built pavement layers. Lo et al. (2022), Costin et al. (2018), and Patel et al. (2021) reported that BIM adoption in road construction is not as widely common as vertical infrastructure [3,13,42]. Consequently, Lo et al. (2022) detected the base course layer of road structure using a UAV-enabled 3D reconstruction technique. Specifically, the proposed method compared the 3D point cloud (as-built) data and AutoCAD files (road geometric design with schedule data) to assess the elevation change at the location for inferring with the road layer [13]. However, it only investigated the soil material for base course layer detection, and the proposed framework was limited to detecting the asphalt layer. Moreover, this approach was also criticized because it cannot be implemented in segment-wise road construction [13,15].

2.3. Machine Learning Techniques in Pattern Detection

Traditional machine learning techniques were researched to detect and classify the patterns of feature data such as random forest (RF), support vector machine (SVM), k-nearest neighbors (k-NN), decision tree (DT), artificial neural network (ANN), and convolutional neural network (CNN) [43]. For example, various labor and equipment activities were recognized in research studies such as [22,29,44,45,46]. The literature review confirms that advanced machine learning techniques, such as deep learning techniques, are capable enough to classify motion sensor-based activity classification with high performance [47,48,49,50]. In particular, research studies such as Slaton et al. (2020), Hernandez and Akhavian (2020), Rashid and Louis (2019), and Deng et al. (2023) [31,51,52,53] have analyzed and confirmed that LSTM and its variant perform well on sequential data with their high capabilities compared to traditional machine learning [47,54]. Moreover, Zhao and Obonyo (2020) utilized convolutional LSTM models [55], and Kim and Cho (2020) used an LSTM network classifier for activity recognition of construction resources [30]. It is evident from previous studies that sensor data were assessed to recognize the activity only, but it is not explored for the as-built completed physical component for CPM.

3. Research Methodology for Layer Change Classifier

As shown in Figure 1, the main three scenarios of pavement layer change commonly observed in road pavement construction are layer not changed, layer up, and layer down. Hence, road layer detection can be carried out with the layer change information and geometric design of road construction.

In this research study, a layer change classifier model was developed for detecting the road layer changes along the as-built road using sensors mounted on a UGV and deep learning algorithms. Figure 2 shows the overall framework of the research methodology to detect the road layer change classification. To develop a pavement layer change classifier, the research methodology was proposed and described with four main sections: (1) data acquisition, (2) data preprocessing and preparations, (3) constructing supervised machine learning algorithms, and (4) classifier model performance measures for optimum layer change detection classifier.

3.1. Dataset Acquisition

Currently, there is no dataset available to train the machine learning model for developing a road layer change classifier from sensor data. To develop the road layer change classifier, a dataset was generated to train and develop the road layer classifier. The dataset generation was carried out in three steps: (1) configuration of the data collection apparatus, (2) experiment setup, and (3) data collection.

3.1.1. Configuration of Data Collection Apparatus (Sensor Prototype Mounted on UGV)

Motion sensors, e.g., the accelerometer and gyroscope, are also widely equipped to collect data on the kinematic (motion) pattern for activity detection, resource tracking, and construction status [10,21]. Hence, to automate the data collection (layer change) of as-built pavement, sensors were configured on a UGV. During the data collection, sensors mounted UGV collected the motion sensor data and the vertical distance of ToF sensors to the ground to detect the road layer change. Figure 3 illustrates the prototype of the sensors mounted on the unmanned ground vehicle for data acquisition to develop a layer change classifier. In the sensor prototype, four sensors were embedded in the ESP32 circuit: the ToF sensor (VL53L3CX), GPS sensor (NEO-6M), accelerometer, and gyroscope (the accelerometer and gyroscope were combined into MPU6050.). A Wi-Fi module was enabled to transmit and download the data in real time. It should be noted that all sensors data were synchronized.

• Embedded sensors
  • ToF sensor: Measures the vertical distance from the ToF sensor to the surface of the pavement layer
  • Accelerometer: Measures the rate of velocity change of UGV around three axes (x,y,z)
  • Gyroscope: Measures angular velocity of UGV around three axes (x,y,z)
  • GPS sensor: Measures the geolocation of UGV with longitude and latitude

3.1.2. Experimental Setup

During road pavement construction, three main events were observed for layer change from typical road construction: layer up, layer down, and layer not changed. Road layers can be detected using logical inter-relationships between road pavement layers based on these three events. A large comprehensive dataset is required to train using a deep learning algorithm collected in various pavement construction scenarios consisting of variables to predict the road layer change. With respect to data generation for three main events, typical road construction consists of three main variables (v): road layers (l), corresponding thicknesses (t), and slopes (m), where v ∈ {road layers, thickness, slopes), l∈ {layer₁, layer₂, layer₃, …, layer_e), t ∈ {thickness₁,thickness₂,thickness₃, …, thickness_g}, m ∈ {slope₁,slope₂,slope₃, …, slope_f}. Road layers and their corresponding thicknesses are designed as per the local pavement structure design standards. However, the slope at the edge adjoining two layers is arbitrary and depends on construction practice. Therefore, various slopes and thicknesses were considered to generate the road construction scenarios for dataset generation.

Figure 4 represents the environment of dataset generation, consisting of different road construction scenarios where different road layers with thicknesses were constructed with adjoining slopes to generate the dataset of three events (layer up, layer not changed, and layer down). Three different types of road layer change events were mainly targeted by data generation.

3.1.3. Data Collection

Consequently, sensors equipped UGV were run over the various road construction scenarios constructed by different combinations of slopes and road layers (with different thicknesses), registering various sensor data of road layer change events (ToF sensor, accelerometer, gyroscope, and GPS) during its operation. After collecting the data for each scenario, the collected data were transferred through a wireless connection to the workstation. Here, sensor data were converted into CSV format using the standard uniform protocol. Subsequently, data labeling was carried out as layer not changed (same layer), layer down, and layer up according to registered changes with a stopwatch after each scenario.

3.2. Data Preprocessing and Preparations

After the data collection, data preparation was carried out. The collected sensor data were stored in an ESP32 chip embedded in the sensor prototype, and they were transmitted to the laptop for data processing. Subsequently, data labeling was carried out to implement the supervised machine learning algorithm to recognize the pattern of feature classes. Data preparation was carried out in three steps, as described below.

3.2.1. Sensors Data Combination: Input Vector

The collected dataset featured various sensor data (ToF sensors, accelerometer, and gyroscope) consisting of the sequential data pattern for layer change events. Each sensor combination developed a unique dataset that impacted the investigation of layer change detection differently. The investigation process included the implementation of four deep learning algorithms (LSTMs) with sensor combinations to examine the performance of layer change detection for assessing the effectiveness of various sensor combinations.

As illustrated in Figure 5, the combination of each sensor created a feature set that was comprised of seven values: acceleration (three values), angular velocity (three values), and distance (one value). Features sets from various sensor combinations were concatenated to create the input vector. The input vectors were defined individually on the basis of sensors utilized in each sensor combination. There were three input vector formations based on sensor combinations: (1) accelerometer (a_x, a_y, a_z) + gyroscope (g_x, g_y, g_z); (2) accelerometer (a_x, a_y, a_z) + gyroscope (g_x, g_y, g_z) + distance; (3) 1 axis of the accelerometer (a_z) + 1 axis of the gyroscope (g_x) + distance. It should be noted that all sensor data were concatenated by aligning independent data with timestamps. To evaluate the machine learning classifiers, training and testing data were taken randomly from the generated input vectors, as shown in Figure 5. The sensor combinations were represented in the following manner.

accelerometer (a_x, a_y, a_z) + gyroscope (g_x, g_y, g_z) = a_{xyz +} g_xyz.

(1)

accelerometer (a_x, a_y, a_z) + gyroscope (g_x, g_y, g_z) + distance= a_xyz + g_xyz + D.

(2)

1 axis of accelerometer (a_z) + 1 axis of gyroscope (g_x) + distance= a_z + g_x+ D.

(3)

3.2.2. Data Normalization

In collected sensor data, the distance data had a different scale than the data of acceleration and angular velocity due to a different data range of ToF sensors. Different ranges of the dataset may create the model weight error [38]. Therefore, min–max scaling as a data normalization technique was applied to rescale the data to obtain the same range of data between −1 to 1. Min–max scaling subtracts the minimum value from the highest value of each sensor’s data and then divides it by the data range.

3.2.3. Data Augmentation

The generated dataset may be imbalanced and can create a bias toward a more dominating class with numbers. Therefore, the random oversampling technique was implemented to generate large synthetic data for less populated data classes to balance and augment the dataset [56,57,58]. This process may improve the model generalization by adding data variability without altering the class labels [39]. It may potentially enhance the generalization of the proposed model by introducing data variability without changing the labels [58].

3.2.4. Data Segmentation with Window Sliding Techniques

The data segmentation technique restructures data to form a supervised learning problem for the sequence of time-series dataset numbers. This study implements a sliding window technique to convert segment data from discrete data. Basically, for the considered time-series data s_i = (s_i,1, s_i,₂, …, s_i,l), this techniques segments s_i into smaller time segments < s_i,x−d, s_i,t−x+1, …, s_i,x, …, s_i,x+d−1, s_i,x+d > in x size of the window as the window slides by s (here, d = (x − 1)/2) [31]. The window size may greatly impact the final sequential classification result. Therefore, various window sizes were experimented with to select the size.

3.3. Constructing Supervised Machine Learning Algorithms

Machine learning techniques are broadly researched to classify the pattern of events on the basis of sequential data collected by high-performing kinemetric sensors [47,48]. Sensor-collected data here consisted of the sequential pattern based on features that could classify the main three events (layer not changed, layer up, and layer down) with machine learning algorithms. To categorize various layer change events, supervised deep learning classifiers were utilized here to understand specific patterns. Traditional machine learning algorithms interpret single data into classes, such as SVM classifier, random forest, artificial neural network, k-nearest neighbor, and decision tree. However, the combination of sequential data forms a class with a unique data pattern. Hence, a previous study also confirmed the superiority of recurrent neural networks (such as the LSTM model) compared to the traditional machine learning model [30,59,60,61,62]. Specifically, LSTM and its variant are broadly popular due to their high performance in data mining of sequential sensor data [31,47,63,64]. The performance of these classifiers may vary according to the data and feature types [31,65,66]. Consequently, LSTM, BiLSTM, CNN-LSTM, and ConvLSTM models were implemented and compared to classify the road layer change for this study, as discussed below.

3.3.1. Long Short-Term Memory Network

The LSTM is a widely known technique for analyzing sequential data (time-series data) in predicting trends and classifying features with great performance [67]. In the architecture of the LSTM model classifier, the two-staked LSTM was constructed consisting of two connected LSTM memory cells to deepen the network. The sensor-collected data were sent to a fully connected layer (FCN), which was succeeded by the rectified linear unit (ReLU) layer. Subsequently, two layers of LSTM cells were provided. Output from this process was transmitted to a fully connected layer because the classifier model detects the layer changes at the end of the layer change sequences. Lastly, the generated output from the fully connected layer was forwarded to the Softmax layer to transform the prediction value of the feature class into probability for generating the final detection of the layer change class with the highest probability.

3.3.2. BiLSTM

The Bidirectional Long Term-Short-Term memory (BiLSTM), a sequence processing technique, comprises two LSTMs [66]. Unlike LSTM, a network of BiLSTM consists of two parallel layers in forward and backward propagation directions [68]. The forward LSTM layer obtains preceding features, whereas the backward LSTM layer extracts succeeding features. BiLSTM captures past and forthcoming information to enhance the available context of the proposed algorithm using two individual hidden layers, the forward layer and the backward layer [69]. Subsequently, both hidden states were merged into the final output. The BiLSTM network consisted of the same architecture as the LSTM network; however, the BiLSTM layers (two) existed instead of both LSTM layers in the model network architecture.

3.3.3. CNN-LSTM

In order to achieve the advantages of CNN and LSTM, both networks were combined as a CNN-LSTM model. CNN-LSTM, a hybrid of CNN and LSTM networks, uses CNN layers to extract the feature on input data joined with LSTMs to carry out sequence prediction [63]. This model network contains two phases; phase 1 consists of convolutional layers with pooling layers, while phase 2 includes the LSTM layer. Unlike the CNN model, the sequence of deep features extracted using pooling layers is passed directly to the LSTM layer [70]. The global and local information of the feature data is encoded through convolutional layers, and this encoded information is decoded by LSTM layers [66].

3.3.4. ConvLSTM

ConvLSTM is the extension of LSTM with the dense operation altered by convolutional layers [71]. Unlike the LSTM model, which interprets the data directly to calculate state transition and internal state, and unlike CNN-LSTM, which reads the output generated from CNN, ConvLSTM utilizes convolutional operations directly as part of interpreting into the units of LSTM themselves [72]. Similar to fully connected LSTM, ConvLSTM is also used as a building block to manage complex sequence operations [66]. The structure of ConvLSTM, where layers encode the input-refined sequence consisting of the defined size that is propagated forward into LSTM [72]. ConvLSTM applies convolutions for the input parameters to hidden connections. In ConvLSTM, convolutional operations are used to replace matrix multiplications of RNN, and these layers determine which information needs to be stored or to be forgotten from the preceding cell state through the forget gate [66]. The ConvLSTM classifier consists of a similar network architecture to the LSTM classifier model; only ConvLSTM layers (two layers) exist instead of both LSTM layers.

3.4. Classifier Model Performance Measures for Optimum Layer Change Detection Classifier

The performance measure of classification relies on the nature of data characteristics and feature detection abilities of the algorithm. The deep learning model may not have the same consistent output for the different datasets. Therefore, it is important to select the most optimum road layer classifier from the performance evaluation of all layer change classifiers of various sensor combinations. To evaluate the performance of implemented deep learning models (LSTMs) [47], a confusion matrix and model learning curves with four main performance measures are analyzed, i.e., overall accuracy, precision, recall, and F1 score [73]. Outwardly, any implemented deep learning method is evaluated by the most general measure as classification accuracy. However, overall classification accuracy is insufficient to represent the complete picture of the robustness and reliability of classifiers, especially for sequential data [46]. Consequently, other performance matrices, namely, precision, recall, and F1 score of the implemented classifiers, were examined in this study. In Equations (4) to (6), true positive (TP) is the total number of correct positive predictions generated by a positive model, and true negative (TN) is the total number of correct negative predictions generated by a negative model, whereas false positive (FP) is the number of feature classes predicted incorrectly as positive for the negative model, and false negative (FN) is the number of feature classes predicted incorrectly as negative for the positive model. All TP, TF, FP, and FN values for multiclass classification were generated through a confusion matrix. Moreover, the precision value suggests how often the prediction of a specific class is right, whereas recall indicates the rightly predicted rate of a specific class. The F1 score evaluates the performance of the model by avoiding the biased influence of the dominant class in the case of imbalanced classes [46]. The precision, recall, and F1 can be calculated using Equation (4), Equation (5), and Equation (6), respectively [73].

$$\mathrm{Precision}=100\times \frac{\mathrm{True} \mathrm{positive} \left(\mathrm{TP}\right)}{\left(\mathrm{True} \mathrm{positive} \left(\mathrm{TP}\right)+ \mathrm{False} \mathrm{positive} \left(\mathrm{FP}\right)\right)}$$

(4)

$$\mathrm{Recall}=100\times \frac{\mathrm{True} \mathrm{positive} \left(\mathrm{TP}\right)}{\left(\mathrm{True} \mathrm{positive} \left(\mathrm{TP}\right)+ \mathrm{False} \mathrm{negative} \left(\mathrm{TN}\right)\right)}$$

(5)

$$\mathrm{F}1 \mathrm{score}=2\times \frac{\left(\mathrm{precision}\times \mathrm{Recall}\right) }{\left(\mathrm{precision}+ \mathrm{Recall}\right)}$$

(6)

4. Experiment

4.1. Controlled Environment Setup for Data Generation

A large dataset is required to develop and train the machine learning classifier model. However, data collection in real construction is expensive in terms of time and cost. This study developed a controlled environment to create the three main events with various road construction scenarios, as shown in Figure 6. For layer change detection, road layers, connecting slopes, and thicknesses of road layers can act as important variables. Initially, four main road design layers were chosen with the most local common practice in Canterbury, New Zealand: (1) surface layer, (2) base course (AP 40), (3) subbase (AP 65), and (4) subgrade. Three trials with various scenario generations were constructed to replicate the real road construction practice: trial 1 (surface layer as upper layer and base course as bottom layer), trial 2 (base course as upper layer and sub-base course as bottom layer), and trial 3 (sub-base as upper layer and subgrade as bottom layer), as seen in Figure 6. Different combinations of road layers with different thicknesses and slopes were designed to generate various scenarios. In particular, different thicknesses of the upper pavement layer and slopes of the edge connecting two layers were used to develop various sub-scenarios. Subsequently, the main three-layer change events ((1) layer up, (2) layer down and (3) no layer change) were created with each created sub-scenario. In this experiment, 54 sub-scenarios were created to represent the main three road layer change events by altering the road layer, thicknesses, and adjoining slopes of two layers.

Table 1. Considered variable values for data collection.

Table	Thickness (mm)	Slopes
Trail 1	Layer thickness (ts) = 40, 50, and 70	Slope ms = high, medium, and low
Trail 2	layer thickness (tb) = 150,175, and 200	Slope mb = high, medium, and low
Trail 3	layer thickness (tc) = 150, 175, and 200	Slope mc = high, medium, and low

shows the various considered values of variables, i.e., thickness and slope for each trail.

4.2. Data Collection in a Controlled Environment

In a controlled environment, sensors mounted on a UGV were run for all 54 scenarios to collect the dataset. Mainly, it consists of the three events of layer change (i.e., layer not changed, layer up and layer down), as shown in Figure 7. Then, these data were downloaded and transferred to the laptop after each sub-scenario. The subsequent manual labeling process was carried out according to the timestamp. In the collected data, the data points of each event were different, as sensors acquired data for each feature for the variant time. This imbalanced dataset of layer change features classes represents the real case. Random sampling was implemented to balance the data by augmenting the data of two feature classes (layer down and layer up), to avoid the biased classification result toward the dominating class. Consequently, the final number of data points of each feature class registered was 48,000.

4.3. Implementation of Proposed Models

The deep learning model-based classifiers were developed using LSTM networks to detect the road layer change and classification. For that, acquired data were divided into training and testing datasets as 80% and 20% of the complete generated dataset in a controlled environment. Then, LSTM models were trained using a training dataset and tested on testing datasets to measure the deep learning performance. For that, Keras was used to build LSTMs classifiers, which provide TensorFlow to construct neural networks. Jupyter notebook was used to execute the LSTM model building process with Python 3.0. NVIDIA CUDA was utilized as a computing platform and API to allow software to use graphic processing units for general-purpose computing on GPU. The selected hyperparameters were selected during the implementation of models, as shown in

Table 2. Hyperparameter information.

Hyperparameter	Selected Parameters
Stochastic gradient	Adam
Regularization	L1, L2, dropout
Batch size	64
Epochs	100
Learning rate	0.001

. The window size was kept to 0.833 s after experimenting with various window sizes such as 0.66, 1.0, 1.2, and 1.4 s. The stochastic gradient as an optimizer minimizes the losses by helping the attributes to change in the model network, such as weights and learning rate. The regularization function supports the model robustness and minimizes the model complexity by adding the penalty to the loss function to eliminate the overfitting issues. Epochs define the total completed number of passes through the complete training dataset, which is delivered forward and backward through the neural network, whereas batch size refers to the sample number transmitted before the updating process of internal model parameters [37].

5. Result

5.1. Performance Result Comparison Models on Various Sensors Data

To examine the performance of proposed LSTMs models for various sensors combinations and determine the best sensor combination data, LSTMs models were constructed utilizing data of different sensor combinations, namely, (a_xyz + g_xyz), (a_xyz + g_xyz + D) and (a_z + g_x + D). The performance measures were generated through the implementation of LSTMs on sensors combination. Figure 8 illustrates the comparison of overall accuracy, precision (P), recall (R), and F1 score for sensor combinations. The comparative result shows that the (a_z + g_x + D) model obtained the highest accuracy as 97.88%, followed by (a_xyz + g_xyz + D) through the ConvLSTM model. Furthermore, the LSTM implementation results of average precision, recall, and F1 score with the considered sensor combinations suggest that the combination of (a_z + g_x + D) outperformed other sensor combinations in all model results. The model of acceleration + gyroscope performed poorly compared to (a_xyz + g_xyz + D) and (a_z + g_x + D) for all proposed algorithms.

5.2. Classifier Results for (a_z + g_x + D) Sensor Combination Data

According to the result of Section 5.1, it can be concluded that a_z + g_x + D was the best sensor combination to classify road layer change events. Consequently, this section analyzes and compares the classification of road layer change classes results generated from using four implemented LSTM models for a_z + g_x + D sensor data. This section consists of the precision, recall, and F1 score of each feature class and the overall accuracy, confusion matrix (CF), and accuracy–loss curve of each implemented model.

(1)

Figure 9 represents the confusion matrices of the proposed different LSTM models for a_z + g_x + D sensor data. In a confusion matrix, the diagonal cells represent correctly identified events, and off-diagonal classes show misclassified events of specific layer classes. In the resultant confusion matrix of the proposed algorithms, X- and Y-axes depict the predicted and true classes. As shown in Figure 9, the proposed ConvLSTM outperformed the other LSTM networks with the fewest identification errors.

(2)

Model learning curves were plotted to examine the performance of models and obtain the bias–variance tradeoff [37]. Model learning curves illustrate the correlation between accuracy progress with epochs. Figure 10 and Figure 11 shows the recorded accuracy, loss for training, and validation datasets over the iterations for implemented LSTM models. These curves depict the tuning process of parameters with adjustment of the value to examine the accuracy and optimization loss of the considered validation dataset and training dataset. As observed from Figure 10 and Figure 11, the highest test accuracy and least test loss were obtained as 97.88% and 8.33%, respectively, for ConvLSTM. On the contrary, CNN-LSTM registered the lowest accuracy at 95.36% and the highest loss at 22.86%.

(3)

Table 3. Precision, Recall and F1 score.

Class	LSTM			BiLSTM			CNN-LSTM			ConvLSTM
	P	R	F1	P	R	F1	P	R	F1	P	R	F1
Layer up	96	92	94	96	94	95	88	97	92	97	96	97
Layer down	93	97	95	95	96	96	97	90	93	96	99	98
Layer not changed	100	100	100	100	100	100	100	99	100	100	98	99

represents the precision, recall, and F1 score for each class of proposed algorithms. It can be observed from Table 1 that layer not changed received the highest precision at 100% with all LSTMs, whereas layer up attained the lowest precision at 88% with CNN-LSTM. Moreover, the highest recall value was observed for layer not changed at 100% with LSTM and BiLSTM, and the lowest recall value was registered for layer down at 90% with CNN-LSTM. Similarly, the highest F1 score was generated for layer not changed at 100 with LSTM, BiLSTM, and CNN-LSTM, and the lowest F1 score was evaluated for layer up at 92 with CNN-LSTM.

6. Discussion

This study proposed a methodology for automated road layer change detection using LSTM algorithms from collected data through a sensor-equipped UGV. This research attempted to examine the capabilities of the sensors mounted UGV as an automated data collection technique and LSTM algorithms as automated data processing and analyzing techniques for automating the road layer change detection, which can lead toward automated road construction progress during the construction stage.

The common challenges of deep learning algorithms are the requirement of a large set of data and an imbalanced dataset, and these challenges were resolved by the data oversampling technique ‘random sampling’. This technique helped to achieve high training accuracy and robustness of the models. The results of the proposed study proved 0.833 s as the optimum window size according to the results of various experiments. First, this study examined the performance of three sensors combinations of (a_{xyz +} g_xyz)), (a_xyz + g_xyz +D) sensors, and (a_z + g_x + D) to detect road layer change using LSTM, BiLSTM, CNN-LSTM, and ConvLSTM. The result of the study confirmed the (a_z + g_x + D) sensor data as the best vector inputs and the (a_xyz + g_xyz + D) combination as a satisfactory result, whereas the combination of accelerometer with gyroscope sensors underperformed following implementation with different LSTMs to achieve accuracy for the real-time road layer change classifications. This outcome implies that (a_z + g_x + D) can efficiently classify road layer changes from collected data through the developed sensor prototype, which may potentially be used to measure as-built road progress. Further results were generated to examine the ability of LSTM implementation on (a_z + g_x + D) sensor data to detect the layer changes. The consequent generated results suggest that ConvLSTM achieved the best accuracy of 97.88%, lowest loss of 8.33%, and minimization of overfitting issues for layer change classification compared to other implemented models. ConvLSTM showed great ability to predict spatiotemporal features of road layer change data using convolution structures which consist of state-to-state and input-to-state transitions. The presented convolutional in ConvLSTM aids in determining the future state of the certain grid according to the past states and inputs of the local neighbors. ConvLSTM is able to capture motions of UGV with transitional capture. Moreover, BiLSTM and LSTM also achieved good accuracy with a satisfactory fitting trend of learning curves, but CNN-LSTM showed an overfitting trend of learning curves. As observed from the learning curves of all proposed models, the values of validation loss and training loss converged for ConvLSTM, which indicates good model fitting on training and validation data. However, model learning curves illustrate a decrease in validation loss after 50 epochs for CNN-LSTM and BiLSTM. Overall, the implemented models obtained the desired accuracy. Nevertheless, the learning curves of the proposed models illustrated in Figure 10 depict that ConvLSTM predicted smoother classes with respect to ground-truth data, confirming it to be the superior model. In addition, the resultant high values of precision, recall, and F1 score of implemented models on sensor-collected data (a_z + g_x + D) confirm the viability of the approach for real-time road layer change detection for as-built road progress measurement. The best performance for classifying the road layer change was detected for layers not changed with an accuracy of 100% with all LSTMs. In contrast, layer up achieved the lowest accuracy at 88% with CNN-LSTM. For layer up, ConvLSTM registered high values of precision (97%), recall (99%), and F1 (97%), whereas LSTM and BiLSTM achieved the highest precision, recall, and F1 score for layer not changed at 100%, 100%, and 100, respectively. It is also noted that CNN-LSTM received the highest precision for layer down as (97%), and ConvLSTM received the highest recall and F1 as 99% and 98, respectively. The obtained highest performance measures of layer not changed suggest the lowest errors when recognizing this class using the proposed approach.

Despite the promising performance of the models, the effect of the various hyperparameters was still obvious due to result variations according to various combinations of the parameters of algorithms such as classifier type, window size, neuron number, type of optimizer, regularizer type, and corresponding value. Finalizing the best parameter combinations of classifiers is a difficult task due to some reasons, such as variations in objectives behind road layer change detection on the same level implications, experimental protocol, type of attached sensor, and the nature of slope and thickness of the experiments. The use of sensor-armed unmanned ground vehicles showed great potential to recognize/detect road layers from collected data using distance, accelerometer, and gyroscope sensors in the control environment in this research. The result of performance measures depicts the improved robustness of the algorithm to fulfil the aim of automation in road progress monitoring. Road layer detection can be potentially carried out with layer change information and geometric design of road construction. The proposed approach will be potentially extended to the as-built measurement of pavement construction using the road layer change and spatial information (location of starting, ending, and change point) to calculate the as-built length of constructed road layers from their starting and finishing points.

7. Conclusions

An automated and real-time road layer detection framework for road construction enables a fundamental platform to assess and monitor pavement construction progress. This study explored the adoption of the low-cost sensor-equipped UGV to develop a road layer change classifier to detect the as-built road detection. In this study, the ToF ranging sensor, accelerometer, and gyroscope sensors were mounted on the UGV to record the sensor pattern data change for road layer change created in various scenarios created by changing the layers, thickness, and slope of layers. From this study, the random oversampling technique proved successful in data augmentation and data balancing to enhance the robustness of models. Subsequently, various deep learning algorithms, such as LSTM, BiLSTM, ConvLSTM, and CNN-LSTM, were implemented and compared for various combinations of sensors such as (a_z + g_x + D), (a_xyz + g_xyz + D), and (a_xyz + g_xyz) to classify road layer change for automating the road layer changes detection process. The result proved a_z + g_x + D as the best sensor combination with high-performance measures, followed by (a_xyz + g_xyz + D). However, algorithms performed poorly on the combination of (a_xyz + g_xyz), depicting the insufficiency of this sensor data combination to classify road layer change. Apparently, ConvLSTM stood out as the best-performing algorithm for (a_z + g_x + D), with the highest overall accuracy at 97.88% and the lowest loss at 8.33%. Layer not changed obtained the highest value, whereas layer up and down achieved decent values of performance measures. The result of the study confirms the feasibility of the proposed classifier model with high-performance measures to predict layer change classes. This novel attempt confirms the potential successful application of the proposed sensor-equipped unmanned ground vehicle approach in road layer detection. This approach enables real-time automated monitoring solutions, which will relieve road construction companies from tedious and time-consuming manual methods.

8. Limitations and Future Study

This research is a stepping stone to automating pavement construction monitoring. However, it also had several limitations. The authors only considered 54 construction scenarios to develop a layer change classifier; however, more potential combinations of considered variables should be investigated with real cases. In this study, UGV-related variables were not considered, such as wheel size and various speeds of UGV. Further investigations can address this issue. This study was limited to developing a road layer classifier for the automated detection of road layer changes. However, this work should be extended to develop the complete approach of automated progress monitoring of road construction using sensors mounted on a UGV. Furthermore, automated navigation will be featured with UGV for planning data collection according to the schedule of activity and operation of machinery considering spatial conflicts.