Recently, the monitoring of road surface conditions has become considerably important. Well-maintained road surfaces increase road user safety and comfort levels. Therefore, it is essential to monitor road conditions continuously to enhance the transportation system in terms of driving safety and comfort. For instance, in Canada, authorities responsible for road surface maintenance have to deal with complaints concerning the poor surface conditions of roadways, particularly during the winter months. One of the main indicators used to determine road surface conditions is the density of road surface anomalies [1
]. Municipalities typically rely on statistical data derived from collected road surface information, visual field inspections, or vehicles outfitted with special instruments which measure and monitor road surface conditions. For example, ARAN (Automated Road Analyzer), which is widely used for road monitoring in Canada, and ROMDAS (Road Measurement Data Acquisition System), which is used to monitor road surfaces in New Zealand, both use lasers combined with ultrasonic and video sensors for high-level road quality assessment [1
]. However, these methods are labor-intensive, costly, and often suffer from insufficient data coverage to generate a complete picture of road surface conditions in large cities, such as Toronto, Canada. For the above reasons, road maintenance authorities are looking for a low-cost, higher-efficiency detection method. They also desire a centralized information system able to monitor the road status in real time. Traditionally, there have been three main approaches for road surface monitoring: 3D reconstruction, vibration, and vision-based [2
A 3D reconstruction approach relies on 3D laser scanning to create accurate surface models. These models are then compared to a base model to detect road surface anomalies. In this approach, a 3D laser scanner uses reflected laser pulses, which create accurate 3D digital models of existing objects, such as road surface anomalies. Subsequently, the distress features (road surface anomalies) are extracted from the created point clouds (i.e., a collection of points that represent a 3D shape of road surface distress). This approach was widely investigated by Kelvin [3
], Vijay and Arya [4
], Salari et al. [5
], Moazzam et al. [6
], Hou et al. [7
], Kim and Ryu [8
], Wang et al. [9
], and Yan et al. [10
]. However, the aforementioned approaches require high-cost laser scanners [8
] and are very costly when monitoring large-scale road networks.
A vision-based approach relies on image processing analysis, such as texture extraction and comparison using captured photographs depicting pavement distress features. The principle of this approach primarily utilizes geotagged images captured by a camera/video system mounted facing downward toward the road surface on a moving vehicle. Any suspicious road surface distress features, including potholes and cracks, can be automatically detected from the collected geotagged video images [11
] by applying, for example, a Canny edge detection process [12
]. Vision-based approaches were extensively evaluated by Koch et al. [13
], Jog et al. [14
], Huidrom et al. [15
], Lokeshwor et al. [16
], and Yan and Yuan [11
]. Even though these approaches are cost-effective compared to 3D reconstruction approaches, they depend on certain environmental conditions, such as lighting and shadow influence, etc.
With a vibration-based approach, road surface anomalies are detected from the rate of moving vehicles’ vibrations captured by motion sensors (e.g., accelerometers or gyroscopes). Theoretically, a vehicle, when passing over any road surface anomaly, such as a pothole, crack, manhole, or expansion joint, will vibrate more than when passing over smooth road surfaces.
Over the last few years, smartphone-based sensing has become an important supplemental technique for detecting road surface anomalies in order to monitor the surface of the road segments or the surface of bicycle and pedestrian lanes [17
]. Participatory sensing is anticipated to be an emerging area in which smartphone-based measurements seem to be particularly attractive, as they are not only widespread but also equipped with several sensing capabilities [18
]. Measuring and analyzing motion sensors’ signal data from various types of moving smartphones may diverge depending on many factors, including the sensor’s characteristics and quality, smartphone device’s position, vehicle’s suspension system, and speed [19
This paper aims to provide a comprehensive review of the existing studies that investigate how smartphone sensors may help detect road surface anomalies by synthesizing the problems, major issues, and challenges in current development, as well as identifying the existing research voids for further research studies. Therefore, the inclusion criteria for the articles in this paper were the evaluation of existing approaches using smartphone sensor data for road surface anomaly detection. In addition, articles that have no focus on the application of smartphone sensors or those using handheld computer devices other than smartphones, such as PDAs (personal digital assistants), or those using low-end accelerometer kits were excluded from this review. It is important to note that there are other studies that also address the application of monitoring road and traffic conditions using smartphones. However, only the parts of these studies that relate to the scope of this paper are considered and reviewed.
This paper is organized as follows. Section 2
compares related work according to the overall workflow of detecting road surface anomalies from smartphone sensors, followed by a review of parameters affecting the performance of the detection and integration approaches. Section 3
presents a brief discussion of the reviewed articles and their limitations as a comprehensive approach. Conclusions and the challenges of detecting road surface anomalies from smartphone sensors, as well as potential future research areas, are given in Section 4
2. Related Work
Measuring the signals from smartphone sensors is challenging due to dissimilar sensor properties between various models of smartphones, as well as differences in vehicle size, weight, length, and suspension systems. There is also a difference in a vehicle’s vertical motion induced by road surface anomalies—the length, depth, and shape of the potholes on the road surface and the road curvature’s effect on the vertical movement rate of different vehicles. In fact, different vehicles passing over a specific pothole would not generate an identical signal pattern [20
]. In addition, vehicle velocities affect the vertical movement rate, which then leads to different patterns of the vibration response to any road surface anomaly.
Road surface anomaly detection approaches consist mainly of five steps: (1) sensing (data collection), (2) preprocessing, (3) processing for feature extraction, (4) post-processing, and (5) performance evaluations. Figure 1
illustrates the overall process for road surface anomaly detection using smartphone sensors.
There are different kinds of sensor data that can be obtained from smartphone sensors. Motion sensor types include: accelerometer, gyroscope, linear accelerometer, and rotation. Position sensor types include: GPS, manometer, and rotation. The next step is the preprocessing of sensor data, with the aim of filtering noises that contaminated the sensor data. The other goal of preprocessing sensor data is to reorient them from the device coordinate system to another geographic coordinate system. The preprocessed sensor data are then analyzed to discover and extract desired information based on predefined rules (feature extraction). After that, the processed sensor data should be transferred to the central server for data post-processing, including integration with other data from different sources (concept of crowdsourcing). Finally, the performance of the proposed process should be evaluated to determine its functionality and reliability.
Sensor Data Collection
The smartphone sensor framework has open access to many types of built-in sensors. Some of these sensors are hardware-based (physical) and some are software-based (virtual). Hardware-based sensors are the physical, built-in sensors, such as accelerometers, gyroscopes, magnetometers, light, temperature, etc. Physical sensors measure motion, orientation, and environmental conditions, such as acceleration force, physical position of device, illumination, etc. In contrast, software-based sensors use data from one or more of the hardware-based sensors and virtually calculate real-time values based on the desired outcome, such as linear acceleration, rotation, gravity, etc. In general, smartphone sensors can be categorized into three different types: motion sensors, position sensors, and environmental sensors.
Motion sensors are suitable for monitoring a device’s movement and vibration, tilt, shake, rotation, or swing. The movements can directly reflect user interaction, as typically happens in game applications (i.e., a user steering a car or a controlling a ball in a game). However, they can also reflect where the device is sitting (i.e., moving with the occupant while they drive their car). With direct user interaction, device movement is monitored relative to the device’s coordinate system or a defined local application frame. With physical environmental conditions, the device movement is monitored relative to the local-level coordinate system (Figure 2
Position sensors are suitable for specifying a device’s physical position in the local-level coordinate system. In fact, the geomagnetic field sensor, in combination with accelerometer sensor data, can determine a device’s position relative to the local-level coordinate system. Environmental sensors measure the environmental properties of the surrounding area, such as temperature, humidity, ambient pressure, and illuminance. This type of sensor has a very limited contribution to road surface anomaly detection, since there does not seem to be any direct relationship/impact between these factors and the formation of road surface anomalies.
For each smartphone-based application, various combinations of sensors (physical or virtual) may be used depending on the desired application criteria. To develop an application for road surface anomaly detection, motion sensor data can be tracked to detect any possible shake or tilt caused by road surface anomalies in a moving vehicle. Previous studies investigating road surface anomaly detection using smartphone sensors have widely employed motions sensors (accelerometers and gyroscopes). Accelerometer sensors measure acceleration force, including gravity force, applied to a device on all three physical axes (m/s2
). Gyroscope sensors measure a device’s rate of rotation around each of the three physical axes (rad/s). Previous studies have frequently used accelerometer sensor data to detect anomalies because road surface anomalies have more influence on and are mainly detectable from the acceleration force applied to the vehicles, rather than the rotation rate caused by vehicles’ vibration. Only a few studies, including Yagi [21
] and Douangphachanh and Oneyama [22
], have investigated gyroscope sensor data, particularly frequency domain combined with accelerometer sensor data, which increases the accuracy of detection (as complementary sensor data).
To determine the current location information of smartphone or mobile devices, including latitude, longitude, bearing of moving direction, and velocity of movement, the location API (application program interface) provides the best available location information of the devices based on the currently available location providers, such as GPS (Global Positioning System) and/or Wi-Fi (Global Positioning System). In addition, the API provides the accuracy for each provided location information data.
summarizes and compares a list of sensors commonly found in smartphones for the application of road surface anomalies [23
]. These sensors are used either directly or indirectly for road surface anomaly detection.
The data preprocessing step involves transforming raw data derived from smartphone sensors into a clean and organized data set prior to analysis [24
]. One of the goals of preprocessing is to smooth the raw sensor data and filter the noise. There are three major types of smoothing and filtering approaches: moving-average filtering, low-/high-pass filtering, and band-pass filtering. Moving-average filtering is the most common filter in digital signal processing, mainly because it is the easiest to understand and implement since there is no need to have any prior information regarding the sensor data [25
]. Despite its simplicity, this kind of filter is ideal for some common tasks, such as reducing random noise while retaining major information content. Low-/high-pass filters remove some undesired parts of signals based on a predetermined cut of frequencies. A band-pass filter passes portions of the signals within a certain range of frequencies and removes the other parts of signals that are outside of that range.
Preprocessing may also aim to reorient sensor data values from a device coordinate system (Figure 2
c) to the local-level coordinate system (Figure 2
a). An example would be a body-frame (i.e., moving platform) coordinate system (Figure 2
]. This process can be accomplished by completing a rotation (using Euler angles) around each of the three axes. The reorientation process reduces issues related to smartphone placement.
The processing step analyzes the preprocessed sensor values to detect road surface anomalies. The signal pattern is tracked to detect any abnormal changes in sensor values using three main approaches:
The threshold-based approach uses simple predefined threshold values based on experiments to detect road surface anomalies from sensor data.
The machine learning approach uses more advanced techniques to detect road surface anomalies. Studies Bhoraskar et al. [27
] have investigated unsupervised approaches, such as k-means clustering, in which a predetermined number of clusters are identified, and data are classified to the same number of clusters. Some other studies, including Perttunen et al. [28
] and Jain et al. [29
], have explored supervised approaches, such as support vector machine (SVM) clustering. In the case of supervised approaches, some training data sets should be collected to train the algorithm. The test data are then classified based on the trained data set.
Another approach that was recently investigated is dynamic time warping (DTW). This approach is predominantly employed in speech recognition studies. DTW compares incoming signal data with predefined templates and measures the similarity between the data sets.
Regarding smartphone applications, the preprocessing and processing steps can be accomplished using two different modes: online and offline. In an offline mode, sensor data are collected, and the preprocessing and processing steps are applied locally on computers. In fact, any application developed for sensor data collection is able to collect and store sensor data while a car is passing over potholes or bumps. Next, the sensor data are extracted for further processing. In an online mode, data collection, preprocessing, and processing are performed simultaneously as the car is passing over potholes or bumps. Ideally, a specific smartphone application should be designed and developed to collect, preprocess, and process the sensor data in an online mode.
Post-processing includes crowdsourcing and integrating data from multiple sources (users’ collaborations, geographic datasets) to increase the accuracy of detection and scalability. Detection results from various users can be integrated (data fusion), enabling more reliable and precise detection. Due to the dynamic behavior of road surface anomalies, integrating the detection results from various users at different times can help evaluate a road surface anomaly’s condition more precisely than in the spatiotemporal domain. Moreover, other geographic data, such as road networks, manhole, and catchment basin location data, can be integrated through filtration or data analysis to increase the accuracy of the results. As an example, manholes and road joints behave similarly to road surface anomalies in sensor data. Therefore, if the geolocation of man-made anomalies is integrated, sensor-detected road surface anomalies (i.e., the manholes or joints in this case) are subsequently able to be filtered.
The central server stores incoming data while also processing parallel incoming data from multiple sources. The completed central processed data can be presented as a geospatial information system (GIS) web-based map to the general users or authorities dealing with road surface maintenance.
2.1. Sensor Data Collection
A major step in developing a viable approach that can detect road surface anomalies is the collection of sample sensor data from smartphones’ sensors. As discussed earlier, motion sensors, such as accelerometers and gyroscopes, have been widely used in the collection and processing of data for road surface anomaly detection. Data collection from previous studies (refer to Table 2
) differ in terms of the types of sensors being employed, sampling rates, variety of vehicle, and devices considered for the data collection. These indicated factors are the most critical parameters that impact the performance of a smartphone’s ability to detect road surface anomalies. Table 2
summarizes the sensor data collection properties reviewed in the selected studies.
According to Table 2
, accelerometer sensors have been widely investigated as a means to develop an approach to detect road surface anomalies. In most previous studies, accelerometer sensor data were investigated in the time domain for detecting road surface anomalies. However, gyroscope sensor data were transformed to the frequency domain for feature extraction (road surface anomaly detection). Moreover, most of the previous studies only employed accelerometer sensors to detect road surface anomalies. However, Yagi [21
], Douangphachanh and Oneyama [22
], and Mohamed et al. [30
] attempted to combine gyroscope and accelerometer sensor data to increase detection accuracy using a data fusion technique.
Data sampling rates play a significant role in the processing of any detected event. Choosing an appropriate sampling rate is a design decision affected by multiple factors, such as: available resources, required accuracy, and the type of data being used for event recognition [31
]. As an example, if only frequency domain features are used for monitoring road anomalies, the sampling rate should be high enough to capture all relevant frequencies. According to Douangphachanh and Oneyama [32
], the road anomalies most likely have the frequency range of 40–50 Hz are captured in accelerometer data.
A higher sampling rate increases the chances of capturing and detecting road surface distress features. However, it also increases the battery usage of a smartphone, as well as the required capacity to store and process data. Finding proper sampling rates is related to the speed of movement, as well as the mechanical properties of a vehicle. Sinharay et al. [19
] investigated the use of a low sampling rate to develop their approach to road surface anomaly detection.
Various models of vehicles and smartphone devices are factors considered by previous studies when collecting data. According to Table 2
, Douangphachanh and Oneyama [22
] and Jain et al. [29
] studied both different vehicles (e.g., sedan, SUV, trucks) and smartphones (i.e., different manufactures) to ensure their approaches functioned equally in different circumstances.
2.2. Sensor Data Preprocessing
The preprocessing of sensor data values is important for two major reasons: filtering noise that distorts parts of the signal; data cleaning and sensor data reorientation. Not all the reviewed studies preprocessed the sensor data. Some reviewed studies only conducted noise filtering and data smoothing approaches as their preprocessing step while some other studies only conducted signal data transformation before processing them. Sebestyen et al. [33
] and Seraj et al. [34
] utilized two different filters: one for eliminating noise, and one for amplifying acceleration variation caused by road anomalies. Douangphachanh and Oneyama [22
] used a high-pass filter to detect low-frequency information, such as changing speed and vehicle maneuvering and turning, which have lower frequencies than road surface anomalies from sensor data. Mohamed et al. [30
] applied the second order high-pass butterworth filter proposed by Butterworth [35
]. Harikrishnan and Gopi [36
] collected data segmented into groups of n-samples. Then, a filtering process was conducted to preserve data samples induced by potholes or speed bumps, as well as to minimize the parts of sensor data corresponding to normal roads. To smooth the signal data, Singh et al. (2017) applied a simple moving-average and band-pass filter to smooth accelerometer sensor data values before processing them. The approaches proposed by Mohan et al. [37
], Bhoraskar et al. [27
], Vittorio et al. [38
], Sebestyen et al. [33
], Wang et al. [9
], and Singh et al. [39
] entailed the application of Euler angles (rotation angles) calculated from accelerometer sensor data to transform the signal data values from device coordinate systems to the local-level coordinate system orientation using a coordinate transformation technique. Silva et al. [40
] applied a data cleaning process by eliminating all null sensor data values and inconsistent timestamp data values before processing them.
2.3. Sensor Data Processing
Processing sensor values for the application of road surface anomaly detection has three main approaches: threshold-based, machine learning, and DTW. Table 3
summarizes the approaches used by previous studies when processing sensor data and detecting abnormal changes in signal data. Data processing in this application can be reviewed in terms of the feature extraction approach (e.g., threshold-based, machine learning, and DTW), ability to classify road surface anomalies (anomaly classification capability), smartphone orientation dependency in detection, and speed dependency in detection.
2.3.1. Feature Extraction Approach
In order to detect road surface anomalies using threshold-based approaches, sensors’ changing data value patterns or statistical values (e.g., standard deviations) taken from sensor data values were analyzed. The amplitude of the accelerometer signal was monitored and the anomaly’s patterns in the signal were identified (an anomaly’s patterns in digital signals occur when the power of the signal exceeds a specific value). Threshold-based approaches were reviewed from three different perspectives: length of interval for a window function, fixed vs. flexible threshold determination, and amplitude of signal vs. other properties of signal amplitude (e.g., mean and standard deviation).
Determining the interval length for a window function in spectral analysis is challenging, as it is related to various factors, such as the speed of vehicles and the distance from the front to rear wheels. A window function considers predefined intervals of the signal data for analysis and feature extraction as opposed to looking at signal data individually. Table 3
summarizes the length of interval for a window function for each of the studies that explored window functions. Defining proper threshold values in a statistical approach is an intensive process since road anomaly data values are affected by variable conditions. The suspension system of a car, sensor properties of smartphones, and smartphone placement all affect how smartphones sense a single anomaly. Studies conducted by Mohan et al. [37
], Mednis et al. [41
], Sinharay et al. [19
], and Yi et al. [43
] determined fixed-threshold values from experiments studying road surface anomaly detection. However, studies conducted by Sebestyen et al. [33
], Wang et al. [9
], and Harikrishnan and Gopi [36
] utilized dynamic threshold values to overcome unsteady signal patterns caused by various sensor and mechanical properties. Dynamically assigned threshold values are desirable when creating methods for detecting road surface anomalies, as they can be adapted to different circumstances.
Mohan et al. [37
], Mednis et al. [41
], Sebestyen et al. [33
], Wang et al. [9
], and Harikrishnan and Gopi [36
] determined thresholds based on the amplitude of the signal. However, other studies, such as Yagi [21
], Nomura and Shiraishi [42
], Vittorio et al. [38
], and Yi et al. [43
] determined thresholds based on the statistical values (such as the standard deviation) derived from signal values. Mednis et al. [41
] confirmed that the standard deviation is the most important parameter for detecting road surface anomalies from accelerometer sensor data.
Machine Learning Approach
There are two prevalent approaches using machine learning techniques: supervised learning and unsupervised learning [44
]. The reviewed studies that involved machine learning techniques can also be categorized based on these two approaches. Bhoraskar et al. [27
] used k-means, an unsupervised method, to classify sensor data on smooth and bumpy roads, as well as to train the SVM algorithm. In this approach, the classes (bumpy or smooth) were manually labeled. Moreover, Mohamed et al. [30
] utilized the proposed threshold-based approach to label classes (smooth or speed bump) and train an SVM algorithm with three different kernel functions to distinguish between bumps and smooth road. Perttunen et al. [28
], Jain et al. [29
], Seraj et al. [34
], and Mohamed et al. [30
] employed SVM to classify sensor data. Captured videos, images, sounds, or field inspection were utilized to collect the ground truth and label the anomalies for classification purposes. Silva et al. [40
] investigated the performance of various supervised machine learning approaches, such as gradient boosting (GB), decision tree (DT), multilayer perceptron classifier (MPL), Gaussian Naive Bayes (GNB), and linear SVC to classify road surface distresses into different classes, including short bump, long bump, unleveled manholes, and others. Although these methods successfully classified the sensor data, a sample of labeled data was required to train the supervised algorithm first, which is impractical for real-time or near real-time applications.
Dynamic Time Warping Approach (DTW)
In time series signal processing, the DTW approach measures the similarity between any two patterns of signals and extracts features from signal data [45
]. For example, Singh et al. [39
] proposed a DTW-based approach to detect road surface anomalies from accelerometer sensor data. In this approach, time series values captured accelerometer sensor data for every pothole and bump, and the data were then stored in a central server as templates. Next, incoming sensor data were compared with the stored templates to detect similarities. The accuracy of this approach was greatly correlated to the quality of the reference template. Therefore, this approach was both computationally intensive and unreliable, as it required reference templates for each different condition (i.e., various vehicles, road conditions, speed of driving).
2.3.2. Differentiating Various Forms of Road Surface Anomalies
Anomalies existing on road surfaces can be categorized into two major forms:
Actual road surface anomalies, including potholes and cracks;
Man-made road surface anomalies, including manholes, road joints, catchment basins, and speed bumps.
A comprehensive pothole detection approach should be able to differentiate actual road surface anomalies (such as potholes and cracks) successfully from a variety of man-made anomalies (such as manholes and speed bumps). However, this is challenging as they both generate similar signal patterns, especially in the case of manholes and catchment basins.
In an approach proposed by Sebestyen et al. [33
], potholes can be distinguished from man-made speed bumps. If a car runs over a pothole, the car first drops and then climbs back up. Conversely, if a car runs over a man-made bump, the car first climbs and then drops. Therefore, by setting these rules within the signal pattern, these anomalies were detected and separated. Sinharay et al. [19
] suggested that the standard deviation of values calculated from sensor data can be used to distinguish potholes from bumps. Seraj et al. [34
] classified detected road surface anomalies to three different classes: severe anomalies, mild anomalies, and spans. Harikrishnan and Gopi [36
] used an X-Z filter proposed by Eriksson et al. [46
] to differentiate between potholes and speed bumps. Eriksson et al. [46
] claimed that potholes are mainly caused by an impact on one side of the vehicle, resulting in a relatively large variation on the x-direction of the accelerometer sensor data. However, speed bumps cause an impact on both sides of a vehicle, leading to small variations on the x-direction of accelerometer sensor data values. Such a mechanism can then be used to distinguish between potholes and speed bumps.
2.3.3. Smartphone Orientation Dependency
Road anomaly detection results are sensitive to the sensors’ orientation. Most of the reviewed studies, such as Yagi [21
], Mednis et al. [41
], Perttunen et al. [28
], and Sinharay et al. [19
], assumed fixed and predetermined positions for analyzing smartphone sensor data. They required users to place their mobile device at a specific orientation and restricted them from using their mobile devices freely. As such, the smartphones had a lack of orientation independence. In order to find a practical road surface anomaly detection solution, smartphones should be freely placed. To develop an approach independent from smartphone orientation, two methods have been investigated:
Signal transformation: In this method, the sensors’ values are transferred from the device coordinate system to another geometric coordinate system (e.g., local-level coordinate system or body-frame coordinate system).
Orientation-independent features: In this method, the magnitude of the sensor data values on all three axes is considered instead of considering their individual values on three separate axes.
The methods proposed by Mohan et al. [37
], Bhoraskar et al. [27
], Vittorio et al. [38
], Sebestyen et al. [33
], Wang et al. [9
], and Singh et al. [39
] applied a signal transformation method, which uses the Euler angles computed from accelerometer sensor data for coordinate transformations. Conversely, the approaches proposed by Jain et al. [29
], Sinharay et al. [19
], and Yi et al. [43
] utilized orientation-independent features of acceleration data (i.e., vector sum, mean, standard deviation) to become independent from smartphone orientation.
2.3.4. Speed Dependency
Another factor that influences road anomaly detection using smartphone sensors is the speed of the vehicle. Douangphachanh and Oneyama [32
] demonstrated that average speed plays an important role in road roughness estimation. When a car passes over a specific road anomaly, such as a pothole, at different speeds, the amplitude of the collected acceleration signal reacts in a different manner, which should be modeled [47
]. Fox et al. [20
] investigated the effect of velocity as a component for detecting road surface anomalies from an onboard accelerometer sensor. Their investigations revealed that at high speeds, discriminating between normal roads and pothole regions was difficult. Sebestyen et al. [33
] collected sensor data at three different speeds: 15, 30, and 60 km/h. All values from the different speeds were normalized to a value of 30 km/h to develop and verify the proposed approach. Yi et al. [43
] examined the effect of vehicle velocity discussed in their approach by creating a lookup table, as well as categorizing the speed into different ranges. Then, each event was indexed according to the ratio of standard deviation, as well as the standard deviation of stable periods of the speed interval during which the event had been detected. Seraj et al. [34
] used speed data logged from the GPS sensor to demodulate accelerometer sensor data and amplify part of the signal caused by anomalies by considering the speed of the vehicles. Sinharay et al. [19
] normalized the feature values based on the speed of the vehicle. Speed was categorized in three ways: lower than 2 km/h, between 2 and 30 km/h, and more than 30 km/h. In addition, Perttunen et al. [28
] adopted an approach by Tanaka et al. [48
] that removed the effects of speed on signal data. Mednis et al. [41
] used different algorithms for different speeds. For instance, for a speed of less than 25 km/h, the z-sus algorithm was implemented. For the speed of more than 25 km/h, the z-peak algorithm was implemented. Mohan et al. [37
] used the z-sus method for speeds of less than 25 km/h and z-peak for speeds of more than 25 km/h. To minimize the false positive detection rate, Harikrishnan and Gopi [36
] proposed specifying a velocity-dependent variable. This variable was adjusted to the threshold value based on the current velocity of the vehicle. Different studies have used various methods to deal with the effects of vehicle speed on the performance of their approaches to road anomaly detection. However, none of them provided a technique robust enough to account for the effect of a vehicle’s velocity.
2.4. Post-Processing of Detected Events (Road Surface Anomalies)
In this section, studies that investigated data integration and approaches used to process detected road surface anomalies from various users (i.e., data fusions) are reviewed and discussed. Chen et al. [49
] and Fox et al. [20
] transferred identified potholes’ information for each selected region in their study to the cloud for further analysis in their proposed approach. Then, a voting algorithm was applied to the study area for final decision making. In fact, the voting algorithm counts the number of reports made for each phenomenon from different sources. If the number of reported anomalies from various sources, for each predefined slice of the road, is more than a predefined threshold, those anomalies are considered true detection. Otherwise, they are rejected and assumed false detection. Fox et al. [20
] considered a sliding window of 10 m for evaluating the number of reports from smartphones on-board vehicles in order to minimize the false positive rate of detection.
Unfortunately, for both studies, this simple voting algorithm ignores the fact that sources have different degrees of trustworthiness [50
]. In addition, this binary-based algorithm does not consider the temporal and probabilistic nature of the anomalies. Results conducted by Fox et al. [20
] indicated that at least 10 vehicles operating at a speed of 50 km/h were required for data collection in order to reach an accuracy of 90%. Furthermore, Chen et al. [49
] claimed 90% accuracy with nearly zero false positive alarms for their crowdsourcing-based road surface monitoring (CRSM) system.
Yi et al. [43
] adopted a grid-based clustering algorithm called DENCLUE (DENsity CLUstering) to filter out false detections using the reporting frequency of events in a 5-m grid zone. Neighboring grids were grouped together if the frequency of the reported anomalies for each neighbor grid was more than three. Otherwise, the grid was assumed to be noisy and was removed. The drawback of this strategy is that some anomalies close to each other were treated as a single anomaly. In addition, threshold-based approaches for classifications are similar to the voting algorithm, which suffers as a result of considering the temporal and probabilistic nature of any road surface anomaly detecting by smartphones. Moreover, there is always a trade-off between reducing the false detection rate and missing the detection of anomalies at the same time. In this study, only the position accuracy of detected road surface anomalies was investigated, and the overall accuracy of detection was not studied.
Alessandroni et al. [51
] proposed the “SmartRoadSense” system. Roughness information was collected in a central server. Then, the average of all roughness values within the predefined buffers of the detected location was considered to be the roughness value for that region. Sebestyen et al. [33
] proposed a method in which an average was taken to combine incoming information from multiple users. In this method, a weighted sum between the available evaluations was computed, as well as integrated multiple surveys from various incidents. This method, similar to the voting algorithm, suffers from the ignorance of the temporal aspect of road surface anomalies and the inherent uncertainty of incoming data. In addition, defining the proper buffer distance is challenging because it significantly affects the detection rate.
2.5. Performance Evaluations
To evaluate the performance of road surface anomaly detection approaches, performance metrics are required, including accuracy ratio, precision, false positives ratio, and false negatives ratio. The choice of a specific performance metric or a combination of different performance metrics depends on the type of application needed, as well as its performance requirements. Table 4
summarizes overall performance evaluations for each approach based on the provided performance metrics. In addition to the overall accuracy of the analysis, some studies investigated performance evaluations with different smartphone placements.
One of the challenges in road surface anomaly detection is that smartphone sensors are sensitive to the placement of the device. In most studies, the location of mobile phones was considered to be fixed to a mount on the windshield or attached to the dashboard. However, few studies have investigated the performance of their approaches with smartphones in different locations in the vehicle, such as in the driver’s pocket or in the console near the gearbox. Different drivers have different habits, and the ideal approach should consider any circumstance that could result in a change in the location of a smartphone and its impact on the loss of recognition performance. Table 5
indicates that only three studies investigated the performance of their respective approaches with different smartphone placements in moving vehicles, namely, [29
]. Douangphachanh and Oneyama [32
] confirmed that smartphones located in a driver’s pocket or in the console caused lower detection rates.
In this review, the existing research investigating approaches for road surface anomaly detection using smartphone sensors is reviewed and compared. The existing approaches are compared using five primary criteria: sensor data collection, preprocessing, processing, post-processing, and performance evaluations.
Data collection is one of the most important considerations when developing any approach to road surface anomaly detection with smartphone sensors. It is essential that the detection process considers all relevant aspects, including smartphone sensor properties, smartphone mounting location, vehicle suspension system, driving behavior, and speed. Therefore, to guarantee that an approach is compatible with different conditions, sensor data should be collected in various situations: various models of cars, smartphones, and speeds. However, due to the limitation of resources, only certain vehicles and smartphone devices have been selected and used for data collection in the reviewed research studies.
Preprocessing is also an important task for any application using sensor data to extract features such as road surface anomalies. Preprocessing has two major objectives:
To smooth the sensor data, to amplify the parts of the sensor data caused by the event (road surface anomaly), and to attenuate or remove parts of sensor data caused by noise or undesired input.
To reorient the sensor data from a device coordinate system to the body-frame (vehicle coordinate system) or local-level coordinate system.
In fact, the preprocessing step can increase the accuracy of detection and decrease the false detection rate. In most reviewed studies, it was assumed that the general location of a smartphone was in a rigid holder. It was also assumed to be fixed, both position- and rotation-wise, with respect to the body-frame (vehicle) coordinate system. Unfortunately, when the smartphone is positioned in its holder or placed on the dashboard, there is no guarantee that the direction of the sensor data measurements will align with the vehicle’s body-frame coordinate system. The relative direction between the device’s coordinate system and the body frame’s coordinate system is therefore considered unknown [52
]. In fact, the desired approach should give freedom to users concerning smartphone placement. Several studies have considered sensor data reorientation using accelerometer sensor data to approximate rotation angles (employing Euler angles). However, the calculated rotation angles from accelerometer sensor data are both biased and contaminated by variant noise caused by thermal and mechanical fluctuations inside the sensor. Most modern smartphones using microelectromechanical sensors (MEMS), such as inertial measurement units (IMU), which contain a three-axis gyroscope for measuring angular velocities around three axes (i.e., pitch, roll, and heading), a three-axis accelerometer for measuring acceleration, and a three-axis magnetometer for measuring magnetic fields. The data from the IMU can be fused to obtain unbiased rotation angles that can then be applied for the purpose of coordinate system transformations.
Developing processing algorithms to detect road surface anomalies from smartphone sensor data is quite challenging. Smartphone sensor properties, car suspension systems, driving behavior, and speed affect the signal pattern when passing over any road anomaly since they are contaminated with biases and noise. Threshold-based approaches have been examined to minimize these problems, and they have been evaluated in several studies. The results of these studies were not reliable as a robust and inclusive detection approach. The machine-learning approach, which has been applied by some studies, was able to overcome some limitations encountered by threshold-based approaches. However, the proposed methods were not inclusive and did not yield a robust solution. For example, supervised approaches, such as SVM, required many trained data sets to cover all possible scenarios for classification. However, by integrating both approaches (threshold-based and machine learning-based approaches), a hybrid approach is developed which can potentially overcome the limitations of each individual approach to detect road surface anomalies from smartphone sensors.
Additionally, as seen in Table 1
, most of the studies employed a single sensor (e.g., an accelerometer) to detect road surface anomalies. Figure 3
illustrates all available motion sensors on currently trending smartphones. Linear acceleration and gravity are the new software-based (virtual) sensors which have been recently integrated into high-end smartphone devices. To improve the system’s performance, sensor fusion techniques can be used. For example, gyroscope or gravity sensors can be combined with accelerometer sensor data to strengthen the detection of road surface anomalies. In addition, accelerometer, magnetic, and gyroscope sensor data are combined in order to derive the isolated gravity vector and to exclude it from accelerometer data. Most new and high-end smartphone devices are capable of calculating linear acceleration from their sensors. Linear acceleration is the effect of acceleration on the smartphone devices excluding the earth’s gravity. As a result, the actual acceleration of the device can be determined irrespective of the device orientation.
Most studies reviewed have implemented and verified their methods in an offline mode. However, with the continued release of powerful smartphones, few studies have both developed and verified their approaches in an online mode. Mednis et al. [41
] and Wang et al. [9
] implemented the proposed method on an Android OS for real-time pothole detection. However, in major studies, entire preprocessing and processing steps which have been done on computers and smartphones have only been used for sensor data collection. Due to the popularity of smartphones embedded with high-performance sensors, as well as the recent increase in the enhanced capability of smartphones, complex analysis and processing of streamed data from smartphone sensors in real time are now possible and practical [53
]. For online road surface anomaly detection approaches, the feasibility of implementing an online mode for smartphones should be investigated. In addition, a proper evaluation of implemented approaches for road surface monitoring on smartphones is desirable. Smartphone resource consumption analysis, such as CPU, memory, and battery usage are topics of further interest.
Due to the complexity of the processing step, post-processing using multiple sources is able to increase both the detection accuracy and decrease the rate of false detection. Some studies investigated data crowdsourcing techniques for road surface monitoring and achieved considerable improvement of the detection accuracy rate. Unfortunately, their recommended approaches were in the very early stages of development, and they suffered from the unreliability of smartphone detection and the variable nature of road surface anomalies.
Moreover, the participatory sensing from smartphone applications presents challenges, such as flawed client-server communication due to unreliable vehicular networks, limited connectivity time, and high packet rate [54
]. GPS errors also complicate data accumulation because of erratic sampling [20
]. In fact, the detected location derived from the GPS sensors of smartphones have uncertainty. For instance, some existing studies proved that the cellular and/or GPS positioning can result in errors ranging from several meters to 100 m (MOK et al. [55
]; Zandbergen [56
]; Pun-Cheng et al. [57
4. Conclusions and Challenges
Efforts have been made to implement various methods which detect road surface anomalies using data from smartphone sensors. However, these approaches continue to face some challenges. Regarding the performance of the associated algorithms, it is difficult to compare the accuracy and performance of various approaches due to restricted availability of the reported algorithms and data sets. For many threshold-based approaches, the manner in which the threshold was set remains unclear. In addition, methods that use supervised or unsupervised learning methods need large amounts of data to train their detection model.
Therefore, a hybrid approach, which would be able to continually detect and distinguish various road surface anomalies using real-time data streaming from smartphone sensors and other geographic data, should be developed. Sensor data values should be smoothed and reoriented to allow smartphone users more freedom, as well as to increase the accuracy of detection. The ideal approach should be self-adapting and self-learning; it would be able to reconcile itself to any platform, the dynamic behavior of different vehicles, as well as various road surface conditions.
A free, cross-platform smartphone application must be developed. This application would be fully automatic and not require any user interaction, as well as allow the driver to maintain focus on the road. In addition, the power consumption of the proposed application should be minimized, perhaps by reducing the usage of GPS sensors. Specific techniques, such as robust sensor calibration readings, should also be investigated because the amount of sensor noise and accessibility of the sensor data vary in different smartphones.
Due to the increase in the number of smartphone holders, continuous monitoring and reporting of road surface anomaly events are achievable by the public. In fact, the effectiveness of road monitoring systems could be substantially improved with effective data-crowdsourcing techniques. Smartphones generally are embedded by various wireless interfaces, such as Bluetooth, Wi-Fi, and cellular networks, making them ideal for crowdsourcing purposes and pushing notifications to road users. In fact, detected anomalies can be transferred to the central server through Wi-Fi or cellular networks for further processing and driver/authority awareness. Data integration in the server would increase system accuracy by training the machine learning algorithms and data classification operations. It would also provide further opportunities to notify drivers about the road surface anomalies ahead, and government agencies about the current road surface condition for potential maintenance and rehabilitation.
Moreover, vehicular network technologies, including V2V (vehicle to vehicle) and C-V2V (Cellular-V2V) communication technologies, are trending nowadays and can play a critical role in transportation management and specifically on driving safety [58
]. By more vehicles being connected to the Internet and to each other, there is potential for enhancing the processes of (1) data collection by obtaining the most updated information regarding the road surface conditions from all connected vehicles and (2) data exchange among road users by notifying drivers approaching to road surface anomalies detected by other road users beforehand through V2V and C-V2V technologies. However, these technologies are still under investigation and are not widely deployed on vehicles due to the absence of a road infrastructure [59