The Design of a Mobile Sensing Framework for Road Surfaces Based on Multi-Modal Sensors ^†

Abstract

Road surface information, encompassing aspects like road surface damages and facility distributions, is vital for maintaining and updating roads in smart cities. The proposed mobile sensing framework uses multi-modal sensors, including a GPS, gyroscope, accelerometer, camera, and Wi-Fi, integrated into a Jetson Nano to collect comprehensive road surface information. The collected data are processed, stored, and analyzed on the server side, with results accessible via RESTful APIs. This system enables the detection of road conditions, which are visualized through the web mapping technique. Based on this concept, the Mobile Sensor Framework for Road Surface analysis (MSF4RS) is designed, and its use significantly enhances road surface data acquisition and analysis. Key contributions include (1) the integration of multi-modal IoT sensors to capture comprehensive road surface data; (2) the development of a software environment that facilitates robust data processing; and (3) the execution of experiments using the MSF4RS, which synergistically combines hardware and software components. The framework leverages advanced sensor technologies and server-based computational methods and offers a user-friendly web interface for the dynamic visualization and interactive exploration of road surface conditions. Experiments confirm the framework’s effectiveness in capturing and visualizing road surface data, demonstrating significant potential for smart city applications.

1. Introduction

In the current digital age, the systematic acquisition and analysis of road data are paramount to the development of smart cities and intelligent transportation systems. Road networks, as critical infrastructure, not only physically connect diverse urban areas but also facilitate economic and social exchanges, underpinning daily life and commercial activity. However, vehicular traffic inevitably compromises the integrity of these networks, leading to surface damage such as potholes, cracks, and other distresses [1,2]. Furthermore, the deterioration of critical urban infrastructure components, including speed bumps and drainage systems, poses significant risks to vehicle safety [3,4]. Therefore, monitoring bump features on road surfaces (BFRS) is crucial for maintaining a safe and efficient transportation system.

Current road surface information extraction technologies can be categorized into three primary types based on their data acquisition methods and research approaches [5,6,7]. The first category encompasses image-based methods, which rely on images captured by onboard cameras. These methods leverage advanced computer vision and machine learning algorithms to identify textural features within the images, enabling accurate road surface recognition [8,9,10]. For instance, Li et al. employed a Back-Propagation Neural Network (BPNN) to detect cracks [11], while Robet et al. proposed a refined U-net architecture incorporating Atrous Spatial Pyramid Pooling (ASPP) to enhance contextual understanding and extraction accuracy [12]. However, external factors such as weather conditions and road shadows can degrade image quality, negatively impacting extraction results. The second category comprises methods employing specialized equipment like LiDAR and photogrammetry, which significantly enhance data collection accuracy [13,14]. For instance, Bhatt et al. integrated Terrain-based Laser Scanning (TLS) with a GPS, cameras, and accelerometers to detect road anomalies [15]. Similarly, Sharma et al. utilized ultrasonic sensors and Dynamic Time Warping (DTW)-based image processing to identify road damage [16]. Despite their advantages, the high costs and specialized expertise required for these methods hinder their wider adoption. The third category consists of sensor-based methods, which utilize onboard sensors to capture anomalous vibration signals generated by vehicle–road interactions [17,18,19]. These signals are then processed using sophisticated algorithms to extract parameters such as acceleration and are often integrated with machine learning techniques for effective feature recognition. For example, Du et al. employed an advanced Gaussian background model combined with a k-nearest neighbor approach for feature classification [20]. Similarly, Chen et al. transformed sensor data into the frequency domain for analysis, utilizing an energy entropy-based algorithm to detect anomalies like potholes and bumps [21]. Multisensory road information acquisition techniques have been widely researched and deployed, demonstrably enhancing both the accuracy and reliability of data collection. Integrating diverse data sources, such as GPS, video, and accelerometer data, facilitates comprehensive road information acquisition, thereby supporting the maintenance and development of robust and resilient transportation systems.

To detect BFRS using these multi-modal sensors, this article presents a novel Multi-modal Sensor Framework for Road Surface analysis (MSF4RS), integrating both hardware and software components. The MSF4RS leverages state-of-the-art sensor technologies to optimize road surface data acquisition, incorporates advanced server-based computational methods for robust data processing, and provides a web-based interface for the effective visualization and interactive exploration of road surface information.

2. Methodology

2.1. Overall Design of Mobile Sensing Framework

The MSF4RS is engineered to meet the escalating requirements for automated road data acquisition. As a critical foundation for applications in road surface data collection, the architectural design of the MSF4RS is delineated into three principal components: data collection, server processing, and data visualization, as illustrated in Figure 1. This tripartite structure ensures an exhaustive management of the entire workflow—from the initial data capture through to the final user interaction—thereby providing a robust infrastructure for the transportation systems.

2.2. Sensor Integration and Data Collection

The MSF4RS is equipped with a comprehensive array of multi-modal sensors, including a GPS, IMU, camera, Wi-Fi, and a Jetson Nano processing board. This integrated configuration allows the framework to efficiently acquire and synthesize extensive road surface data from diverse sensors. This robust and cohesive data collection approach is essential for the effective management and analysis of road conditions. Within the MSF4RS, the GPS sensor captures precise location data, formatted as <time, latitude, longitude>. The IMU connects to the Jetson Nano via a USB to TTL converter, with data formatted as <3D axis acceleration, 4-dimensional quaternions>. The camera sensor, employed for capturing two-dimensional road surface images, connects through a USB connector and offers a 120-degree wide-angle field of view, with data formatted as <time, BFRS type>. The Jetson Nano serves as the central edge computing device, interfacing with sensors such as the GPS, IMU, and camera through USB connections. Additionally, a Wi-Fi module facilitates wireless data transmission, with detection results transmitted to the database server using the MQTT protocol.

2.3. Data Transmitting and Server Processing

Sensor data are transmitted in real-time to the server using the MQTT protocol and are received by EMQX, a high-performance message broker that serves as a central hub. This broker effectively manages multiple sensor data streams, directing them to the appropriate processing nodes based on predefined topics. Sensor data retrieval from EMQX is facilitated through subscriptions managed by Django Web services which trigger a sequence of data processing operations. These operations commence with data cleaning, which includes noise removal and the imputation of missing values to preserve data integrity. Following this, data transformation is performed to align the data format with the requirements of storage and subsequent applications. The processed data are then stored in a relational database designed to support complex queries and analyses. Additionally, Django provides a suite of REST APIs that enable web clients to access and utilize the data, thereby supporting flexible data visualization and application integration.

2.4. Web Visualization and Interaction

The web interface is crafted as a platform for road information management and analysis, integrating map services like OSM and Google Maps to visualize road information and detection results. Users can interact with the platform through operations such as zooming and dragging to explore road conditions in various locations. The platform facilitates the viewing and management of both raw data and identification results, providing features to filter and export data based on criteria such as time range, geographic location, and data type. To accommodate the varied needs of different usage scenarios, the interface includes capabilities for flexible system parameter configuration. Users are able to adjust settings such as the data collection frequency and model recognition thresholds to meet specific real-world demands. Moreover, the application of Node.js technology enables the real-time pushing of data, ensuring that the latest road detection results and analytical data are promptly visualized.

3. Experiment

3.1. Multi-Modal Sensor Configuration

The initial setup involves mounting the GPS antenna on the top of the vehicle to optimize satellite signal reception. The antenna, attached via a magnetic base, transmits data to the onboard GPS sensor board through a USB interface. The IMU sensor (JY901B, manufactured by WIT Motion at Shenzhen, China) is connected to the Jetson Nano using a USB to TTL converter, which facilitates synchronized data collection from the various sensors. A USB camera, positioned on the vehicle’s rear-view mirror, captures images of the road surface along the driving path. Furthermore, to ensure a stable network connection, a Wi-Fi module is integrated into the Jetson Nano. The vehicle’s power supply provides ample electrical support for the entire data collection framework. The sampling rates are established as follows: 1 Hz for GPS, 100 Hz for IMU, and 30 frames per second (FPS) for the camera. The configuration and integration of these multi-modal sensors are depicted in Figure 2.

3.2. Road Surface Information Detection

To detect BFRS using multi-modal sensor data, the YOLO v5 model is utilized for analyzing video images to identify uneven areas, as depicted in Figure 3a. This model, which processes images through convolutional networks, excels at detecting and outlining irregular sections on road surfaces. For the analysis of acceleration data, an LSTM model is employed. The data first undergo a pose space transformation to align with a reference frame perpendicular to the road surface, effectively mitigating disturbances from changes in vehicle posture. The transformed acceleration data (as depicted in Figure 3b) are then input into an LSTM model equipped with 100 hidden units. This setup allows for the capturing of temporal dynamics and corresponding BFRS features, thus enhancing the accuracy of BFRS detection.

3.3. Interaction and Visualization

The BFRS detection process within the MSF4RS is implemented as a Web Processing Service (WPS) which is accessible and manipulable via various web requests. The configuration of detection parameters and data retrieval is facilitated through GET and POST requests. Through these requests, parameters essential for training the YOLO or LSTM models, such as the execution environment, initial learning rate, maximum number of epochs, and minimum batch size, can be accessed and modified. This interactive parameter configuration is executed through the RESTful APIs, operable directly within a web browser. Adjustments made to the settings are transmitted to the data server for execution and corresponding APIs are invoked to initiate the BFRS detection service. For example, users can specify parameters like the model, sensor type, and dataset in the URL to activate the detection service via the RESTful APIs. Furthermore, BFRS are visualized through the web interface. Leaflet.js is employed to manage various layers, including base maps, detection results, regions of interest (ROIs), and annotations. These layers are interactively controllable and can be visualized based on selected filter values, offering dynamic visualization of road surface information, as illustrated in Figure 4.

The experiments demonstrate that the MSF4RS harnesses multi-modal sensors to gather road surface information, demonstrating robust performance in data acquisition using the mobile platform. Data are transmitted to the server via the MQTT protocol, enabling efficient BFRS detection. Beyond mere data collection and transmission, the framework supports the user-driven configuration of parameters and data visualization through a dedicated web browser interface. This functionality substantially improves the operational convenience and enhances the usability associated with the framework.

4. Conclusions

The collection and analysis of road information are pivotal in the development of smart cities and intelligent transportation systems. This paper introduces the MSF4RS, a framework designed for the comprehensive acquisition and analysis of road surface information. The framework integrates GPS, IMU, camera, and Wi-Fi modules to ensure thorough data collection and transmission. Utilizing models such as YOLO v5 and LSTM for data analysis enables precise identification of BFRS. Furthermore, the MSF4RS employs a robust server and an intuitive web interface, enhancing user engagement through RESTful APIs with convenient data management, parameter configuration, and visualization functionalities. Ultimately, the MSF4RS presents a novel approach and methodology for the automated and intelligent perception of road information and is poised to significantly enhance road management and promote a safer, more convenient transportation system.