Sensors

Research

20 pages, 2853 KiB

Open AccessArticle

MHFS-FORMER: Multiple-Scale Hybrid Features Transformer for Lane Detection

by Dongqi Yan and Tao Zhang

Sensors 2025, 25(9), 2876; https://doi.org/10.3390/s25092876 - 2 May 2025

Viewed by 263

Although deep learning has exhibited remarkable performance in lane detection, lane detection remains challenging in complex scenarios, including those with damaged lane markings, obstructions, and insufficient lighting. Furthermore, a significant drawback of most existing lane-detection algorithms lies in their reliance on complex post-processing [...] Read more.

Although deep learning has exhibited remarkable performance in lane detection, lane detection remains challenging in complex scenarios, including those with damaged lane markings, obstructions, and insufficient lighting. Furthermore, a significant drawback of most existing lane-detection algorithms lies in their reliance on complex post-processing and strong prior knowledge. Inspired by the DETR architecture, we propose an end-to-end Transformer-based model, MHFS-FORMER, to resolve these issues. To tackle the interference with lane detection in complex scenarios, we have designed MHFNet. It fuses multi-scale features with the Transformer Encoder to obtain enhanced multi-scale features. These enhanced multi-scale features are then fed into the Transformer Decoder. A novel multi-reference deformable attention module is introduced to disperse the attention around the objects to enhance the model’s representation ability during the training process and better capture the elongated structure of lanes and the global environment. We also designed ShuffleLaneNet, which meticulously explores the channel and spatial information of multi-scale lane features, significantly improving the accuracy of target recognition. Our method has achieved an accuracy score of 96.88%, a real-time processing speed of 87 fps on the TuSimple dataset, and an F1 score of 77.38% on the CULane dataset. Compared with the methods based on CNN and those based on Transformer, our method has demonstrated excellent performance. Full article

(This article belongs to the Special Issue AI-Driving for Autonomous Vehicles)

► Show Figures

Figure 1

25 pages, 27458 KiB

Open AccessArticle

A Comparative Study and Optimization of Camera-Based BEV Segmentation for Real-Time Autonomous Driving

by Woomin Jun and Sungjin Lee

Sensors 2025, 25(7), 2300; https://doi.org/10.3390/s25072300 - 4 Apr 2025

Viewed by 591

Abstract

This study addresses the optimization of a camera-based bird’s eye view (BEV) segmentation technique that operates in real-time within an embedded system environment while maintaining high accuracy despite limited computational resources. Specifically, it examines three technical approaches for BEV segmentation in autonomous driving: [...] Read more.

This study addresses the optimization of a camera-based bird’s eye view (BEV) segmentation technique that operates in real-time within an embedded system environment while maintaining high accuracy despite limited computational resources. Specifically, it examines three technical approaches for BEV segmentation in autonomous driving: depth-based methods, MLP-based methods, and transformer-based methods, focusing on key techniques such as lift–splat–shoot, HDMapNet, and BEVFormer. A mathematical analysis of these methods is conducted, followed by a comparative performance evaluation using the nuScenes dataset. The optimization process was carried out in three stages: accuracy improvement, latency reduction, and model size optimization. In the first stage of the process, the three modules for BEV segmentation (encoder, view transformation, and decoder) were selected with the goal of maximizing mIoU performance. In the second stage, environmental variable optimization was performed through input resolution adaptation and data augmentation to improve accuracy. Finally, in the third stage, model compression was applied to minimize model size and latency for efficient deployment on embedded systems. Experimental results from the first stage show that the lift–splat–shoot view transformation model, based on the InternImage-B encoder and EfficientNet-B0 decoder, achieved the highest performance with 54.9 mIoU at an input image size of

448 \times 800

. Notably, the lift–splat–shoot view transformation model with the InternImage-T encoder and EfficientNet-B0 decoder demonstrated performance of 53.1 mIoU while achieving high efficiency (51.7 ms and 159.5 MB, respectively). The application of the second stage revealed that increasing the input resolution does not always lead to improved accuracy, and there is an optimal resolution size depending on the model. In this study, the best performance was achieved with an input image size of

448 \times 800

. During the third stage, FP16 quantization enabled a 50% reduction in memory size and decreased latency while maintaining similar or identical mIoU performance. When deployed on the NVIDIA AGX Orin device, which operates under power constraints, energy efficiency improved, although it resulted in higher latency under certain power supply conditions. As a result, the InternImage encoder-based lift–splat–shoot technique was shown to achieve the highest accuracy performance relative to latency and model size. This approach outperformed the original method by achieving a 29.2% higher mIoU while maintaining similar latency performance and reducing memory size by 32.2%. Full article

(This article belongs to the Special Issue AI-Driving for Autonomous Vehicles)

► Show Figures

Figure 1

17 pages, 3398 KiB

Open AccessArticle

A Double-Layer LSTM Model Based on Driving Style and Adaptive Grid for Intention-Trajectory Prediction

by Yikun Fan, Wei Zhang, Wenting Zhang, Dejin Zhang and Li He

Sensors 2025, 25(7), 2059; https://doi.org/10.3390/s25072059 - 26 Mar 2025

Viewed by 326

Abstract

In the evolution of autonomous vehicles (AVs), ensuring safety is of the utmost significance. Precise trajectory prediction is indispensable for augmenting vehicle safety and system performance in intricate environments. This study introduces a novel double-layer long short-term memory (LSTM) model to surmount the [...] Read more.

In the evolution of autonomous vehicles (AVs), ensuring safety is of the utmost significance. Precise trajectory prediction is indispensable for augmenting vehicle safety and system performance in intricate environments. This study introduces a novel double-layer long short-term memory (LSTM) model to surmount the limitations of conventional prediction methods, which frequently overlook predicted vehicle behavior and interactions. By incorporating driving-style category values and an improved adaptive grid generation method, this model achieves more accurate predictions of vehicle intentions and trajectories. The proposed approach fuses multi-sensor data collected by perception modules to extract vehicle trajectories. By leveraging historical trajectory coordinates and driving style, and by dynamically adjusting grid sizes according to vehicle dimensions and lane markings, this method significantly enhances the representation of vehicle motion features and interactions. The double-layer LSTM module, in conjunction with convolutional layers and a max-pooling layer, effectively extracts temporal and spatial features. Experiments conducted using the Next Generation Simulation (NGSIM) US-101 and I-80 datasets reveal that the proposed model outperforms existing benchmarks, with higher intention accuracy and lower root mean square error (RMSE) over 5 s. The impact of varying sliding window lengths and grid sizes is examined, thereby verifying the model’s stability and effectiveness. Full article

(This article belongs to the Special Issue AI-Driving for Autonomous Vehicles)

► Show Figures

Figure 1

20 pages, 3171 KiB

Open AccessArticle

Multimodal Trajectory Prediction for Diverse Vehicle Types in Autonomous Driving with Heterogeneous Data and Physical Constraints

by Maoning Ge, Kento Ohtani, Ming Ding, Yingjie Niu, Yuxiao Zhang and Kazuya Takeda

Sensors 2024, 24(22), 7323; https://doi.org/10.3390/s24227323 - 16 Nov 2024

Cited by 1 | Viewed by 2017

Abstract

The accurate prediction of vehicle behavior is crucial for autonomous driving systems, impacting their safety and efficiency in complex urban environments. To address the challenge of multi-agent trajectory prediction, we propose a novel model integrating multiple input modalities, including historical trajectories, map data, [...] Read more.

The accurate prediction of vehicle behavior is crucial for autonomous driving systems, impacting their safety and efficiency in complex urban environments. To address the challenge of multi-agent trajectory prediction, we propose a novel model integrating multiple input modalities, including historical trajectories, map data, vehicle features, and interaction information. Our approach employs a Conditional Variational Autoencoder (CVAE) framework with a decoder that predicts control actions using the Gaussian Mixture Model (GMM) and then converts these actions into dynamically feasible trajectories through a bicycle model. Evaluated on the nuScenes dataset, the model achieves great performance across key metrics, including minADE₅ of 1.26 and minFDE₅ of 2.85, demonstrating robust performance across various vehicle types and prediction horizons. These results indicate that integrating multiple data sources, physical models, and probabilistic methods significantly improves trajectory prediction accuracy and reliability for autonomous driving. Our approach generates diverse yet realistic predictions, capturing the multimodal nature of future outcomes while adhering to Physical Constraints and vehicle dynamics. Full article

(This article belongs to the Special Issue AI-Driving for Autonomous Vehicles)

► Show Figures

Figure 1

18 pages, 13017 KiB

Open AccessArticle

DeployFusion: A Deployable Monocular 3D Object Detection with Multi-Sensor Information Fusion in BEV for Edge Devices

by Fei Huang, Shengshu Liu, Guangqian Zhang, Bingsen Hao, Yangkai Xiang and Kun Yuan

Sensors 2024, 24(21), 7007; https://doi.org/10.3390/s24217007 - 31 Oct 2024

Viewed by 1270

Abstract

To address the challenges of suboptimal remote detection and significant computational burden in existing multi-sensor information fusion 3D object detection methods, a novel approach based on Bird’s-Eye View (BEV) is proposed. This method utilizes an enhanced lightweight EdgeNeXt feature extraction network, incorporating residual [...] Read more.

To address the challenges of suboptimal remote detection and significant computational burden in existing multi-sensor information fusion 3D object detection methods, a novel approach based on Bird’s-Eye View (BEV) is proposed. This method utilizes an enhanced lightweight EdgeNeXt feature extraction network, incorporating residual branches to address network degradation caused by the excessive depth of STDA encoding blocks. Meantime, deformable convolution is used to expand the receptive field and reduce computational complexity. The feature fusion module constructs a two-stage fusion network to optimize the fusion and alignment of multi-sensor features. This network aligns image features to supplement environmental information with point cloud features, thereby obtaining the final BEV features. Additionally, a Transformer decoder that emphasizes global spatial cues is employed to process the BEV feature sequence, enabling precise detection of distant small objects. Experimental results demonstrate that this method surpasses the baseline network, with improvements of 4.5% in the NuScenes detection score and 5.5% in average precision for detection objects. Finally, the model is converted and accelerated using TensorRT tools for deployment on mobile devices, achieving an inference time of 138 ms per frame on the Jetson Orin NX embedded platform, thus enabling real-time 3D object detection. Full article

(This article belongs to the Special Issue AI-Driving for Autonomous Vehicles)

► Show Figures

Figure 1

12 pages, 14397 KiB

Open AccessArticle

Attention-Linear Trajectory Prediction

by Baoyun Wang, Lei He, Linwei Song, Rui Niu and Ming Cheng

Sensors 2024, 24(20), 6636; https://doi.org/10.3390/s24206636 - 15 Oct 2024

Viewed by 1541

Abstract

Recently, a large number of Transformer-based solutions have emerged for the trajectory prediction task, but there are shortcomings in the effectiveness of Transformers in trajectory prediction. Specifically, while position encoding preserves some of the ordering information, the self-attention mechanism at the core of [...] Read more.

Recently, a large number of Transformer-based solutions have emerged for the trajectory prediction task, but there are shortcomings in the effectiveness of Transformers in trajectory prediction. Specifically, while position encoding preserves some of the ordering information, the self-attention mechanism at the core of the Transformer has its alignment invariance that leads to the loss of temporal information, which is crucial for trajectory prediction. For this reason, we design a simple and efficient strategy for temporal information extraction and prediction of trajectory sequences using the self-attention mechanism and linear layers. The experimental results show that the strategy can improve the average accuracy by 15.31%, effectively combining the advantages of the linear layer and the self-attention mechanism, while compensating for the shortcomings of the Transformer. Additionally, we conducted an empirical study to explore the effectiveness of the linear layer and sparse self-attention mechanisms in trajectory prediction. Full article

(This article belongs to the Special Issue AI-Driving for Autonomous Vehicles)

► Show Figures

Figure 1

21 pages, 12925 KiB

Open AccessEditor’s ChoiceArticle

Learning-Based Control of Autonomous Vehicles Using an Adaptive Neuro-Fuzzy Inference System and the Linear Matrix Inequality Approach

by Mohammad Sheikhsamad and Vicenç Puig

Sensors 2024, 24(8), 2551; https://doi.org/10.3390/s24082551 - 16 Apr 2024

Cited by 5 | Viewed by 2415

Abstract

This paper proposes a learning-based control approach for autonomous vehicles. An explicit Takagi–Sugeno (TS) controller is learned using input and output data from a preexisting controller, employing the Adaptive Neuro-Fuzzy Inference System (ANFIS) algorithm. At the same time, the vehicle model is identified [...] Read more.

This paper proposes a learning-based control approach for autonomous vehicles. An explicit Takagi–Sugeno (TS) controller is learned using input and output data from a preexisting controller, employing the Adaptive Neuro-Fuzzy Inference System (ANFIS) algorithm. At the same time, the vehicle model is identified in the TS model form for closed-loop stability assessment using Lyapunov theory and LMIs. The proposed approach is applied to learn the control law from an MPC controller, thus avoiding the use of online optimization. This reduces the computational burden of the control loop and facilitates real-time implementation. Finally, the proposed approach is assessed through simulation using a small-scale autonomous racing car. Full article

(This article belongs to the Special Issue AI-Driving for Autonomous Vehicles)

► Show Figures

Figure 1

17 pages, 1135 KiB

Open AccessArticle

Goal-Guided Graph Attention Network with Interactive State Refinement for Multi-Agent Trajectory Prediction

by Jianghang Wu, Senyao Qiao, Haocheng Li, Boyu Sun, Fei Gao, Hongyu Hu and Rui Zhao

Sensors 2024, 24(7), 2065; https://doi.org/10.3390/s24072065 - 23 Mar 2024

Cited by 3 | Viewed by 1870

Abstract

The accurate prediction of the future trajectories of traffic participants is crucial for enhancing the safety and decision-making capabilities of autonomous vehicles. Modeling social interactions among agents and revealing the inherent relationships is crucial for accurate trajectory prediction. In this context, we propose [...] Read more.

The accurate prediction of the future trajectories of traffic participants is crucial for enhancing the safety and decision-making capabilities of autonomous vehicles. Modeling social interactions among agents and revealing the inherent relationships is crucial for accurate trajectory prediction. In this context, we propose a goal-guided and interaction-aware state refinement graph attention network (SRGAT) for multi-agent trajectory prediction. This model effectively integrates high-precision map data and dynamic traffic states and captures long-term temporal dependencies through the Transformer network. Based on these dependencies, it generates multiple potential goals and Points of Interest (POIs). Through its dual-branch, multimodal prediction approach, the model not only proposes various plausible future trajectories associated with these POIs, but also rigorously assesses the confidence levels of each trajectory. This goal-oriented strategy enables SRGAT to accurately predict the future movement trajectories of other vehicles in complex traffic scenarios. Tested on the Argoverse and nuScenes datasets, SRGAT surpasses existing algorithms in key performance metrics by adeptly integrating past trajectories and current context. This goal-guided approach not only enhances long-term prediction accuracy, but also ensures its reliability, demonstrating a significant advancement in trajectory forecasting. Full article

(This article belongs to the Special Issue AI-Driving for Autonomous Vehicles)

► Show Figures

Figure 1

20 pages, 6982 KiB

Open AccessArticle

Learn to Bet: Using Reinforcement Learning to Improve Vehicle Bids in Auction-Based Smart Intersections

by Giacomo Cabri, Matteo Lugli, Manuela Montangero and Filippo Muzzini

Sensors 2024, 24(4), 1288; https://doi.org/10.3390/s24041288 - 17 Feb 2024

Cited by 3 | Viewed by 1344

Abstract

With the advent of IoT, cities will soon be populated by autonomous vehicles and managed by intelligent systems capable of actively interacting with city infrastructures and vehicles. In this work, we propose a model based on reinforcement learning that teaches to autonomous connected [...] Read more.

With the advent of IoT, cities will soon be populated by autonomous vehicles and managed by intelligent systems capable of actively interacting with city infrastructures and vehicles. In this work, we propose a model based on reinforcement learning that teaches to autonomous connected vehicles how to save resources while navigating in such an environment. In particular, we focus on budget savings in the context of auction-based intersection management systems. We trained several models with Deep Q-learning by varying traffic conditions to find the most performance-effective variant in terms of the trade-off between saved currency and trip times. Afterward, we compared the performance of our model with previously proposed and random strategies, even under adverse traffic conditions. Our model appears to be robust and manages to save a considerable amount of currency without significantly increasing the waiting time in traffic. For example, the learner bidder saves at least 20% of its budget with heavy traffic conditions and up to 74% in lighter traffic with respect to a standard bidder, and around three times the saving of a random bidder. The results and discussion suggest practical adoption of the proposal in a foreseen future real-life scenario. Full article

(This article belongs to the Special Issue AI-Driving for Autonomous Vehicles)

► Show Figures

Figure 1

17 pages, 4414 KiB

Open AccessArticle

Coordinated Decision Control of Lane-Change and Car-Following for Intelligent Vehicle Based on Time Series Prediction and Deep Reinforcement Learning

by Kun Zhang, Tonglin Pu, Qianxi Zhang and Zhigen Nie

Sensors 2024, 24(2), 403; https://doi.org/10.3390/s24020403 - 9 Jan 2024

Cited by 6 | Viewed by 2040

Abstract

Adaptive cruise control and autonomous lane-change systems represent pivotal advancements in intelligent vehicle technology. To enhance the operational efficiency of intelligent vehicles in combined lane-change and car-following scenarios, we propose a coordinated decision control model based on hierarchical time series prediction and deep [...] Read more.

Adaptive cruise control and autonomous lane-change systems represent pivotal advancements in intelligent vehicle technology. To enhance the operational efficiency of intelligent vehicles in combined lane-change and car-following scenarios, we propose a coordinated decision control model based on hierarchical time series prediction and deep reinforcement learning under the influence of multiple surrounding vehicles. Firstly, we analyze the lane-change behavior and establish boundary conditions for safe lane-change, and divide the lane-change trajectory planning problem into longitudinal velocity planning and lateral trajectory planning. LSTM network is introduced to predict the driving states of surrounding vehicles in multi-step time series, combining D3QN algorithm to make decisions on lane-change behavior. Secondly, based on the following state between the ego vehicle and the leader vehicle in the initial lane, as well as the relationship between the initial distance and the expected distance with the leader vehicle in the target lane, with the primary objective of maximizing driving efficiency, longitudinal velocity is planned based on driving conditions recognition. The lateral trajectory and conditions recognition are then planned using the GA-LSTM-BP algorithm. In contrast to conventional adaptive cruise control systems, the DDPG algorithm serves as the lower-level control model for car-following, enabling continuous velocity control. The proposed model is subsequently simulated and validated using the NGSIM dataset and a lane-change scenarios dataset. The results demonstrate that the algorithm facilitates intelligent vehicle lane-change and car-following coordinated control while ensuring safety and stability during lane-changes. Comparative analysis with other decision control models reveals a notable 17.58% increase in driving velocity, underscoring the algorithm’s effectiveness in improving driving efficiency. Full article

(This article belongs to the Special Issue AI-Driving for Autonomous Vehicles)

► Show Figures

Figure 1

15 pages, 5807 KiB

Open AccessArticle

Performance Verification of Autonomous Driving LiDAR Sensors under Rainfall Conditions in Darkroom

by Jaeryun Choe, Hyunwoo Cho and Yoonseok Chung

Sensors 2024, 24(1), 14; https://doi.org/10.3390/s24010014 - 19 Dec 2023

Cited by 1 | Viewed by 1885

Abstract

This research aims to assess the functionality of the VLP-32 LiDAR sensor, which serves as the principal sensor for object recognition in autonomous vehicles. The evaluation is conducted by simulating edge conditions the sensor might encounter in a controlled darkroom setting. Parameters for [...] Read more.

This research aims to assess the functionality of the VLP-32 LiDAR sensor, which serves as the principal sensor for object recognition in autonomous vehicles. The evaluation is conducted by simulating edge conditions the sensor might encounter in a controlled darkroom setting. Parameters for environmental conditions under examination encompass measurement distances ranging from 10 to 30 m, varying rainfall intensities (0, 20, 30, 40 mm/h), and different observation angles (0°, 30°, 60°). For the material aspects, the investigation incorporates reference materials, traffic signs, and road surfaces. Employing this diverse set of conditions, the study quantitatively assesses two critical performance metrics of LiDAR: intensity and NPC (number of point clouds). The results indicate a general decline in intensity as the measurement distance, rainfall intensity, and observation angles increase. Instances were identified where the sensor failed to record intensity for materials with low reflective properties. Concerning NPC, both the effective measurement area and recorded values demonstrated a decreasing trend with enlarging measurement distance and angles of observation. However, NPC metrics remained stable despite fluctuations in rainfall intensity. Full article

(This article belongs to the Special Issue AI-Driving for Autonomous Vehicles)

► Show Figures

Figure 1

27 pages, 6199 KiB

Open AccessArticle

End-to-End Autonomous Navigation Based on Deep Reinforcement Learning with a Survival Penalty Function

by Shyr-Long Jeng and Chienhsun Chiang

Sensors 2023, 23(20), 8651; https://doi.org/10.3390/s23208651 - 23 Oct 2023

Cited by 6 | Viewed by 2777

Abstract

An end-to-end approach to autonomous navigation that is based on deep reinforcement learning (DRL) with a survival penalty function is proposed in this paper. Two actor–critic (AC) frameworks, namely, deep deterministic policy gradient (DDPG) and twin-delayed DDPG (TD3), are employed to enable a [...] Read more.

An end-to-end approach to autonomous navigation that is based on deep reinforcement learning (DRL) with a survival penalty function is proposed in this paper. Two actor–critic (AC) frameworks, namely, deep deterministic policy gradient (DDPG) and twin-delayed DDPG (TD3), are employed to enable a nonholonomic wheeled mobile robot (WMR) to perform navigation in dynamic environments containing obstacles and for which no maps are available. A comprehensive reward based on the survival penalty function is introduced; this approach effectively solves the sparse reward problem and enables the WMR to move toward its target. Consecutive episodes are connected to increase the cumulative penalty for scenarios involving obstacles; this method prevents training failure and enables the WMR to plan a collision-free path. Simulations are conducted for four scenarios—movement in an obstacle-free space, in a parking lot, at an intersection without and with a central obstacle, and in a multiple obstacle space—to demonstrate the efficiency and operational safety of our method. For the same navigation environment, compared with the DDPG algorithm, the TD3 algorithm exhibits faster numerical convergence and higher stability in the training phase, as well as a higher task execution success rate in the evaluation phase. Full article

(This article belongs to the Special Issue AI-Driving for Autonomous Vehicles)

► Show Figures

Figure 1

Journal Menu

Journal Browser

AI-Driving for Autonomous Vehicles

Share This Special Issue

Special Issue Editors

Special Issue Information

Keywords

Benefits of Publishing in a Special Issue

Published Papers (12 papers)

Research

Further Information

Guidelines

MDPI Initiatives

Follow MDPI