Sensors

Research

Jump to: Review

15 pages, 3818 KiB

Open AccessArticle

Snow-CLOCs: Camera-LiDAR Object Candidate Fusion for 3D Object Detection in Snowy Conditions

by Xiangsuo Fan, Dachuan Xiao, Qi Li and Rui Gong

Sensors 2024, 24(13), 4158; https://doi.org/10.3390/s24134158 - 26 Jun 2024

Cited by 6 | Viewed by 1944

Abstract

Although existing 3D object-detection methods have achieved promising results on conventional datasets, it is still challenging to detect objects in data collected under adverse weather conditions. Data distortion from LiDAR and cameras in such conditions leads to poor performance of traditional single-sensor detection [...] Read more.

Although existing 3D object-detection methods have achieved promising results on conventional datasets, it is still challenging to detect objects in data collected under adverse weather conditions. Data distortion from LiDAR and cameras in such conditions leads to poor performance of traditional single-sensor detection methods. Multi-modal data-fusion methods struggle with data distortion and low alignment accuracy, making accurate target detection difficult. To address this, we propose a multi-modal object-detection algorithm, Snow-CLOCs, specifically for snowy conditions. In image detection, we improved the YOLOv5 algorithm by integrating the InceptionNeXt network to enhance feature extraction and using the Wise-IoU algorithm to reduce dependency on high-quality data. For LiDAR point-cloud detection, we built upon the SECOND algorithm and employed the DROR filter to remove noise, enhancing detection accuracy. We combined the detection results from the camera and LiDAR into a unified detection set, represented using a sparse tensor, and extracted features through a 2D convolutional neural network to achieve object detection and localization. Snow-CLOCs achieved a detection accuracy of 86.61% for vehicle detection in snowy conditions. Full article

(This article belongs to the Special Issue Multi-modal Sensor Fusion and 3D LiDARs for Vehicle Applications)

► Show Figures

Figure 1

11 pages, 3889 KiB

Open AccessArticle

New Scheme of MEMS-Based LiDAR by Synchronized Dual-Laser Beams for Detection Range Enhancement

by Chien-Wei Huang, Chun-Nien Liu, Sheng-Chuan Mao, Wan-Shao Tsai, Zingway Pei, Charles W. Tu and Wood-Hi Cheng

Sensors 2024, 24(6), 1897; https://doi.org/10.3390/s24061897 - 15 Mar 2024

Viewed by 2080

Abstract

A new scheme presents MEMS-based LiDAR with synchronized dual-laser beams for detection range enhancement and precise point-cloud data without using higher laser power. The novel MEMS-based LiDAR module uses the principal laser light to build point-cloud data. In addition, an auxiliary laser light [...] Read more.

A new scheme presents MEMS-based LiDAR with synchronized dual-laser beams for detection range enhancement and precise point-cloud data without using higher laser power. The novel MEMS-based LiDAR module uses the principal laser light to build point-cloud data. In addition, an auxiliary laser light amplifies the single-noise ratio to enhance the detection range. This LiDAR module exhibits the field of view (FOV), angular resolution, and maximum detection distance of 45° (H) × 25° (V), 0.11° (H) × 0.11° (V), and 124 m, respectively. The maximum detection distance is enhanced by 16% from 107 m to 124 m with a laser power of 1 W and an additional auxiliary laser power of 0.355 W. Furthermore, the simulation results show that the maximum detection distance can be up to 300 m with laser power of 8 W and only 6 W if the auxiliary laser light of 2.84 W is used, which is 35.5% of the laser power. This result indicates that the synchronized dual-laser beams can achieve long detection distance and reduce laser power 30%, hence saving on the overall laser system costs. Therefore, the proposed LiDAR module can be applied for a long detection range in autonomous vehicles without requiring higher laser power if it utilizes an auxiliary laser light. Full article

(This article belongs to the Special Issue Multi-modal Sensor Fusion and 3D LiDARs for Vehicle Applications)

► Show Figures

Figure 1

20 pages, 4551 KiB

Open AccessArticle

Extrinsic Calibration of Thermal Camera and 3D LiDAR Sensor via Human Matching in Both Modalities during Sensor Setup Movement

by Farhad Dalirani and Mahmoud R. El-Sakka

Sensors 2024, 24(2), 669; https://doi.org/10.3390/s24020669 - 20 Jan 2024

Cited by 2 | Viewed by 2694

Abstract

LiDAR sensors, pivotal in various fields like agriculture and robotics for tasks such as 3D object detection and map creation, are increasingly coupled with thermal cameras to harness heat information. This combination proves particularly effective in adverse conditions like darkness and rain. Ensuring [...] Read more.

LiDAR sensors, pivotal in various fields like agriculture and robotics for tasks such as 3D object detection and map creation, are increasingly coupled with thermal cameras to harness heat information. This combination proves particularly effective in adverse conditions like darkness and rain. Ensuring seamless fusion between the sensors necessitates precise extrinsic calibration. Our innovative calibration method leverages human presence during sensor setup movements, eliminating the reliance on dedicated calibration targets. It optimizes extrinsic parameters by employing a novel evolutionary algorithm on a specifically designed loss function that measures human alignment across modalities. Our approach showcases a notable

4.43 %

improvement in the loss over extrinsic parameters obtained from target-based calibration in the FieldSAFE dataset. This advancement reduces costs related to target creation, saves time in diverse pose collection, mitigates repetitive calibration efforts amid sensor drift or setting changes, and broadens accessibility by obviating the need for specific targets. The adaptability of our method in various environments, like urban streets or expansive farm fields, stems from leveraging the ubiquitous presence of humans. Our method presents an efficient, cost-effective, and readily applicable means of extrinsic calibration, enhancing sensor fusion capabilities in the critical fields reliant on precise and robust data acquisition. Full article

(This article belongs to the Special Issue Multi-modal Sensor Fusion and 3D LiDARs for Vehicle Applications)

► Show Figures

Figure 1

35 pages, 20635 KiB

Open AccessArticle

Leveraging LiDAR-Based Simulations to Quantify the Complexity of the Static Environment for Autonomous Vehicles in Rural Settings

by Mohamed Abohassan and Karim El-Basyouny

Sensors 2024, 24(2), 452; https://doi.org/10.3390/s24020452 - 11 Jan 2024

Cited by 2 | Viewed by 1770

Abstract

This paper uses virtual simulations to examine the interaction between autonomous vehicles (AVs) and their surrounding environment. A framework was developed to estimate the environment’s complexity by calculating the real-time data processing requirements for AVs to navigate effectively. The VISTA simulator was used [...] Read more.

This paper uses virtual simulations to examine the interaction between autonomous vehicles (AVs) and their surrounding environment. A framework was developed to estimate the environment’s complexity by calculating the real-time data processing requirements for AVs to navigate effectively. The VISTA simulator was used to synthesize viewpoints to replicate the captured environment accurately. With an emphasis on static physical features, roadways were dissected into relevant road features (RRFs) and full environment (FE) to study the impact of roadside features on the scene complexity and demonstrate the gravity of wildlife–vehicle collisions (WVCs) on AVs. The results indicate that roadside features substantially increase environmental complexity by up to 400%. Increasing a single lane to the road was observed to increase the processing requirements by 12.3–16.5%. Crest vertical curves decrease data rates due to occlusion challenges, with a reported average of 4.2% data loss, while sag curves can increase the complexity by 7%. In horizontal curves, roadside occlusion contributed to severe loss in road information, leading to a decrease in data rate requirements by as much as 19%. As for weather conditions, heavy rain increased the AV’s processing demands by a staggering 240% when compared to normal weather conditions. AV developers and government agencies can exploit the findings of this study to better tailor AV designs and meet the necessary infrastructure requirements. Full article

(This article belongs to the Special Issue Multi-modal Sensor Fusion and 3D LiDARs for Vehicle Applications)

► Show Figures

Figure 1

22 pages, 32270 KiB

Open AccessArticle

Exploring Adversarial Robustness of LiDAR Semantic Segmentation in Autonomous Driving

by K. T. Yasas Mahima, Asanka Perera, Sreenatha Anavatti and Matt Garratt

Sensors 2023, 23(23), 9579; https://doi.org/10.3390/s23239579 - 2 Dec 2023

Cited by 4 | Viewed by 2259

Abstract

Deep learning networks have demonstrated outstanding performance in 2D and 3D vision tasks. However, recent research demonstrated that these networks result in failures when imperceptible perturbations are added to the input known as adversarial attacks. This phenomenon has recently received increased interest in [...] Read more.

Deep learning networks have demonstrated outstanding performance in 2D and 3D vision tasks. However, recent research demonstrated that these networks result in failures when imperceptible perturbations are added to the input known as adversarial attacks. This phenomenon has recently received increased interest in the field of autonomous vehicles and has been extensively researched on 2D image-based perception tasks and 3D object detection. However, the adversarial robustness of 3D LiDAR semantic segmentation in autonomous vehicles is a relatively unexplored topic. This study expands the adversarial examples to LiDAR-based 3D semantic segmentation. We developed and analyzed three LiDAR point-based adversarial attack methods on different networks developed on the SemanticKITTI dataset. The findings illustrate that the Cylinder3D network has the highest adversarial susceptibility to the analyzed attacks. We investigated how the class-wise point distribution influences the adversarial robustness of each class in the SemanticKITTI dataset and discovered that ground-level points are extremely vulnerable to point perturbation attacks. Further, the transferability of each attack strategy was assessed, and we found that networks relying on point data representation demonstrate a notable level of resistance. Our findings will enable future research in developing more complex and specific adversarial attacks against LiDAR segmentation and countermeasures against adversarial attacks. Full article

(This article belongs to the Special Issue Multi-modal Sensor Fusion and 3D LiDARs for Vehicle Applications)

► Show Figures

Figure 1

24 pages, 4257 KiB

Open AccessArticle

Multi-Level Optimization for Data-Driven Camera–LiDAR Calibration in Data Collection Vehicles

by Zijie Jiang, Zhongliang Cai, Nian Hui and Bozhao Li

Sensors 2023, 23(21), 8889; https://doi.org/10.3390/s23218889 - 1 Nov 2023

Cited by 3 | Viewed by 3054

Abstract

Accurately calibrating camera–LiDAR systems is crucial for achieving effective data fusion, particularly in data collection vehicles. Data-driven calibration methods have gained prominence over target-based methods due to their superior adaptability to diverse environments. However, current data-driven calibration methods are susceptible to suboptimal initialization [...] Read more.

Accurately calibrating camera–LiDAR systems is crucial for achieving effective data fusion, particularly in data collection vehicles. Data-driven calibration methods have gained prominence over target-based methods due to their superior adaptability to diverse environments. However, current data-driven calibration methods are susceptible to suboptimal initialization parameters, which can significantly impact the accuracy and efficiency of the calibration process. In response to these challenges, this paper proposes a novel general model for the camera–LiDAR calibration that abstracts away the technical details in existing methods, introduces an improved objective function that effectively mitigates the issue of suboptimal parameter initialization, and develops a multi-level parameter optimization algorithm that strikes a balance between accuracy and efficiency during iterative optimization. The experimental results demonstrate that the proposed method effectively mitigates the effects of suboptimal initial calibration parameters, achieving highly accurate and efficient calibration results. The suggested technique exhibits versatility and adaptability to accommodate various sensor configurations, making it a notable advancement in the field of camera–LiDAR calibration, with potential applications in diverse fields including autonomous driving, robotics, and computer vision. Full article

(This article belongs to the Special Issue Multi-modal Sensor Fusion and 3D LiDARs for Vehicle Applications)

► Show Figures

Figure 1

20 pages, 6446 KiB

Open AccessArticle

AFTR: A Robustness Multi-Sensor Fusion Model for 3D Object Detection Based on Adaptive Fusion Transformer

by Yan Zhang, Kang Liu, Hong Bao, Xu Qian, Zihan Wang, Shiqing Ye and Weicen Wang

Sensors 2023, 23(20), 8400; https://doi.org/10.3390/s23208400 - 12 Oct 2023

Cited by 5 | Viewed by 3412

Abstract

Multi-modal sensors are the key to ensuring the robust and accurate operation of autonomous driving systems, where LiDAR and cameras are important on-board sensors. However, current fusion methods face challenges due to inconsistent multi-sensor data representations and the misalignment of dynamic scenes. Specifically, [...] Read more.

Multi-modal sensors are the key to ensuring the robust and accurate operation of autonomous driving systems, where LiDAR and cameras are important on-board sensors. However, current fusion methods face challenges due to inconsistent multi-sensor data representations and the misalignment of dynamic scenes. Specifically, current fusion methods either explicitly correlate multi-sensor data features by calibrating parameters, ignoring the feature blurring problems caused by misalignment, or find correlated features between multi-sensor data through global attention, causing rapidly escalating computational costs. On this basis, we propose a transformer-based end-to-end multi-sensor fusion framework named the adaptive fusion transformer (AFTR). The proposed AFTR consists of the adaptive spatial cross-attention (ASCA) mechanism and the spatial temporal self-attention (STSA) mechanism. Specifically, ASCA adaptively associates and interacts with multi-sensor data features in 3D space through learnable local attention, alleviating the problem of the misalignment of geometric information and reducing computational costs, and STSA interacts with cross-temporal information using learnable offsets in deformable attention, mitigating displacements due to dynamic scenes. We show through numerous experiments that the AFTR obtains SOTA performance in the nuScenes 3D object detection task (74.9% NDS and 73.2% mAP) and demonstrates strong robustness to misalignment (only a 0.2% NDS drop with slight noise). At the same time, we demonstrate the effectiveness of the AFTR components through ablation studies. In summary, the proposed AFTR is an accurate, efficient, and robust multi-sensor data fusion framework. Full article

(This article belongs to the Special Issue Multi-modal Sensor Fusion and 3D LiDARs for Vehicle Applications)

► Show Figures

Figure 1

Review

Jump to: Research

44 pages, 9114 KiB

Open AccessReview

A Review of Deep Learning-Based LiDAR and Camera Extrinsic Calibration

by Zhiguo Tan, Xing Zhang, Shuhua Teng, Ling Wang and Feng Gao

Sensors 2024, 24(12), 3878; https://doi.org/10.3390/s24123878 - 15 Jun 2024

Cited by 3 | Viewed by 4355

Abstract

Extrinsic parameter calibration is the foundation and prerequisite for LiDAR and camera data fusion of the autonomous system. This technology is widely used in fields such as autonomous driving, mobile robots, intelligent surveillance, and visual measurement. The learning-based method is one of the [...] Read more.

Extrinsic parameter calibration is the foundation and prerequisite for LiDAR and camera data fusion of the autonomous system. This technology is widely used in fields such as autonomous driving, mobile robots, intelligent surveillance, and visual measurement. The learning-based method is one of the targetless calibrating methods in LiDAR and camera calibration. Due to its advantages of fast speed, high accuracy, and robustness under complex conditions, it has gradually been applied in practice from a simple theoretical model in just a few years, becoming an indispensable and important method. This paper systematically summarizes the research and development of this type of method in recent years. According to the principle of calibration parameter estimation, learning-based calibration algorithms are divided into two categories: accurate calibrating estimation and relative calibrating prediction. The evolution routes and algorithm frameworks of these two types of algorithms are elaborated, and the methods used in the algorithms’ steps are summarized. The algorithm mechanism, advantages, limitations, and applicable scenarios are discussed. Finally, we make a summary, pointing out existing research issues and trends for future development. Full article

(This article belongs to the Special Issue Multi-modal Sensor Fusion and 3D LiDARs for Vehicle Applications)

► Show Figures

Figure 1

Journal Menu

Journal Browser

Multi-modal Sensor Fusion and 3D LiDARs for Vehicle Applications

Share This Special Issue

Special Issue Editor

Special Issue Information

Keywords

Benefits of Publishing in a Special Issue

Published Papers (8 papers)

Research

Review

Further Information

Guidelines

MDPI Initiatives

Follow MDPI