Integrating Knowledge Graphs into Autonomous Vehicle Technologies: A Survey of Current State and Future Directions

Abstract

Autonomous vehicles (AVs) represent a transformative innovation in transportation, promising enhanced safety, efficiency, and sustainability. Despite these promises, achieving robustness, reliability, and adherence to ethical standards in AV systems remains challenging due to the complexity of integrating diverse technologies. This survey reviews literature from 2017 to 2023, analyzing over 90 papers to explore the integration of knowledge graphs (KGs) into AV technologies. Our findings indicate that KGs significantly enhance AV systems by providing structured semantic understanding, improving real-time decision-making, and ensuring compliance with regulatory standards. The paper identifies that while KGs contribute to better environmental perception and contextual reasoning, challenges remain in their seamless integration with existing systems and in maintaining processing speed. We also address the ethical dimensions of AV decision-making, advocating for frameworks that prioritize safety and transparency. This review underscores the potential of KGs to address critical challenges in AV technologies, offering a hopeful and optimistic outlook for the development of robust, reliable, and socially responsible autonomous transportation solutions.

1. Introduction

Autonomous vehicles (AVs) lead a transportation revolution, promising enhanced safety, reduced traffic congestion, and improved energy efficiency [1,2]. AVs are commonly defined as driverless vehicles capable of operating on standard road infrastructure without human intervention. However, the term AV can encompass a range of automation levels [3]. According to the Society of Automotive Engineers (SAE), vehicle automation is classified into six levels, from Level 0 (no automation) to Level 5 (full automation), as illustrated in Figure 1. Our paper focuses on AVs operating at Levels 3 to 5. At these levels, AVs can perform most or all driving tasks, with Level 5 representing full autonomy in any environment and Level 3 requiring occasional human intervention. While the default definition of an AV aligns most closely with Levels 4 and 5, this paper also considers a broader view, addressing AV systems that may include vehicles operating at Level 3 automation, where human intervention is occasionally required.

The journey from early prototypes in the mid-20th century to today’s advanced AV systems has been marked by significant research and technological breakthroughs. In the United States, major tech companies and automakers, such as Waymo [4] and General Motors [5], are pioneering innovations and conducting extensive real-world testing. Despite these advancements, regulatory hurdles and public acceptance remain critical challenges, as noted by the National Highway Traffic Safety Administration (NHTSA) [6].

In China, rapid AV development is propelled by strong government support and a vast market, with large-scale data availability and extensive testing positioning the country as a global leader. Notably, China led the development of the first international standard for autonomous driving test scenarios, as reported by the Ministry of Industry and Information Technology (MIIT) [7].

Meanwhile, Japan’s approach emphasizes safety and the integration of AVs with existing transportation systems, addressing the needs of an aging population and advancing smart city initiatives. Reports from Japan’s Ministry of Economy, Trade and Industry (METI) [8] and the Japan Automobile Manufacturers Association (JAMA) [9] highlight Japan’s significant progress in developing AVs capable of navigating complex urban environments. These global advancements underscore the critical need for sophisticated systems that encompass perception, decision-making, control, and communication to navigate the complexities of real-world environments successfully. However, transitioning from human-driven to fully autonomous vehicles presents significant challenges across technology, regulation, and public acceptance.

Knowledge graphs (KGs) have emerged as a powerful tool to manage complex, interconnected data, offering significant potential to address key challenges in AV technologies. By enhancing data integration, contextual understanding, and operational efficiency, KGs can revolutionize how AVs interpret and navigate their environments. Current AV perception systems, relying on sensors like cameras, LiDAR, and radar, often struggle with accurately understanding complex driving scenarios, especially in unpredictable conditions such as varying weather, lighting, or sudden obstacles [10,11,12]. Misinterpretations—whether due to false positives, incorrect object classification, or delayed processing—can compromise vehicle safety and reliability [13]. KGs help mitigate these issues by providing a structured representation of diverse data, allowing AVs to better understand and interpret their surroundings [14]. KGs improve object recognition and scene interpretation, especially in challenging situations like distinguishing between a stationary object and a pedestrian about to cross the street.

In addition to perception, AVs must make rapid, safe, and ethically sound decisions in complex environments like urban streets, where pedestrians, cyclists, and other vehicles create dynamic scenarios [15]. These decisions must minimize harm while aligning with societal values, such as prioritizing pedestrian safety, even at the risk of a collision with another vehicle. AVs also face challenges related to uncertainty (incomplete or unclear sensor data) and ambiguity (situations that can be interpreted in multiple ways, like whether a pedestrian is about to cross). Traditional rule-based systems [16], while effective in predictable environments, struggle to adapt to unexpected situations, such as sudden pedestrian movements or unclear traffic signals. KGs address these limitations by enabling AVs to connect real-time sensor inputs with pre-established knowledge, making context-aware decisions in complex, uncertain scenarios [14,17]. This integration could significantly enhance the safety, adaptability, and trustworthiness of autonomous driving systems.

Existing research on AVs often focuses on individual system components, such as perception, decision-making, and control. However, there is a gap in understanding how KGs can significantly enhance these systems by improving how they process and interpret information. Despite their potential, only some reviews address how KGs can be cohesively integrated with AV technologies. Yurtsever et al. [18] comprehensively review general AV technologies, covering core functions, system architectures, and key challenges. While their work establishes a solid foundation for understanding AV systems, it does not explore emerging technologies like KGs. Similarly, Badue et al. [19] offer an in-depth examination of AV technologies, discussing aspects like architecture, perception, decision-making, and control mechanisms while highlighting key advancements. However, this survey also could focus on KGs. Zhao et al. [20] present an extensive overview of AV technologies, addressing critical components such as perception, localization, mapping, planning, and control. Their work emphasizes challenges related to real-time decision-making, safety, scalability, and sensor integration but does not delve into the potential role of KGs in enhancing these systems. While these surveys provide valuable insights, there is a clear need for more comprehensive reviews that specifically examine how KGs could transform AV technologies.

Luettin et al. [14] focus specifically on the role of KGs in AVs, exploring their application in tasks like perception, decision-making, validation, and scene understanding. However, their survey does not extensively address the challenges involved in implementing KGs, leaving a gap in understanding the practical hurdles of integration.

Our survey addresses a gap in the literature by specifically exploring the role of KGs in AV systems. Unlike other works, such as those by Yurtsever et al. [18], Badue et al. [19], and Zhao et al. [20], which do not delve into the potential of KGs, our work emphasizes their importance in providing semantic understanding, decision-making support, and the ability to tackle complex AV challenges. While Luettin et al. [14] provide a good starting point by discussing the application of KGs in AV tasks like perception and decision-making, their work does not sufficiently address the challenges involved in implementing KGs in real-world systems. Our contribution fills this gap by examining not only the benefits but also the practical hurdles of integrating KGs into AV systems, such as scalability, data availability, and the complexity of real-time decision-making. The comparison between our work and closely related survey studies is presented in

Table 1. Comparison between our work and related survey studies.

Survey Coverage	[18]	[19]	[20]	[14]	Ours
Perception	√	√	√	√	√
Localization	√	√	√	√	√
Mapping	√	√	√	√	√
Moving Object Detection and Tracking	√	√	√	√	√
Traffic Signalization Detection		√
Path Planning		√	√		√
Behavior Selection		√
Motion Planning		√		√	√
Obstacle Avoidance		√
Control		√			√
Sensors and Hardware	√
Road and Lane Detection	√
Assessment	√
Decision-Making	√	√	√	√	√
Human–Machine Interaction	√				√
Datasets and Tools	√
Semantic Segmentation			√
Trajectory Prediction			√
Simulator and Scenario Generation			√		√
KGs Applied to AVs				√	√
Object Detection				√
Semantic Segmentation				√	√
Mapping				√
Scene Understanding				√	√
Object Behavior Prediction				√
Motion Planning				√
Validation				√
Scene Representation					√
Object Tracking					√
Road Sign Detection					√
Scene-Graph Augmented Risk Assessment					√
Scene Creation					√
Current Challenges and Limitations			√		√
Future Directions					√
Development in Industry		√
Ethical and Practical Considerations in AV Technologies					√

√ indicates that the survey study includes coverage of the respective topic.

.

This paper comprehensively reviews state-of-the-art AV technologies and the current state of integrating KGs into AV technologies. Moreover, we identify research challenges and propose future directions for developing robust, reliable, and socially responsible autonomous transportation solutions. By synthesizing existing knowledge and shedding light on emerging trends, this survey aims to contribute to the advancement of AV technologies that are proficient, ethically sound, and accepted by society.

The paper is organized as follows. Section 2 presents the review strategy. Section 3 covers fundamental concepts and recent advancements in AV technologies, such as perception, localization, path planning, and decision-making. Section 4 examines the role of KGs in enhancing AV systems, focusing on their contributions to semantic understanding, data integration, and real-time decision-making. In addition, it investigates the practical integration of KGs with AV technologies, addressing challenges and impacts on performance. Section 5 explores the ethical dimensions of AV decision-making, emphasizing the need for tailored ethical frameworks. Finally, Section 6 presents the conclusions.

2. Review Strategy

Defining the review strategy is a fundamental aspect of a systematic review [17]. This section details the review strategies employed in this paper. The review strategy we propose is structured to explore studies that contribute to the integration of KGs into AV technologies. It is composed of three key elements: research questions, publications retrieval, and articles in review. The following research questions, as presented in

Table 2. Overview of research questions and focus areas.

Research Question	Focus
What are the key applications in AV technologies and KGs?	Application
Which aspects of KG integration does the article address?	Contribution
What methods do the article discuss for integrating KGs with AV systems?	Methodology
What are the limitations of AVs, and what future research does the article suggest?	Limitations and Future Work

, were acquired to guide the article analysis.

For a comprehensive review, we retrieved publications from reputable databases, including IEEE Xplore and Google Scholar. These databases were selected for their broad coverage of relevant journals and conferences, ensuring a thorough review. Keywords used in the search include “knowledge graphs”, “autonomous vehicles”, and “integration”. The initial search yielded a significant number of results from the databases, with publications dated between 2017 and 2023. Articles were then selected for detailed review based on the following filtering process:

• Relevance: Articles were selected not only for focus on the integration of KGs with autonomous vehicle technologies but also for their contribution to understanding the background and fundamentals of AV technologies.
• Manual Screening: Abstracts of the identified articles were manually reviewed to assess alignment with the research questions. Only studies that were directly relevant to AV technologies and KG integration or offered valuable insights into AV technologies were retained for further analysis.
• Institutional Expertise: To capture cutting-edge research and institutional expertise, we specifically included sources from Toyota Research Institute, Kanazawa University, The University of Tokyo, and the National Institute of Advanced Industrial Science and Technology (AIST) as part of the filtering process.
• Peer-Review Status: We ensured that the review is based on peer-reviewed sources. Preprint articles from repositories such as ArXiv were excluded unless their final, peer-reviewed versions were available. Articles with discrepancies between preprint and published versions were cross-checked, and only the final published versions were retained for analysis.

After applying these criteria, approximately 85 articles were identified as most relevant for inclusion in this review. Figure 2 illustrates the percentage of papers published each year between 2017 and 2024, reflecting the final set of articles selected for analysis.

3. Background and Fundamentals

This section aims to provide a comprehensive overview of the key components of AV technologies. In support of this analysis, we reference around 50 journals and conferences that have made significant contributions to AV research. Notable among them are the IEEE/CVF International Conference on Computer Vision (ICCV), IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), Robotics: Science and Systems (RSS), IEEE Transactions on Intelligent Transportation Systems, IEEE Robotics and Automation Letters, and the International Conference on Robotics and Automation (ICRA). These venues are renowned for their substantial impact on AV-related topics, as summarized in

Table 3. Selected conferences and journals contributing to AV technologies.

Conferences/Journals	Publisher	References/Published Year
IEEE/CVF International Conference on Computer Vision (ICCV)	IEEE	[1] 2021, [21] 2023, [22] 2023, [23] 2019
Workshop on the Algorithmic Foundations of Robotics	Springer	[24] 2018
Autonomous Robots	Springer	[25] 2016
Conference on Robot Learning (CoRL)	PMLR	[26] 2020, [27] 2019
IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS)	IEEE	[28] 2023, [29] 2020
IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)	IEEE	[2] 2020, [10] 2023, [30] 2019, [31] 2018
International Conference on Robotics and Automation (ICRA)	IEEE	[32] 2020, [33] 2019, [34] 2018, [35] 2022, [36] 2024, [37] 2022
International Conference on Learning Representations (ICLR)	ICLR	[38] 2018, [39] 2020
IEEE/International Conference on Intelligent Transportation Systems (ITSC)	IEEE	[40] 2020
IEEE Robotics and Automation Letters	IEEE	[41] 2022, [42] 2021, [43] 2020
Robotics: Science and Systems (RSS)	MIT Press	[44] 2019, [45] 2020
IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP)	IEEE	[46] 2018
IEEE Winter Conference on Applications of Computer Vision (WACV)	IEEE	[47] 2019
Remote Sensing	MDPI	[48] 2022, [49] 2021, [50] 2022
Sensors (Basel, Switzerland)	MDPI	[11] 2023, [17] 2023, [51] 2020, [52] 2022, [53] 2024 [54] 2023, [55] 2020,
International Association of Traffic and Safety Sciences (IATSS)	Elsevier	[56] 2019
International Conference on Multisensor Fusion and Integration for Intelligent Systems	IEEE	[57] 2017
International Conference on Intelligent Informatics and Biomedical Sciences (ICIIBMS)	IEEE	[58] 2017
IEEE Transactions on Cybernetics	IEEE	[59] 2022
IEEE Transactions on Intelligent Transportation Systems	IEEE	[60] 2022, [61] 2022
Accident Analysis and Prevention	Elsevier	[62] 2023
International Conference on Pattern Recognition (ICPR)	IEEE	[63] 2020

PMLR: Proceedings of Machine Learning Research, IEEE: Institute of Electrical and Electronics Engineers, MIT: Massachusetts Institute of Technology, MDPI: Multidisciplinary Digital Publishing Institute.

, which is based on a selection of surveys referencing these notable sources.

Furthermore, key contributors to the AV field, such as V.C. Guizilini, A. Gaidon, and R. Ambrus, have collectively authored significant papers on essential AV topics. Their extensive work underscores the depth of research covered in our survey.

Table 4. Contributions of scholars in AV technologies.

Scholar	Research Focus	No. of Published Papers	Conferences/ Journals
V.C. Guizilini	Semantic segmentation [1,2,39]; Monocular Depth Estimation [21,26,30,39]; Sparse View Synthesis [22]; Calibration [28]; Ego-Motion Estimation [27]; Occupancy Prediction [33]	10	ICCV, IROS, ICLR, CVPR, CoRL, ICRA
A. Gaidon	Semantic segmentation [1,2,39]; Monocular Depth Estimation [21,26,30,39]; Sparse View Synthesis [22]; Calibration [28]; Object Detection [31]; Flow Estimation [29]; Ego-Motion Estimation [27]; Occupancy Prediction [33]; Sparse Visual Odometry [44]; Pedestrian Locomotion Forecasting [47]; Near-Accident Driving [45]; Behavior Cloning [23]; Pedestrian Intent Prediction [43]	18	ICCV, IROS, ICLR, CVPR, CoRL, ICRA, IEEE Robotics and Automation Letters, RSS, WACV
R. Ambrus	Semantic segmentation [1,39]; Monocular Depth Estimation [21,26,30,39]; Sparse View Synthesis [22]; Calibration [28]; Multi-Object Tracking [32]; Ego-Motion Estimation [27]	9	ICCV, CoRL, IROS, ICRA, ICRL, CVPR

outlines the primary research contributions made by these scholars.

To further elucidate the landscape of AV research, we analyze the popularity and focus areas of key components in AV technologies.

Table 5. Popularity and reasons for key components of AV technologies.

Key Component	Popularity	Reasons
Sensors and perception systems	Very high	High research volume on sensor accuracy and data processing.
Localization and mapping	High	Significant focus on SLAM and GPS-based localization.
Path planning	High	Advancements in planning algorithms for efficient AV navigation.
Control systems	Moderate to high	Ongoing research in control mechanisms integral to AV operation.
Decision-making	High	Significant focus on machine learning and AI-based decision-making.
Human–machine interface (HMI)	Moderate	Increasing considerations for user experience.
Communication systems	Moderate	Growing interest in 5G and V2X technologies.
Safety and redundancy	Moderate to high	Substantial interest in AV reliability and public acceptance.
Ethical and legal considerations	Moderate	Rising importance due to regulatory and societal impact.
Societal impact and infrastructure	Moderate	Long-term AV integration amid growing policy discussions.

details the popularity of these components and the reasons behind their emphasis on research. This analysis highlights which components are receiving significant attention and are experiencing substantial growth. Among these, we briefly discuss the components where the research focus is either already high or is experiencing significant growth through Section 3.1 to Section 3.6. By examining the interrelationships between these components, we aim to demonstrate how advancements in each area contribute to AV systems’ performance and reliability. For a clear and concise overview of the research focus across different AV technologies, refer to

Table 6. Overview of research focus in AV technologies.

Key Component	Research Focus
Perception	Segmentation [1,2,38]; Street-View Change Detection [25]; Monocular Depth Estimation [21,26,30,39]; Sparse View Synthesis [22]; Calibration [28,57]; Multi-Object Tracking [32]; Object Detection [31]; Flow Estimation [29]; Ego-Motion Estimation [27]; Occupancy Prediction [33]; Visual Odometry and Image Registration [44]; Driver Alertness Detection [46]; Pedestrian Locomotion Prediction [47,63]; Traffic Lights and Arrow detection [51]; Interpreting Environmental Conditions [56]; Recognition and Matching Road Surface Features [52]; Turn Signal Recognition [58]; Gaze Tracking [62]; Spatio-Temporal Image Representation [61]
Localization and Mapping	Updating and Maintaining Maps [25]; Depth-Aware Map [26]; Multi-Camera Maps [28]; Ego-Motion Estimation [27]; Creating and Updating Occupancy Maps [33]; Visual Odometry [41,44]; Probabilistic Localization [34]; Mapping with GNSS/INS-RTK [48]; Transferring Lane Graphs and Different Map Representation [53]; Generating 2.5D maps using LIDAR and Graph SLAM [49]; 2.5D Maps for Multilevel Environments and Vehicle Localization [50]; Map Generation and Localization [57]; 3D LiDAR Mapping and Location [35]; 3D Mapping [36,42]
Path Planning	Flow Estimation [29]; Point-to-Point Navigation [34]; Control-Aware Prediction [37]; Planning Near-Accident Driving Scenarios [45]; Safety Trajectory Generation [58]; Driver’s Target Trajectory [54]; Interactive Trajectory Prediction [60]; Lane-Change Styles Classification [64]
Control	Generating Control Commands [34]; Control-Aware Prediction [37]; Controlling Vehicle’s Actions [45]; Automated Lane-Change Control [64]; Safety Verification [24]; Interpretable Policies [40]; Behavior Cloning [23]; Chassis Performance [59]; Controlling Vehicle Steering [55]
Decision-Making	Weather Conditions in Decision-Making [56]; Predicted Probability Distribution [34]; Control-Aware Prediction [37]; Switching Between Different Driving Modes [45]; Decision-Marking on Predicted Driver’s Behavior [58]; Driver’s Target Trajectory [54]; Decision-Making on Lane-Changes [64]; Interpretable Policies [40]; Behavior Cloning [23]; Chassis Performance [59]; Decision-Based Dynamic Traffic Conditions [61]; Pedestrian Intent Prediction [43]; Monitoring Duration on Takeover Time [62]
Human–Machine Interaction (HMI)	Sparse View Synthesis and Scene Visualization [22]; Real-Time Feedback based on Driver State [46]; Driving Simulation and User Interaction [64]; Controlling Vehicle Steering [55]; Monitoring Duration and Eye Tracking [62]

.

3.1. Perception

The perception of AVs has evolved significantly, incorporating advanced techniques to handle the complex and dynamic environments encountered during operation. These systems rely on data from various sensors, including cameras and LiDAR. Approaches include using cameras alone, LiDAR alone, or combining both through sensor fusion to enhance perception. Since sensors are essential for determining both the state of the AV and its environment,

Table 7. A summary of common types of sensors used in AVs.

Sensor Type	Placement in Automated Car	Some Exampled Use Cases
Cameras	Front, sides, rear, roof	Lane-keeping, pedestrian detection, object recognition
LiDAR 1	Roof, bumpers, sides	3D object detection, terrain mapping, localization
Radar 2	Front and rear bumpers	Adaptive cruise control, collision detection
Ultrasonic Sensors	Front and rear bumpers, sides	Parking assistance, close-range obstacle detection
GPS	Roof, dashboard	Route planning, navigation, localization
IMU	Integrated in-vehicle systems	Stabilization, motion tracking, localization
Odometry Sensors	Wheels or chassis	Localization, motion planning, distance tracking
V2X 3 Sensors	Roof, exterior antennas	Traffic management, safety alerts, cooperative driving
Infrared (IR) Sensors	Front bumper, roof	Night vision, obstacle detection in low visibility
Magnetic Sensors	Bottom of the vehicle	Lane-keeping in autonomous shuttles
Barometric Pressure Sensors	Inside vehicle sensor suite	Altitude measurement, terrain planning
Laser Rangefinders	Front and rear of the vehicle	Object detection, parking assistance
Proximity Sensors	Front and rear bumpers	Parking, collision avoidance
Environmental Sensors	Exterior, often on the roof	Adjusting driving in response to weather

¹ LiDAR: light detection and ranging; ² Radar: radio detection and ranging; ³ V2X: vehicle-to-everything.

provides an overview of the different sensors integrated into autonomous vehicles.

Building on this foundational sensor data, key areas of focus in perception include segmentation [1,2,38], which divides visual data into meaningful parts, enabling the identification and localization of objects. Street-view change detection [25] enhances these systems by recognizing environmental alterations that could impact navigation. Monocular depth estimation [21,26,30,39] helps the system understand the 3D structure of a scene using just one camera, which is especially important when other sensors, like LiDAR, are not available.

Innovations such as sparse view synthesis [22] and rigorous calibration methods [28,57] ensure that sensor data are both accurate and comprehensive, aiding in reliable object detection and scene interpretation. Multi-object tracking [32], 3D object detection [31,63], and LIDAR-based flow estimation in bird’s eye view (BeV) [29] enhance obstacle state estimation by accurately predicting the dynamic state of objects from consecutive point cloud data, enabling AV systems to maintain situational awareness and predict potential hazards in real time.

Advanced techniques like ego-motion estimation [27], occupancy prediction [33], and visual odometry [44] further enhance the vehicle’s understanding of its surroundings, ensuring robust navigation even in challenging conditions. Additionally, the detection of driver alertness [46], pedestrian locomotion [47,63], and traffic signals [51,57] contribute to the system’s ability to interact safely and effectively with human operators and other road users.

By recognizing and adapting to environmental conditions [56] and road surface features [52], these systems can adjust their behavior to ensure safe operation. However, the reliability of perception systems remains paramount, with ongoing research addressing challenges such as sensor noise, adverse weather, poor lighting, and high traffic density, all of which can affect the accuracy and safety of autonomous driving systems.

3.2. Localization and Mapping

Localization ensures that an AV accurately identifies its position in the environment. This is typically achieved by fusing data from a global positioning system (GPS), inertial measurement units (IMUs), and visual-based sensors, such as LiDAR and cameras. Localization approaches generally fall into two categories: simultaneous localization and mapping (SLAM), which performs localization and mapping concurrently, and offline mapping, where the map is constructed separately. Mapping, essential for creating detailed HD maps, can be classified into online and offline approaches. LiDAR remains a primary sensor for HD map generation, although vision-based methods also contribute through visual SLAM and deep learning techniques. HD map generation typically involves collecting and aligning point clouds, labeling map elements, and frequent updates, making it labor intensive.

Updating and maintaining maps [25] is necessary to ensure that the system’s understanding of its environment remains accurate over time, especially in dynamic settings. Techniques such as depth-aware mapping [26] integrate depth information from sensors like LiDAR to enhance map detail and accuracy, while multi-camera approaches [28] leverage data from multiple cameras to create comprehensive map representations.

Ego-motion estimation [27] updates the vehicle’s position on the map and is further refined using probabilistic localization methods [34] that estimate the most likely position based on sensor data. Similarly, creating and updating occupancy maps [33], which delineate occupied and accessible spaces, is vital for safe navigation and obstacle avoidance. Visual odometry [41,44] estimates movement through the environment, complementing other mapping techniques.

Advanced mapping methods that integrate global navigation satellite system/inertial navigation system–real-time kinematic (GNSS/INS-RTK) positioning [48] offer high-precision localization. Different map representations [53]—such as grid maps and topological maps—affect how the system navigates and localizes itself in various scenarios.

Additionally, generating 2.5D maps using LiDAR and Graph SLAM [49] provides a simplified yet effective 3-dimensional representation of the environment, particularly useful in multilevel environments [50], like parking garages, for precise vehicle localization. The dual process of map generation and localization [57] underpins the system’s ability to navigate effectively, with 3D LiDAR mapping [35] offering detailed environmental scans that support robust localization even in complex terrains. Comprehensive 3D mapping frameworks [36,42] integrate these diverse techniques, facilitating the creation and continuous updating of maps critical for autonomous navigation.

3.3. Path Planning

Path planning in autonomous systems involves generating safe and efficient trajectories for navigating complex environments. Control-aware prediction [37] enhances this process by considering vehicle dynamics and control constraints when predicting future states and planning paths. Flow estimation [29] also plays a role in understanding traffic patterns and the movement of other vehicles, which is crucial for effective path planning and collision avoidance.

In scenarios where potential accidents are a concern, planning near-accident driving scenarios [45] ensures that the vehicle can respond appropriately to sudden environmental changes or hazards. Point-to-point navigation [34] calculates optimal routes between specific locations, ensuring the vehicle reaches its destination efficiently.

Safety trajectory generation [58] is a crucial aspect of path planning, aiming to create trajectories that minimize risk and enhance overall safety. This involves considering the movements of the vehicle and other road users to avoid collisions and ensure smooth operation. The driver’s target trajectory [54] provides a reference for planning and aligning the vehicle’s trajectory with the expected path of human drivers.

Interactive trajectory prediction [60] involves adapting the planned trajectory based on real-time interactions with other road users, such as adjusting to changes in traffic conditions or the actions of nearby vehicles. Additionally, lane-change style classification [64] helps the system understand different lane-change behaviors, which can inform planning strategies and improve the overall driving experience. Together, these advanced methods address the challenges of real-time decision-making and long-term prediction, significantly enhancing both path and trajectory planning in autonomous driving systems.

3.4. Control

Control in autonomous systems encompasses various strategies and technologies to ensure the vehicle operates safely and effectively. Safety verification [24] is crucial for validating that the control systems adhere to safety standards and prevent dangerous behaviors. Control-aware prediction [37] is also a key strategy that integrates vehicle dynamics into the prediction process. This ensures that the control commands are not just realistic but also feasible, given the vehicle’s capabilities and constraints, resulting in a better alignment of the planned trajectories with the vehicle’s control limits.

Interpretable policies [40] are also a significant aspect of control strategies. These policies are designed to be transparent and understandable, fostering better analysis and trust in the decision-making process. They guide the vehicle’s actions in different situations, making the control system more robust and reliable.

Additionally, controlling a vehicle’s actions [45] focuses on implementing control strategies to execute the planned maneuvers, ensuring that the vehicle performs the desired actions accurately. This includes generating control commands [34] that translate high-level plans into specific actions for the vehicle. Behavior cloning [23] involves training control systems using data from human drivers to replicate their driving behaviors, which can enhance the vehicle’s ability to handle complex driving scenarios.

Chassis performance [59] refers to the system’s ability to manage and optimize the vehicle’s physical movements, including steering and acceleration. Controlling vehicle steering [55] is a specific aspect of control that involves adjusting the vehicle’s direction to ensure precise maneuvering. Automated lane-change control [64] involves managing the vehicle’s transitions between lanes, typically integrating with path planning to ensure smooth and safe lane changes. Controlling lateral and longitudinal vehicle movements [61] covers the broader scope of managing both the vehicle’s side-to-side and forward/backward movements, which is essential for maintaining desired trajectories and ensuring smooth operation.

In summary, effective control strategies integrate predictive models, transparent decision-making, and precise maneuver execution to enhance AVs’ overall safety and performance.

3.5. Decision-Making

Decision-making in autonomous systems involves complex processes to ensure safe and effective driving by interpreting data and predicting outcomes. Control-aware prediction [37] integrates vehicle dynamics into the decision-making process, helping ensure that decisions are feasible and align with the vehicle’s capabilities. Pedestrian intent prediction [43] enables the vehicle to make informed decisions to avoid potential collisions and ensure safety. This involves analyzing pedestrian movements and predicting their future actions.

Moreover, switching between different driving modes [45] allows the vehicle to adapt to different driving conditions and scenarios, enhancing its flexibility and responsiveness. This capability is vital for transitioning between manual and autonomous driving modes or different levels of automation. Behavior cloning [23] leverages data from human drivers to replicate their decision-making processes, providing a basis for the vehicle to handle complex driving scenarios by mimicking human behavior. Another technique, predicted probability distribution [34], helps evaluate the likelihood of various outcomes based on current data, aiding in making more informed decisions by assessing potential risks and benefits.

Weather conditions in decision-making [56] involve incorporating environmental factors such as rain, snow, or fog into the decision-making process, ensuring that the vehicle can adapt to changing weather conditions and maintain safe operation. Additionally, decision-making on predicted driver behavior [58] focuses on anticipating and responding to the behavior of other drivers, enhancing the vehicle’s ability to interact safely and effectively in mixed-traffic scenarios. The driver’s target trajectory [54] aligns the vehicle’s actions with the driver’s intended path, ensuring that the vehicle’s decisions support the overall driving goals.

Monitoring the duration of takeover time [62] involves assessing the time required for a driver to take control of the vehicle from autonomous mode, which is crucial for ensuring a smooth transition and maintaining safety. Additionally, decision-making on lane changes [64] encompasses strategies for safely and efficiently changing lanes, integrating with path planning and control systems to execute smooth lane transitions. Decision-making based on dynamic traffic conditions [61] involves adapting decisions based on real-time traffic data, ensuring that the vehicle can effectively navigate through varying traffic scenarios.

Despite advancements, decision-making in autonomous systems faces challenges such as accurately predicting complex driver behaviors, adapting to dynamic and adverse conditions, and ensuring seamless transitions between different driving modes, all of which are critical for enhancing overall system reliability and safety.

3.6. Human–Machine Interaction (HMI)

HMI in semi-autonomous systems optimizes communication between the driver and the vehicle, improving safety and usability. In driver assistance systems (Levels 2–3), the vehicle relies on driver inputs for tasks like steering and monitoring the environment. Technologies such as sparse view synthesis and scene visualization [22] help enhance the vehicle’s perception of its surroundings, which is then communicated to the driver through real-time feedback systems [46]. This feedback includes driver state monitoring to ensure the driver is engaged, enabling timely interventions when needed. Furthermore, eye tracking and monitoring duration [62] provide insights into driver attention patterns to refine interaction interfaces.

In fully autonomous systems (Levels 4–5), the HMI’s main role is to keep the passenger informed about the route, the vehicle’s status, and any important conditions. Since the vehicle handles all driving tasks, the passenger does not need to take control. Instead, the HMI focuses on providing clear updates, such as navigation and system performance. Studies involving driving simulations and user interactions [64] help improve these interfaces, making sure they are easy to use and provide passengers with the information they need in a simple and intuitive way.

4. Knowledge Graphs and Ontologies

KGs are pivotal in structuring and representing knowledge through graph formats, where nodes represent entities and edges capture relationships between them. According to Hogan et al. [65], a KG is “a graph of data with the objective of accumulating and conveying real-world knowledge”, with entities being either tangible objects or abstract concepts and relationships indicating how these entities are connected. KGs utilize factual triples (e.g., Albert Einstein, WinnerOf, Nobel Prize), formalized under the Resource Description Framework (RDF). Formally, a KG is represented as G = (H, R, T), where H denotes a set of entities, T includes entity-literal pairs, and R represents the relationships connecting these elements.

Different graph models can be used to construct KGs, including directed labeled graphs, which feature nodes connected by directed, labeled edges to represent binary relationships intuitively; hyper-relational graphs, which are directed multigraphs with nodes and edges that can have associated key–value pairs for more complex representations; and hypergraphs, which extend the concept of binary edges to model multiple and complex relationships, including nested graphs within hypernodes [14]. These models provide flexible frameworks for capturing diverse data structures.

Building on these fundamental structures, knowledge graph embeddings (KGEs) convert the symbolic information in KGs into low-dimensional vectors that reflect the semantic meanings of entities and relationships [14]. Unsupervised KGE methods generate embeddings based on the inherent structure and attributes of the KG, using approaches such as statistical relational learning and embedding techniques [66,67]. In contrast, supervised KGE methods optimize embedding for specific tasks using labeled data and include techniques such as graph neural networks (GNNs), graph convolutional networks (GCNs), and graph attention networks (GATs) [68,69]. Recent surveys, including Monka et al. [70], illustrate the growing interest in integrating KGs with machine learning methods, highlighting their applications in fields like automated driving. Thus, while the graph models provide the structure and organization of knowledge, KGEs enhance its practical application by translating this structured data into meaningful representations for different tasks.

In the context of KGs, ontologies provide a formal schema that defines the types of entities and relationships within a specific domain. Philosophically, ontology refers to the study of categories and their interrelations, but in computer science, it represents a formal and explicit specification of a shared conceptualization [14]. Ontologies address semantic heterogeneity and enable interoperability by establishing a common understanding of terms and their interrelationships. In the domain of automated driving (AD), ontologies are essential for modeling and integrating knowledge related to vehicles, drivers, routes, and driving environments. They ensure consistent interpretations of concepts such as vehicle types, driver behaviors, and traffic scenarios.

Building upon this base, specific ontologies tailored to different aspects of AD provide structured frameworks for integrating domain-specific knowledge. For example, Vehicle Model Ontologies enhance data interoperability by representing vehicle-related concepts, including types, components, and sensors. Driver Model Ontologies focus on characterizing driver behaviors and styles, supporting driver assistance systems, while Driver Assistance Ontologies underpin decision-making and information sharing related to driver assistance features and human–machine interactions. Additionally, Routing and Context Model Ontologies facilitate route planning and contextual modeling by addressing elements like road geometry, traffic signs, and environmental conditions. Cross-Cutting Ontologies encompass different aspects of driving automation, such as different levels of automation, risk assessment, and interactions with human–driven vehicles. Collectively, these ontologies promote semantic consistency across systems and enhance the capabilities of AD systems by enabling the comprehensive integration of complex scenarios and diverse data sources [14].

4.1. Knowledge Graphs Integrated into AV Technologies

KGs have been increasingly recognized for their crucial role in autonomous driving, particularly in enhancing situation comprehension. The integration of KGs into these domains provides a structured approach to handling and interpreting complex driving scenarios and sensor data, thereby improving the decision-making process of autonomous vehicles. We have been presented with Section 3, in which foundational understanding sets the stage for exploring the challenges and research opportunities that KGs can address when integrated into AV technologies. In the following subsection, we delve into KGs’ contributions to AVs from 12 papers. These contributions highlight how KGs enable more efficient and explainable models, ultimately leading to safer and more reliable AV systems.

Table 8. Key contributions of knowledge graphs in AV applications.

Research Focus	Approach	Key Contributions
Scene Representation	CoSI [71]	Integrates heterogeneous sources into a unified KG structure for situation classification, difficulty assessment, and trajectory prediction using GNN architecture.
	roadscene2vec [72]	Generates scene graphs for risk assessment, collision prediction, and model explainability.
	Semantic Scene Graph [73]	Captures traffic participants’ interactions and relative positions.
	nSKG [74]	Represents scene participants and road elements, including semantic and spatial relationships.
Object Tracking	3D multi-object tracking [75]	Graph structures integrate detection and track states to improve 3D multi-object tracking accuracy and stability.
Road Sign Detection	KGs with VPE [76]	Combines KGs with variational prototyping encoder (VPE) for improved road sign classification and accurate annotation.
Scene Graph-Augmented Risk Assessment	Scene graph sequence [77]	Scene graphs with multi-relation GCN, LSTM, and attention layers assess driving maneuver risks, improving object recognition and scene comprehension.
Scene Creation	Ontologies [78]	Ontologies model expert knowledge for generating diverse traffic scenes and enhancing scenario creation for AV testing.
	AGO [79]	Automotive global ontology (AGO) as a knowledge organization system (KOS) for semantic labeling and scenario-based testing.
Lane Graph Estimation	LaneGraphNet [80]	Estimates lane geometry from BEV images by framing it as a graph estimation problem.
	TopoNet [81]	Uses a scene graph neural network to model relationships in driving scenes, understanding traffic element connections.

provides an overview of KGs’ contributions to AV applications.

4.1.1. Scene Representation

The research effort [71] on context and situation intelligence (CoSI) introduces a KG-based framework aimed at enhancing situation comprehension in driving scenarios, as illustrated in Figure 3. This framework integrates diverse information sources—such as driver state, destination, personal preferences, and the surrounding environment—into a unified KG structure. Figure 3 provides an excerpt of the CoSI knowledge graph (CKG), demonstrating how environmental information is represented through instances (assertional box) of ontological concepts (terminological box). The CoSI ontology, which underpins this KG, models the key aspects as follows:

• Scene: Refers to a snapshot of the environment, including both static and dynamic elements, as well as the self-representations of actors and observers and the relationships among these entities.
• Situation: Represents the complete set of circumstances considered when choosing an appropriate behavioral pattern at a specific moment. It includes all relevant conditions, options, and factors influencing behavior.
• Scenario: Describes the progression over time across multiple scenes, including actions, events, and goals that define this temporal development.
• Observation: Involves the process of performing a procedure to estimate or determine the value of a property of a feature of interest.
• Driver: A user with attributes specific to the driving context.
• Profile: Structured representation of user characteristics.
• Preference: A concept used in psychology, economics, and philosophy to describe a choice between alternatives. For instance, a person shows a preference for A over B if they would opt for A rather than B.

This is a great example of ontology, which helps integrate and interpret both sensor data and driver-related information, including preferences and abilities. By representing this information as entities and their inter-relationships and incorporating semantic axioms, the KG enables advanced reasoning and inference capabilities. The use of axiomatic rules and KG embedding techniques in a GNN architecture has shown significant improvements in situation classification, difficulty assessment, and trajectory prediction.

Additionally, Wickramarachchi et al. [82] highlight the benefits of knowledge graph embeddings (KGEs) in facilitating neuro-symbolic fusion. This approach improves the predictive performance of machine learning models in autonomous driving by integrating symbolic reasoning with neural network capabilities.

Further, Wang et al. [83] presented an approach to predicting pedestrian trajectories. As illustrated in Figure 4, the framework is composed of two main modules: the spatio-temporal interaction aware module and the trajectory distribution aware module, followed by a trajectory decoder. First, the spatio-temporal interaction-aware module captures both spatial and temporal relationships between pedestrians. This module includes spatial self-attention, which focuses on the most relevant pedestrian interactions at each time step, and a graph convolutional network (GCN) that models pedestrian interactions by treating them as nodes in a graph, with edges representing interactions between them. Additionally, it employs a temporal asymmetric network to process the temporal evolution of trajectories, with a stronger emphasis on recent movements, and a temporal self-attention mechanism to highlight the most critical time steps. The module also incorporates a spatial asymmetric network, which accounts for the unequal influence pedestrians have on each other based on their positions and movements. Residual connections ensure that important features are preserved as they pass through the model. Second, the trajectory distribution aware module handles uncertainty in trajectory prediction by encoding both observed trajectories (past movements) and ground truth trajectories (actual future paths). This allows the model to capture the underlying distribution of possible future movements, enabling it to generate potential future trajectories that reflect the inherent variability in pedestrian behavior. Lastly, the trajectory decoder uses a combination of the temporal asymmetric CNN (TA-CNN), which focuses on the temporal dynamics of predicted trajectories, and the trajectory embedding CNN (TE-CNN), which converts the learned features into spatial coordinates. This results in the generation of multiple plausible future trajectories, offering possible pedestrian paths. By modeling both spatio-temporal interactions and trajectory distributions, the framework enhances the understanding of pedestrian movement in complex environments, improving decision-making and planning.

Road scene-graph representations, when combined with graph learning techniques, have recently surpassed deep learning methods in action classification, risk assessment, and collision prediction. Ref. [72] introduced roadscene2vec, an open-source tool designed to facilitate research into road scene-graph applications. roadscene2vec provides capabilities for generating scene graphs from video clips or CARLA (Car Learning to Act) simulator data, creating spatio-temporal embeddings with various models, and visualizing and analyzing scene graphs. It supports risk assessment, collision prediction, transfer learning, and model explainability evaluation. Similarly, Zipfl et al. [73] introduced a semantic scene graph model where traffic participants are represented as nodes, and their relationships are captured as semantically classified edges. This model provides a structured way to describe traffic scenes beyond just the road geometry, focusing on traffic participants’ interactions and relative positions. Using graph-based representations for dynamic objects in traffic scenes, such as pedestrians and vehicles, highlights an approach to organizing and interpreting complex scene data.

In addition, the SemanticFormer [84] approach uses a semantic traffic scene graph to represent and process high-level information about traffic participants, road topology, and traffic signs. The ontologies of the traffic scene are shown in Figure 5. The ontology depicted in the figure represents a comprehensive semantic framework for modeling traffic scenes, particularly designed for applications such as trajectory prediction, behavior analysis, and decision-making in autonomous driving. This structured KG integrates static and dynamic entities in a traffic scene, capturing their relationships and interactions over time. The graph follows a modular approach, where different components represent the environment (road structures, lanes, intersections) and participants (vehicles, pedestrians, obstacles). These components, when interconnected through predefined relationships, form a foundation for understanding and predicting real-world driving scenarios.

The elements in the ontology include the EgoVehicle, representing the autonomous vehicle navigating the scene, and Participants, which encapsulate all dynamic entities, such as Humans, Vehicles, and MovableObjects. The human participants are further subclassed into entities like Children, Adults, and Police Officers, reflecting their distinct behaviors and interactions in a traffic environment. Similarly, vehicles are divided into types like Cars, Bicycles, Trucks, and Motorcycles, capturing the diverse range of interactions and speeds these entities exhibit on the road. In addition to dynamic entities, the ontology models static objects such as RoadSegments, LaneDividers, Intersections, and CarparkAreas. These elements define the structure of the road environment and are essential for path planning and navigation. Relationships like switchVia or hasNextLaneSnippet model how vehicles can move between lanes, offering information into potential lane-switching behaviors or turns at intersections. The inclusion of StaticObjects, such as BicycleRacks or TrafficCones, also ensures that the autonomous vehicle can account for obstacles that may affect route planning or cause obstructions.

Geometric and positional data, represented through entities like Point and Polygon, offer spatial information about road elements and participants. This information is crucial for defining the location and shape of lanes, intersections, or other key objects in the scene. The integration of Geometry into the ontology allows the autonomous vehicle to map its environment and understand the spatial constraints in real time. The temporal aspect of the ontology is managed through elements like Scene and Sequence. A Scene represents a snapshot of the traffic environment, containing all entities and their relationships at a given moment. Multiple scenes can be grouped into a Sequence, forming a temporal chain that models the evolution of a traffic situation over time. This temporal structure is critical for predicting the future trajectories of participants, as the system can understand not only the current state of the environment but also how it is likely to change. In summary, this ontology provides a structured framework that captures the complexities of real-world traffic environments, integrating spatial, temporal, and relational data. Modeling both the static infrastructure and the dynamic behaviors of participants serves as a foundational tool for enhancing the decision-making capabilities of autonomous vehicles, enabling them to navigate safely and effectively in diverse and unpredictable scenarios.

Additionally, [74] introduces an approach using KGs to model various entities and their semantic connections within traffic scenes. It presented the nuScenes knowledge graph (nSKG), which explicitly models scene participants and road elements, including their semantic and spatial relationships. Similarly, Sun et al. [85] investigate the effects of spatial resolution, the relationship between graphs and trajectory predictions, and methods for embedding knowledge into graphs.

Urbieta et al. [79] introduced and formalized knowledge-based entity prediction (KEP), which improves scene understanding by predicting potentially missing entities using a knowledge-infused learning approach. The proposed solution includes (1) a dataset-agnostic ontology for describing driving scenes, (2) a comprehensive scene representation with knowledge graphs, and (3) a novel mapping of KEP to link prediction (LP) using KGE. Evaluations are performed with real urban driving data.

4.1.2. Object Tracking

Focused on 3D multi-object tracking, Zaech et al. [75] use graph structures to integrate detection and track states, improving tracking accuracy and stability. This approach aligns with enhancing scene understanding through dynamic object tracking. The learning-based graph approach integrates object detections and tracks, creating a unified representation of the environment that supports effective scene understanding.

4.1.3. Road Sign Detection

Accurate road sign annotation is crucial for AI-based road sign recognition (RSR) systems but is often hindered by annotators’ difficulties with diverse road sign systems. Ref. [76] propose a novel method combining knowledge graphs with a machine learning algorithm—variational prototyping encoder (VPE)—to enhance road sign classification. Annotators use the road sign knowledge graph to query visual attributes, receiving candidate suggestions from the VPE model.

4.1.4. Scene Graph Augmented Risk Assessment

Despite significant autonomous driving progress, navigating complex road conditions remains challenging. There is notable evidence that assessing the subjective risk level of different decisions can improve AD safety in normal and complex driving scenarios. Traditional deep learning methods often fail in modeling traffic interactions and need more explainability. Thus, [77] proposes a novel approach using scene graphs with a multi-relation graph convolution network, long-short term memory network, and attention layers to assess driving maneuver risks. By leveraging KGs, this research demonstrates how KGs can enhance object recognition and scene comprehension, aligning with scene graphs to model relationships between traffic participants. This integration supports improved object detection algorithms and a more comprehensive understanding of driving environments.

4.1.5. Scene Creation

To address the challenges of identifying diverse scenarios, Bagschik et al. [78] proposed the use of ontologies to model expert knowledge and generate a wide array of traffic scenes. Ontologies facilitate the creation of detailed and varied scenarios by translating specialist knowledge into structured formats that can be used for computer-aided processing. This approach enhances the scenario creation process, ensuring that automated vehicles undergo rigorous testing across a comprehensive set of scenarios, contributing to their safety and development. Furthermore, [79] proposed the automotive global ontology (AGO) as a knowledge organization system (KOS) implemented with the Neo4j graph database. The AGO is demonstrated through two use cases—semantic labeling and scenario-based testing.

4.1.6. Lane Graph Estimation for Urban Driving

Lane-level scene annotations are crucial for trajectory planning in autonomous vehicles but are labor intensive and costly. Zurn et al. [80] propose a novel method for estimating lane geometry from bird’s-eye-view images by framing it as a graph estimation problem. Lane anchor points are represented as graph nodes, and lane segments as graph edges. The model, trained on multimodal data from the NuScenes dataset, estimates lane shapes and connections, resulting in a directed lane graph. The LaneGraphNet model demonstrates strong performance on urban scenes in the NuScenes dataset, offering a promising step towards automated HD lane annotation. Traditional methods need help with lane connectivity and often overlook interaction modeling, while traffic element-to-lane assignments remain limited to the image domain. To address these challenges, Li et al. [81] introduced TopoNet, which uses a scene graph neural network to model relationships in driving scenes. This end-to-end framework abstracts traffic knowledge and understands the connections between traffic elements and lanes beyond traditional perception tasks. Figure 6 shows how TopoNet built the connections between traffic elements and lanes.

4.2. Challenges and Potential Solutions in Existing Work

In this section, we outline the limitations identified in integrating KGs and semantic technologies as presented in Section 4.1, followed by potential solutions and directions for future research.

4.2.1. Maturity of Semantic Technologies

Semantic technologies, which enable machines to understand and reason with human language, have made significant strides. However, their integration into automotive applications still presents challenges, particularly regarding performance and interoperability. In AVs, processing large amounts of data in real time remains an issue, and the task of merging different systems and data sources, such as sensors and maps, is complicated by varying formats and standards. Triple stores, specialized databases for storing and querying data relationships, play a key role in addressing these challenges [71]. The balance between virtual data access, which allows for flexible, on-the-fly queries, and materialization, which speeds up queries by pre-storing data, is crucial for optimizing performance. Advances in triple stores are helping to improve query efficiency, but further research is needed to refine these techniques. Future work needs to focus on optimizing this balance to more effectively handle the vast and complex data involved in AD.

4.2.2. Knowledge Graph Embeddings (KGE) and Data Preparation

Preparing data for KGE can be complex, particularly when information is dispersed across multiple relationships [71,77]. Special queries and the creation of views can help manage this complexity, but more advanced methods are needed. Future research should explore automated techniques for optimizing KGE in dynamic, multi-relational contexts. Additionally, current datasets often lack the volume and variety necessary for complex semantic inferences. Expanding datasets to include risk and accident scenarios and employing advanced methods like scene captioning and risk classification could address these limitations. Scalability remains a concern as datasets grow larger, requiring more efficient data processing methods.

4.2.3. Handling Long Frame Gaps and Occlusions in Tracking

Tracking systems often struggle with long frame gaps and occlusions, leading to ID switches and false positives. Developing adaptive methods that intelligently manage these trade-offs is crucial for improving tracking continuity. Advanced occlusion models and temporal context-aware algorithms present promising avenues for AV research [75].

4.2.4. Intersection and Lane Detection Challenges

Complex intersections and lane configurations present significant challenges for lane detection models, often leading to inaccuracies. Enhancing these models with additional contextual information (traffic signal and sign information, road markings and signage, surrounding vehicle behavior, GPS and map data, road geometry), improving data augmentation techniques, and expanding training datasets to cover a broader range of scenarios can improve model robustness in these challenging environments.

4.2.5. Expanding Predictive Capabilities and Scalability

While current predictive systems excel in specific scenarios like “pedestrian crossing” and “lane changes”, they fall short in more complex situations, such as near-miss maneuvers. Expanding predictive models to include a wider range of use cases, particularly in diverse cultural contexts, is necessary for more comprehensive behavior prediction in autonomous vehicles. Integration with AV behavior planners is a critical next step [72]. Additionally, the scalability of deploying GNNs on large graphs, as seen in dataset-specific ontologies, is another critical challenge [74]. Techniques to manage the increased computational complexity and memory requirements associated with large graphs are necessary to fully realize the potential of GNNs in AV applications.

4.2.6. Validation of Automated Driving Systems

One of the primary challenges in validating highly automated driving systems is the impractical requirement of millions of test kilometers to cover critical situations [71]. This challenge can be addressed by utilizing simulation data for validation, which is expected to generalize well to real-world scenarios. However, the effectiveness of simulation data needs further exploration, such as validation with real-world data, transfer learning techniques, and human-in-the-loop testing, to ensure reliability in diverse and unpredictable real-world conditions.

5. Discussion for Ethical and Practical Considerations in AV Technologies

In this section, we discuss ethical and practical considerations in AV technologies.

5.1. Challenges of Knowledge Graphs in AVs

As we presented in Section 3 through Section 4, KGs are known for their ability to represent complex information in a way that helps humans understand relationships between objects and situations. However, their exact role in helping AV systems recognize and make decisions is still debated. While KGs can improve decision-making by organizing information about the environment and objects, it is unclear how essential or effective they are when it comes to the fast, real-time decisions that AV systems need to make.

Recent research has explored how KGs can serve as valuable tools for contextual enrichment in AV systems, particularly in areas such as environmental perception and situation awareness. For instance, KGs have been employed to improve the semantic understanding of traffic environments, support sensor data fusion, and provide a knowledge base for real-time decision-making. However, integrating KGs into AV systems must balance improving decision accuracy with maintaining the processing speed needed for safe operations. For instance, KGs can help an AV interpret “traffic signs” or predict “pedestrian behavior” in contexts like “construction zones” or “school areas”, but this information must be processed quickly enough for the vehicle to respond in real time, such as “stopping for a crossing pedestrian”. KGs may not serve as a standalone solution but rather as a component within a broader decision-making framework, particularly in machine learning pipelines where they enhance prediction accuracy and contextual reasoning.

Future research should focus on evaluating the real-time performance of KGs in AV decision processes and determining whether they offer tangible benefits compared to machine learning approaches. Key questions include how well KGs scale with the complexity of AV environments and how they integrate with high-performance machine learning algorithms that enable fast decision-making.

5.2. Ethical Decision-Making in AVs: Moving Beyond Human Analogy

One of the critical aspects of AV decision-making lies in its ethical dimension. Unlike human drivers, who rely on intuition and often make decisions under time pressure with limited information, AVs are pre-programmed to follow specific rules and logic. However, this raises an important concern: how are AVs programmed to make ethically sound decisions in real-world scenarios, particularly when their perception of the environment differs fundamentally from that of humans?

In AVs, decision-making algorithms need to consider the limitations of machine perception, which depends on sensors like cameras, lidar, and radar. Unlike human senses, which directly interpret the environment, these sensors generate data streams that the AV must process. Because machines perceive the world differently from humans, the ethical frameworks guiding AV decisions need to reflect the specific strengths and weaknesses of these sensors. Instead of just copying human decision-making, the ethical guidelines for AVs should be tailored to how these machines actually see and understand their surroundings.

Recent incidents in AV development, such as the Boeing MCAS (maneuvering characteristics augmentation system) disaster [86], underscore the risks of poorly designed decision-making systems that fail to account for complex, real-world situations. In AVs, ethical decisions should be tightly coupled with safety regulations, focusing on minimizing harm while adhering to legal standards. The debate around whether AVs should be programmed to make difficult ethical choices (e.g., who to protect in an unavoidable accident) often leads to the spread of “fake ethics” that are not grounded in the practical realities of AV technology.

To mitigate such concerns, developing decision-making algorithms that prioritize risk avoidance and minimize harm is crucial, rather than only replicating human ethical dilemmas in AV systems. Furthermore, the ethical reasoning framework of AVs should be transparent and accountable, ensuring that decisions can be traced and reviewed in the event of an incident.

5.3. Addressing Fake Ethics in AVs

In discussions about ethics and AVs, speculative scenarios often emerge, such as an AV having to choose between “saving a mother and a child” or “avoiding a cat”. These hypothetical situations, while interesting in ethical debates, do not reflect the real operational challenges AVs face. Such scenarios lead to “fake ethics”, which distract from the real goal of AV development—preventing accidents and ensuring safe driving.

For example, human drivers are expected to follow traffic laws and avoid dangerous situations, like speeding through a crowded intersection. AVs should be programmed the same way: to avoid risks entirely rather than needing to make moral decisions in split seconds. The focus should be on designing AV systems that can recognize and react safely to potential hazards before they escalate into accidents. Take a real-world case: instead of asking an AV to choose between hitting a pedestrian or swerving into oncoming traffic, the goal should be to avoid either scenario in the first place by programming the vehicle to slow down when approaching a crowded area. This proactive approach avoids the need for moral decision-making altogether. Additionally, concerns about AVs being programmed to take risks that human drivers would avoid, such as speeding through yellow lights to save time. Rather than forcing AVs to make risky decisions, developers should ensure that AVs always prioritize safety, such as stopping at yellow lights even when they could technically make it through. Ultimately, the ethical focus for AVs should be on preventing accidents and minimizing harm, not on programming them to make difficult moral choices in high-pressure moments.

5.4. Accountability in AV Decision-Making

A critical concern raised in AV ethics is accountability—specifically, who or what is responsible when an AV makes a decision that results in harm or an accident. Unlike human drivers, who may act unpredictably or make judgment calls, AVs operate under strict programming, meaning that every decision can be traced back to the algorithms that govern their behavior.

There are growing concerns that if AVs are programmed to take risks that human drivers would not be expected to take, the responsibility for accidents could shift toward the manufacturers or software developers. For example, if an AV is designed to prioritize the safety of its passengers over pedestrians in certain scenarios, such programming could be interpreted as premeditated risk-taking, raising legal and ethical questions about liability.

To address these concerns, regulations must ensure that AVs follow responsible driving behaviors similar to the standards and legal rules that apply to human drivers. Developers of AV systems need to be responsible for making sure their algorithms do not create unnecessary risks on the road, such as making unsafe lane changes or failing to prioritize emergency vehicles. Additionally, AV decision-making processes should be reviewed and checked regularly. In addition to legal responsibility, there is also a need for AVs to be equipped with fail-safe mechanisms that can respond appropriately when unexpected situations arise. This could include handing control back to a human driver or executing predefined safety maneuvers when the system cannot confidently make an ethically sound decision.

6. Conclusions

This paper has surveyed the integration of knowledge graphs (KGs) into autonomous vehicle (AV) technologies, addressing the research questions outlined in Section 2. In response to the first question—what are the key applications in AV technologies and KGs—we discussed the critical components of AV systems in Section 3, including perception, localization, path planning, and decision-making, and highlighted the significant challenges these systems face in real-world environments. In Section 4, we explored the specific ways in which KGs can enhance these components by providing a structured framework for organizing and interpreting complex environmental data, contributing to improved perception, localization, path planning, and decision-making within AV systems.

For the second question—which aspects of KG integration does the article address—we discussed the application of KGs in AV systems in Section 4, demonstrating how they can enhance perception and decision-making by providing structured and contextualized data from diverse sources. We highlighted that while KGs have the potential to improve AVs’ ability to interpret traffic environments and make more informed decisions, their practical integration still faces challenges, particularly in terms of scalability, robustness, and real-time performance across various driving conditions.

In relation to the third question—what methods do the article discuss for integrating KGs with AV systems—we presented the approaches for KG integration, particularly in enhancing sensor data processing and real-time decision-making. These methods involve employing KGs to provide contextualized data that enhances the performance of existing AV systems. Future work is required to evaluate and refine these methods in diverse and unpredictable driving scenarios.

Lastly, in response to the question—what are the limitations of AVs, and what future research does the article suggest—we discussed the limitations, including the need to improve the scalability and real-time performance of KG-based systems. In Section 5, we also explored the ethical considerations in AV decision-making, emphasizing that AVs should prioritize safety and risk minimization over replicating human ethical reasoning. Transparency and accountability in AV decision-making are crucial, as is ensuring that manufacturers take responsibility for any pre-programmed risks embedded in AV systems.

In conclusion, while the integration of KGs into AV technologies holds great potential for addressing the complexities of perception and decision-making, significant research gaps remain. Enhancing the scalability, robustness, and real-time processing capabilities of KGs is essential to meet the stringent demands of AV systems. Future research should focus on real-world evaluations of KG performance across diverse scenarios, as well as deeper integration with machine learning pipelines. Ethical concerns, particularly those related to safety and accountability, must remain central to the development of AV technologies. By addressing these challenges, KGs could unlock new possibilities for creating more reliable, ethical, and safe autonomous driving systems.