A Survey of Scenario Generation for Automated Vehicle Testing and Validation
Abstract
1. Introduction
2. Terminology
2.1. Operational Design Domain (ODD)
2.2. Description of Scenario
2.3. Scenario Types
3. Scenario Generation Methods
3.1. Non-Adaptive Test Scenario Generation Methods
3.1.1. Knowledge-Based Generation Method
3.1.2. Data-Driven Generation Methods
3.1.3. Scenario Library Generation Method
3.2. Adaptive Test Scenario Generation Methods
3.2.1. Reinforcement-Learning-Based Methods
3.2.2. Importance-Sampling-Based Method
3.2.3. Imitation-Learning-Based Method
4. Discussion
5. Conclusions
Author Contributions
Funding
Conflicts of Interest
References
- National Highway Traffic Safety Administration. 2015 Motor Vehicle Crashes: Overview. Traffic Saf. Facts Res. Note 2016, 2016, 1–9. [Google Scholar]
- European Commission. 2018 Road Safety Statistics: What Is Behind the Figures? European Commission: Brussels, Belgium, 2018. [Google Scholar]
- Ding, W.; Xu, C.; Arief, M.; Lin, H.; Li, B.; Zhao, D. A survey on safety-critical driving scenario generation—A methodological perspective. IEEE Trans. Intell. Transp. Syst. 2023, 24, 6971–6988. [Google Scholar] [CrossRef]
- Koopman, P.; Wagner, M. Autonomous vehicle safety: An interdisciplinary challenge. IEEE Intell. Transp. Syst. Mag. 2017, 9, 90–96. [Google Scholar] [CrossRef]
- Kalra, N.; Paddock, S.M. Driving to safety: How many miles of driving would it take to demonstrate autonomous vehicle reliability? Transp. Res. Part A Policy Pract. 2016, 94, 182–193. [Google Scholar] [CrossRef]
- Zhu, Y.; Wang, J.; Guo, X.; Meng, F.; Liu, T. Functional testing scenario library generation framework for connected and automated vehicles. IEEE Trans. Intell. Transp. Syst. 2023, 24, 9712–9724. [Google Scholar] [CrossRef]
- SAE International. Taxonomy and definitions for terms related to driving automation systems for on-road motor vehicles. SAE Int. 2018, 4970, 1–5. [Google Scholar]
- On-Road Automated Driving (Orad) Committee. Taxonomy & Definitions for Operational Design Domain (ODD) for Driving Automation Systems. 2021. Available online: https://www.sae.org/standards/content/j3259/ (accessed on 15 October 2024).
- Koopman, P.; Fratrik, F. How many operational design domains, objects, and events? In Proceedings of the Safe AI 2019: AAAI Workshop on Artificial Intelligence Safety, Honolulu, HI, USA, 27–28 January 2019. [Google Scholar]
- Czarnecki, K. Operational design domain for automated driving systems. In Taxonomy of Basic Terms; Waterloo Intelligent Systems Engineering (WISE) Lab, University of Waterloo: Waterloo, ON, Canada, 2018. [Google Scholar]
- Thorn, E.; Kimmel, S.C.; Chaka, M.; Hamilton, B.A. A Framework for Automated Driving System Testable Cases and Scenarios; Technical Report; United States Department of Transportation, National Highway Traffic Safety Administration: Washington, DC, USA, 2018. [Google Scholar]
- Zhang, Y.; Sun, B.; Li, Y.; Zhao, S.; Zhu, X.; Ma, W.; Ma, F.; Wu, L. Research on the Physics–Intelligence Hybrid Theory Based Dynamic Scenario Library Generation for Automated Vehicles. Sensors 2022, 22, 8391. [Google Scholar] [CrossRef]
- PEGASUS Project. Scenario Description and Knowledge-Based Scenario Generation. Available online: https://www.pegasusprojekt.de/files/tmpl/Pegasus-Abschlussveranstaltung/05_Scenario_Description_and_Knowledge-Based_Scenario_Generation.pdf (accessed on 12 June 2024).
- Ma, Y.; Jiang, W.; Zhang, L.; Chen, J.; Wang, H.; Lv, C.; Wang, X.; Xiong, L. Evolving testing scenario generation method and intelligence evaluation framework for automated vehicles. arXiv 2023, arXiv:2306.07142. [Google Scholar]
- Schelter, S.; Lange, D.; Schmidt, P.; Celikel, M.; Biessmann, F.; Grafberger, A. Automating large-scale data quality verification. Proc. VLDB Endow. 2018, 11, 1781–1794. [Google Scholar] [CrossRef]
- So, J.J.; Park, I.; Wee, J.; Park, S.; Yun, I. Generating traffic safety test scenarios for automated vehicles using a big data technique. KSCE J. Civ. Eng. 2019, 23, 2702–2712. [Google Scholar] [CrossRef]
- Sagiroglu, S.; Sinanc, D. Big data: A review. In Proceedings of the 2013 International Conference on Collaboration Technologies and Systems (CTS), San Diego, CA, USA, 20–24 May 2013; pp. 42–47. [Google Scholar]
- Rana, A.; Malhi, A. Building safer autonomous agents by leveraging risky driving behavior knowledge. In Proceedings of the 2021 International Conference on Communications, Computing, Cybersecurity, and Informatics (CCCI), Virtual Event, 3–5 March 2021; pp. 1–6. [Google Scholar]
- Puterman, M.L. Markov decision processes. In Handbooks in Operations Research and Management Science; Elsevier: Amsterdam, The Netherlands, 1990; Volume 2, pp. 331–434. [Google Scholar]
- Arulkumaran, K.; Deisenroth, M.P.; Brundage, M.; Bharath, A.A. Deep reinforcement learning: A brief survey. IEEE Signal Process. Mag. 2017, 34, 26–38. [Google Scholar] [CrossRef]
- Arief, M.; Glynn, P.; Zhao, D. An accelerated approach to safely and efficiently test pre-production autonomous vehicles on public streets. In Proceedings of the 2018 21st International Conference on Intelligent Transportation Systems (ITSC), Maui, HI, USA, 4–7 November 2018; pp. 2006–2011. [Google Scholar]
- Zhao, D.; Peng, H.; Bao, S.; Nobukawa, K.; LeBlanc, D.J.; Pan, C.S. Accelerated evaluation of automated vehicles using extracted naturalistic driving data. In Proceedings of the 24th International Symposium of Vehicles on Road and Tracks, Gothenburg, Sweden, 17–21 August 2015. [Google Scholar]
- Zhao, D.; Lam, H.; Peng, H.; Bao, S.; LeBlanc, D.J.; Nobukawa, K.; Pan, C.S. Accelerated evaluation of automated vehicles safety in lane-change scenarios based on importance sampling techniques. IEEE Trans. Intell. Transp. Syst. 2016, 18, 595–607. [Google Scholar] [CrossRef] [PubMed]
- Glynn, P.W.; Iglehart, D.L. Importance sampling for stochastic simulations. Manag. Sci. 1989, 35, 1367–1392. [Google Scholar] [CrossRef]
- Tan, S.; Wong, K.; Wang, S.; Manivasagam, S.; Ren, M.; Urtasun, R. Scenegen: Learning to generate realistic traffic scenes. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, Nashville, TN, USA, 19–25 June 2021; pp. 892–901. [Google Scholar]
- Shi, X.; Chen, Z.; Wang, H.; Yeung, D.-Y.; Wong, W.-K.; Woo, W.-C. Convolutional LSTM network: A machine learning approach for precipitation nowcasting. Adv. Neural Inf. Process. Syst. 2015, 28, 802–810. [Google Scholar]
- Hochreiter, S.; Schmidhuber, J. Long short-term memory. Neural Comput. 1997, 9, 1735–1780. [Google Scholar] [CrossRef]
- Yegnanarayana, B. Artificial Neural Networks; PHI Learning Pvt. Ltd.: Delhi, India, 2009. [Google Scholar]
- O’Shea, K. An Introduction to Convolutional Neural Networks. arXiv 2015, arXiv:1511.08458. [Google Scholar]
- Liu, H.; Zhang, L.; Hari, S.K.S.; Zhao, J. Safety-Critical Scenario Generation via Reinforcement Learning Based Editing. In Proceedings of the 2024 IEEE International Conference on Robotics and Automation (ICRA), Yokohama, Japan, 13–17 May 2024; pp. 14405–14412. [Google Scholar]
- Kaelbling, L.P.; Littman, M.L.; Moore, A.W. Reinforcement learning: A survey. J. Artif. Intell. Res. 1996, 4, 237–285. [Google Scholar] [CrossRef]
- Doersch, C. Tutorial on variational autoencoders. arXiv 2016, arXiv:1606.05908. [Google Scholar]
- Wu, Z.; Pan, S.; Chen, F.; Long, G.; Zhang, C.; Yu, P.S. A comprehensive survey on graph neural networks. IEEE Trans. Neural Netw. Learn. Syst. 2020, 32, 4–24. [Google Scholar] [CrossRef]
- Schulman, J.; Wolski, F.; Dhariwal, P.; Radford, A.; Klimov, O. Proximal policy optimization algorithms. arXiv 2017, arXiv:1707.06347. [Google Scholar]
- Feng, S.; Feng, Y.; Yu, C.; Zhang, Y.; Liu, H.X. Testing scenario library generation for connected and automated vehicles, part I: Methodology. IEEE Trans. Intell. Transp. Syst. 2020, 22, 1573–1582. [Google Scholar] [CrossRef]
- Feng, S.; Feng, Y.; Sun, H.; Bao, S.; Zhang, Y.; Liu, H.X. Testing scenario library generation for connected and automated vehicles, part II: Case studies. IEEE Trans. Intell. Transp. Syst. 2020, 22, 5635–5647. [Google Scholar] [CrossRef]
- Feng, S.; Feng, Y.; Sun, H.; Zhang, Y.; Liu, H.X. Testing scenario library generation for connected and automated vehicles: An adaptive framework. IEEE Trans. Intell. Transp. Syst. 2020, 23, 1213–1222. [Google Scholar] [CrossRef]
- Haklay, M.; Weber, P. Openstreetmap: User-generated street maps. IEEE Pervasive Comput. 2008, 7, 12–18. [Google Scholar] [CrossRef]
- Monahan, G.E. State of the art—A survey of partially observable Markov decision processes: Theory, models, and algorithms. Manag. Sci. 1982, 28, 1–16. [Google Scholar] [CrossRef]
- Kuutti, S.; Bowden, R.; Jin, Y.; Barber, P.; Fallah, S. A survey of deep learning applications to autonomous vehicle control. IEEE Trans. Intell. Transp. Syst. 2020, 22, 712–733. [Google Scholar] [CrossRef]
- Jin, Y. Does level-k behavior imply level-k thinking? Exp. Econ. 2021, 24, 330–353. [Google Scholar] [CrossRef]
- Crawford, V.P.; Iriberri, N. Level-k auctions: Can a nonequilibrium model of strategic thinking explain the winner’s curse and overbidding in private-value auctions? Econometrica 2007, 75, 1721–1770. [Google Scholar] [CrossRef]
- Fujimoto, S.; Hoof, H.; Meger, D. Addressing function approximation error in actor-critic methods. In Proceedings of the International Conference on Machine Learning, Stockholm, Sweden, 10–15 July 2018; pp. 1587–1596. [Google Scholar]
- Ge, J.; Xu, H.; Zhang, J.; Zhang, Y.; Yao, D.; Li, L. Heterogeneous driver modeling and corner scenarios sampling for automated vehicles testing. J. Adv. Transp. 2022, 2022, 8655514. [Google Scholar] [CrossRef]
- Wei, Z.; Huang, H.; Zhang, G.; Zhou, R.; Luo, X.; Li, S.; Zhou, H. Interactive Critical Scenario Generation for Autonomous Vehicles Testing Based on In-depth Crash Data Using Reinforcement Learning. IEEE Trans. Intell. Veh. 2024. [Google Scholar] [CrossRef]
- Sun, H.; Feng, S.; Yan, X.; Liu, H.X. Corner case generation and analysis for safety assessment of autonomous vehicles. Transp. Res. Rec. 2021, 2675, 587–600. [Google Scholar] [CrossRef]
- Roderick, M.; MacGlashan, J.; Tellex, S. Implementing the deep q-network. arXiv 2017, arXiv:1711.07478. [Google Scholar]
- Abdi, H.; Williams, L.J. Principal component analysis. Wiley Interdiscip. Rev. Comput. Stat. 2010, 2, 433–459. [Google Scholar] [CrossRef]
- Krishna, K.; Murty, M.N. Genetic K-means algorithm. IEEE Trans. Syst. Man Cybern. Part B 1999, 29, 433–439. [Google Scholar] [CrossRef]
- Khan, K.; Rehman, S.U.; Aziz, K.; Fong, S.; Sarasvady, S. DBSCAN: Past, present and future. In Proceedings of the the Fifth International Conference on the Applications of Digital Information and Web Technologies (ICADIWT 2014), Bangalore, India, 17–19 February 2014; pp. 232–238. [Google Scholar]
- Chen, B.; Chen, X.; Wu, Q.; Li, L. Adversarial evaluation of autonomous vehicles in lane-change scenarios. IEEE Trans. Intell. Transp. Syst. 2021, 23, 10333–10342. [Google Scholar] [CrossRef]
- Tokdar, S.T.; Kass, R.E. Importance sampling: A review. Wiley Interdiscip. Rev. Comput. Stat. 2010, 2, 54–60. [Google Scholar] [CrossRef]
- Feng, S.; Yan, X.; Sun, H.; Feng, Y.; Liu, H.X. Intelligent driving intelligence test for autonomous vehicles with naturalistic and adversarial environment. Nat. Commun. 2021, 12, 748. [Google Scholar] [CrossRef]
- Hussein, A.; Gaber, M.M.; Elyan, E.; Jayne, C. Imitation learning: A survey of learning methods. ACM Comput. Surv. (CSUR) 2017, 50, 21. [Google Scholar]
- Jia, L.; Yang, D.; Ren, Y.; Qian, C.; Feng, Q.; Sun, B.; Wang, Z. A dynamic test scenario generation method for autonomous vehicles based on conditional generative adversarial imitation learning. Accid. Anal. Prev. 2024, 194, 107279. [Google Scholar] [CrossRef]
- Ho, J.; Ermon, S. Generative adversarial imitation learning. In Advances in Neural Information Processing Systems 29; Curran Associates Inc.: Red Hook, NY, USA, 2016. [Google Scholar]
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2024 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Wang, Z.; Ma, J.; Lai, E.M.-K. A Survey of Scenario Generation for Automated Vehicle Testing and Validation. Future Internet 2024, 16, 480. https://doi.org/10.3390/fi16120480
Wang Z, Ma J, Lai EM-K. A Survey of Scenario Generation for Automated Vehicle Testing and Validation. Future Internet. 2024; 16(12):480. https://doi.org/10.3390/fi16120480
Chicago/Turabian StyleWang, Ziyu, Jing Ma, and Edmund M-K Lai. 2024. "A Survey of Scenario Generation for Automated Vehicle Testing and Validation" Future Internet 16, no. 12: 480. https://doi.org/10.3390/fi16120480
APA StyleWang, Z., Ma, J., & Lai, E. M.-K. (2024). A Survey of Scenario Generation for Automated Vehicle Testing and Validation. Future Internet, 16(12), 480. https://doi.org/10.3390/fi16120480