A Multi-Scale Airspace Sectorization Framework Based on QTM and HDQN
Abstract
1. Introduction
2. Methodology
2.1. Airspace Gridding
2.1.1. Multi-Resolution Airspace Grid Modeling Based on QTM
2.1.2. Workload Gridding
- (1)
- Monitoring Workload Model
- (2)
- Conflict Workload Model
- (3)
- Coordination Workload Model
2.2. Sector Construction Based on DGGS Grid Voronoi Diagrams
2.3. Airspace Sector Optimization Strategy Based on Hierarchical Reinforcement Learning
2.3.1. HDQN Framework for Airspace Sectorization
2.3.2. Action Space and State Space
- (1)
- State Space of the Top-Level DQN
- (2)
- The state space of the bottom-level DQN
- (3)
- The action space of the top-level DQN
- (4)
- The action space of the bottom-level DQN
2.3.3. Reward Function
- (1)
- The reward function of the top-level DQN
- (2)
- The reward function of the bottom-level DQN
2.3.4. Prior Knowledge
- (1)
- Initialization
- (2)
- The plasticity of the reward function
- (3)
- Priority Adjustment
3. Results
3.1. Experimental Environment
3.2. Workload Calculation Efficiency Comparison
3.3. Effectiveness Comparison Experiment
- (1)
- Comparative Experiment between Hierarchical Reinforcement Learning and Single-Layer Reinforcement Learning
- (2)
- Comparative Experiment on Effectiveness and Efficiency between Multi-Scale Grids and Single-Scale Grids
- (3)
- Comparative Experiment with Incorporation of Prior Knowledge
3.4. Temporal Testing and Validation
4. Discussion
4.1. Airspace Design as a Complex System Problem
4.2. Selection of Base Grids
5. Conclusions
5.1. Innovations
- (1)
- The construction of a QTM-based airspace grid; leveraging the properties and coding of grids, the workload calculation model based on airspace grids has been optimized. Experimental results demonstrate that, compared to traditional methods, the grid-based workload calculation model for airspace management exhibits significant advantages in efficiency.
- (2)
- A multi-scale airspace sectorization framework is constructed based on multi-resolution grids and hierarchical reinforcement learning models. The framework decomposes the airspace sectorization task into two levels: global control area delineation and local sectorization. These tasks are handled by top-level and bottom-level reinforcement learning strategies, respectively. It considers both global airspace information and local details to improve local sectorization results. It also adopts multi-resolution grids to increase the interaction efficiency between the model and the environment. Furthermore, it incorporates prior knowledge to enhance training efficiency and optimization accuracy. Experimental results show that the framework demonstrates significant advantages in both efficiency and effectiveness.
5.2. Future Work
- (1)
- This study is based on a two-dimensional spherical space. While much current research focuses on three-dimensional airspace sector partitioning, two-dimensional sector partitioning remains a cutting-edge research problem with significant importance [22]. Future research will explore the potential of multi-scale volumetric grids in dynamic airspace partitioning and integrate them with reinforcement learning algorithms. For instance, three-dimensional grids could be applied in high-density urban flight areas, while planar grids could be used in other regions, thereby enhancing the intelligence level of three-dimensional airspace management.
- (2)
- Predicting airspace workload changes in advance is another critical focus in airspace management [67]. This study conducts airspace partitioning based on historical data, but in practical applications, adjusting sector partitions based on predicted workloads would be more forward-looking and practical. Therefore, future research will use workload data from airspace grids and apply spatiotemporal convolutional models (e.g., ConvLSTM) to forecast dynamic workload changes. These predictions will support real-time optimization of sector partitioning.
Author Contributions
Funding
Data Availability Statement
Conflicts of Interest
Abbreviations
QTM | Quaternary Triangular Mesh |
HDQN | Hierarchical deep Q-network |
DGGS | Discrete Global Grid System |
References
- Bichot, C.-E.; Durand, N. A tool to design functional airspace blocks. In Proceedings of the ATM 2007, 7th USA/Europe Air Traffic Management Research and Development Seminar, Barcelona, Spain, 2–5 July 2007. [Google Scholar]
- Kökhan, A.; Kökhan, S.; Gökdalay, M. An algorithmic application with flexible airspace approach. Aircr. Eng. Aerosp. Technol. 2023, 95, 1045–1053. [Google Scholar] [CrossRef]
- Liu, Z.; Cai, K.; Zhu, Y. Civil unmanned aircraft system operation in national airspace: A survey from Air Navigation Service Provider perspective. Chin. J. Aeronaut. 2021, 34, 200–224. [Google Scholar]
- Pang, B.; Dai, W.; Hu, X.; Dai, F.; Low, K.H. Multiple air route crossing waypoints optimization via artificial potential field method. Chin. J. Aeronaut. 2021, 34, 279–292. [Google Scholar] [CrossRef]
- Wang, X.; Peng, J.; Tang, J.; Lu, Q.; Li, X. Investigating the impact of adding new airline routes on air transportation resilience in China. Transp. Policy 2022, 125, 79–95. [Google Scholar] [CrossRef]
- Ahrenhold, N.; Gerdes, I.; Mühlhausen, T.; Temme, A. Validating Dynamic Sectorization for Air Traffic Control Due to Climate Sensitive Areas: Designing Effective Air Traffic Control Strategies. Aerospace 2023, 10, 405. [Google Scholar] [CrossRef]
- Mogtit, A.; Aribi, N.; Lebbah, Y.; Lagha, M. Equitable optimized airspace sectorization based on constraint programming and OWA aggregation. Aircr. Eng. Aerosp. Technol. 2020, 92, 1225–1243. [Google Scholar] [CrossRef]
- Schultz, M.; Gerdes, I.; Standfuß, T.; Temme, A. Future airspace design by dynamic sectorization. In Proceedings of the Air Traffic Management and Systems III: Selected Papers of the 5th ENRI International Workshop on ATM/CNS (EIWAC2017), Tokyo, Japan, 16 November 2019; pp. 19–34. [Google Scholar]
- Serhan, D.; Yoon, S.W.; Chung, S.H. Dynamic reconfiguration of terminal airspace during convective weather: Robust optimization and conditional value-at-risk approaches. Comput. Ind. Eng. 2019, 132, 333–347. [Google Scholar] [CrossRef]
- Standfuß, T.; Gerdes, I.; Temme, A.; Schultz, M. Dynamic airspace optimisation. CEAS Aeronaut. J. 2018, 9, 517–531. [Google Scholar] [CrossRef]
- Sui, D.; Liu, K.; Li, Q. Dynamic Prediction of Air Traffic Situation in Large-Scale Airspace. Aerospace 2022, 9, 568. [Google Scholar] [CrossRef]
- Wong, C.S.Y.; Sundaram, S.; Sundararajan, N. CDAS: A cognitive decision-making architecture for dynamic airspace sectorization for efficient operations. IEEE Trans. Intell. Transp. Syst. 2018, 20, 1659–1668. [Google Scholar] [CrossRef]
- Yang, L.; Huang, J.; Gao, Q.; Zhou, Y.; Hu, M.; Xie, H. Dynamic Boundary Optimization of Free Route Airspace Sectors. Aerospace 2022, 9, 832. [Google Scholar] [CrossRef]
- Zhang, W.; Hu, M.; Yin, J.; Li, H.; Du, J. Multi-Objective 3D Airspace Sectorization Problem Using NSGA-II with Prior Knowledge and External Archive. Aerospace 2023, 10, 216. [Google Scholar] [CrossRef]
- Ghorpade, S. Airspace configuration model using swarm intelligence based graph partitioning. In Proceedings of the 2016 IEEE Canadian Conference on Electrical and Computer Engineering (CCECE), Vancouver, BC, Canada, 15–18 May 2016; pp. 1–5. [Google Scholar]
- Yin, C.W.S.; Venugopalan, T.; Suresh, S. A multi-objective approach for 3D airspace sectorization: A study on Singapore regional airspace. In Proceedings of the 2016 IEEE Symposium Series on Computational Intelligence (SSCI), Athens, Greece, 6–9 December 2016; pp. 1–8. [Google Scholar]
- Gianazza, D.; Durand, N. Ant colony systems for optimizing sequences of airspace partitions. In Proceedings of the 2020 International Conference on Artificial Intelligence and Data Analytics for Air Transportation (AIDA-AT), Singapore, 3–4 February 2020; pp. 1–10. [Google Scholar]
- Chen, Y.; Zhang, D. Dynamic airspace configuration method based on a weighted graph model. Chin. J. Aeronaut. 2014, 27, 903–912. [Google Scholar] [CrossRef]
- Ma, C.; Guleria, Y.; Alam, S.; Li, M.Z. Deep reinforcement learning-based air traffic flow coordination in flow-centric airspace. Adv. Eng. Inform. 2025, 65, 103342. [Google Scholar] [CrossRef]
- Jägare, P.; Flener, P.; Pearson, J. Airspace sectorisation using constraint-based local search. In Proceedings of the Tenth USA/Europe Air Traffic Management Research and Development Seminar (ATM2013), Chicago, IL, USA, 10–13 June 2013. [Google Scholar]
- Sergeeva, M.; Delahaye, D.; Mancel, C. 3D airspace sector design by genetic algorithm. In Proceedings of the 2015 International Conference on Models and Technologies for Intelligent Transportation Systems (MT-ITS), Budapest, Hungary, 3–5 June 2015; pp. 499–506. [Google Scholar]
- Ye, Z.; Kong, F.; Zhang, B.; Gao, W.; Mao, J. A method framework for automatic airspace reconfiguration-monte Carlo method for eliminating irregular sector shapes generated by region growth method. Sensors 2019, 19, 3934. [Google Scholar] [CrossRef]
- Tang, J.; Alam, S.; Lokan, C.; Abbass, H.A. A multi-objective approach for dynamic airspace sectorization using agent based and geometric models. Transp. Res. Part C Emerg. Technol. 2012, 21, 89–121. [Google Scholar] [CrossRef]
- Flener, P.; Pearson, J. Automatic airspace sectorisation: A survey. arXiv 2013, arXiv:1311.0653. [Google Scholar]
- Sergeeva, M.; Delahaye, D.; Mancel, C.; Vidosavljevic, A. Dynamic airspace configuration by genetic algorithm. J. Traffic Transp. Eng. 2017, 4, 300–314. [Google Scholar] [CrossRef]
- Deng, C.; Cheng, C.; Qu, T.; Li, S.; Chen, B. A Method for Managing ADS-B Data Based on a 4D Airspace-Temporal Grid (GeoSOT-AS). Aerospace 2023, 10, 217. [Google Scholar] [CrossRef]
- Chen, Y.; Xu, Y.; Hu, M. General multi-agent reinforcement learning integrating heuristic-based delay priority strategy for demand and capacity balancing. Transp. Res. Part C Emerg. Technol. 2023, 153, 104218. [Google Scholar] [CrossRef]
- Razzaghi, P.; Tabrizian, A.; Guo, W.; Chen, S.; Taye, A.; Thompson, E.; Bregeon, A.; Baheri, A.; Wei, P. A survey on reinforcement learning in aviation applications. Eng. Appl. Artif. Intell. 2024, 136, 108911. [Google Scholar] [CrossRef]
- Chen, Y.; Zhao, Y.; Fei, F.; Yang, H. Optimizing Large-Scale Demand and Capacity Balancing in Air Traffic Flow Management Using Deep Neural Networks. Aerospace 2024, 11, 966. [Google Scholar] [CrossRef]
- Chen, Y.; Hu, M.; Xu, Y.; Yang, L. Locally generalised multi-agent reinforcement learning for demand and capacity balancing with customised neural networks. Chin. J. Aeronaut. 2023, 36, 338–353. [Google Scholar]
- Spatharis, C.; Bastas, A.; Kravaris, T.; Blekas, K.; Vouros, G.A.; Cordero, J.M. Hierarchical multiagent reinforcement learning schemes for air traffic management. Neural Comput. Appl. 2023, 35, 147–159. [Google Scholar] [CrossRef]
- Mas-Pujol, S.; Salamí, E.; Pastor, E. Image-Based Multi-Agent Reinforcement Learning for Demand—Capacity Balancing. Aerospace 2022, 9, 599. [Google Scholar] [CrossRef]
- Ribeiro, M.; Ellerbroek, J.; Hoekstra, J. Using reinforcement learning to improve airspace structuring in an urban environment. Aerospace 2022, 9, 420. [Google Scholar] [CrossRef]
- Granberg, T.A.; Polishchuk, T.; Polishchuk, V.; Schmidt, C. Integer Programming—Based Airspace Sectorization for Terminal Maneuvering Areas with Convex Sectors. J. Air Transp. 2019, 27, 169–180. [Google Scholar] [CrossRef]
- Basu, A.; Mitchell, J.S.B.; Sabhnani, G.K. Geometric algorithms for optimal airspace design and air traffic controller workload balancing. ACM J. Exp. Algorithmics 2010, 14, 2.3-2.28. [Google Scholar] [CrossRef]
- Hidila, Z.; Belhaous, S.; Bentaleb, M.; Naji, A.; Mestari, M. Airspace sectorization and comparison via computational geometry, openCV and networkx. In Proceedings of the 2019 Third International Conference on Intelligent Computing in Data Sciences (ICDS), Marrakech, Morocco, 28–30 October 2019; pp. 1–7. [Google Scholar]
- Cai, M.; Wan, L.; Jiao, Z.; Lv, M.; Gao, Z.; Qi, D. Clustering Method of Large-Scale Battlefield Airspace Based on Multi A* in Airspace Grid System. Appl. Sci. 2022, 12, 11396. [Google Scholar] [CrossRef]
- Qian, C.; Yi, C.; Cheng, C.; Pu, G.; Wei, X.; Zhang, H. Geosot-based spatiotemporal index of massive trajectory data. ISPRS Int. J. Geo-Inf. 2019, 8, 284. [Google Scholar] [CrossRef]
- Kotsuki, S.; Miyoshi, T.; Terasaki, K.; Lien, G.Y.; Kalnay, E. Assimilating the global satellite mapping of precipitation data with the Nonhydrostatic Icosahedral Atmospheric Model (NICAM). J. Geophys. Res. Atmos. 2017, 122, 631–650. [Google Scholar] [CrossRef]
- Ventrella, J. Glider Dynamics on the Sphere: Exploring Cellular Automata on Geodesic Grids. J. Cell. Autom. 2011, 6, 245–256. [Google Scholar]
- Chen, C.; Xiao, F. Shallow water model on cubed-sphere by multi-moment finite volume method. J. Comput. Phys. 2008, 227, 5019–5044. [Google Scholar] [CrossRef]
- Robertson, C.; Chaudhuri, C.; Hojati, M.; Roberts, S.A. An integrated environmental analytics system (IDEAS) based on a DGGS. ISPRS J. Photogramm. Remote Sens. 2020, 162, 214–228. [Google Scholar] [CrossRef]
- Rawson, A.; Sabeur, Z.; Brito, M. Intelligent geospatial maritime risk analytics using the Discrete Global Grid System. Big Earth Data 2022, 6, 294–322. [Google Scholar] [CrossRef]
- Li, M.; McGrath, H.; Stefanakis, E. Integration of heterogeneous terrain data into Discrete Global Grid Systems. Cartogr. Geogr. Inf. Sci. 2021, 48, 546–564. [Google Scholar] [CrossRef]
- Li, M.; McGrath, H.; Stefanakis, E. Multi-resolution topographic analysis in hexagonal Discrete Global Grid Systems. Int. J. Appl. Earth Obs. Geoinf. 2022, 113, 102985. [Google Scholar] [CrossRef]
- Li, M.; McGrath, H.; Stefanakis, E. Geovisualization of hydrological flow in hexagonal grid systems. Geographies 2022, 2, 227–244. [Google Scholar] [CrossRef]
- Yongwen, Z.; Fan, P. Principle and application of airspace spatial grid identification. J. Beijing Univ. Aeronaut. Astronaut. 2021, 47, 2462–2474. [Google Scholar]
- Dutton, G. Modeling locational uncertainty via hierarchical tessellation. In Accuracy of Spatial Databases; CRC Press: Boca Raton, FL, USA, 1989; pp. 125–140. [Google Scholar]
- Wang, Z.; Zhao, X.; Sun, W.; Luo, F.; Li, Y.; Duan, Y. Correlation analysis and reconstruction of the geometric evaluation indicator system of the discrete global grid. ISPRS Int. J. Geo-Inf. 2021, 10, 115. [Google Scholar] [CrossRef]
- Lin, B.; Zhou, L.; Xu, D.; Zhu, A.-X.; Lu, G. A discrete global grid system for earth system modeling. Int. J. Geogr. Inf. Sci. 2018, 32, 711–737. [Google Scholar] [CrossRef]
- Dutton, G.H. A Hierarchical Coordinate System for Geoprocessing and Cartography; Springer: Berlin/Heidelberg, Germany, 1999; Volume 79. [Google Scholar]
- Gianazza, D. Learning air traffic controller workload from past sector operations. In Proceedings of the ATM Seminar, 12th USA/Europe Air Traffic Management R&D Seminar, Seattle, WA, USA, 27–30 June 2017. [Google Scholar]
- Delahaye, D.; Ma, C.; Alam, S.; Cai, Q. Air Traffic Flow Representation and Prediction using Transformer in Flow-Centric Airspace. In Proceedings of the SESAR Innovation Days, Seville, Spain, 27–30 November 2022. [Google Scholar]
- Li, C.; Hu, Y. Controller’s Workload and Sector Capacity Assessment Based on 4D Track. In Artificial Intelligence in China, Proceedings of the 2nd International Conference on Artificial Intelligence in China, Hangzhou, China, 26–28 June 2021; Springer: Singapore, 2021; pp. 285–295. [Google Scholar]
- Mohammed, G.; El Bekkaye, M. Fuzzy Dynamic Airspace Sectorization Problem; Springer International Publishing: Cham, Switzerland, 2021; pp. 229–250. [Google Scholar]
- Pham, D.-T.; Alam, S.; Duong, V. An air traffic controller action extraction-prediction model using machine learning approach. Complexity 2020, 2020, 1659103. [Google Scholar] [CrossRef]
- Pham, D.-T.; Alam, S.; Su, Y.-L.; Duong, V.N. A Machine Learning Approach on Past ADS-B Data to Predict Planning Controller’s Actions; Nanyang Technological University: Singapore, 2018. [Google Scholar]
- Wieland, F.; Rebollo, J.; Gibbs, M.; Churchill, A. Predicting sector complexity using machine learning. In Proceedings of the AIAA AVIATION 2022 Forum, Chicago, IL, USA, 27 June–1 July 2022; p. 3754. [Google Scholar]
- Ozgur, M.; Cavcar, A. 0–1 integer programming model for procedural separation of aircraft by ground holding in ATFM. Aerosp. Sci. Technol. 2014, 33, 1–8. [Google Scholar] [CrossRef]
- Zhai, W.; Tong, X.; Miao, S.; Cheng, C.; Ren, F. Collision detection for UAVs based on GeoSOT-3D grids. ISPRS Int. J. Geo-Inf. 2019, 8, 299. [Google Scholar] [CrossRef]
- Allignol, C.; Barnier, N.; Flener, P.; Pearson, J. Constraint programming for air traffic management: A survey1: In memory of Pascal Brisset. Knowl. Eng. Rev. 2012, 27, 361–392. [Google Scholar] [CrossRef]
- Liu, Q.; Zhao, X.; Duan, Y.; Qin, M.; Xie, W.; Sun, W. Dynamic Construction of Spherical Raster Voronoi Diagrams Based on Ordered Dilation. ISPRS Int. J. Geo-Inf. 2024, 13, 202. [Google Scholar] [CrossRef]
- Laudeman, I.V.; Shelden, S.G.; Branstrom, R.; Brasil, C. Dynamic Density: An Air Traffic Management Metric; (NASA/TM-1998-112226); NASA: Moffett Field, CA, USA, 1998. [Google Scholar]
- Yin, J.; Liu, J.; Zeng, W.; Wei, W.; Tian, W.; Zhang, Y. Spatiotemporal Dynamic Graph Convolutional Networks for Air Route Network Congestion Prediction. J. Aerosp. Inf. Syst. 2024, 22, 177–188. [Google Scholar] [CrossRef]
- Li, B.; Li, Z.; Chen, J.; Yan, Y.; Lv, Y.; Du, W. MAST-GNN: A multimodal adaptive spatio-temporal graph neural network for airspace complexity prediction. Transp. Res. Part C Emerg. Technol. 2024, 160, 104521. [Google Scholar] [CrossRef]
- Li, Y.; Zhao, X.; Sun, W.; Wang, G.; Luo, F.; Wang, Z.; Duan, Y. A GtoG Direct Coding Mapping Method for Multi-Type Global Discrete Grids Based on Space Filling Curves. ISPRS Int. J. Geo-Inf. 2022, 11, 595. [Google Scholar] [CrossRef]
- Arardhi, V.P.; Hazra, S.; Singh, U.; Nithya, P.; Sundaram, S.; Sooraj, K.; Menon, P.; Padhi, R. Dynamic Air Traffic Model for Controlled Large Airspace. J. Aerosp. Inf. Syst. 2025; articles in advance. [Google Scholar] [CrossRef]
RL | Grids | Prior Knowledge | Time Per Cycle | Reward |
---|---|---|---|---|
HDQN | multi-resolution | None | 6.724 s | −2.67 × 105 |
Local-region DQN | multi-resolution | None | - | −3.63 × 105 |
Global-region DQN | multi-resolution | None | - | −4.92 × 105 |
HDQN | single resolution | None | 14.08 s | −2.71 × 105 |
HDQN | multi-resolution | Yes | - | −1.59 × 105 |
Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content. |
© 2025 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/).
Share and Cite
Liu, Q.; Zhao, X.; Wang, X.; Qin, M.; Sun, W. A Multi-Scale Airspace Sectorization Framework Based on QTM and HDQN. Aerospace 2025, 12, 552. https://doi.org/10.3390/aerospace12060552
Liu Q, Zhao X, Wang X, Qin M, Sun W. A Multi-Scale Airspace Sectorization Framework Based on QTM and HDQN. Aerospace. 2025; 12(6):552. https://doi.org/10.3390/aerospace12060552
Chicago/Turabian StyleLiu, Qingping, Xuesheng Zhao, Xinglong Wang, Mengmeng Qin, and Wenbin Sun. 2025. "A Multi-Scale Airspace Sectorization Framework Based on QTM and HDQN" Aerospace 12, no. 6: 552. https://doi.org/10.3390/aerospace12060552
APA StyleLiu, Q., Zhao, X., Wang, X., Qin, M., & Sun, W. (2025). A Multi-Scale Airspace Sectorization Framework Based on QTM and HDQN. Aerospace, 12(6), 552. https://doi.org/10.3390/aerospace12060552