Transformer-Augmented MCTS for Aircraft Landing Problem

Abstract

The aircraft landing problem (ALP) poses significant challenges for traditional Monte Carlo Tree Search (MCTS) due to its vast search space and reliance on inefficient random simulations. To overcome these limitations, this paper proposes a novel Transformer-Augmented Monte Carlo Tree Search (TMCTS) algorithm. Our approach integrates a reinforcement learning framework that incorporates key operational constraints, including wake turbulence separation and time windows, and employs a cost function aimed at minimizing both delay time and fuel consumption. A core innovation is the replacement of the conventional random simulation phase in MCTS with a Transformer-based value predictor. This leverages the Transformer’s superior ability to model sequences and capture global dependencies among flights, thereby dramatically accelerating search convergence. Specifically, we designed a two-head Transformer network (comprising policy and value heads) to provide informed prior knowledge, which effectively guides the selection and expansion steps of the MCTS tree. The model is trained within an Actor–Critic framework, utilizing behavior cloning for pre-training followed by reinforcement learning for fine-tuning. Experimental evaluations on the standard OR-Library benchmark demonstrate that our TMCTS method significantly reduces scheduling deviation compared to state-of-the-art baselines (including FCFS, DPALO+GA, DPALO+PSO, and CPLEX). Moreover, it achieves a 93.7% reduction in computation time relative to the CPLEX method, highlighting its superior efficiency and practical applicability for real-time scheduling.

1. Introduction

In 2024, China’s civil aviation industry achieved a total transport turnover of 1485.17 billion t-km, 730 million passenger trips, and 8.9816 million t of cargo and mail volume, representing year-on-year increases of 25.0%, 17.9%, and 22.1%, respectively. These figures reflect steady growth in the industry’s transportation operations [1]. During the same year, Chinese passenger airlines operated a cumulative total of 5.0031 million flights, of which 4.3577 million were punctual, resulting in an average flight punctuality rate of 87.1%. Runway resource constraints have been identified as a primary cause of flight delays at hub airports, and so this decline in punctuality underscores a pressing need to enhance runway throughput and scheduling efficiency [2].

To mitigate the risks posed by wake turbulence during approach and landing phases, aircraft must maintain a minimum separation distance from preceding flights. The International Civil Aviation Organization (ICAO) categorizes aircraft into three classes based on maximum takeoff weight (MTOW):

• Light (L): MTOW less than 7 t;
• Medium (M): MTOW between 7 and 136 t;
• Heavy (H): MTOW greater than 136 t.

The wake vortices generated by a leading aircraft can significantly affect the takeoff and landing performance of following aircraft. Therefore, during the approach phase, the required separation between aircraft must satisfy both the wake turbulence separation minima and the standard aircraft safety separation requirements.

In 2010, Balakrishnan [3] reformulated the wake separation distance criteria into equivalent wake time separation intervals, as detailed in

Table 1. Wake separation interval(s).

The Aircraft Ahead	The Aircraft Behind
	Heavy	Medium	Light
Heavy	96	157	196
Medium	60	69	131
Light	60	69	82

. This conversion facilitates integration into runway scheduling models.

The core optimization problem addressed in this paper is the determination of an optimal aircraft sequencing schedule. This schedule must do the following:

• Satisfy all prescribed wake time separation constraints;
• Minimize the total completion time for takeoff and landing operations;
• Reduce associated landing costs.

According to references [4,5], the aircraft landing problem (ALP) is subject to two primary constraints:

• Time Window Constraint: Each flight must land within a specified time interval bounded by an Earliest Landing Time (ELT) and a Latest Landing Time (LLT).
• Wake separation constraint: Successive landings must adhere to the minimum wake turbulence separation intervals.

The Earliest Landing Time (ELT) is defined as the earliest theoretical time at which the flight could arrive and land without any delays. Conversely, the Latest Landing Time (LLT) represents the maximum allowable time an aircraft can remain airborne, typically involving holding patterns, before it must land [6].

Deviations from an ideal schedule incur operational costs. Landing earlier than planned can increase airport congestion and air traffic controller workload, leading to higher associated labor costs. Landing later typically results in increased fuel consumption due to extended flight time. Consequently, for each flight i, there exists an Optimal Landing Time (T_i) that minimizes the total operational cost. The cost function for flight i is typically modeled as a piecewise linear function:

$$C_i\left(t_i\right)=\left\{\begin{array}{ccc}{g}_i\times \left(T_i-t_i\right)\hfill & \mathrm{i}\mathrm{f}\hfill & t_{\pi }^i<T_{\pi }^i (\mathrm{e}\mathrm{a}\mathrm{r}\mathrm{l}\mathrm{y} \mathrm{l}\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{g})\hfill \\ 0\hfill & \mathrm{i}\mathrm{f}\hfill & t_{\pi }^i=T_{\pi }^i\hfill \\ h_i\times \left(t_i-T_i\right)\hfill & \mathrm{i}\mathrm{f}\hfill & t_{\pi }^i>T_{\pi }^i (\mathrm{l}\mathrm{a}\mathrm{t}\mathrm{e} \mathrm{l}\mathrm{a}\mathrm{n}\mathrm{d}\mathrm{i}\mathrm{n}\mathrm{g})\hfill \end{array}\right.$$

(1)

where t_i is the actual landing time, E_i and L_i are the Earliest and Latest Landing Time, and g_i and h_i are the unit costs for earliness and tardiness. This creates a V-shaped convex cost profile centered at T_i, as illustrated in Figure 1.

The ALP is a well-known NP-hard combinatorial optimization problem characterized by multiple operational constraints and strict time-sensitivity requirements. Extensive research has been conducted over the past two decades, leading to the development of various models and solution methodologies. Beasley [7,8] established a foundational mixed-integer programming (MIP) model for the ALP and created the widely used OR-Library benchmark dataset. This dataset formally defines the key temporal parameters—Earliest Landing Time (ELT), Optimal Landing Time (OLT), and Latest Landing Time (LLT)—enabling the optimization of objectives for different stakeholders.

To tackle the computational complexity, numerous heuristic algorithms have been proposed. H. Pinol [9] introduced a population-based heuristic, demonstrating its capability on a large-scale instance with 500 aircraft and 5 runways, though its performance degraded with over 50 aircraft. Sheng-Peng Yu [10] developed a cellular automata-based optimization algorithm, which achieved optimal solutions for small-scale cases (test cases 1–7) and near-optimal solutions for large-scale datasets from the OR-Library. Gui D et al. [11] proposed a novel meta-heuristic for the ALP, integrating tabu search to improve solution quality and a rolling time horizon method to enhance computational speed. Daniel A et al. [12] presented a fast greedy heuristic, the Closest Aircraft Sequence with Time Windows (CAS-TW), designed to minimize total delay while respecting operational constraints through a discretization strategy.

Recognizing the multi-faceted nature of the problem, several studies have incorporated multiple objectives. Chen Xiaodao et al. [13] formulated the problem as an integer linear program and transformed it into a multi-objective optimization, proposing an uncertainty-aware scheduling algorithm. Zhang Junfeng et al. [14] constructed a multi-objective ranking and scheduling model for a Parallel Machine Scheduling (PMS) analogy and proposed a multi-objective Imperial Competition Algorithm (ICA), showing significant improvement over First-Come-First-Served (FCFS). Wang Jianzhong et al. [15] proposed a multi-objective time index model, using a genetic algorithm to minimize overall delay and controller workload by weighting different objectives. Chen K et al. [16] established a multi-objective model for the complex interdependent runway ASP (ASP-CIR) and developed an Improved Multi-Objective Restarted Variable Neighborhood Search (IMORVNS) algorithm.

Recent work has increasingly focused on fairness and integrated arrival–departure operations. Liu Jixin et al. [17] addressed four combined operational scenarios, introducing fairness metrics—“equalized delay time for arrivals” and “equalized satisfaction for departures”—and established a two-level planning model to balance fairness and efficiency. Zhou Dingkai et al. [18] applied reinforcement learning to a static terminal area sequencing model, incorporating “flight satisfaction” to enhance fairness in resource allocation between airports and airlines.

Advanced computational methods have been employed to address large solution spaces and real-time requirements. Kang Rui [19] integrated an EoR (Earliest on Route) approach with sequencing strategies and proposed an S-shaped function-based Adaptive Particle Swarm Optimization (SA-PSO) algorithm to minimize flight delay. Zhang C et al. [20] introduced two prescription analysis methods—Estimation-Trial-Optimization (ETO) and Estimation-Trial-Distributionally Robust Optimization (ETDRO)—to improve the efficiency and robustness of sequencing under uncertainty. Dönmez K et al. [21] applied a Multi-Criteria Decision-Making (MCDM) approach, using the MEREC method for criteria weighting and the MARCOS technique to rank solutions, aiming to find a single objective representing an optimal compromise for all stakeholders. Feng Xiaorong et al. [6] developed a time-window-constrained optimization model and proposed a compact subsequence-based algorithm. This method demonstrated high accuracy and speed for small datasets but exhibited lower computational efficiency on large-scale problems.

Although the aforementioned methods have achieved satisfactory performance in addressing the ALP, their high computational complexity often leads to substantial processing time when handling large-scale instances. This limitation poses significant challenges for real-time scheduling and dynamic adjustments in practical operational environments. Consequently, improving algorithmic efficiency and reducing computational overhead remain critical and unresolved issues in this field of research.

In 2006, French computer scientist Coulom [22] first proposed Monte Carlo Tree Search (MCTS), a heuristic search algorithm designed for optimal decision-making in complex scenarios. MCTS efficiently approximates optimal policies through iterative random sampling and tree structure optimization, without requiring complete prior knowledge of the state space. Initially gaining prominence in artificial intelligence, MCTS has achieved groundbreaking success in domains such as the game of Go [23]. A notable milestone was reached in 2016 when AlphaGo, leveraging MCTS, defeated world champion Lee Sedol 4–1, marking the first time an AI system surpassed top human performance in Go.

Beyond game playing, MCTS has been effectively adapted to various optimization and scheduling problems. For instance, Song Wenshuai et al. [24] addressed the dynamic adjustment of reservoir surface sorting strategies under changing environmental conditions by proposing a method based on deep Monte Carlo Tree Search, which integrates deep neural networks to predict prior action probability and evaluate policies. In railway operations, Wang Guanghui et al. [25] introduced a hybrid reinforcement learning–Monte Carlo Tree Search approach for route optimization, employing a breadth-first followed by depth-first selection strategy to balance global and local benefits. For military logistics, Peng Jian et al. [26] designed a heuristic algorithm incorporating MCTS to optimize task sequences for fleet support, using Monte Carlo Trees to record evaluation outcomes and guide subsequent search directions. Similarly, Yu Ze et al. [27] applied an improved MCTS method to unmanned aerial vehicle (UAV) task planning in complex three-dimensional environments, introducing normalized reward ranges to handle parameter uncertainty and balance exploration versus exploitation. Pang et al. [28] proposed a machine learning-enhanced traveling salesman problem framework aimed at reducing total landing time compared to the FCFS approach.

For ALP, existing Arrival Management (AMAN) systems primarily achieve time sequencing through terminal area spatial path expansion and employ speed management when sufficient planning scope is available. In practice, it typically requires air traffic controllers (ATCs) to perform unified, globally optimized sequencing and scheduling of multiple incoming flights when they enter the radar range (radar visibility) of the airport ATC system, assigning each flight a landing time and corresponding time slot. The most widely used strategy for the ALP is the FCFS rule, which schedules aircraft based on their order of entry into the radar range. Current mainstream algorithms predominantly rely on heuristics [7,8,9,10,11,12,29]. There is a demand for aircraft landing scheduling algorithms based on deep reinforcement learning to minimize landing costs. Therefore, this paper proposes a Transformer-Augmented Monte Carlo Tree Search (TMCTS) algorithm to solve the flight sequencing scheduling problem. By considering constraints such as wake separation standards and time windows, combined with estimated arrival times in the terminal area, the algorithm aims to minimize the total cost of the flight sequence. The TMCTS method is employed to calculate the optimal landing sequence and landing time for each flight, The main contributions of this work are threefold:

• Model Formulation: A reinforcement learning-based optimization model is established for aircraft sequencing, framing the scheduling task as a sequential decision-making process.
• Algorithm Design: A novel flight scheduling methodology is proposed, which integrates the structural strengths of the Transformer architecture with the strategic search capability of MCTS.
• Architecture Innovation: A specialized two-head output module is designed, comprising a policy head, and a value head, to effectively capture the unique dependencies and priority constraints inherent in flight scheduling.

The proposed TMCTS algorithm operates as a deterministic search method, demonstrating strong robustness and stability in handling large-scale scheduling instances. We employed the Airland test case from the OR-Library benchmark to train the model and validate the effectiveness and efficiency of our proposed algorithm. Results were compared against FCFS, heuristic algorithms (GA, PSO, CAO), and a commercial mixed-integer programming (MIP) solver (CPLEX).

The remainder of this paper is organized as follows: Section 2 details the workflow of the proposed TMCTS framework, including its algorithmic architecture and model components. Section 3 describes the experimental setup, covering data sources, parameter configurations, and training procedures. Section 4 presents the computational results and provides a comparative analysis with benchmark methods. Finally, Section 5 concludes the paper by summarizing the findings and suggesting directions for future research.

2. Methodology

2.1. Algorithm Framework

Figure 2 illustrates a deep reinforcement learning-based research framework for flight scheduling, which comprises the following three core components.

• Transformer-based Scheduling Architecture: Flight data are initially encoded into feature vectors. These vectors are then processed through multiple encoder layers, each consisting of multi-head attention mechanisms and feed forward neural networks, to simultaneously capture the constraints of paired wake separation and the relationship between delay time and cost.

After multi-layer transformations, two output heads generate prior knowledge and state-value estimates, respectively.

2.

MCTS-Guided Strategy Optimization: Leveraging the prior probabilities provided by the Transformer, MCTS iteratively refines flight scheduling strategies through a cyclic process of selection, expansion, simulation, and backpropagation. At each decision node, the improved policy and corresponding value derived from MCTS serve as supervisory signals to train the Transformer network.

3.

Actor–Critic Reinforcement Learning with Transformer Core: An Actor–Critic architecture, integrated with policy gradient methods, employs the Transformer as its central model. Decision-making is guided by MCTS during exploration, while collected trajectory data are used to update the Transformer parameters, thereby facilitating continuous policy optimization for the scheduling agent.

This integrated framework enables synergistic interaction between representation learning, strategic search, and policy improvement, enhancing both the efficiency and robustness of flight scheduling under complex operational constraints.

2.2. Aircraft Landing Issues and Reinforcement Learning Models

2.2.1. Mathematical Model for ALP

This paper focuses on the single-runway, arrival-only aircraft landing problem, where the runway is dedicated to landing operations and departure flights are not considered. The objective of the time-window-constrained multi-flight optimization model is to determine an optimal scheduling sequence ${\pi }^{*}$ for N flights and compute the corresponding optimal landing time $t_{{\pi }^{*}}^i$ for each flight under sequence ${\pi }^{*}$, such that the total cost $T{C}_{{\pi }^{*}}^N$ for all N flights is minimized. This objective can be formulated as follows:

$$T{C}_{{\pi }^{*}}^N=\mathit{argmin}t c_{\pi }\left(\pi \in \Pi \right)$$

(2)

where N denotes the total number of aircraft to be scheduled, $\pi$ denotes a candidate landing sequence, $\Pi$ represents the set of all feasible sequences of N flights, and $t{c}_{\pi }$ (total cost) is the minimum total landing cost under sequence $\pi$, which can be expressed as

$$t{c}_{\pi }=\mathit{min}{\displaystyle \sum _{i=1}^N}f\left(t_{\pi }^i\right)$$

(3)

Here, $f\left(t_{\pi }^i\right)$ represents the cost incurred when the i-th flight in sequence $\pi$ lands at time $t_{\pi }^i$, expressed as

$$f\left(t_{\pi }^i\right)=g_{\pi }^i\ast max\left({0,T}_{\pi }^i-t_{\pi }^i\right)+h_{\pi }^i\ast max\left(0,t_{\pi }^i-T_{\pi }^i\right)$$

(4)

where $T_{\pi }^i$ denotes the optimal landing time for flight i under sequence $\pi$, and $g_{\pi }^i$ and $h_{\pi }^i$ are the unit cost coefficients for earliness and tardiness, respectively, relative to $T_{\pi }^i$.

The model is subject to the following operational constraints:

• Wake separation interval constraint:

$$t_{\pi }^j-t_{\pi }^i\ge S_{\pi \left(i,j\right)},\forall i,j\in \left\{1,\cdots ,N\right\},i\ne j$$

(5)

where $S_{\pi (i,j)}$ denote the minimum wake turbulence separation interval between flights i and j under sequence $\pi$.

2.

Earliest Landing Time constraint:

$$t_{\pi }^i\ge E_{\pi }^i,\hspace{1em}\forall i\le N$$

(6)

where $E_{\pi }^i$ is the earliest permissible landing time for flight i.

3.

Latest Landing Time constraint:

$$t_{\pi }^i\le L_{\pi }^i,\hspace{1em}\forall i\le N$$

(7)

where $L_{\pi }^i$ is the latest permissible landing time for flight i.

These constraints ensure that the scheduling sequence respects safety, operational, and time window requirements while minimizing the total landing cost.

2.2.2. Markov Decision Process (MDP) Modeling

The ALP is formulated as a Markov Decision Process (MDP), defined by the quadruple (S, A, R, P):

• State space

The state space integrates all essential aircraft information and scheduling progress. Each aircraft is represented by a 7-dimensional state vector, defined as

$$s_i\left(t\right)=\left[E_{\pi }^i\left(t\right),T_{\pi }^i\left(t\right),L_{\pi }^i\left(t\right),g_{\pi }^i\left(t\right),h_{\pi }^i\left(t\right),m_{\pi }^i\left(t\right),d_{\pi }^i\left(t\right)\right]$$

(8)

where $E_{\pi }^i$, $T_{\pi }^i$, and $L_{\pi }^i$ denote the time window information; $g_{\pi }^i$ and $h_{\pi }^i$ are penalty coefficients; $m_{\pi }^i$ is a binary indicator of whether flight i has been scheduled; and $d_{\pi }^i$ represents the required wake separation between unscheduled flight i and the last scheduled flight.

2.

Action Space

The action space comprises the set of aircraft that can be selected for landing in the current state. A valid action must satisfy two conditions: the aircraft has not yet been assigned a landing, and it complies with safety separation constraints relative to the preceding aircraft. The action space is formally expressed as

$$A\left(t\right)=\left\{a_i\left(t\right)|a_i\left(t\right)\in \{0,1\},\sum _n{a}_i\left(t\right)=1\right\}$$

(9)

Here a_i(t) denotes the action for the i-th flight. a_i(t) = 0 indicates that the i-th flight is not selected, while a_i(t) = 1 indicates selection.

3.

Reward function

The reward function R(t) is designed to guide the agent toward minimizing cost and completing the scheduling task. It consists of three components:

• Cost penalty: A negative reward proportional to the cumulative cost incurred at the current scheduling step, with dynamically adjusted weighting throughout the process.
• Completion reward: A positive reward granted upon successful scheduling of all flights. The global reward is computed using the method described in the previous section for fixed-order ALP, yielding the total cost under the constructed sequence.
• Deviation Bonus: An additional reward provided when the final scheduling cost outperforms the historical best recorded value.

4.

State transition.

For the current flight scheduling state S(t), after executing action a_i(t), the i-th aircraft is marked as “scheduled”; the DPALO module is invoked to calculate the landing time for that aircraft; the flight sequence and the set of unscheduled flights are updated; the wake separation constraints between each unscheduled aircraft and the last scheduled aircraft are calculated; and a new state S(t + 1) is formed. In this paper, action selection follows a stochastic policy with probabilities summing to 1. At each decision step, the agent samples an action a(t) from the action distribution, with the action probability provided by the Transformer’s policy head. The state transition is deterministic: given the current state and the selected next arriving flight, the next state (the set of scheduled flights and the set of remaining unscheduled flights) is uniquely determined, i.e., P(S(t + 1)|S(t),a(t)) = 1. The time period from t to t + 1 represents a discrete decision step in the MCTS process.

The overall reinforcement learning-based decision-making process for flight scheduling is illustrated in Figure 3.

2.3. Solving for Aircraft Landing Times Under Fixed Sequence

ALP can be decomposed into two subproblems: the first is to determine optimal landing time given a sequence, and the second is to find the optimal landing sequence. In the proposed algorithm, MCTS selects an aircraft and appends it to the current landing sequence, and then computes the cost of the resulting partial sequence. At the end of a complete round, the final cost is normalized to the value target range [−1, 1]. This target is then used to train the value head of the Transformer network using mean squared error loss.

To address the problem of determining aircraft landing times under a fixed sequence, we used the Dynamic Programming Approach to Limit Optimization (DPALO) [29]. First, the landing scheduling problem with a fixed flight sequence is formulated as an integer programming problem for determining the landing time of each flight. This problem is then solved using the Dynamic Programming Algorithm (DPA), which can determine the cost of this sequence with certainty. The DP model consists of N stages (each stage corresponds to a single aircraft). In each stage i, the number of states equals the number of feasible discrete landing times within the time window and interval constraints, with an upper bound determined by the reduced time window length; assuming 1-s discretization, the number of states per stage is ∆T. The total time complexity for solving this fixed-order subproblem is O(N × ∆T).

Let $F\left(t_{\pi }^i\right)$ denote the minimum cumulative total cost generated by the first i flights when the i-th flight in sequence $\pi$ lands at time $t_{\pi }^i$. Given that the cost of the first flight in the sequence is $F\left(t_{\pi }^1\right) = \mathrm{f}\left(t_{\pi }^1\right)$, the cost of the i-th flight is $F\left(t_{\pi }^i\right)=f\left(t_{\pi }^i\right)+\mathit{min}\left(F\left(t_{\pi }^{i-1}\right)\right)$. Then, establish a time-based recursive function for $F\left(t_{\pi }^i\right)$ as follows:

$$\mathrm{F}\left(t_{\pi }^i\right)=\left\{\begin{array}{cc}\mathrm{f}\left(t_{\pi }^i\right)\hfill & i=1\\ \mathrm{f}\left(t_{\pi }^i\right)+\underset{E_{\pi }^{i-1}\le t_{\pi }^{i-1}\le \min \left(t_{\pi }^i-S_{\pi (i-1,i)},L_{\pi }^{i-1}\right)}{\min }\left(\mathrm{F}\left(t_{\pi }^{i-1}\right)\right)\hfill & 1<i\le n\end{array}\right.$$

(10)

Equation (2) can be further simplified to Equation (11).

$$\mathrm{t}{\mathrm{c}}_{\pi }=\mathit{min}\left(F\left(t_{\pi }^i\right)\right)\hspace{1em}\left(\pi \in \Pi \right)$$

(11)

Based on Equations (5) and (6), the range of values for dependent variable $t_{\pi }^i$ of function $f\left(t_{\pi }^i\right)$ is determined to be $\left[E_{\pi }^i,L_{\pi }^i\right]$. Considering Equation (4) and the optimal value of objective function $t{c}_{\pi }$, the range of values for dependent variable $t_{\pi }^i$ can be further narrowed. This section will further analyze the range of values for dependent variable $t_{\pi }^i$. Figure 4 shows the flowchart of the dynamic programming algorithm for solving the ALP. In Figure 4, $f\pi \left[i\right]\left[t\right]$ equals $f\left(t_{\pi }^i\right)$, where $f\pi \left[i\right]\left[t\right]$ represents the cost of landing a flight at that time. For example, $f\left(E_{\pi }^i\right)$ denotes the cost of landing the i-th flight in sequence $\pi$ at its earliest possible landing time. $F\pi \left[i\right]\left[t\right]$ equals $F\left(t_{\pi }^i\right)$, where $F\pi \left[i\right]\left[t\right]$ represents the landing time of a flight at that moment and the minimum cumulative total cost generated by all preceding flights. For example, $F\left(E_{\pi }^i\right)$ denotes the minimum cumulative total cost generated by the preceding i − 1 flights when flight i lands at its earliest possible landing time. By calculation, the optimal landing time and total cost for flights under a fixed sequence can be determined.

Figure 5 illustrates the search range for flight landing costs under the discretized $\pi$ sequence, using three flights as an example. Flight landing times must simultaneously satisfy constraints (4), (5), and (6). When $E_{\pi }^2>E_{\pi }^1+S_{\pi (1,2)}$ occurs, the second flight cannot land at time $\left(E_{\pi }^2,{ E}_{\pi }^1+S_{\pi (1,2)}\right)$, as shown in dashed box 1. When $E_{\pi }^3<E_{\pi }^2+S_{\pi (2,3)}$ occurs, the third flight cannot land at time $\left(E_{\pi }^3, E_{\pi }^2+S_{\pi (2,3)}\right)$, as shown in dashed box 2. When $L_{\pi }^2>L_{\pi }^1+S_{\pi (1,2)}$ occurs, since deviation costs increase linearly with time, $f\left(L_{\pi }^2\right)>f\left(L_{\pi }^1+S_{\pi (1,2)}\right)$ holds. Therefore, when calculating the total landing cost for the first two flights, $L_{\pi }^1+S_{\pi (1,2)}, L_{\pi }^2$ within the second flight’s time window can be omitted from the calculation, as shown in dashed box 3. When $L_{\pi }^1+S_{\pi (1,2)},{ L}_{\pi }^2$ occurs, the third flight cannot land at time $L_{\pi }^3,{ L}_{\pi }^2+S_{\pi (2,3)}$, as indicated by dashed box 4.

Proposition 1.

Let $\pi$ denote a fixed landing sequence for flights. Under constraints (3), (4), and (5), sequence $\mathit{t}=\left\{{t_{\pi }^i}^{*}\right|i=1,2,3,\dots ,n\left)\right\}$ represents the landing times $t{c}_{\pi }={\sum }_{i=1}^{n}f({t_{\pi }^i}^{*})=\mathit{min}\left(F\left(t_{\pi }^n\right)\right)=F\left({t_{\pi }^n}^{*}\right)$ for each flight that minimize $t{c}_{\pi }$. Then, ${t_{\pi }^1}^{*}\le T_{\pi }^1$. Prove by contradiction as follows: First assume that under sequence $\pi$, there exists sequence $\mathit{t}\prime =\left\{{t_{\pi }^1}^{\prime }\right|i=1,2,3,\dots ,n\}$ such that the minimum value $t{c}_{\pi }$ is obtained, $t{c}_{\pi }={\sum }_{i=1}^{n}f({t_{\pi }^i}^{\prime })$, and ${t_{\pi }^1}^{\prime }>T_{\pi }^1$. Then, the landing time of the first flight is advanced from ${t_{\pi }^1}^{\prime }$ to $T_{\pi }^1$. Since this adjustment does not affect the safety intervals or landing times of subsequent flights, their arrival times remain unchanged. Thus, the $\pi$ sequence lands in the order determined by $\mathit{t}”=\{T_{\pi }^1,{t_{\pi }^1}^{\prime },\dots ,{t_{\pi }^n}^{\prime }\}$. At this point, $t{c}_{\pi }=f(T_{\pi }^1)+{\sum }_{i=2}^{n}f({t_{\pi }^i}^{\prime })=0+{\sum }_{i=2}^{n}f({t_{\pi }^i}^{\prime })<h_{\pi }^1\times \left({t_{\pi }^1}^{\prime }-T_{\pi }^1\right)+{\sum }_{i=2}^{n}f({t_{\pi }^i}^{\prime })$, and sequence $\mathit{t}″$ can achieve a lower cost. Assuming this contradicts Proposition 1, ${t_{\pi }^1}^{\prime }\le T_{\pi }^1$ is proven. The time window for the first flight can be optimized to $\left[E_{\pi }^1, T_{\pi }^1\right]$.

Inference 1.

Similarly, by contradiction, the optimal landing time for the last flight n cannot be earlier than its Earliest Landing Time. Therefore, the time window for the nth flight can also be constrained to optimize to $\left[T_{\pi }^n, L_{\pi }^n\right]$ 

Inference 2.

Based on Proposition 1, Corollary 1, Constraint (3), and Constraint (4), Equations (12) and (13) can be derived.

$$e_{\pi }^i=\left\{\begin{array}{cc}{T}_{\pi }^i\hfill & i=n\\ \mathit{max}\left(E_{\pi }^i,\mathit{min}\left(T_{\pi }^i,e_{\pi }^{i+1}-S_{\pi (i+1,i)}\right)\right)\hfill & i<n\end{array}\right.$$

(12)

$$l_{\pi }^i=\left\{\begin{array}{cc}{T}_{\pi }^i\hfill & i=1\\ \mathit{min}\left(L_{\pi }^i,\mathit{max}\left(T_{\pi }^i,l_{\pi }^{i-1}+S_{\pi (i-1,i)}\right)\right)\hfill & i>1\end{array}\right.$$

(13)

Inference 2 provides the landing time window $\left[e_{\pi }^i,l_{\pi }^i\right]$ after boundary optimization. Figure 6 shows the search range after boundary optimization, using three flights as an example. For Flight 1, the search range is narrowed to $\left[E_{\pi }^1,T_{\pi }^1\right]$; for Flight 3, it is narrowed to $\left[T_{\pi }^3,L_{\pi }^3\right]$. For Flight 2, if $T_{\pi }^2>e_{\pi }^3-S_{\pi (2,3)}$, $E_{\pi }^2>e_{\pi }^3-S_{\pi (2,3)}$ and $T_{\pi }^2>l_{\pi }^1+S_{\pi (1,2)}$ occurs, its search range is narrowed to $\left[E_{\pi }^2,T_{\pi }^2\right]$. The dashed box indicates the search range eliminated compared to DPA optimization, while the dashed line denotes omitted search steps. The search strategy for the remaining portions remains unchanged. The search range required to find the minimum landing cost for the first i + 1 flights also correspondingly decreases, $t{c}_{\pi }^{i+1}=\mathit{min}\left(F\left(e_{\pi }^{i+1}\right),\dots ,F\left(l_{\pi }^{i+1}\right)\right)$. Compared to DPA, this approach significantly reduces computational steps and simplifies algorithmic complexity.

Figure 7 shows the initial time windows [$E_{\pi }^i$,$L_{\pi }^i$] and the boundary-optimized time windows [$e_{\pi }^i$,$l_{\pi }^i$] for 10 flights in the Airland1 dataset from OR-Library [7]. The dashed lines indicate the search range for DPA, while the solid lines represent the search range for DPALO. Boundary optimization significantly reduces the computational scope of the algorithms. It reduces ∆T and lowers the algorithm’s complexity.

2.4. Solving for the Optimal Landing Sequence

2.4.1. Monte Carlo Tree Search

To solve for the optimal landing sequence, the TMCTS approach is employed. The landing cost computed using the method in Section 2.3 serves as the primary optimization metric for evaluating candidate landing sequences during the MCTS.

MCTS is a heuristic search algorithm based on random simulation and tree structures, particularly suitable for scenarios with large state spaces where exhaustive search methods are computationally infeasible. Figure 8 illustrates the workflow of the MCTS process. The specific steps are as follows:

(1) Selection. Starting from the root node, where a “node” represents a scheduling state at a given decision step. The algorithm traverses downward through the tree structure to identify a node worthy of further exploration. The core action selection principle is based on the Upper Confidence Bound (UCB), which balances exploration and exploitation. The UCB formula is defined as follows:

$$\mathit{UCB}= {\displaystyle \frac{Q\left(n\right)}{N\left(n\right)}}+C\times \sqrt{{\displaystyle \frac{\mathit{ln}{N}_p\left(n\right)}{N\left(n\right)}}}$$

(14)

Among these, Q(n) represents the cumulative reward of node n; N(n) denotes the number of visits to node n; ${\displaystyle \frac{Q\left(n\right)}{N\left(n\right)}}$ is the utilization term; C is the exploration coefficient; N_p(n) indicates the number of visits to the parent node; $\sqrt{{\displaystyle \frac{\mathit{ln}{N}_p\left(n\right)}{N\left(n\right)}}}$ is the exploration term.

Traditional MCTS relies on fully random simulations to evaluate node values, resulting in numerous futile searches that reduce computational efficiency. This paper employs a Transformer to generate action policies, state value estimates, and specialized decisions, providing MCTS with prior probabilities. This optimizes search logic during expansion and simulation phases, reducing ineffective exploration. The optimized UCB formula is

$$\mathit{UCB}={\displaystyle \frac{Q\left(n\right)}{N\left(n\right)}}+C\times p\left(a\right)\times \sqrt{{\displaystyle \frac{\mathit{ln}{N}_p\left(n\right)}{N\left(n\right)}}}$$

(15)

Among these, p(a) is the prior probability of selecting action a at node n.

(2) Expansion. When node n is not in a terminal state and unexplored actions exist, perform tree extension on node n. Randomly select an action a from the unexplored action set of node n, generate the corresponding child node n′, and initialize the utility and exploration values of the new node. After simulation, proceed with scoring. The scoring calculation process for each node is as follows:

$$Q=Va{l}_{nn}\times e^{-{\displaystyle \frac{c-c_g}{c_g}}}$$

(16)

Among these, Val_nn is the normalized cost advantage of the value-head output (with a range of [−1, 1]); a higher value indicates that the expected final cost of the current sub-sequence is more favorable than the Target Time Sequence (TTS) baseline. c represents the cost of selecting this node, and c_g represents the cost of TTS at this step. TTS is a sorting method that arranges flights to land in the target time order and serves as the baseline for evaluation.

(3) Simulation. The potential reward of a newly expanded node n′ is rapidly evaluated through simulation. Starting from node n′, actions are continuously executed according to a predefined policy until a terminal state is reached, ultimately yielding the reward value R for that trajectory. By repeatedly performing simulations, MCTS progressively explores different ordering strategies.

(4) Backpropagation. During the backpropagation phase, the reward R obtained from the simulation is propagated backward through all ancestral nodes along the path (from n′ back to the root node). This process updates the node ratings and visit counts of each node, providing evaluation criteria for subsequent selection phases.

By iteratively executing these four steps, a search tree is progressively constructed. Through a finite number of iterations, the tree structure gradually converges toward high-value paths, ultimately yielding an optimal sequencing result.

2.4.2. Transformer Architecture

The ALP involves multiple constraints, strong interdependencies, and sequential dependencies. This paper proposes a Transformer-based flight sequencing algorithm. The Transformer architecture is primarily employed to address aircraft scheduling decisions by capturing the features and interdependencies of each aircraft within the sequence. It outputs action policies and state values, providing prior probabilities for MCTS.

The Transformer architecture, as illustrated in Figure 9, primarily consists of two components:

(1) Transformer Encoder

The model architecture consists of stacked encoder layers, each comprising a multi-head attention module, a feed forward neural network, residual connections, and layer normalization. Through multi-layer stacking, feature encoding progressively becomes more abstract and global in scope.

The Embeddings Projections Layer maps the multidimensional features of each flight into a fixed-dimensional space via linear transformation, generating flight feature vectors. This enables the model to process features as numerical representations. Sine–cosine positional encoding is applied to incorporate the positional information of flights within the sequence, endowing the model with awareness of temporal order. Positional encodings are combined with state embeddings before being fed into the model, as defined by the following equations:

$$P{E}_{pos,2i}=\mathit{sin}({\displaystyle \frac{pos}{{10000}^{2i/d_{\mathrm{model}}}}})$$

(17)

$$P{E}_{pos,2i+1}=\mathit{cos}({\displaystyle \frac{pos}{{10000}^{2i/d_{model}}}})$$

(18)

where pos denotes the position of a flight within the sequence, and i represents the feature dimension index.

The Multi-Headed Self-Attention layer enables each flight feature to attend to all other flight features, capturing global dependencies. It allows the model to identify relationships between aircraft while simultaneously handling interactions between individual and multiple flights. Flight features, after positional encoding and embedding, are used as Query, Key, and Value inputs. Attention weights are computed between each flight and every other flight in the sequence.

The feed forward network layer applies a nonlinear transformation to the output vectors from the self-attention mechanism, enhancing the model’s expressive capacity to learn more complex feature patterns. Residual connections and layer normalization are incorporated to mitigate gradient vanishing during training, ensuring stable feature extraction for flight states.

(2) Multi-Task Output Layer

Based on the learned decision features, the model outputs two types of result. The policy head outputs a probability distribution over aircraft to be selected as the next action, which guides the MCTS in choosing subsequent moves. The value head estimates the value of the current state. This value is used to compute node scores within MCTS.

3. Model Training

3.1. Training Framework

During training, an Actor–Critic framework guided by MCTS is employed. As illustrated in Figure 10, the Transformer simultaneously outputs a policy distribution and a state value estimate. By interacting with the environment to collect data optimized through MCTS, the network is updated via the joint optimization of policy loss and value loss, enabling the agent to improve its policy within the target environment.

First, flight information and state representations are encoded into feature vectors by the Transformer encoder, which outputs both a policy distribution and a value estimate for the current state. The policy output provides prior action probabilities for MCTS, while the value estimate guides MCTS in action selection. MCTS constructs a search tree and ultimately outputs an MCTS policy distribution, a ground-truth value target, and a selected action.

A multi-objective loss function is defined to jointly optimize the policy network and the value network. The policy loss employs Kullback–Leibler (KL) divergence, encouraging the policy network’s prior probabilities to converge toward the optimal action distribution discovered by MCTS through deep search. Value loss is trained using Mean Squared Error (MSE) to bring the value head’s predictions closer to the calculated target values. A target value is calculated for each complete round based on the final scheduling result. If the round generates a feasible landing sequence, the final total cost of that sequence is compared with the cost of the TTS baseline. The difference between the two is scaled to the range [−1, 1] via a tanh transformation: when the final cost is significantly lower than the TTS baseline, the target value approaches +1; when it is on par with the baseline, it approaches 0; it approaches −1 when it is significantly higher than the baseline. If no feasible schedule can be generated, the target value is directly set to −1. The value head is trained to predict this target from any intermediate state, thereby learning to estimate the normalized cost advantage of the current partial sequence.

Network parameters are optimized via gradient descent, with Transformer parameters updated through backpropagation. Through multiple training iterations, the network progressively approximates the decision-making capability and value estimation accuracy of MCTS.

3.2. Experimental Data

The experimental data were obtained from the ALP dataset in J.E. Beasley’s OR-Library, accessible via http://people.brunel.ac.uk/~mastjjb/jeb/orlib/airlandinfo.html (accessed on 20 January 2024). The dataset consists of 13 flight sequences, each containing a varying number of flights. Sequences 1 through 8 constitute the small-scale subset, with 10, 15, 20, 20, 20, 30, 44, and 50 flights, respectively. The remaining five sequences form the large-scale subset, comprising 100, 150, 200, 250, and 500 flights, respectively.

Each flight in the dataset is characterized by seven attributes. The final attribute specifies the required wake separation time interval between the current flight and its predecessor. This information is structured as a matrix where n denotes the number of flights, and n_ij represents the wake separation interval required for flight j to follow flight i. The value of n_ij depends on the aircraft types of both flights.

3.3. Parameter Settings

In the flight scheduling analysis based on deep MCTS, a Transformer model combined with an MCTS strategy is employed, with the corresponding parameters listed in

Table 2. Parameter settings.

Model	Parameter	Value
MCTS	Number of simulations	128
	Exploration constant	1.414
	Batch search quantity	32
Transformer	Basic feature dimensions	256
	Number of attention heads	8
	Number of encoder stacking layers	6
	Dropout	0.1
	Initial learning rate	0.0003
	Experience replay buffer capacity	10,000
	Number of training batches	128

.

The parameter design of the model balances the complexity of the ALP with computational efficiency. For the search strategy, the MCTS module uses a simulation search depth of 128 and a batch size of 32, enabling thorough exploration and exploitation of the decision space with limited computational resources. The exploration constant is set to 1.414, achieving a balanced trade-off between exploration and exploitation based on the classic Upper Confidence Bound (UCB) formula, thereby preventing premature convergence to local optima.

Regarding the model architecture, a Transformer-based encoder structure is adopted with an embedding dimension of 256. It consists of six encoding layers, each equipped with eight self-attention heads. Each attention head focuses on feature representations within a 32-dimensional subspace, enabling the parallel capture of multidimensional scheduling features such as time window constraints, penalty cost weights, and separation requirements. The hidden layer of the feed forward network has a dimension of 512. During training, dropout regularization is applied with a rate of 0.1 to mitigate overfitting. Positional encoding supports sequences of up to 500 elements, accommodating large-scale scheduling scenarios.

The experimental setup employs the Adam optimizer with a learning rate of 3 × 10⁻⁴, an experience replay buffer capacity of 10,000, and a batch size of 128.

3.4. Training Methods

The training process adopts a hybrid approach combining behavior cloning and reinforcement learning. It is divided into two main stages: pre-training (distillation phase) and reinforcement learning training.

Given the limited volume of data in existing datasets and the fact that Transformer models require substantial data to learn effective strategies, a pre-training phase is conducted before the main training. A large amount of flight-like data was generated based on the original dataset. For ALPs of different scales, genetic algorithms were used to produce extensive trajectory data. These trajectories include sequences of decisions from initial to terminal states, along with corresponding expert actions for each decision. The Transformer network then underwent behavior cloning training on this trajectory data, learning to imitate the decision-making of the genetic algorithm. The goal of this phase is to enable the network to initially capture basic scheduling decision patterns, providing a solid foundation for subsequent reinforcement learning. In this work, 200,000 simulated flight data points were generated, and the dataset was trained for 300 epochs.

Following pre-training, MCTS is employed as the policy improvement algorithm. Data is generated through the “select-expand-simulate-backtrack” process, and this data is used to further train the network. At each training step, the current network guided the MCTS. The network was then updated based on the policy distribution obtained from the MCTS and the final outcome. An experience replay buffer was used to store training data, with samples periodically drawn from the buffer for training. Curriculum learning was employed, starting with a small dataset and gradually increasing its size as model performance improved.

4. Results and Discussion

4.1. Results

The experimental setup includes an Intel Core i5-12400F processor (Intel Corporation, Santa Clara, CA, USA) and an NVIDIA GeForce RTX 4060 Ti graphics card (NVIDIA Corporation, Santa Clara, CA, USA), and 16 GB of RAM. All implementations were developed in Python 3.10.16.

Figure 11 illustrates the loss curves of the TMCTS model over 300 training epochs. The policy network loss converged to 0.0275, the value network loss converged to 0.0046, and the total loss converged to 0.0229. The value network loss decreased rapidly, converging and stabilizing within the first 10 epochs. The policy network loss gradually converged to 0.0275 as training progressed, stabilizing after approximately 250 epochs.

For the ALP, MCTS should prioritize depth-first over breadth-first search logic. The decision tree structure generated by MCTS is illustrated in Figure 12, which depicts the action selection process. A key characteristic of ALP lies in the sequential dependency of decisions: selecting a specific aircraft for landing not only affects the scheduling window of adjacent aircraft but also alters the feasible time intervals for distant aircraft through the propagation of separation constraints. Therefore, the search algorithm must thoroughly explore the complete outcomes of decision sequences. While a breadth-first strategy can evaluate more actions at shallow levels, fully exploring the scheduling sequences of each branch leads to an exponential increase in computational complexity.

The number of MCTS simulations influences both model accuracy and computational time. In this study, MCTS simulations were performed with 32, 64, 128, and 256 iterations. The results obtained for each simulation count, compared against the best-known values (BKV) [30], are presented in

Table 3. Computational results for different simulation runs.

Group	BKV	TMCTS
		Num = 32		Num = 64		Num = 128		Num = 256
	TC	TC	t (s)	TC	t (s)	TC	t (s)	TC	t (s)
1	700	700	0.75	700	0.977	700	1.573	700	2.517
2	1480	1480	0.351	1480	0.683	1480	1.511	1480	2.804
3	820	820	0.457	820	0.962	820	1.919	820	3.677
4	2520	2520	0.536	2520	0.884	2520	2.032	2520	3.675
5	3100	6260	0.451	3100	0.889	3100	2.111	3100	3.656
6	24,442	24,442	3.114	24,442	6.301	24,442	12.844	24,440	25.559
7	1550	1550	1.569	1550	3.214	1550	6.686	1550	13.819
8	1950	2285	1.221	2025	2.499	1860	5.196	1860	9.972
9	5611	5907.85	5.821	5861.51	11.423	5748.73	24.811	5706.44	58.409
10	12,329	14,142.58	11.902	13,696.76	22.52	12,703.58	53.382	12,793.52	119.577
11	12,418	12,698.92	20.691	12,675.54	39.035	12,503.92	85.725	12,503.92	172.669
12	16,209	17,192.69	31.114	17,318.36	66.428	16,725.44	121.36	16,803.22	256.906
13	44,832	38,134.56	145.918	38,549.12	302.504	38,629.45	644.307	38,444.89	1221.15
Average difference	——	13.277	——	−247.9	——	−398.298	——	−403	——
Average time	——	——	17.22	——	35.255	——	74.112	——	145.722
Average time for the first 8 groups	——	——	1.056	——	2.051	——	4.234	——	8.21
Average time for the last 5 groups	——	——	43.089	——	88.382	——	185.917	——	365.742

.

As shown in Table 3, when the number of simulations is 128, the average difference reaches −398.298, and the computation time is relatively fast.

On small-sample datasets (first 8 datasets), achieving BKV is possible with 128 simulations, and outperforming BKV occurs in the 8th dataset. When simulations exceed 128, computational results remain unchanged while average computation time increases from 4.234 s to 8.21 s—a 93.9% rise significantly reducing computational efficiency. When the simulation count is below 128, computational efficiency improves only marginally while compromising accuracy. For the eighth dataset, a simulation count of 64 yields a cost of 2025, exceeding the cost of 1860 achieved at 128 simulations.

For large-scale datasets (the last five datasets), the number of simulations substantially affects computation time, which approximately doubles with each doubling of the simulation count. Comparisons indicate that 128 simulations provide both shorter computation times and higher computational accuracy. Therefore, the results obtained with 128 simulations are selected for subsequent comparative analysis.

To verify whether the MCTS-guided reinforcement learning fine-tuning phase can provide incremental gains for pre-trained policies, we compared the performance of T-only (which uses only the Transformer policy head to directly output actions without MCTS) and TMCTS across different-scale cases. The results are presented in

Table 4. Performance comparison between T-only and TMCTS.

Group	T-Only	TMCTS	Improvements
9	6892.78	5748.73	16.60%
10	18,060.00	12,703.58	29.66%
11	13,417.16	12,503.58	6.81%
12	17,594.72	16,725.44	4.94%
13	39,435.04	38,629.45	2.04%
Average	19,079.94	17,262.22	12.01%

.

The results show that, in large-scale complex scheduling scenarios, TMCTS achieved lower total costs than T-only across all test cases, with an average improvement of 12.01%. This demonstrates that the MCTS-guided RL fine-tuning phase can effectively optimize pre-trained strategies and overcome their performance bottlenecks in large-scale problems.

We compared five algorithms—FCFS, DPALO+GA [29], DPALO+PSO [29], and CPLEX [5]—against the proposed TMCTS approach. The mean interpolation values and average computation times of these algorithms, along with BKV, were calculated on the same dataset. The comparison results are presented in

Table 5. Comparison of results from various algorithms.

Group	BKV	FCFS	TMCTS		DPALO+GA		DPALO+PSO		CPLEX
			TC	t (s)	TC	t (s)	TC	t (s)	TC	t (s)
1	700	1280	700	1.573	700	0.004	700	0.006	700	0.8
2	1480	1790	1480	1.511	1480	0.028	1482	0.046	1480	0.8
3	820	1790	820	1.919	820	0.025	820	0.058	820	1.3
4	2520	4890	2520	2.032	2520	0.004	2520	0.006	2520	2.3
5	3100	6470	3100	2.111	3100	0.043	3244	0.159	3100	3.7
6	24,442	24,442	24,442	12.844	24,442	0.004	24,442	0.006	24,442	0.7
7	1550	1550	1550	6.686	1550	0.005	1550	0.007	1550	1.2
8	1950	18,870	1860	5.196	1863.75	0.056	1863.25	0.185	1950	2.2
9	5611	18,937.16	5748.73	24.811	5694.52	2.541	5675.01	3.223	5611.7	932.2
10	12,329	27,660	12,703.58	53.382	13,826.42	5.727	14,799.94	4.615	12,292.2	3600
11	12,418	35,532.91	12,503.92	85.725	12,671.36	4.706	12,922.42	5.506	12,418.3	3600
12	16,209	46,471.72	16,725.44	121.36	17,249.11	6.72	17,626.09	6.849	16,122.2	3600
13	44,832	99,776.58	38,629.45	644.307	38,724.21	21.91	41,323.23	12.052	37,848.9	3600
Average difference	——	12,423.03	−398.298	——	−255.36	——	77.457	——	−546.6	——
Average time	——	——	——	74.112	——	3.214	——	2.517	——	1180.4

.

Table 4 demonstrates that the TMCTS approach achieves a significant improvement in average deviation compared to FCFS. Adhering to FCFS under large sample sizes would incur substantial landing costs—specifically, a landing cost of 99,776.58 when handling 500 flights, more than double that of the adjusted methods. Compared to heuristic algorithms DPALO+GA and DPALO+PSO, improvements of 142.938 and 475.755 were achieved, respectively. In terms of computational time, TMCTS achieved a 93.7% improvement over the MIP solver CPLEX, though it still lags behind other algorithms. This is primarily because relying solely on the Transformer’s direct output leads to significant computational inaccuracies. On certain datasets, it may fail to complete the sequencing task, requiring MCTS to search for the optimal ordering sequence. During MCTS simulation, the complexity of the action space and state space grows exponentially. To achieve higher computational accuracy, substantial time is required for MCTS to explore better paths.

When the sample size is less than 50 (first 8 datasets), optimal results can be provided directly. All algorithms achieved the BKV value in the first 7 datasets. For the eighth dataset, the TMCTS computation yielded 1860, outperforming both BKV’s 1950 and previous algorithms while simultaneously ensuring computational accuracy and speed. When the sample size exceeded 100 (the last five datasets), the TMCTS results showed minimal deviation from existing algorithms, maintaining computational accuracy. However, computational efficiency was limited due to algorithmic constraints.

4.2. Discussion

The hybrid learning framework proposed in this paper consists of two stages: behavior cloning pre-training based on genetic algorithm (GA) trajectories, and MCTS-guided reinforcement learning fine-tuning. A potential concern is that, as a stochastic heuristic algorithm, GA may converge to a local optimum, and the trajectory data it generates may impart this “expert bias” to the pre-trained policy. Therefore, it is necessary to verify whether the RL fine-tuning stage can effectively overcome this bias. Theoretically, the UCB formula in the MCTS process encourages the algorithm to actively deviate from the pre-trained prior policy and explore under-visited regions of the action space. When exploratory selections lead to better scheduling outcomes, positive reward signals are backpropagated through the Actor–Critic framework to update the Transformer’s policy network. In terms of experimental evidence, if RL fine-tuning fails to effectively overcome pre-training bias, the performance of TMCTS should not significantly outperform the GA that generated the pre-training data. However, the comparison results in Table 4 show that in Case 8, TMCTS (total cost 1860) outperformed the DPALO+GA results reported in the literature (1863.75); in Case 13, TMCTS (38,629.45) also outperformed DPALO+GA (38,724.21). Furthermore, in Cases 10–12, TMCTS achieved lower total costs than DPALO+GA. These results indicate that MCTS-guided RL fine-tuning successfully guided the agent to explore policy spaces not fully covered by the GA pre-training data, thereby discovering optimal scheduling sequences.

A notable result is evident in the performance of Case 8: the total cost of the TMCTS algorithm proposed in this paper is only 1860, surpassing the known optimal value (BKV) of 1950. Looking at the sequence results, all flights in Case 8 satisfy the minimum separation requirements. The change in cost is due to a shift in the position of Flight 15. Flight 15 has a target landing time of 409, with early arrival and late arrival penalties of 10 each. In the 1950 sequence, Flight 15 lands at 387, with a cost of 220. In the 1860 sequence, Flight 15 is advanced to land at 422, with a cost of 130, reducing the total cost by 90. This also results in the separation matrix violating the triangle inequality (i.e., S1860(25,24) > S1950(25,15) + S1950(15,24)). The structural property that the separation matrix follows the triangle inequality may pose a challenge to classical solution methods (such as CPLEX) that rely on convexity assumptions, and may also mislead certain heuristic algorithms that depend on transitive relationships. The TMCTS algorithm proposed in this paper, based on a Transformer architecture, exhibits stronger robustness to such data anomalies. The multi-head self-attention mechanism enables the model to directly learn and represent dependencies between flights without the need to satisfy assumptions such as the triangle inequality. The MCTS then explores the sequence space based on these learned representations.

5. Conclusions

For the ALP, this paper proposes a deep reinforcement learning method based on MCTS and Transformers. The model outputs the landing sequence and landing times for flights to minimize landing costs. The approach formulates flight scheduling as an MDP and designs a dual-head Transformer network comprising policy and value heads. Leveraging its global attention mechanism, the network captures dependencies among flights and provides prior policy guidance for MCTS. Training is conducted within an Actor–Critic framework, combining behavior cloning pretraining with reinforcement learning optimization.

The experiment utilized the OR-Library dataset. Results indicate that the TMCTS approach achieved a significant improvement in average deviation compared to FCFS, with relative improvements of 142.938 and 475.755 over DPALO+GA and DPALO+PSO, respectively. In terms of computation time, TMCTS achieved a 93.7% improvement over the MIP solver CPLEX, though it remains slower than other benchmark algorithms. The proposed method is able to approximate or outperform known optimal solutions in most cases. However, the number of MCTS simulations constrains the algorithm’s search efficiency. Future improvements could involve expanding training data and extending training time to enhance the quality of the initial policy, thereby reducing the reliance on MCTS simulations and improving overall algorithmic efficiency. The proposed method can assist the AMAN system. First, by statistically analyzing historical flight trajectory information and integrating aircraft performance data with the terminal area route network, it calculates the landing time window (i.e., the earliest and latest possible landing times) for each flight. Second, by receiving real-time flight trajectory data, it computes the landing sequence and timing with the minimum cost for each flight. Compared to traditional rule-based optimization methods, the proposed approach is data-driven and capable of identifying optimization patterns from historical data, resulting in a flight sequence with a lower total cost. This provides new technical support for practical air traffic arrival management and holds significant reference value.

Future improvements could involve expanding the training dataset with more diverse flight scenarios and extending the training epochs to further enhance the quality of the initial policy. This would reduce the reliance on MCTS simulations during inference and improve overall algorithmic efficiency.