A Hybrid Deep Reinforcement Learning Framework for Vehicle Path Optimization with Time Windows

Abstract

The vehicle routing problem with time windows (VRPTW) is a core challenge in logistics optimization, requiring the minimization of transportation costs under constraints such as time windows and vehicle capacity. Deep reinforcement learning (DRL) provides an effective approach for solving such complex combinatorial optimization problems. However, existing DRL methods still suffer from shortcomings, including insufficient modeling of spatiotemporal correlations among customer nodes, inadequate capture of path temporal dependencies, and policy exploration prone to local optima. To address these issues, this paper proposes an end-to-end hybrid DRL framework: the encoder employs a graph attention network (GATv2) with adaptive gating to effectively model the coupling between customer spatial proximity and time window constraints; the decoder integrates multi-head attention (MHA) and a dynamic context-aware long short-term memory network (LSTM) to synergistically enhance the overall quality and constraint feasibility of route solutions; during the training phase, an improved proximal policy optimization (PPO) algorithm and a constraint-aware composite reward function are used to enhance optimization stability. Experiments on random instances, Solomon benchmark datasets, and real-world logistics datasets show that, compared to mainstream DRL methods and classical heuristic algorithms, the proposed framework reduces transportation costs by 2–10%, achieves a demand fulfillment rate exceeding 99%, and exhibits a performance degradation of only 3.2% in cross-distribution testing. This study provides an integrated DRL solution paradigm for combinatorial optimization problems with complex constraints, promoting the application of DRL in the field of intelligent logistics.

1. Introduction

The vehicle routing problem (VRP), as a classic combinatorial optimization problem, has been extensively studied in the fields of transportation and logistics management [1]. High-quality route planning not only enables the fulfillment of more customer demands within a limited time, thereby improving service levels, but also effectively reduces operational costs, leading to significant economic benefits [2]. This problem aims to schedule a fleet of vehicles to serve a set of given customer requests without violating constraints such as vehicle capacity and energy consumption, while minimizing overall travel costs [3]. The vehicle routing problem with time windows (VRPTW) is an important variant of VRP, which adds the constraint that customers must be served within specific time windows, making the problem more realistic and challenging in practical applications [4]. In actual delivery systems, customers not only require services to be completed within specified time frames, but factors such as traffic conditions, time management, and vehicle scheduling also need to be considered, increasing the complexity and uncertainty of the problem. Therefore, VRPTW not only requires optimizing vehicle routes but also must satisfy time window constraints to enhance delivery efficiency, reduce costs, and improve customer satisfaction. It holds significant practical application value in various fields, including package delivery, time-sensitive and emergency services [5], robot path planning [6], and electric vehicle services [7].

In recent years, deep reinforcement learning (DRL) technology, leveraging its powerful self-learning capability and advantages in efficiently modeling complex problems, can generate higher-quality solutions within reasonable running times. This has had a profound impact on solution methods for combinatorial optimization problems like the vehicle routing problem (VRP) [8]. More importantly, compared to traditional solutions, DRL-based approaches, after a single training session, can handle tasks of different scales but with the same structure without requiring additional retraining [9]. This also makes DRL-based solutions more flexible and adaptable when dealing with dynamic real-world scenarios like VRPTW, which involve time window constraints. The design of the encoder–decoder architecture is critical for capturing the spatiotemporal dependencies in VRPTW. Currently, research on solving VRPTW using end-to-end deep learning models primarily revolves around three core architectures: sequence modeling based on recurrent neural networks (RNN) [10], attention mechanisms based on Transformer [11], and graph neural network (GNN) architectures [12]. Since the node set in VRPTW can be represented as a graph, the structural information of the graph can be embedded into continuous node representations to generate solutions within these models. Among these, GNN architectures have attracted significant attention for their ability to explicitly model the spatial relationships between customer nodes. Through graph embedding techniques, they encode node features and topological structures into a continuous vector space, providing a structured representational foundation for subsequent policy networks [13]. However, despite the successes achieved by these methods, they still face key bottlenecks under complex time window constraints, such as insufficient feasibility modeling, inadequate utilization of structural information, and limited policy exploration capability. Specifically, this is reflected in: firstly, graph structure modeling is often limited to shallow aggregation of neighborhood information, making it difficult to capture high-order interaction features between customer nodes; secondly, existing sequential decoding mechanisms inadequately model the dynamic perception of contextual information and spatio-temporal dependencies during route generation, easily leading to violations of time window constraints and thus affecting the quality of generated routes; lastly, traditional exploration strategies exhibit a significant imbalance between search efficiency and diversity in the solution space, making it difficult to achieve a dynamic balance between solution quality and computation time, and challenging to obtain a near-optimal solution within a relatively short timeframe.

To address the above issues, this study proposes an end-to-end hybrid deep reinforcement learning (DRL) framework for the Vehicle Routing Problem with Time Windows (VRPTW). The framework adopts an adaptive gated GATv2 as the encoder to jointly learn the features of nodes and edges, effectively characterizing the graph topology; the decoder integrates a multi-head attention (MHA) mechanism and a long short-term memory (LSTM) network to perceive global dependencies and dynamic constraints. The training process is based on the proximal policy optimization (PPO) [14] algorithm to achieve stable optimization. The main contributions are summarized as follows:

(1)

An end-to-end hybrid Deep Reinforcement Learning (DRL) framework is constructed, which enables unified modeling of graph topology, spatio-temporal constraints, and routing decisions in VRPTW through the synergistic integration of an encoder–decoder architecture. This system-level design provides an integrated modeling paradigm for route optimization under complex constraints.

(2)

An innovative synergistic encoder–decoder structure is proposed: the encoder incorporates a GATv2 with an adaptive gating mechanism to enhance node-edge interaction modeling and suppress neighborhood noise; the decoder integrates MHA with a dynamic context-aware LSTM to capture global dependencies and model the dynamic characteristics of temporal constraints during route generation, thereby improving solution feasibility and adaptability.

(3)

A stable training framework based on an improved Proximal Policy Optimization (PPO) algorithm is established. To address the sparse rewards and constraint characteristics of VRPTW, techniques such as advantage function normalization, dynamic value loss adjustment, and a constraint-aware reward function are introduced to enhance training stability and exploration efficiency. Evaluations on random instances, the Solomon benchmark dataset, and real-world datasets show that compared to existing mainstream DRL methods, the proposed approach reduces transportation costs by 2–10% while maintaining inference speeds on the order of seconds.

The remainder of this paper is organized as follows: Section 2 reviews in detail traditional methods for the vehicle routing problem with time windows and related work on DRL methods. Subsequently, Section 3 provides a clear definition of VRPTW and details the architecture of the model proposed in this paper. Section 4 presents the experimental results of the model. Finally, Section 5 concludes the research of this paper.

2. Related Work

2.1. VRPTW Solution Methods Based on Conventional Algorithms

The vehicle routing problem and its extension with time windows (VRPTW) have long been primarily solved using traditional optimization algorithms, with related research broadly categorized into exact algorithms and heuristic algorithms. Exact algorithms are typically based on rigorous mathematical programming models, such as linear programming [15] and branch-and-bound methods [16], which can obtain globally optimal solutions for small-scale or structurally simpler VRP instances. However, as the problem scale expands and constraints such as time windows and capacity increase, the computational complexity of exact algorithms rises rapidly, making it difficult to obtain feasible solutions within a reasonable time. Consequently, they face significant limitations in practical VRPTW applications.

To enhance the scalability of algorithms under complex constraints, researchers have proposed numerous heuristic and hybrid metaheuristic methods. Wei et al. [17] designed a hybrid adaptive large neighborhood search (ALNS) algorithm to improve the solution accuracy for the vehicle routing problem with time window constraints. This method demonstrates good solution quality on VRPTW instances with relatively loose time windows, but its computational efficiency remains somewhat limited. Building on this, Liu et al. [18] proposed a parallel adaptive large neighborhood search algorithm (Spark ALNS) based on the Spark platform, which accelerates the solution process through parallelization strategies and partially mitigates the issue of algorithms getting trapped in local optima, though there is still room for improvement in solution quality. Maroof et al. [19] introduced a hybrid genetic algorithm combined with the Solomon insertion constructive heuristic. This method exhibits strong adaptability and robustness across various VRPTW scenarios, but its evolutionary process is still influenced by initial solutions and operator design, often leading to suboptimal solutions.

Overall, traditional exact algorithms struggle to scale to medium- and large-scale instances in complex VRPTW problems. While heuristic methods can achieve high solution quality in specific scenarios, they often rely on manual rules and parameter tuning, making it difficult to balance solution speed, stability, and generalization ability across different instances. This has motivated researchers to explore more adaptive, learning-based solution paradigms.

2.2. Deep Reinforcement Learning-Based Path Optimization Methods

In recent years, learning-based methods, particularly deep reinforcement learning (DRL), have gradually become a research hotspot for solving combinatorial optimization problems like VRPTW [20]. By modeling path planning as a sequential decision-making process, DRL methods enable models to adaptively learn decision-making strategies through interaction, thus demonstrating the potential to surpass traditional heuristic methods under complex spatiotemporal constraints and large-scale instances. Existing DRL-based VRPTW research primarily revolves around three directions: sequence modeling methods, Transformer-based methods, and graph neural network (GNN) architectures.

2.2.1. DRL Methods Based on Sequence Modeling

Early learning-based path planning research often relied on recurrent neural networks (RNNs) for sequence modeling. Vinyals et al. [21] proposed the Pointer Network (PtrNet), which selects output nodes from the input sequence via a softmax probability distribution, providing a general solution approach for the Traveling Salesman Problem (TSP) and other combinatorial optimization tasks. Building on this, Kong et al. [22] replaced the LSTM encoder of PtrNet with element-level graph embedding representations and combined it with a pointer mechanism-based decoder to generate solutions for multi-trip VRP instances. Subsequently, Bogyrbayeva et al. [23] proposed a DRL framework based on attention-enhanced LSTM, which, to some extent, improved the efficiency and solution quality of sequence models in solving VRP. Although the aforementioned methods have achieved some success in route sequence generation, the vehicle routing problem inherently possesses significant graph structural characteristics. Relying solely on sequence modeling often struggles to explicitly capture spatial proximity relationships, topological structures between nodes, and their dynamic correlations that change with decision stages. This leads to the model easily falling into suboptimal solutions in high-dimensional or densely distributed node scenarios, and limits its ability to express complex constraint information.

2.2.2. Transformer-Based Methods

Inspired by the successful application of Transformers in the field of natural language processing, recent years have seen numerous studies attempting to introduce it into combinatorial optimization problems. Fellek et al. [24] proposed an attention-based end-to-end DRL model for solving VRP. By embedding edge information between nodes into a representation space and leveraging Transformers’ powerful parallel attention capabilities for feature modeling, this approach outperforms traditional RNN methods in both convergence speed and solution quality. However, the model’s ability to capture relationships between edges in the graph remains relatively limited. Tang et al. [25] proposed a Transformer-based end-to-end DRL framework for the Electric Vehicle Routing Problem (EVRP). By introducing a feature embedding module between the encoder and decoder to fully utilize contextual information, it achieved performance superior to heuristic methods and other learning-based methods on EVRP tasks. Wu et al. [26] proposed a DRL framework for VRPTW, combining a Transformer-based policy network with NSGA II. While it demonstrated advantages in solution quality, its computational cost grows rapidly with the number of vehicles and problem scale, limiting its application in real-time or large-scale scenarios. Meng et al. [27] further introduced an edge-based Transformer model, comprehensively modeling global relationships between edges through parallel encoding strategies. During the decoding phase, they incorporated parallel contextual embeddings and a multi-query mechanism, achieving more efficient route construction. Overall, Transformer-based methods offer significant advantages in parallel computing and global dependency modeling. However, their computational complexity typically grows quadratically with the number of nodes, and they lack explicit modeling for route feasibility and constraint satisfaction, posing certain challenges in strongly constrained VRPTW scenarios.

2.2.3. DRL Methods Based on Graph Neural Networks

Given the rich structural information inherent in combinatorial optimization problems, recent years have seen a growing number of studies utilizing Graph Neural Networks (GNNs) to model relationships between nodes and edges in VRPTW. Chen et al. [28] proposed an encoder–decoder framework to solve the Vehicle Routing Problem with Mixed Delivery and Pickup (VRPMDP). The encoder uses GNNs to extract instance features, while the decoder constructs routes step-by-step via attention mechanisms. Experimental results validated the effectiveness of the method, though there remains room for further optimization in capturing relationships between nodes. Wang et al. [29] proposed an adaptive graph attention sampling and edge fusion framework, generating node embeddings by focusing on highly relevant neighborhoods and critical edge information, achieving performance improvements of 2.08–6.23% across multiple VRP instances. For the Multi-Depot Vehicle Routing Problem (MDVRP), Zhang et al. [30] developed an encoder–decoder model based on Graph Attention Networks (GAT), improving routing effectiveness by exploiting complex spatiotemporal correlations within time windows. However, its modeling of edge information remains relatively limited. Wang et al. [31] proposed an end-to-end DRL framework for the EVRP scenario, employing an improved GAT encoder to jointly model node and edge information. During decoding, a two-layer attention mechanism was introduced to generate solutions. Experimental results show that this method outperforms traditional methods and some DRL baselines, though its ability to suppress neighborhood noise during encoding is still insufficient. Bai et al. [32] proposed a Residual Edge Demand-integrated Graph Attention Network encoder combined with a Multi-attention hybrid decoder (REDGAT-MA). By integrating node and edge features and incorporating real-time constraint dynamic decoding, this work further validates the advantages of GNNs in modeling complex VRP tasks. Overall, GNN-based methods hold a clear advantage in capturing complex inter-node relationships. However, existing studies mostly rely on static graph attention structures, lacking adaptive modeling for neighborhood noise and dynamic spatiotemporal relationships. Additionally, there remains room for further improvement in the synergistic design between encoded representations and sequential decoding.

In summary, while existing methods have achieved rich results in solving VRPTW, there remain shortcomings in the fine-grained modeling of spatiotemporal correlations among customer nodes, capturing long-term temporal dependencies during route generation, and the stability of policy optimization. Precisely to address these issues, this paper proposes a hybrid deep reinforcement learning framework that integrates an adaptive gated GATv2 encoder, a dynamic context-aware decoding mechanism, and an improved PPO algorithm, aiming to achieve efficient solving of complex constrained VRPTW problems.

3. Methods

In this section, we begin by clearly defining the Vehicle Routing Problem with Time Windows (VRPTW) and outlining the basic premises and optimization objectives of the study. Subsequently, we systematically introduce the research methods employed, with a detailed explanation focusing on the structural design of the encoder and decoder, as well as the overall algorithmic workflow.

3.1. Problem Definition

VRPTW is defined on a complete directed graph G = {$V$, $E$}. The node set is $V=\left\{v_0,v_1,\cdots ,v_n\right\}$, where $v_0$ is the depot, and the remaining nodes are customers. Each node $v_i$ possesses several features, including its coordinates $\left(x_i,y_i\right)$, time window ${\tau }_i$, and demand $q_i$ that must be satisfied. The time window ${\tau }_i$ is an interval, denoted as $\left[a_i,b_i\right]$, where $a_i$ and $b_i$ represent the lower and upper bounds of the permissible service time interval, respectively. If a vehicle arrives at customer $v_i$ later than $b_i$, it will incur a corresponding penalty. For the depot node, we set $d_0$ = 0 and $t_0$ = $[a_0$, $b_0]$, where $a_0$ and $b_0$ represent the earliest departure time and the latest return time for vehicles at the depot, respectively. $\mathrm{E}=\left\{e_{ij}|v_i,v_jϵ\mathrm{V}\right\}$ is the edge set of graph G. Each edge $e_{ij}$ has two features: the travel distance $d_{ij}$ and the travel time $t_{ij}$, which are asymmetric, i.e., $d_{ij}$ ≠ $d_{ji}$ and $t_{ij}$ ≠ $t_{ji}$. For ease of description, we summarize the notations to be used in Table 1.

Table 1. Symbol Description.

Notation	Definition
$V=\left\{v_0,v_1,\cdots ,v_n\right\}$	set of vertex, $v_0$ for depot
N	number of customer nodes
$v_i,i\ne 0$	customer nodes
$\left(x_i,y_i\right)$	coordinate of $v_i$
$q_i$	demand of $v_i$
${\tau }_i=\left[a_i,b_i\right]$	time window of $v_i$
$\mathrm{E}=\left\{e_{ij}|v_i,v_jϵ \mathrm{V}\right\}$	edge set
$d_{ij}$	traveling distance of $e_{ij}$
$t_{ij}$	traveling time of $e_{ij}$
$r_{ij}$	binary decision variable to indicate edges in solution
$s_i$	service time of $v_i$
${slack}_{ij}$	time window surplus from node $i$ to $j$
Q	initial vehicle capacity
M	number of vehicles
K	the number of time steps
$t$	current time step
h	node embedding vector
$\pi$	solution sequence
${\pi }_t$	solution node at step $t$
S	current state of the DRL environment
$\mathsf{\theta }$	neural network parameters
$p_{\mathsf{\theta }}\left(·\right)$	probability under neural network parameter θ
L	total solution length

The scenario of the problem is set as follows: there is a fleet of vehicles, each with an initial capacity Q > 0, and their work schedule is denoted as T. The vehicles depart from the depot, serve customers under the constraints of capacity and time windows with the objective of minimizing the total route length, and finally return to the depot. The constraints of VRPTW are as follows:

$$\mathit{min}{\sum }_{i\in V}{\sum }_{j\in V,j\ne i}{d}_{ij}{r}_{ij}, \forall i,j\in \mathrm{V},{\mathrm{r}}_{ij}\in \left\{\mathrm{0,1}\right\}$$

(1)

$${\sum }_{i\in V}{r}_{ij}=1, \forall j\in V\backslash \left\{0\right\}$$

(2)

$${\sum }_{j\in V}{r}_{ij}=1, \forall i\in V\backslash \left\{0\right\}$$

(3)

$${\sum }_{i\in V}{r}_{i0}=M$$

(4)

$${\sum }_{j\in V}{r}_{0,j}=M$$

(5)

$$a_i\le t_i\le b_i\hspace{1em}\forall i\in V\backslash \left\{0\right\}$$

(6)

$$t_j\ge t_i+s_i+t_{ij}-\mathrm{C}\left(1-r_{ij}\right)\forall i,j\in V\backslash \left\{0\right\}$$

(7)

$${\sum }_{i\in R_k}{q}_i\le Q \forall k=1,\cdots ,\mathrm{M}$$

(8)

$${\sum }_{i\notin P}{\sum }_{j\in P}{r}_{ij}\ge r\left(P\right),\hspace{1em}\hspace{1em}\forall P\subseteq V\backslash \left\{0\right\}, P\ne \mathrm{\varnothing }$$

(9)

where $r_{ij}$ is a binary decision variable indicating whether edge $e\left(i,j\right)$ is part of the solution, and $d_{ij}$ is the cost (distance) of using $e\left(i,j\right)$. M is the number of vehicles being used, and $r\left(P\right)$ is the minimum number of vehicles required to serve the customer set. C is a sufficiently large constant used to linearize the temporal continuity constraints. Equation (1) provides a compact mathematical expression of the VRPTW objective. In the DRL framework of this paper, multiple vehicles are implicitly modeled through multiple returns to the depot within a single solution sequence. Constraints (2) and (3) ensure each customer is visited exactly once. Constraints (4) and (5) guarantee the satisfaction of the vehicle count. Constraints (6) and (7) ensure the vehicle’s arrival time falls within the time window and specify the vehicle’s arrival time at the next customer. Constraint (7) applies only to scenarios where the vehicle visits customer nodes $i$ and $j$ consecutively without returning to the depot for replenishment. If the remaining capacity of the vehicle after serving node $i$ is insufficient to meet the demand of node $j$, then a direct transfer from $i$ to $j$ is deemed infeasible during the decoding stage; the vehicle must first return to the depot for replenishment before proceeding to subsequent customer nodes. Constraint (8) ensures that the capacity of each vehicle does not exceed its maximum capacity, where $R_k$ denotes the set of customer nodes served by vehicle $k$. Constraint (9) ensures the demands of all customers are fully satisfied. Therefore, given a VRPTW environment $G$ and a solution π, the objective under the DRL architecture is to minimize the total solution length $L\left(\pi |G\right)$:

$$L\left(\pi |G\right)={{\sum }_{i=0}^{\left|\pi \right|-1}\Vert c_{\pi \left(i\right)}-c_{\pi \left(i+1\right)}\Vert }_2+{\Vert c_{\pi \left(\left|\pi \right|\right)}-c_0\Vert }_2$$

(10)

where $\Vert ·\Vert$ calculates the $L_2$ distance between node pairs, $c_0=c_{\pi +1}$, and $\left|\pi \right|$ denotes the sequential length of solution $\pi$. $c_{\pi \left(i\right)}$ denotes the coordinate vector of the $i$-th element in the route sequence $\pi$. The sequence $\pi$ starts from the depot, i.e., ${\pi }_0=0$, with coordinates $c_0$. The elements from ${\pi }_1$ to ${\pi }_{\left|\pi \right|}$ represent the successively visited customer nodes. The fundamental components of the overall DRL architecture are as follows:

• Status

The system state can be divided into static and dynamic features. Static features include the spatial locations of customer nodes, such as their Euclidean coordinates $\left(x_i,y_i\right)$, service demands $q_i$, and respective time window constraints $\left[a_i,b_i\right]$. Dynamic features evolve with the decision-making process, mainly including the vehicle’s current position, remaining capacity, current time, and status indicators for whether each customer node has been served. For example, once a vehicle serves a customer, its capacity Q decreases, and when the vehicle returns to the depot node $v_0$, its load is reset, and the time may also be updated.

2.

Action

At each time step, the system selects the next customer node to visit from the set of admissible actions based on the current state. A node becomes an admissible action if it meets the following conditions: it has not been served, the vehicle’s remaining capacity is sufficient to meet its demand, and the vehicle’s arrival time at that node after serving the previous node is not later than the node’s time window deadline. The policy network π is responsible for generating a probability distribution over the set of admissible nodes, sampling an action ${\pi }_t$ according to this distribution, with its conditional probability being $p\left({\pi }_t|{\pi }_1,\cdots ,{\pi }_{t-1},\mathrm{S}\right)$, where S represents the current state.

3.

Rewards

The reward function is designed to guide the agent in finding the shortest path that satisfies all constraints. The total reward R is defined as the negative value of the total travel distance of the complete path. This is because we aim to optimize the shortest visitation route. After executing a policy according to the probability distribution, the total return is calculated as the path length after satisfying all node constraints. In other words, the total return equals $-L\left(\pi |G\right)$, where $L\left(\pi |G\right)$ denotes the path length on graph $G$ according to policy $\pi$. This paper adopts a hard time window assumption, dynamically filtering inadmissible actions during the decoding process through an action masking mechanism to ensure generated paths strictly satisfy time window constraints. Meanwhile, a capacity overload penalty is introduced to further enforce feasibility. This design, combining the dual mechanisms of masking and penalty, ensures the rapid elimination of infeasible trajectories during early training stages. It is also tightly integrated with the decoder’s dynamic context awareness, enhancing the model’s ability to comply with hard time window constraints. The reward function can be expressed as

$$\mathrm{R}=-L\left(\pi \right|G)-1000·(\mathit{overload\_total}+\mathit{unvisited\_node})$$

(11)

where $\mathit{overload\_total}$ is the sum of all overload amounts, and $\mathit{unvisited\_node}$ is the number of unvisited customer nodes. The coefficient 1000 is derived by referencing the specific range from other deep reinforcement learning studies [33], followed by preliminary tuning on small-to-medium Solomon problem instances. This penalty assigns a highly negative reward to any infeasible solution, thereby rapidly eliminating infeasible trajectories during the early stages of training. In this framework, the VRPTW is treated as a single-agent reinforcement learning problem: the agent constructs complete routes through autoregressive sequence generation, and the number of vehicles is implicitly determined by the count of returns to the depot in the sequence, rather than being modeled as independent agents collaborating in a multi-agent system.

In summary, the objective of the policy $p_{\theta }\left(\pi |G\right)$ is to generate a path $\pi$ that minimizes the total travel distance while satisfying all constraints of the VRPTW. The training process involves adjusting the neural network parameters $\theta$ to find a solution given the VRPTW graph $G$. This policy can be expressed as follows:

$$p_{\theta }\left(\pi |G\right)={\prod }_{t=1}^{\left|\pi \right|}{p}_{\theta }\left({\pi }_t|G,{\pi }_{t^{\prime }},\forall t^{\prime }<t\right)$$

(12)

3.2. Model Framework

3.2.1. Model Architecture Description

This study constructs the model based on the classic encoder–decoder framework [34]. The encoder employs a GATv2 graph attention network with adaptive gated aggregation. This design effectively enhances the model’s ability to extract key information from the graph, enabling the transformation of input instances into high-quality high-dimensional embeddings (see Figure 1 for the specific architecture). The decoder combines a multi-head attention (MHA) mechanism with an LSTM incorporating dynamic context awareness. The former is used to capture global dependencies, while the latter is responsible for modeling sequential context. They work synergistically to progressively generate the probability distribution for the next node to visit. Specifically, the LSTM is primarily responsible for modeling constraint states that evolve monotonically over time during the path generation process, such as vehicle remaining capacity and time window slack. These constraints exhibit explicit temporal dependencies. Conversely, the MHA focuses on modeling the global relevance and ranking of all candidate nodes at each decision step based on the current context embedding. This design of separated responsibilities allows the dynamic evolution of constraints and the combinatorial node selection process to be explicitly and effectively decoupled, thereby improving the decoding stability and efficiency of the model in complex constraint scenarios.

Figure 1. Encoder Architecture Diagram in the Model.

Throughout the solving process, the encoder is responsible for comprehending the problem, extracting static structural information and feasibility priors unrelated to the specific path from the input instance. The decoder, acting as the solution generator, outputs a complete node sequence (i.e., the routing plan) in an autoregressive manner, modeling and updating the dynamic feasibility of the path. This decoding process can be viewed as the sequential decision-making process of the agent. Finally, the trained model will be tested on generated data, benchmark data, and real-world data to obtain VRPTW solutions. A specific illustration is shown in Figure 2.

Figure 2. Overall architecture diagram of the model.

3.2.2. Encoder

Unlike the conventional GAT, which uses static attention weights, GATv2 explicitly models the joint feature representation of source and target nodes when computing attention coefficients, thereby generating dynamic attention weights that are relevant to the current node pair [35]. This characteristic is particularly crucial for the VRPTW problem: the correlation between customer nodes depends not only on spatial distance but is also highly influenced by the degree of time window overlap, the potential feasibility of service sequences, and the current vehicle state. Building on this, GATv2 can more effectively capture the spatiotemporal relational structure of customer nodes that changes with context, avoiding the issue where static modeling struggles to distinguish key neighboring nodes at different decision stages. Furthermore, the adaptive gating mechanism introduced in this paper dynamically weights and regulates the aggregated neighborhood information. It can automatically suppress the influence of redundant or noisy neighbors based on node importance, further enhancing the discriminative power and robustness of the graph representation. The synergistic effect of these two components enables the encoder to fully integrate static feature information—such as spatial proximity, time window constraints, and vehicle states—in a high-dimensional embedding space. This approach, while avoiding information leakage and ensuring causal consistency in policy learning, provides high-quality structured representations for the subsequent sequential decision-making in the decoding stage.

Input Layer: As shown in Figure 2, the node information $n_i$ and edge information $e_{ij}$ are first embedded into a d-dimensional vector space via two fully connected (FC) layers. Where $e_{ij}$ is characterized by the Euclidean distance between node $i$ and node $j$, where $i,j ϵ\left\{\mathrm{0,1},2,\cdots ,N\right\}$:

$$x_i^0=BN\left(A_1{n}_i+b_1\right)$$

(13)

$$f_{ij}=\left[e_{ij},t_{ij},{slack}_{ij}\right]$$

(14)

$${\widehat{e}}_{ij}=BN\left(A_2{f}_{ij}+b_2\right)$$

(15)

Here, $A_1$ and $A_2$ denote linear weight matrices, $b_1$ and $b_2$ denote bias matrices, $BN\left(·\right)$ represents batch normalization, $t_{ij}$ denotes the travel time from node $i$ to $j$, and ${slack}_{ij}$ denotes the time window slack, which is a static feasibility prior for the time windows, referring to the feasible slack in the time window for traveling from $i$ to $j$. It is defined as the difference between the upper bound of the time window for node j and the earliest possible arrival time at node $j$ when traveling directly from node $i$. The formula is as follows:

$${slack}_{ij}=b_j-\left(a_i+s_i+t_{ij}\right)$$

(16)

where $\left[a_i,b_i\right]$ and $\left[a_j,b_j\right]$ are the time windows of nodes $i$ and $j$, respectively.

Graph Attention Layer: Let $x_i^l\in {\mathbb{R}}^{d_x}$,$l\in \left\{1,\dots ,L\right\}$ be the feature vector of node $i$ at the $l$-th layer. The attention coefficient ${\alpha }_{ij}^l$ in GATv2 indicates the relevance degree between nodes $i$ and $j$ at the $l$-th layer. By injecting edge features into the attention calculation, the interaction modeling between nodes and edges is enhanced. The attention coefficient ${\alpha }_{ij}^l$ can be expressed by Equation (17) as follows:

$${\alpha }_{ij}^l={\displaystyle \frac{e^{\sigma \left(W_3^{l^T}\left(\left[W_1^l\left(x_i^{l-1}\Vert x_j^{l-1}\right)\Vert W_2^l{\widehat{e}}_{ij}\right]\right)\right)}}{{\sum }_{z=0}^N{e}^{\sigma \left(W_3^{l^T}\left(\left[W_1^l\left(x_i^{l-1}\Vert x_z^{l-1}\right)\Vert W_2^l{\widehat{e}}_{iz}\right]\right)\right)}}}$$

(17)

where $W_1^l$, $W_2^l$, and $W_3^l$ are learnable weight matrices, $\sigma \left(·\right)$ is the LeakyReLU activation function, $\left|\right|$ denotes the concatenation operation, and ${\left(·\right)}^T$ represents the transpose operation. The node feature vector $x_i^l$ is computed as follows:

$$d_i^l={\sum }_{j\in N\left(i\right)}{\alpha }_{ij}^l\left(W_4^l{x}_j^{l-1}+W_5^l{e}_{ij}^{\prime }\right)$$

(18)

$$x_i^l=d_i^l+x_i^{l-1}$$

(19)

where $W_4^l$ and $W_5^l$ are learnable weight matrices, $d_i^l$ is the aggregated information that fuses the contributions of nodes and edges, and the edge features together with the features of neighboring nodes constitute the transmitted information. Equation (19) employs residual connections to alleviate training instability in deep networks. The obtained node feature vector $x_i^l$ is then adjusted and optimized through an adaptive gating mechanism to determine the final update of each node’s features.

$$c_i={\sum }_{j\in N\left(i\right)}{\alpha }_{ij}^l\cdot \left(W_c{x}_j^l\right)$$

(20)

$$g_i^{adapt}=\sigma \left(W_g\cdot \left[x_i^l\Vert c_i\right]+b_g\right)$$

(21)

$$x_i^{out}=g_i^{adapt}\cdot x_i^l+\left(1-g_i^{adapt}\right)\cdot x_i^0$$

(22)

where $c_i$ is the weighted sum of neighboring features using attention weights ${\alpha }_{ij}^l$, $\sigma$ is the Sigmoid activation function, $W_g$ and $b_g$ are the weight and bias of the gating mechanism, respectively, and $g_i^{adapt}$ is the adaptive gating value.

The adaptive gating mechanism quantitatively assesses and perceptually modulates the logical pruning effect of temporal constraints on spatial decisions through the gating value $g_i^{adapt}$. Specifically, the time window slackness is incorporated as an edge feature into the attention computation. When temporal compatibility is satisfied (i.e., ${slack}_{ij}>0$), the attention coefficient increases, thereby promoting the aggregation of $c_i$. Conversely, it weakens attention to temporally infeasible neighborhoods. The gating mechanism learns implicit patterns of temporal constraints based on the joint representation of the current node embedding $x_i^l$ and the aggregated embedding $c_i$. When a conflict is detected, $g_i^{adapt}$ approaches zero, achieving logical pruning and ensuring that temporal constraints dominate spatial decisions. For example, if nodes $i$ and $j$ are spatially proximate but temporally conflicting, a low adaptive gating value is generated, producing a clear inhibitory signal.

Output Layer: Outputs a global feature vector. The node embeddings output by the $L$-th GATv2 layer are denoted as $x_i^{out}$. The graph embedding $\overline{x}=\left({\overline{x}}_{1,\dots ,}{\overline{x}}_{d_x}\right)$ is obtained by performing average pooling on the node embeddings and is expressed as

$${\overline{x}}_d={\displaystyle \frac{1}{N+1}}{\sum }_{i=0}^N{x}_{id}^L,d=1,\dots ,d_x$$

(23)

where $x_{id}^L$ represents the node feature at dimension $d$ in the $L$-th layer.

3.2.3. Decoder

The decoder adopts a three-stage cascaded architecture to achieve progressive generation of the path sequence. First, a dynamic context-aware LSTM module enhances the temporal features of the node embeddings output by the encoder by integrating real-time information such as the global state, time window slack, and capacity constraints. Subsequently, a multi-head attention (MHA) module calculates global node correlation weights based on the hidden states of the LSTM, generating representation vectors rich in contextual information. Finally, a single-head attention (SHA) module focuses the global information and converts it into a precise probability distribution for the next node to visit, completing the autoregressive path planning.

LSTM Layer: As shown in Figure 3, the node embeddings $x^L$ are processed by integrating contextual information such as the global state, time slack, and capacity feasibility. At each decoding step, the LSTM is actively fed with critical constraint information, including the current global state, time window slack, and capacity feasibility. The LSTM processes this information to generate the hidden state $h_t$ and output $x_t$, thereby handling the dynamic constraints. We first construct a low-dimensional dynamic context vector $c_t\in {\mathbb{R}}^{2d+3}$, as shown in the following formula:

$$c_t=\left[h_{last};h_{depot};{\displaystyle \frac{{load}_{remain}}{Q}};{\displaystyle \frac{t_{curr}}{t_{max}}};{\displaystyle \frac{1}{N}}{\sum }_{iϵV}{\displaystyle \frac{b_i-a_i}{t_{max}}}\right]$$

(24)

where $h_{last}\in {\mathbb{R}}^d$ is the embedding of the customer node selected in the previous step (taken as $h_{depot}$ for the first step, which is the encoder embedding of the depot node), ${load}_{remain}$ is the current vehicle’s remaining capacity, $t_{curr}$ is the current time, $t_{max}={max}_i{b}_i$ is the global time normalization constant, and the last term is the average normalized slack of all customer time windows. Subsequently, $c_t$ is projected to the hidden dimension via a single-layer fully connected network:

$$\widetilde{c_t}=W_c{c}_t+b_c W_c\in {\mathbb{R}}^{d\times 2d+3}$$

(25)

where $W_c$ is the learnable weight matrix, and $b_c$ is the bias term. The input to the LSTM is defined as the residual fusion of the candidate node embeddings and the projected dynamic context vector:

$$u_{t,i}=x_i^L+{MLP}_x\left(\widetilde{c_t}\right)$$

(26)

where $x_i^L\in {\mathbb{R}}^d$ is the node embedding output by the encoder, and ${MLP}_x\left(\cdot \right)$ is a single-layer fully connected network with a ReLU activation (output dimension $d$). This dynamic context fusion mechanism employs a low-dimensional constraint injection strategy, enhancing the model’s awareness of capacity and time window constraints with minimal parameter increase. The resulting vector is then processed by the LSTM.

$$f_t=\sigma \left(W_f\cdot \left[h_{t-1},u_t\right]+b_f\right)$$

(27)

$$i_t=\sigma \left(W_i\cdot \left[h_{t-1},u_t\right]+b_i\right)$$

(28)

$${\tilde{C}}_t=\tanh \left(W_C\cdot \left[h_{t-1},u_t\right]+b_C\right)$$

(29)

$$C_t=f_t\cdot C_{t-1}+i_t\cdot {\tilde{C}}_t$$

(30)

$$o_t=\sigma (W_o\cdot [h_{t-1},u_t]+b_o)$$

(31)

$$h_t=o_t\cdot tanh\left(C_t\right)$$

(32)

Figure 3. Decoder Architecture Diagram.

LSTM consists of an input gate, a forget gate, an output gate, and a memory cell, where $\sigma$ is the sigmoid activation function, $W_f$ is the weight matrix of the forget gate, $b_f$, $b_i$, $b_c$, and $b_o$ are bias terms, $f_t$ is the output of the forget gate, $i_t$ is the output of the input gate, and ${\tilde{C}}_t$ is the candidate memory cell.

MHA Layer: The node embedding $x_t$ and the context vector $c_t^1$ processed by the LSTM serve as input to the MHA layer. The multi-head attention mechanism with H heads is used to obtain a new context vector $c_t^2$. The current context vector $c_t^1$ contains global information at decoding step t, including the average node embedding $\overline{x}$, the embedding of the last selected node in the current route $x_{{\widehat{\pi }}_{t-1}}$, and the remaining load $q_t$. It is expressed as follows:

$${\mathrm{c}}_{\mathrm{t}}^1=\{\begin{array}{l}\bar{\mathrm{x}}+{\mathrm{W}}^{\mathrm{x}}\left({\mathrm{x}}_{{\widehat{\pi }}_{t-1}}\Vert \mathrm{Q}\right), \mathrm{i}\mathrm{f} \mathrm{t}=1,\\ \bar{\mathrm{x}}+{\mathrm{W}}^{\mathrm{x}}\left({\mathrm{x}}_{{\widehat{\pi }}_{t-1}}\Vert {\mathrm{q}}_{\mathrm{t}}\right), \mathrm{i}\mathrm{f} \mathrm{t}\ge 2\end{array}$$

(33)

where $W^x$ is a learnable weight matrix. The query vector $q$, key vector $k_i$, and value vector $v_i$ are then computed as

$$q=W^Q{c}_t^1,v_i=W^V{x}_t,k_i=W^K{x}_t$$

(34)

where $W^Q$, $W^V$, and $W^K$ are learnable weight matrices. The attention coefficients are then computed using the key and value vectors, and normalized using the $softmax$ function, as shown below:

$${\widehat{u}}_{i,t}^1=softmax({\displaystyle \frac{q^T{k}_i}{\sqrt{d_v}}})$$

(35)

Then, the outputs of the H heads are concatenated to obtain the updated context vector:

$$c_t^2=W^F\cdot \left({\left|\right|}_{h=1}^H{\sum }_{i=0}^g{\left({\widehat{u}}_{i,t}^1\right)}^h{v}_i^h\right)$$

(36)

where $W^F$ is a learnable weight matrix, ${\left({\widehat{u}}_{i,t}^1\right)}^h$ and $v_i^h$ represent the correlation coefficient and the value vector, respectively, in the $h$-th attention head.

SHA Layer: The context vector and key vectors obtained from the MHA are processed by the SHA to compute node selection for the current step. The correlation between the context vector and the key vectors is calculated and normalized to form a selection probability distribution over the remaining nodes, as shown in the following formula:

$$u_{i,t}^2=C\cdot \mathit{tanh}\left({\displaystyle \frac{{\left(c_t^2\right)}^T{k}_i}{\sqrt{d_v}}}\right)$$

(37)

$$p_{i,t}=softmax\left(u_{i,t}^2\right)$$

(38)

To ensure that the generated route consistently satisfies the hard constraints of VRPTW during the decoding process, this paper introduces a feasibility mask mechanism at each decision step to dynamically prune the action space. Specifically, at step t, only nodes meeting the following conditions are allowed to participate in the probability calculation: (1) the node has not been visited; (2) the current vehicle’s remaining capacity is sufficient to meet its demand; and (3) the vehicle can arrive at the node before its time window deadline after completing the already selected path. For infeasible actions, their attention scores before $softmax$ normalization are forced to $-\mathrm{\infty }$, resulting in a strict probability of 0 after normalization. This mechanism equivalently transforms the combinatorial optimization problem with complex constraints into a policy optimization problem over a constrained action space at the policy learning level, confining the policy network’s search to the feasible solution subspace and effectively preventing the generation of infeasible paths.

Furthermore, to prevent $softmax$ saturation or numerical instability caused by excessively large attention scores, the logit values are truncated using the tanh function and then scaled by a constant factor C (set to 10 in this paper) before normalization. This scaling operation limits the score magnitude while preserving sufficient policy discriminability, thereby ensuring numerical stability and preventing the policy from degenerating into an approximately uniform distribution. This value has been found to exhibit good stability across problem instances of different scales in experiments.

When solving VRPTW, the method proposed in this paper first effectively extracts key information from the graph structure using GATv2 in the encoder, enhancing the model’s understanding of the relationships between nodes (e.g., coordinates, demands) and edges (e.g., travel time). Combined with the adaptive gating mechanism, it dynamically adjusts neighborhood information fusion, suppresses irrelevant noise, and effectively captures the graph structure and node features. In the decoding stage, the dynamic context-aware LSTM module models the sequential dependencies of node selection, while the attention mechanism dynamically adjusts node focus weights based on global information, ensuring that each decoding step concentrates on the most relevant candidate nodes, thereby improving the accuracy of path generation. Combining the aforementioned techniques, the proposed model demonstrates excellent representational capacity and solution effectiveness. Secondly, the end-to-end architecture enables the model to directly generate solutions from instance inputs, bypassing traditional complex iterative search processes and significantly improving solving efficiency. Finally, the adaptive GATv2, LSTM, and attention modules exhibit good robustness to variations in input scale, endowing the model with strong generalization capabilities across VRPTW problems of different sizes. These advantages collectively enable the model to effectively solve the VRPTW problem through graph structure learning and sequence optimization.

3.2.4. Training (Reinforcement Learning Algorithm)

We adopt policy gradient methods to train the proposed model. Policy gradients learn optimal decisions in reinforcement learning by directly optimizing the policy function to maximize cumulative reward [36]. Among these, the Proximal Policy Optimization (PPO) algorithm, as a stable and efficient policy gradient method, effectively mitigates training instability caused by excessively large updates by limiting the magnitude of each policy update. Therefore, we select PPO to train and infer VRPTW instances generated from specified distributions. The PPO algorithm is based on the actor-critic framework, which requires actor and critic networks with parameters denoted as $\mathsf{\theta }$ and $\mathrm{\varnothing }$, respectively. Here, the actor network is our proposed model, and the critic network consists of a three-layer feedforward neural network, where the first two layers are fully connected layers with ReLU activation, and the third layer is a linear layer. We use the total return $R\left(\pi /s\right)$ to obtain an unbiased Monte Carlo estimate of the policy gradient, with the objective of optimizing the parameters $\mathsf{\theta }$ of the actor-network to achieve the optimal policy. The advantage function $A_t$ and the probability ratio $r\left(\mathsf{\theta }\right)$ between the updated policy and the previous policy version are defined as follows:

$$A_t=R\left(\pi /s\right)-v_{\mathrm{\varnothing }}\left(s\right)$$

(39)

$$r(\mathsf{\theta })={\displaystyle \frac{p_{\mathsf{\theta }}\left(\pi \backslash s\right)}{{p_{\mathsf{\theta }}}_{old}\left(\pi /s\right)}}$$

(40)

We use the clipped surrogate objective $L_{CLIP}$($\mathsf{\theta }$) and the policy entropy loss $L_E\left(\mathsf{\theta }\right)$ to train the actor network:

$$L_{CLIP}\left(\mathsf{\theta }\right)=\widehat{\mathbb{E}}\left[min\left\{r\left(\mathsf{\theta }\right)A_t,clip\left(r\left(\mathsf{\theta }\right),1-\epsilon ,1+\epsilon \right)A_t\right\}\right]$$

(41)

$$L_E\left(\mathsf{\theta }\right)=Entropy\left(p_{\mathsf{\theta }}\left(\pi /s\right)\right)$$

(42)

The critic network is trained by minimizing the Mean Squared Error (MSE) loss to update the baseline network, thereby estimating the expected discounted return at state $s$.

$$L_{mse}\left(\mathrm{\varnothing }\right)=MSE\left(R\left(\left(\pi \backslash s\right),v_{\mathrm{\varnothing }}\left(s\right)\right)\right)$$

(43)

Therefore, the combined model training loss function integrating the actor network and the critic network is expressed as

$$Loss\left(\mathrm{\varnothing },\mathsf{\theta }\right)=c_p{L}_{CLIP}\left(\mathsf{\theta }\right)+c_v{L}_{mse}\left(\mathrm{\varnothing }\right)-c_e{L}_E\left(\mathsf{\theta }\right)$$

(44)

where $c_p$, $c_v$, and $c_e$ are the weighting coefficients for $L_{CLIP}$, $L_{mse}$ and $L_E\left(\mathsf{\theta }\right)$, respectively.

In response to the characteristics of sparse rewards, strict constraints, and imbalanced trajectory lengths in the VRPTW decision-making process, this paper introduces a series of policy optimization enhancements highly relevant to the path planning task, based on the standard PPO framework. Specifically, these include: (1) Adopting a critic regression target based on the total return of the complete path to avoid interference from stepwise reward noise on value estimation. (2) Applying batch normalization to the advantage function to mitigate variance fluctuations in policy gradients caused by different instance scales. (3) Introducing a dynamic value function loss weighting adjustment mechanism that adaptively intensifies critic training when the advantage function variance is high, thereby accelerating value estimation convergence and improving policy update stability. The aforementioned designs enable PPO to better adapt to the learning requirements of the high-dimensional state space and strongly constrained decisions in VRPTW. Combined with PPO’s clipped update mechanism, this policy optimization scheme maintains training process stability when facing complex states and constraints, significantly reducing the risk of converging to local optima or policy oscillation.

Algorithm 1 summarizes the pseudocode of the training process.

Algorithm 1 PPO for VRPTW

Input: total epochs $E_t$, PPO epochs $E_p$, PPO steps $T_s$, steps per total epoch $P_e$, batch size $\mathrm{B}$; actor network ${\mathsf{\pi }}_{\mathsf{\theta }}$ with trainable parameters $\mathsf{\theta }$; critic network ${\mathrm{v}}_{\varnothing }$ with trainable parameters $\varnothing$; policy loss coefficient $c_p$, value function loss coefficient $c_v$, entropy loss coefficient $c_e$; clipping ratio $\epsilon$. Initialization: Orthogonal initialization $\mathsf{\theta }$$, \varnothing$$, {\mathsf{\theta }}_{old}$$\leftarrow \mathsf{\theta }$ for each total epoch $\mathrm{e} = 1, ..., E_t$ do     $\mathrm{for} \mathrm{each} \mathrm{step} \mathrm{i}$$= 1, ..., P_e$ do       $s_i$← RandomVRPTWInstance() for all i in {1, …, B}       ${\widehat{\pi }}_i$$~ \mathrm{SampleSolution}({\pi \mathsf{\theta }}_{old}$$({\widehat{\pi }}_i$$|s_i$)) for all i in {1, …, B}       $\mathrm{Compute} \mathrm{R}({\widehat{\pi }}_i$$|s_i$$) \mathrm{via} s_i$$, {\pi ̄}_i$       $\mathrm{Store} s_i$$, {\widehat{\pi }}_i$$, \mathrm{R}({\widehat{\pi }}_i$$|s_i$$), {\pi \mathsf{\theta }}_{old}$$({\widehat{\pi }}_i$$|s_i$) into Memory Buffer       $\mathrm{if} \mathrm{step} \% T_s$ == 0 then         $\mathrm{for} \mathrm{each} \mathrm{PPO} \mathrm{epoch} \mathrm{j} = 1, ..., E_p$ do          $\mathrm{for} \mathrm{each} \mathrm{PPO} \mathrm{step} \mathrm{t} = 1, ..., T_s$ do           ${\widehat{v}}_{\varnothing }$$(s_i$$) \leftarrow { v}_{\varnothing }$$(s_i$) for all i in {1, …, B}           $\mathrm{R}({\widehat{\pi }}_i$$|s_i$$) \leftarrow \mathrm{RewardNormalization}(\mathrm{R}({\widehat{\pi }}_i$$|s_i$)) for all i in {1, …, B}           $\mathrm{Compute} \mathrm{advantage} \mathrm{estimates} {\widehat{A}}_i$$= \mathrm{R}({\widehat{\pi }}_i$$|s_i$$) - {\widehat{v}}_{\varnothing }$$(s_i$)              $r_i$$(\mathsf{\theta }$$) = \pi \mathsf{\theta }$$({\widehat{\pi }}_i$$|s_i$$)/{\pi \mathsf{\theta }}_{old}$$({\widehat{\pi }}_i$$|s_i$)              $L_{CLIP}^i\left(\mathsf{\theta }\right)$$= \widehat{\mathbb{E}}$$\left[\min \right(r_i$$(\mathsf{\theta }$$){\widehat{A}}_i$$, \mathrm{clip}(r_i$$(\mathsf{\theta }$$), 1 - \mathsf{\epsilon }, 1 + \mathsf{\epsilon }){\widehat{A}}_i$)]              $L_{MSE}^i\left(\mathrm{\varnothing }\right)$$= \mathrm{MSE}\left(\mathrm{R}\right({\widehat{\pi }}_i$$|s_i$$),{\widehat{v}}_{\varnothing }$$(s_i$))              $L_E^i\left(\mathsf{\theta }\right)$$= \mathrm{Entropy}(\pi \mathsf{\theta }$$({\widehat{\pi }}_i$$|s_i$))              ${Loss}_i$$(\mathsf{\theta }$$,\mathrm{\varnothing }$$) = {\displaystyle \frac{1}{B}}{\sum }_{i=1}^B\left\{c_p{L}_{CLIP}^i\left(\mathsf{\theta }\right)+c_v{L}_{MSE}^i\left(\mathrm{\varnothing }\right)-c_e{L}_E^i\left(\mathsf{\theta }\right)\right\}$           $\mathrm{Update} \mathsf{\theta }$$,\mathrm{\varnothing }$$\mathrm{by} \mathrm{a} \mathrm{gradient} \mathrm{method} \mathrm{w}.\mathrm{r}.\mathrm{t}. {Loss}_i$$(\mathsf{\theta }$$,\mathrm{\varnothing }$)          end for         end for         ${\mathsf{\theta }}_{old}$$\leftarrow \mathsf{\theta }$         Clear Memory Buffer       end if     end for   end for Output: Trained parameters set $\mathsf{\theta }$ of the actor

In summary, the hybrid deep reinforcement learning framework proposed in this paper achieves efficient representation of the VRPTW graph structure through GATv2 and the adaptive gating mechanism. During the decoding stage, it integrates dynamic context-aware LSTM and attention mechanisms to collaboratively model the sequential dependencies and constraint feasibility in the path generation process. Additionally, by introducing a constraint-aware action masking mechanism and an improved PPO training strategy, the model can directly generate high-quality feasible paths that satisfy the constraints from instance input within an end-to-end framework. This method has undergone customized optimization at both the structural design and training mechanism levels to address the specific characteristics of the VRPTW problem, laying a methodological foundation for subsequent experimental validation of its performance advantages.

4. Experiment

This section will systematically elaborate on the experimental part of the research: First, the datasets used are introduced in detail, providing a solid data foundation for the experiments. Second, the selected comparative methods are outlined and explained, covering classical benchmarks and state-of-the-art algorithms within the field. Then, the specific parameter settings and core evaluation metrics are clearly defined to ensure the reproducibility of the experiments and the objectivity of the evaluation. Finally, multi-dimensional experiments, including comparative experiments and ablation studies, are designed around the proposed method. The results are analyzed in depth to comprehensively validate the performance improvements and intrinsic advantages of the method across various metrics.

4.1. Dataset Introduction

4.1.1. Generated Dataset

Following the data generation rules adopted in reference [27], this study constructs random instances of the Vehicle Routing Problem with Time Windows (VRPTW) for model training and evaluation. Specifically, multiple groups of instances with customer sizes of 20, 50, and 100 are generated to systematically investigate the model’s performance under different problem scales. The spatial coordinates of all customer nodes are randomly sampled from a uniform distribution within the unit square region [0, 1]², with the depot node fixed within this space. The service demand of each customer is uniformly and randomly drawn from the discrete set {0.1, 0.2,…, 0.9} to simulate the diversity of customer demands in real-world delivery scenarios. The service time windows for customers are dynamically generated based on their Euclidean distance to the depot. Specifically, a customer’s earliest service time (ready time) is set as a random value between 0.1 and 1.5 times this distance, while the latest service time adds a predefined minimum time width to the ready time. This time window construction method ensures the challenging nature of the problem constraints while maintaining practical feasibility for route planning, avoiding the generation of a large number of unsolvable instances. Furthermore, to better align with transportation needs under different task scales in real logistics scenarios, this paper introduces multiple vehicle capacity configuration strategies. For instances with customer sizes of 20, 50, and 100, the maximum vehicle load capacity is set to 4, 8, and 16 units respectively, thereby ensuring a reasonable balance between problem difficulty and resource allocation across different scales. Vehicles with load capacities of 4, 8, and 16 units are configured for tasks with 20, 50, and 100 customers respectively to meet different task requirements. Figure 4 illustrates the specific distribution of a dataset with a customer size of 20 nodes.

Figure 4. Distribution of the generated dataset with 20 nodes.

4.1.2. Solomon Benchmark Dataset

The Solomon benchmark dataset is the most renowned and widely used standard test set for the Vehicle Routing Problem with Time Windows (VRPTW) [37]. It aims to provide a unified and fair platform for performance comparison among different VRPTW algorithms. Therefore, this study employs the Solomon benchmark dataset for testing to validate the effectiveness of the proposed work. The Solomon benchmark dataset includes three types: C, R, and RC, which represent clustered distribution, random distribution, and mixed distribution, respectively. The clustered distribution features customer points exhibiting a distinct clustered structure with relatively tight time windows. The random distribution features both customer nodes and time windows that are completely randomly distributed. The RC type combines characteristics of both C and R types, presenting a certain degree of mixture. In this study, experiments are conducted according to different customer scales, divided into multiple scenarios such as 20-C1, 20-R1, 20-RC1, 50-C1, 50-R1, 50-RC1, and so on.

4.1.3. Construction of Real-World Dataset

To further validate the practical effectiveness of the proposed model, this study employs a real-world data collection method consistent with reference [38], acquiring high-precision road network data for the Hakata area of Fukuoka City, Japan, based on OpenStreetMap [39]. The core parameter settings for the experimental dataset are as follows:

Road Network and Path Parameters: The travel distances and travel times between customer nodes are directly extracted from the real geographic information, yielding network path distances to ensure the realism of route features.

Customer Demand Parameters: The goods demand value for each node is randomly sampled from the discrete set {1,…, 9} and standardized according to different problem scales to meet the model’s input requirements.

Time Window Parameters: Following the classification standard in reference [40], time windows are categorized into three basic types: [0, 160], [160, 320], and [320, 480] minutes. Building upon this, and considering the estimated travel time from the depot to each customer and service duration, the upper and lower bounds of the time windows are dynamically adjusted via uniform sampling. A feasibility verification mechanism is simultaneously introduced to filter out invalid instances with time conflicts, ensuring the dataset authentically simulates the daily operational scenarios of delivery companies.

Data Distribution Control: The time window type for each customer is uniformly and randomly selected from the aforementioned three categories, effectively mitigating bias in experimental results caused by uneven distribution of time windows. Figure 5 illustrates the construction process of the real-world dataset, where ‘Hakata’ refers to the Hakata district in Fukuoka City, Japan.

Figure 5. Workflow for generating the dataset in a real-world scenario.

4.2. Introduction to Baseline and Comparative Methods

To comprehensively and systematically validate the effectiveness and superiority of the method proposed in this paper, this study selects classic heuristic algorithms and mainstream deep reinforcement learning algorithms, which have been extensively validated in the field of Vehicle Routing Problems (VRP), as baseline and comparative methods. These methods have been proven by numerous studies to possess excellent performance, capable of generating near-optimal solutions for VRP series tasks. By benchmarking performance against the aforementioned methods, the advantages of our method in core metrics such as solution quality and efficiency can be objectively evaluated. The specific comparative methods are as follows:

OR-Tools [41]: An open-source operations research optimization toolkit proposed by Google, featuring a high-performance heuristic solver specifically for VRP. This toolkit integrates various metaheuristic algorithms and local search strategies such as guided local search, simulated annealing, and tabu search. Through a multi-algorithm collaborative optimization mechanism, it can effectively adapt to VRP and its various variants, generating high-quality feasible solutions. It is a highly authoritative benchmark solver in this field.

VNS/TS [42]: A classic high-performance hybrid heuristic algorithm. Its core strength lies in fusing the broad exploration capability of Variable Neighborhood Search (VNS) with the local optimization advantage of Tabu Search (TS). It achieves a balance between global exploration and local refinement by dynamically adjusting neighborhood structures, demonstrating excellent optimization performance and stability in solving the Vehicle Routing Problem with Time Windows (VRPTW).

PtrNet [43]: A pioneering deep learning model designed for combinatorial optimization problems. Its core innovation is introducing an attention mechanism as a “pointer” to directly select elements from the input sequence to construct the output sequence, breaking through the output dimension limitations of traditional sequence models. This model transforms combinatorial optimization problems into sequence generation tasks and has become a classic deep learning framework for solving VRP and its variants, widely applied in various combinatorial optimization scenarios.

AM [33]: A lightweight VRP-solving model based on attention mechanisms. Its core characteristic is modeling solely based on node features, achieving efficient solving through a purely node-driven decoding mechanism. It discards complex structural designs, significantly improves computational efficiency while ensuring solution accuracy, and has been shown to outperform traditional baseline methods in both solving efficiency and solution quality.

GCN-NPEC [44]: A hierarchical graph neural network model that innovatively combines Graph Convolutional Networks (GCN) with a Neural Controller (NPEC). It employs GCN for deep encoding of the problem’s graph structural features while designing a dedicated Neural Controller (NPEC) to handle sequential decisions, achieving deep integration of structure-aware capability and decision optimization capability. It demonstrates outstanding structural modeling advantages in VRP solving.

E-GAT [45]: An advanced Graph Neural Network architecture. It enhances the modeling capability for graph-structured data by co-learning node and edge features. When solving combinatorial optimization problems like VRP, it can more fully utilize the inherent relationships within the problem, thereby achieving superior performance.

POMO [46]: A general reinforcement learning framework for combinatorial optimization that learns high-quality solution policies without incorporating manually designed heuristics. When applied to the Vehicle Routing Problem (VRP), the framework leverages the inherent symmetry of the problem to conduct parallel learning over multiple optimal solutions, which substantially improves training efficiency and stability and accelerates convergence.

4.3. Parameter Settings and Evaluation Metrics Introduction

The experiments in this study were all conducted on a machine with an Intel(R) Xeon(R) Gold 5218 CPU and an RTX 3090 GPU. The hyperparameters for the proposed model are shown in Table 2.

Table 2. Hyperparameter Value Design.

Parameters	Value
Number of encoder layers	4
Learning rate decay	0.98
Epoch	100
PPO learning rate	3 × 10−4
PPO steps	1
Heads of attention	8
PPO epochs	3
Number of LSTM layers	1
Optimizer	Adam
Node embedding dimension	128
Edge embedding dimension	64

For OR-Tools, we employ the default Guided Local Search strategy to ensure consistency with domain standards [40]. To balance computational efficiency with solution quality, we set different time limits based on problem scale: 60 s for scales of 20 and 50 nodes, and 1800 s for 100 nodes, to fully leverage OR-Tools’ iterative convergence capability on complex instances. Other parameters use configurations known within the domain, including a tabu search threshold of 100 iterations and a simulated annealing cooling rate of 0.95. For VNS/TS, we set iteration thresholds to 5000 (for small-scale 20/50 nodes) and 20,000 (for 100 nodes), with time limits matching those of OR-Tools. Other parameters use known configurations [41], including a neighborhood switching threshold of 10 non-improving iterations and a tabu list length of 50.

This study employs three metrics to evaluate model performance: the average length of the paths obtained by the model on the test set, the optimality gap compared to baseline solutions obtained by optimization solvers, and the model inference time. The optimality gap is calculated as follows:

$$G\mathrm{ap}={\displaystyle \frac{D_{average}-D_{best}}{D_{best}}}\times 100\mathrm{\%}$$

(45)

4.4. Experimental Evaluation and Analysis

4.4.1. Comparative Experiments

During the training process, we randomly generated 512,000 training instances and 1000 evaluation instances. The model was trained for a total of 100 epochs. For the scenario involving 20 customer points and one depot, we validated that the proposed method can obtain feasible solutions by analyzing the trends of the average penalty cost and travel distance. The results are shown in Figure 6 and Figure 7.

Figure 6. Comparison of average distance for 20-node scenarios.

Figure 7. Visitation route map for 20 nodes under a random instance.

Figure 6 illustrates the trend of average distance across training epochs for the model proposed in this paper compared to three other reinforcement learning models in the scenario of 20 nodes. As shown in the figure, the proposed method converges to the lowest average distance and exhibits a stable downward trend, indicating that our model possesses significant advantages in both training stability and path optimization effectiveness compared to other approaches.

Figure 7 displays the specific visitation routes for a scenario with 20 nodes, where different colors represent distinct visiting routes. From a spatial distribution perspective, the route planning avoids obvious crossovers and detours. The sequence of node visits demonstrates effective utilization of geographical proximity, proving the effectiveness of the model’s strategy and indicating that the model achieves good optimization results in balancing distance minimization with operational constraints.

Furthermore, we conducted comparative experiments involving heuristic methods and reinforcement learning methods for scenarios with 20, 50, and 100 nodes. Due to limitations in computational resources, the testing for 100 nodes was performed using the model trained on 50-node instances. The experimental results based on 10,000 test instances are shown in Table 3.

Table 3. Comparison of the average travel distance and time for eight methods.

Model	VRPTW20			VRPTW50			VRPTW100
	Length	Gap (%)	Time	Length	Gap (%)	Time	Length	Gap (%)	Time
OR-Tools	7.51	0.00	11.7 s	15.36	0.00	26.8 s	26.56	8.23	48.5 s
VNS/TS	7.62	1.46	10.4 s	15.83	3.06	52.3 s	24.54	0.00	7 m 14 s
PtrNet(G)	8.36	11.32	0.3 s	16.94	10.29	0.7 s	28.45	15.93	1.4 s
AM(G)	7.91	5.33	0.4 s	16.62	8.20	0.7 s	28.22	15.00	1.5 s
GCN-NPEC(G)	7.82	4.13	0.4 s	15.98	4.04	0.7 s	28.01	14.14	1.4 s
E-GAT(G)	7.67	2.13	0.5 s	15.91	3.58	0.8 s	27.76	13.12	1.5 s
POMO(G)	7.65	1.86	0.4 s	15.86	3.26	0.7 s	27.61	12.52	1.4 s
Ours(G)	7.55	0.53	0.4 s	15.79	2.80	0.8 s	27.48	11.98	1.5 s
PtrNet(S)	8.21	9.32	0.7 s	15.88	3.39	1.3 s	26.46	7.82	2.4 s
AM(S)	7.83	4.26	0.7 s	15.76	2.60	1.3 s	26.33	7.29	2.4 s
GCN-NPEC(S)	7.73	6.92	0.8 s	15.51	0.98	1.4 s	26.21	6.81	2.5 s
E-GAT(S)	7.57	0.8	0.9 s	15.42	0.39	1.4 s	26.03	6.07	2.5 s
POMO(S)	7.55	0.53	0.8 s	15.38	0.13	1.4 s	25.96	5.79	2.5 s
Ours(S)	7.43	−1.1	0.8 s	15.31	−0.33	1.4 s	25.88	5.46	2.5 s

Table 3 compares eight methods on three different scales of VRPTW instances (VRPTW20, VRPTW50, and VRPTW100) in terms of average travel distance (Length), percentage gap relative to the baseline solution (Gap/%), and computation time (Time), where the best results are highlighted in bold. The compared methods include traditional heuristic algorithms (OR-Tools, VNS/TS) and various deep reinforcement learning-based methods, where G denotes the greedy strategy and S denotes the sampling strategy.

Overall, the proposed method demonstrates consistently competitive solution quality across different problem scales while maintaining the high inference efficiency characteristic of deep reinforcement learning approaches. In particular, under the greedy decoding strategy, our method achieves gaps of only 0.53% and 2.80% on VRPTW20 and VRPTW50, respectively, which are the smallest among all deep reinforcement learning methods. Even on the large-scale VRPTW100 instance, our approach still outperforms other learning-based baselines, indicating strong robustness under increased problem complexity.

In addition, comparisons between greedy and sampling strategies reveal that sampling-based decoding generally yields further improvements on medium- and large-scale instances. For example, AM(S) achieves a Gap of 7.29% on VRPTW100, which is significantly better than AM(G)’s 15.00%. Similar trends can be observed for POMO and E-GAT, suggesting that sampling enhances solution diversity and exploration capability in larger search spaces.

Compared with classical heuristic solvers such as OR-Tools and VNS/TS, the proposed method exhibits a favorable trade-off between solution quality and computational efficiency. Although VNS/TS achieves the best average travel distance on VRPTW100, its computation time reaches 7 min and 14 s. In contrast, our method under the greedy strategy produces a travel distance of 27.48 (Gap = 11.98%) with a computation time of only 2.5 s. As the problem scale increases, the computation times of heuristic methods grow substantially, while all deep reinforcement learning methods consistently generate solutions within seconds. Overall, the proposed approach achieves route quality closer to the optimal solution while preserving high solving efficiency, particularly demonstrating clear advantages on medium- and large-scale VRPTW instances.

To further validate the effectiveness of the proposed method, we conducted additional experiments on the Solomon benchmark dataset, evaluating route distances and times across different problem instances. The results are shown in Table 4.

Table 4. Comparison of different methods on the Solomon dataset.

Type	OR-Tools		AM		GCN-NPEC		E-GAT		POMO		Ours
	Length	Time	Length	Time	Length	Time	Length	Time	Length	Time	Length	Time
20-C1	220.7	60.1 s	299.8	0.4 s	296	0.4 s	278	0.5 s	281	0.4 s	269.1	0.4 s
20-R1	436.4	60.2 s	657.7	0.5 s	642.3	0.5 s	612.5	0.6 s	609.4	0.5 s	501.2	0.3 s
20-RC1	320.2	59.8 s	535.8	0.4 s	520.6	0.4 s	486.4	0.5 s	493	0.4 s	360.7	0.3 s
50-C1	362.6	60.2 s	695.2	0.7 s	688.3	0.7 s	643.1	0.8 s	655.2	0.7 s	475.3	0.7 s
50-R1	780.9	60.1 s	1184.5	0.9 s	1172.8	0.9 s	1106.1	0.8 s	1088.3	0.8 s	892.1	0.8 s
50-RC1	731.9	60.8 s	1161.1	0.8 s	1149.6	0.8 s	1088.7	0.8 s	1024.6	0.8 s	827.6	0.8 s
100-C1	831.8	30 m 1 s	1112.5	1.5 s	1090.6	1.5 s	1041.5	1.5 s	1026.1	1.5 s	936.6	1.5 s
100-R1	1481.3	30 m 3 s	1732.6	1.6 s	1705.1	1.6 s	1638.2	1.6 s	1610.5	1.6 s	1559.7	1.5 s
100-RC1	1422.8	30 m 1 s	1718.6	1.6 s	1688.7	1.6 s	1614.7	1.6 s	1593.4	1.6 s	1518.7	1.6 s
20-C2	264.7	60.1 s	445.3	0.4	441.9	0.4 s	408.6	0.5 s	421.2	0.4 s	315.2	0.4 s
20-R2	346.1	60.3 s	576.1	0.5	560.4	0.5 s	522.1	0.5 s	501.2	0.4 s	399.6	0.4 s
20-RC2	277.3	59.3 s	477.9	0.5 s	472.3	0.5 s	436.3	0.4 s	443.0	0.5 s	312.3	0.3 s
50-C2	359.6	60.4 s	681	0.8 s	673.3	0.8 s	626.2	0.8 s	603.1	0.7 s	464.2	0.7 s
50-R2	625.8	59.8 s	969.4	0.8 s	959.8	0.8 s	906.3	0.8 s	884.4	0.8 s	723.5	0.8 s
50-RC2	554.5	59.1 s	892.9	0.8 s	884.1	0.8 s	834.3	0.8 s	862.3	0.8 s	654.8	0.8 s
100-C2	850.7	30 m 1 s	1394.6	1.6 s	1376.8	1.6 s	1326.8	1.6 s	1210.6	1.6 s	1088.7	1.6 s
100-R2	1604.7	30 m 4 s	1728.2	1.7 s	1705.1	1.7 s	1648.3	1.7 s	1638.3	1.7 s	1626.5	1.7 s
100-RC2	1578.3	30 m 3 s	1709.6	1.7 s	1688.7	1.7 s	1628.7	1.6 s	1642.1	1.7 s	1503.4	1.7 s

From Table 4, it can be seen that the proposed method demonstrates outstanding performance in route optimization, achieving higher solution quality and significantly shorter solving times compared to other deep reinforcement learning methods in the vast majority of instances, thereby effectively balancing solution quality and computational efficiency.

Specifically, on small-scale C1-type instances with 20 customers, the route length obtained by our method is 269.1, which outperforms all other deep reinforcement learning approaches. Although this result is slightly inferior to OR-Tools (220.7), the proposed method requires only 0.4 s to generate solutions, whereas OR-Tools consumes over 60 s, clearly highlighting the efficiency advantage of deep reinforcement learning.

As the problem scale increases, the advantage of the proposed method becomes more pronounced. For example, on the R1-type instance with 100 customers, our method achieves a route length of 1559.7, outperforming all other deep reinforcement learning baselines. Meanwhile, the gap with the heuristic solver OR-Tools (1481.3) is substantially narrowed, demonstrating strong scalability and robustness under large-scale and complex time window constraints.

In terms of solving efficiency, all deep reinforcement learning methods significantly outperform traditional heuristic algorithms. The solving time of OR-Tools increases dramatically with problem scale, exceeding 30 min on 100-node instances, whereas the proposed method consistently maintains second-level inference time, requiring only 1.5–1.7 s across all tested scenarios.

Overall, the proposed method not only preserves the high efficiency inherent to deep reinforcement learning approaches but also achieves notable improvements in route quality compared with existing learning-based methods. Furthermore, it demonstrates competitive performance against classical optimization algorithms on large-scale VRPTW instances, validating its ability to strike a favorable balance between solution quality and computational efficiency.

Furthermore, based on real road network data from the Hakata area of Fukuoka City, Japan, we conducted additional experiments using multiple instances under different customer node scenarios (named H20, H50, and H100). The results are shown in Table 5. The visualization results of our model experiments are presented in Figure 8, Figure 9 and Figure 10.

Table 5. Comparison of different methods on Hakata data instances.

Model	H20			H50			H100
	Length	Gap (%)	Time	Length	Gap (%)	Time	Length	Gap (%)	Time
OR-Tools	41.2	0.00	41.1 s	67.89	0.00	65.3 s	116.2	2.2	90.3 s
VNS/TS	45.9	11.4	82.4 s	72.8	7.23	3 m 20 s	113.7	0.00	15 m 6 s
PtrNet(G)	43.4	5.34	0.3 s	70.5	3.84	0.7 s	128.6	13.1	1.3 s
AM(G)	41.8	1.46	0.3 s	69.1	1.78	0.8 s	123.8	8.88	1.4 s
GCN-NPEC(G)	44.2	7.28	0.4 s	71.2	4.88	0.9 s	127.6	12.23	1.5 s
E-GAT(G)	41.9	1.7	0.4 s	68.6	1.05	0.8 s	122.1	7.39	1.4 s
POMO(G)	41.7	1.21	0.3 s	68.9	1.49	0.8 s	119.3	4.93	1.4 s
Ours(G)	41.9	1.7	0.4 s	67.6	−0.43	0.8 s	117.5	3.34	1.4 s
PtrNet(S)	42.7	3.64	0.5 s	67.9	0.01	1.1 s	118.1	3.87	2.1 s
AM(S)	40.9	−0.73	0.5 s	66.7	−1.75	1.1 s	115.9	1.94	2.1 s
GCN-NPEC(S)	43.6	5.83	0.6 s	68.4	0.75	1.3 s	118.4	4.13	2.4 s
E-GAT(S)	40.5	−1.7	0.6 s	65.9	−2.93	1.1 s	114.3	0.53	2.2 s
POMO(S)	40.8	−0.97	0.5 s	65.1	−4.11	1.1 s	113.9	0.18	2.1 s
Ours(S)	39.8	−3.4	0.6 s	64.3	−5.29	1.2 s	109.9	−3.34	2.1 s

Figure 8. Visitation route map for 20 nodes in the H20 instance.

Figure 9. Visitation route map for 50 nodes in the H50 instance.

Figure 10. Visitation route map for 100 nodes in the H100 instance.

Table 5 presents the comparison results of different methods on the Hakata data instances, where the best results are highlighted in bold. The results show that the proposed method achieves the shortest total path lengths in the H20, H50, and H100 scenarios. The negative Gap values indicate that the solutions obtained by our method outperform the reference solutions from the corresponding baseline algorithms under the given time limits, fully demonstrating the solving efficacy of the method in real-world scenarios. Regarding solving efficiency, the proposed method inherits the high-efficiency characteristics of deep learning models, with solving times controlled within a few seconds. It shows significant advantages compared to traditional heuristic algorithms like OR-Tools, indicating broad application prospects in large-scale dynamic logistics scheduling that requires rapid response. Figure 8, Figure 9 and Figure 10 show the route visitation diagrams generated by our model under different customer node scales in the Hakata area. The figures detail the demand and time window information for each node. Faced with diverse scenarios, our model, leveraging its powerful encoding and decoding capabilities, generates multiple feasible delivery routes. This demonstrates a deep understanding of geospatial relationships and effective clustering capabilities, providing solid technical support for intelligent delivery scheduling in complex urban environments.

We also conducted statistics on the training time of the proposed method and other reinforcement learning methods under 20-node and 50-node VRPTW scenarios. This analysis aims to examine the computational efficiency variation in different models as the problem scale increases, thereby evaluating their adaptability to problem scaling. The results are presented in Table 6.

Table 6. Training Time Comparison of Different Methods.

Model	VRPTW20	VRPTW50
	Training Time	Training Time
PtrNet	4.0 h	8.8 h
AM	4.1 h	7.9 h
GCN-NPEC	6.2 h	11.8 h
E-GAT	5.5 h	10.4 h
POMO	4.2 h	8.5 h
Ours	5.0 h	9.2 h

As can be seen from Table 6, as the problem size increases from 20 nodes to 50 nodes, the training time of all methods exhibits a stable upward trend. Among them, PtrNet and AM, due to their relatively simple model structures, require shorter training times. In contrast, the training time of the proposed method is only slightly higher than that of lightweight models such as POMO and AM, but lower than that of more complex models like GCN-NPEC and E-GAT. This indicates that the proposed model achieves a favorable trade-off between training time and solution efficiency.

4.4.2. Ablation Experiments

To verify the effectiveness of each component of the proposed model and its synergistic effects, experiments were conducted using different combinations of the encoder and decoder. These include GAT + MHA, GATv2 + MHA, GATv2 + gating + MHA, and GATv2 + gating + MHA + LSTM. The experiments were carried out under two scenarios: a randomly generated dataset and a real-world dataset, each containing instances with 20 and 50 customer nodes, aimed at investigating the impact of each improvement on the model’s average path distance metric.

The experimental results are shown in Table 7, which fully validates the effectiveness of each component of the proposed model. First, replacing the encoder from GAT to GATv2 led to significant improvements across multiple problem sets on both random and real-world data. For example, on the VRPTW50 problem, the objective value decreased from 16.76 to 16.34, an improvement of 2.5%. In the more complex H50 problem, the objective value also improved from 67.4 to 66.1, fully demonstrating the advantages of GATv2 in capturing complex node dependencies and enhancing feature extraction capability. Secondly, after introducing the adaptive gating mechanism into the GATv2 encoder, performance was further enhanced. On the VRPTW20 problem, the objective value decreased from 7.78 to 7.52, an improvement of 3.3%, highlighting the critical role of the gating mechanism in dynamically filtering information and suppressing noise propagation. Finally, integrating the dynamic context-aware LSTM module into the decoder further enhanced the model’s performance. Notably, even with multiple existing improvements, the introduction of LSTM still provided additional gains. For instance, on the VRPTW50 problem, the objective value decreased from 15.96 to 15.35, a reduction of 3.8%. This fully proves that the LSTM, by integrating constraint-aware contextual information and sequential decision history, effectively handles dynamic constraints and forms a good synergistic effect with other components. Overall, each improvement in the proposed model brought stable performance gains across different problem scales and scenarios, validating the effectiveness of each component’s design and its collaborative effect.

Table 7. Ablation Study Validation.

Model		Problem Sets (VRPTW)
Encoder	Decoder	VRPTW20	VRPTW50	H20	H50
GAT	MHA	8.11	16.76	42.1	67.4
GATv2	MHA	7.78	16.34	41.3	66.1
GATv2 + gating	MHA	7.52	15.96	40.8	65.6
GATv2	MHA + LSTM	7.68	16.12	41.1	65.7
GATv2 + gating	MHA + LSTM	7.45	15.35	40.2	64.4

Furthermore, we conducted an ablation study on the improvements made to the PPO algorithm to verify their effectiveness. Experiments were carried out on randomly generated instances with 20 and 50 customer nodes, and the results are presented in Table 8.

Table 8. Ablation Study for Verification of the PPO Algorithm Improvements.

Problem Sets Result	Constraint-Aware Reward	Normalization of Advantage Function	Dynamic Value Loss Adjustment	Variance	Length
VRPTW20	-	-	-	2.5	7.71
VRPTW20	✓	-	-	2.4	7.55
VRPTW20	✓	✓	-	0.8	7.54
VRPTW20	✓	✓	✓	0.6	7.46
VRPTW50	-	-	-	4.2	16.10
VRPTW50	✓	-	-	3.8	15.51
VRPTW50	✓	✓	-	1.2	15.44
VRPTW50	✓	✓	✓	0.9	15.35

As shown in Table 8, each row progressively adds one improvement component on top of the baseline PPO configuration. The improvement strategies applied to the PPO algorithm enhance the model’s training stability and routing optimization performance on both the VRPTW20 and VRPTW50 datasets. Specifically, the baseline algorithm exhibits relatively high variance (2.5 for VRPTW20 and 4.2 for VRPTW50). After progressively incorporating the constraint-aware reward, the variance shows a slight decrease while the route length is correspondingly optimized. This demonstrates the role of this reward in indirectly reducing reward noise by penalizing infeasible trajectories. Subsequently, introducing advantage function normalization leads to a substantial reduction in variance—dropping to 0.8 (a decrease of approximately 67%) for VRPTW20 and to 1.2 (a decrease of about 68%) for VRPTW50—while further improving the route length. This confirms its core contribution in directly mitigating gradient variance fluctuation and enhancing stability. Finally, adding dynamic value loss adaptation results in a continued decrease in variance and a gradual reduction in route length. These cumulative enhancements demonstrate that the PPOs effectively reduce variance and strengthen training stability while indirectly improving route quality, thereby validating their necessity for addressing the VRPTW problem.

5. Conclusions

To address the challenges in the Vehicle Routing Problem with Time Windows (VRPTW), including the coupling of time window and capacity constraints, complex spatiotemporal correlations among customer nodes, and difficulties in capturing temporal dependencies in routes, this paper proposes an end-to-end hybrid deep reinforcement learning (DRL) framework. This framework provides an efficient and stable integrated solution for combinatorial optimization problems driven by complex constraints. It constructs an encoder–decoder collaborative optimization structure: the encoder employs an adaptive gated graph attention network (GATv2), which dynamically adjusts the information flow between nodes and edges, enhances interaction modeling, and effectively characterizes the coupling relationship between customer spatial proximity and time window constraints; the decoder integrates a multi-head attention (MHA) mechanism with a dynamic context-aware long short-term memory network (LSTM), collaboratively capturing global dependencies of nodes and dynamic temporal features of routes, thereby ensuring the overall quality and constraint feasibility of route generation; during the training phase, an improved proximal policy optimization (PPO) algorithm and a constraint-aware composite reward function are introduced, significantly enhancing the stability and convergence efficiency of model training. Comprehensive experimental validation using randomly generated instances, the Solomon benchmark dataset, and real-world logistics scenarios demonstrates that the proposed framework exhibits excellent solving performance and generalization capability. Compared to mainstream DRL methods and classical heuristic algorithms, it reduces transportation costs by 2–10%, achieves a demand fulfillment rate exceeding 99%, and shows a cross-distribution generalization performance degradation of only 3.2%, fully reflecting its adaptability and reliability across different scales and scenarios.

In future work, we will further refine the existing framework to more effectively capture the underlying structure of VRPTW and improve routing strategies, thereby optimizing the generation of node embeddings and route selection. We will also extend it to multi-objective scenarios such as green logistics, promoting the practical application of the research findings in industrial systems.