Machine Learning-Enhanced Last-Mile Delivery Optimization: Integrating Deep Reinforcement Learning with Queueing Theory for Dynamic Vehicle Routing

Abstract

We present the ML-CALMO framework, which integrates machine learning with queueing theory for last-mile delivery optimization under dynamic conditions. The system combines Long Short-Term Memory (LSTM) demand forecasting, Convolutional Neural Network (CNN) traffic prediction, and Deep Q-Network (DQN)-based routing with theoretical stability guarantees. Evaluation on modern benchmarks, including the 2022 Multi-Depot Dynamic VRP with Stochastic Road Capacity (MDDVRPSRC) dataset and real-world compatible data from OSMnx-based spatial extraction, demonstrates measurable improvements: 18.5% reduction in delivery time and +8.9 pp (≈12.2% relative) gain in service efficiency compared to current state-of-the-art methods, with statistical significance (p < 0.01). Critical limitations include (1) computational requirements that necessitate mid-range GPU hardware, (2) performance degradation under rapid parameter changes (drift rate > 0.5/min), and (3) validation limited to simulation environments. The framework provides a foundation for integrating predictive machine learning with operational guarantees, though field deployment requires addressing identified scalability and robustness constraints. All code, data, and experimental configurations are publicly available for reproducibility.

1. Introduction

The last-mile delivery problem represents a critical challenge in modern logistics, accounting for up to 53% of total shipping costs [1]. The integration of machine learning with operational research methods offers promising solutions but requires careful theoretical grounding to ensure system stability and performance guarantees [2].

As same-day delivery has become the new normal, logistics companies must handle sudden order surges and traffic congestion effectively. Recent empirical studies indicate that delivery failures during peak periods can reach 15–20% in urban areas [3], highlighting the need for adaptive routing systems capable of handling dynamic demand patterns and real-time constraints.

1.1. Problem Statement

Based on our comprehensive analysis of the existing literature and industry challenges, we identify the following critical problems that current last-mile delivery systems face:

• Dynamic Demand Uncertainty: Traditional static optimization methods fail to adapt to real-time demand fluctuations, resulting in inefficient resource allocation and poor service quality during peak periods.
• Priority Order Management: Existing systems lack sophisticated mechanisms to handle priority orders that require immediate attention and can preempt ongoing deliveries, leading to cascading delays and customer dissatisfaction.
• Computational Scalability: Centralized routing systems create computational bottlenecks for large-scale urban networks, while distributed approaches lack coordination, resulting in suboptimal global solutions.
• Theoretical–Practical Gap: Current machine learning applications in logistics operate without theoretical foundations, leading to unpredictable system behavior and inability to provide service guarantees.
• Real-time Adaptability: Existing frameworks cannot effectively incorporate real-time traffic conditions and dynamic service requirements, resulting in outdated routing decisions.

These problems collectively necessitate a novel approach that integrates predictive capabilities with theoretical rigor while maintaining computational efficiency and scalability.

1.2. Theoretical Foundations and Gaps

The application of queueing theory to dynamic routing problems requires careful consideration of time-varying parameters. Classical Jackson networks [4] assume stationary arrival and service processes, which are violated in real-world delivery systems. Recent work on time-varying queues [5] provides theoretical tools, but their integration with machine learning remains unexplored.

Priority queueing systems have been extensively studied [6,7], but existing results assume perfect information and stationary processes. The combination of predictive uncertainty from machine learning with priority service disciplines presents unique theoretical challenges not addressed in the current literature.

Our approach addresses these gaps by developing a piecewise-stationary approximation framework that maintains tractability while handling time-varying parameters. This theoretical contribution, detailed in Section 3, provides the foundation for integrating machine learning predictions with queueing theory guarantees.

This research addresses these gaps by developing the Machine Learning-Enhanced Cloud-Assisted Last-Mile Optimization (ML-CALMO) framework. Our contributions include the following:

• A rigorous theoretical framework using piecewise-stationary approximation to handle time-varying parameters while maintaining tractability.
• Proven stability guarantees through Lyapunov analysis under machine learning predictions.
• Comprehensive experimental validation with statistical significance testing on extended benchmarks.
• Complete reproducibility package including source code and detailed implementation specifications.

2. Literature Review and Research Gap Analysis

Modern Vehicle Routing Benchmarks and Dataset Evolution: Traditional benchmarks (Solomon, Li-Lim) were designed for static problems and fail to capture modern delivery complexities. Recent dataset developments address these limitations: The 2022 Multi-Depot Dynamic VRP with Stochastic Road Capacity (MDDVRPSRC) [8] provides 10 dynamic networks with realistic damage scenarios. OpenStreetMap-based spatial data extraction using OSMnx [9] enables generation of geographically accurate routing instances from actual urban environments. However, current datasets predominantly focus on single-attribute optimization, lacking integrated priority management and computational feasibility analysis for deployment scenarios.

2.1. Priority Queueing in Service Systems

Priority queueing systems have been extensively studied since the seminal work of Cobham [10] and Jaiswal [6]. The analysis of preemptive priority systems, where high-priority customers can interrupt service of low-priority ones, is particularly relevant for delivery systems with urgent orders. Miller [11] developed computational methods for multi-class priority queues, while Takagi [12] provided comprehensive analysis of priority polling systems.

However, these classical results assume perfect information about system parameters and stationary processes. The integration of machine learning predictions introduces estimation errors and time-varying parameters that violate these assumptions. Recent work by Dai and Gluzman [2] explores learning-augmented queueing systems but focuses on single-server settings without priority considerations.

2.2. Integration of Machine Learning and Queueing Theory

The integration of machine learning with queueing theory represents an emerging research direction. Recent work [13] proposed using neural networks to approximate value functions in queueing control problems, achieving near-optimal performance in simulation. Another study [14] developed hybrid frameworks combining deep learning predictions with queueing analysis for cloud resource allocation.

The pioneering work of Nazari et al. [15] demonstrated that attention mechanisms could effectively solve routing problems, while Bertsimas and Van Ryzin [16] established theoretical foundations for dynamic vehicle routing in stochastic environments. Recent advances by Konovalenko and Hvattum [17] show promise in applying deep reinforcement learning to dynamic VRP variants. A comprehensive review by Kuo et al. [18] highlights the growing importance of reinforcement learning in supply chain optimization, though practical deployment challenges remain.

These machine learning approaches exhibit critical limitations that restrict their practical applicability. Most existing ML methods treat routing as a pure combinatorial optimization problem, ignoring the underlying stochastic service dynamics that fundamentally characterize real delivery operations where service times, travel conditions, and customer availability vary unpredictably.

2.3. Key Insights from Literature Survey

Based on our comprehensive literature review, we derive the following critical insights that inform our proposed solution:

• Integration Necessity: Successful last-mile delivery optimization requires seamless integration of predictive (machine learning) and analytical (queueing theory) approaches, rather than treating them as separate optimization layers.
• Dynamic Adaptation: Static optimization methods consistently underperform in real-world scenarios due to their inability to adapt to changing conditions, necessitating continuous learning mechanisms.
• Priority Handling: Commercial logistics operations require sophisticated priority management that existing academic solutions fail to address adequately.
• Theoretical Grounding: Pure machine learning approaches without theoretical foundations cannot provide the service guarantees required for commercial deployment.
• Scalability Requirements: Practical solutions must scale efficiently to handle metropolitan-scale delivery networks with thousands of daily orders.

3. Materials and Methods

3.1. Variable Definitions and Notation

To enhance clarity, we provide comprehensive variable definitions before their first usage.

• N: Number of heterogeneous vehicles in the fleet;
• ${\mathcal{O}}_r$: Set of regular orders with flexible scheduling windows;
• ${\mathcal{O}}_p$: Set of priority orders requiring immediate attention;
• $\lambda \left(t\right)$: Time-varying arrival rate at time t;
• $\mu \left(t\right)$: Time-varying service rate at time t;
• $\widehat{\lambda }\left(t\right),\widehat{\mu }\left(t\right)$: Predicted arrival and service rates;
• $\rho \left(t\right)=\lambda \left(t\right)/\mu \left(t\right)$: System utilization factor;
• $x_{ij}\in \{0,1\}$: Binary assignment variable (order j to vehicle i);
• $T_{ij}^{\mathrm{travel}}\left(t\right)$: Travel time from vehicle i location to order j at time t;
• $T_j^{\mathrm{service}}$: Service time at customer location for order j;
• $T_j^{\mathrm{loading}}$: Loading time for order j;
• $T_j^{\mathrm{waiting}}$: Customer waiting time for order j;
• $C_j^{\mathrm{preempt}}$: Preemption cost for interrupting order j;
• $\Delta t$: Time window duration for piecewise-stationary approximation (15 min);
• $ϵ$: Stability margin parameter (0.05 in our implementation).

3.2. Queueing Model Formulation and System Mapping

To improve clarity about the queueing model specifics, we provide explicit mapping between the routing problem and queueing system:

Queueing System Structure:

• Servers: Each vehicle $i\in \{1,...,N\}$ operates as an independent server with state-dependent service rates ${\mu }_i\left(t\right)$;
• Queues: Two priority classes—priority queue $Q_p$ for urgent orders and regular queue $Q_r$ for standard orders;
• Service Discipline: Preemptive priority with resume—priority orders can interrupt regular order service;
• Arrival Process: Non-homogeneous Poisson process with rates ${\lambda }_p\left(t\right)$ and ${\lambda }_r\left(t\right)$.

The system operates as an $M\left(t\right)/M\left(t\right)/N$ queue with two priority classes, where vehicles are servers and orders are customers. The mapping is formalized as

$$\mathrm{Vehicle}i\leftrightarrow \mathrm{Server}i,\mathrm{Order}j\leftrightarrow \mathrm{Customer}j$$

Integration Mechanism with DQN and Control Architecture

The stability constraint $\rho \left(t\right)<1-ϵ$ is enforced through a two-layer control architecture:

Queueing Controller Layer: Computes stability constraints based on current and predicted system state:

• Before action selection, compute predicted utilization ${\widehat{\rho }}_i^{\mathrm{new}}$ for each potential vehicle-order assignment.
• Identify feasible actions where ${\widehat{\rho }}_i^{\mathrm{new}}<1-ϵ$ (stability preserved).
• Signal to the Dispatcher Module if all actions would violate stability.

Dispatcher Module Layer: Executes assignment decisions based on controller signals:

• Receives feasibility constraints from the Queueing Controller.
• Assigns orders to vehicles that satisfy stability constraints.
• If controller signals constraint violation, rejects lowest-priority orders to preserve system stability.
• Applies penalty $P_{\mathrm{stability}}=100$ to the reward if a stability violation is attempted.

Role Clarification: The Queueing Controller calculates whether assignments are feasible from a stability perspective; the Dispatcher Module uses this information to make final assignment decisions and implement admission control by accepting or rejecting orders.

3.3. System Architecture

The ML-CALMO framework integrates four components:

LSTM Demand Predictor: Generates arrival rate forecasts ${\widehat{\lambda }}_p\left(t\right)$ and ${\widehat{\lambda }}_r\left(t\right)$ for priority and regular orders using historical patterns.

CNN Traffic Analyzer: Predicts travel times ${\widehat{T}}_{ij}\left(t\right)$ from spatiotemporal traffic data.

DQN Route Optimizer: Selects vehicle-order assignments $x_{ij}$ within stability constraints.

Queueing Controller with Dispatcher: This module maintains system stability by monitoring utilization $\rho \left(t\right)<1-ϵ$ and implementing admission control. When utilization approaches the stability threshold, the dispatcher rejects low-priority orders to preserve system stability. This component acts as the central coordination mechanism between ML predictions and routing decisions.

The system operates in synchronized 15 min windows for computational tractability, as illustrated in Figure 1.

3.4. Deep Q-Network Route Optimization

The DQN optimizes routing decisions using state representation $s_t=\{V_t,O_t,T_t,{\rho }_t\}$ where

• $V_t$: Vehicle states (location, capacity);
• $O_t$: Order states (priority, deadline, location);
• $T_t$: Predicted travel time matrix;
• ${\rho }_t$: System utilization factors.

The action space consists of feasible assignments:

$$a_t\in \mathcal{A}:=\{(i,j)\mid \mathrm{capacity}\mathrm{and}\mathrm{stability}\mathrm{constraints}\mathrm{are}\mathrm{satisfied}\}.$$

The reward function includes preemption costs:

$$r_t=-{\alpha }_1{T}_{\mathrm{delivery}}-{\alpha }_2{T}_{\mathrm{waiting}}-{\alpha }_3{P}_{\mathrm{penalty}}+{\alpha }_4{R}_{\mathrm{priority}}-{\alpha }_5{C}_{\mathrm{preempt}}$$

(1)

where the parameters are defined as

• ${\alpha }_1=1.0$: Weight for delivery time minimization;
• ${\alpha }_2=0.5$: Weight for customer waiting time;
• ${\alpha }_3=2.0$: Weight for deadline violation penalties;
• ${\alpha }_4=1.5$: Reward weight for successful priority deliveries;
• ${\alpha }_5=0.8$: Weight for preemption costs.

These weights were determined through hyperparameter tuning on the validation dataset to balance multiple objectives effectively.

The preemption cost $C_{\mathrm{preempt}}$ is calculated as

$$C_{\mathrm{preempt}}={\beta }_1{d}_{\mathrm{reposition}}+{\beta }_2{t}_{\mathrm{delay}}+{\beta }_3$$

(2)

Calibration of $\mathit{\beta }$ parameters: We specify and calibrate the cost coefficients following Equation (2) as follows:

• ${\beta }_1=0.2USD/\mathrm{km}$: Cost per kilometer for vehicle repositioning (grid search range: $0.1$–$0.5$).
• ${\beta }_2=0.5USD/\min$: Cost per minute of delay (grid search range: $0.2$–$1.0$).
• ${\beta }_3=5USD$: Fixed switching overhead (grid search range: 2–10).

The grid search targets an empirical preemption frequency of 8–$10\%$ while minimizing the total operational cost. Additionally,

• $d_{\mathrm{reposition}}$: Distance traveled to switch between orders (km);
• $t_{\mathrm{delay}}$: Time delay caused by order switching (minutes).

These values were selected through validation data to balance preemption frequency with operational efficiency.

Training uses standard experience replay with stability constraint verification before action execution.

Note that Equation (1) provides the immediate reward signal for DQN learning, while Equation (5) defines the overall system cost minimization objective. These serve complementary roles: the DQN reward guides real-time learning decisions, while the cost function evaluates overall operational performance.

3.5. Service Rate Analysis and Optimization

Service Efficiency Definition: Service efficiency is formally defined as the ratio of productive delivery time to total operational time:

$$\mathrm{Service}\mathrm{Efficiency}={\displaystyle \frac{T_{\mathrm{productive}}}{T_{\mathrm{total}}}}={\displaystyle \frac{{\sum }_{j\in \mathcal{O}}(T_j^{\mathrm{travel}}+T_j^{\mathrm{service}})}{{\sum }_{i=1}^N{T}_i^{\mathrm{operational}}}}$$

(3)

where

• $T_{\mathrm{productive}}$: Time spent on actual delivery activities (travel to customers and service);
• $T_{\mathrm{total}}$: Total operational time including idle time, repositioning, and delays;
• $T_i^{\mathrm{operational}}$: Total time vehicle i is in operation.

The service rate for vehicle i serving order j is

$${\mu }_{ij}\left(t\right)={\displaystyle \frac{1}{T_{ij}^{\mathrm{travel}}\left(t\right)+T_j^{\mathrm{service}}+T_j^{\mathrm{loading}}}}$$

(4)

The optimization objective minimizes total system costs

$$min\sum _{i,j}{x}_{ij}\left[{\widehat{T}}_{ij}^{\mathrm{travel}}+T_j^{\mathrm{service}}+{\gamma }_w{\widehat{T}}_j^{\mathrm{waiting}}+{\gamma }_p{C}_j^{\mathrm{preempt}}\right]$$

(5)

subject to assignment, capacity, and stability constraints:

$$\begin{array}{cc}\hfill \sum _i{x}_{ij}& =1\forall j\left(\mathrm{assignment}\right)\hfill \end{array}$$

(6)

$$\begin{array}{cc}\hfill \sum _j{x}_{ij}{D}_j& \le C_i\forall i\left(\mathrm{capacity}\right)\hfill \end{array}$$

(7)

$$\begin{array}{cc}\hfill {\rho }_i\left(t\right)& <1-ϵ\forall i\left(\mathrm{stability}\right)\hfill \end{array}$$

(8)

3.6. Baseline Adaptation for Priority Orders

To ensure fair comparison, all baseline methods were adapted to handle priority orders with preemption:

OR-Tools VRP: Modified to include priority constraints using custom routing dimension with priority-based penalties. Preemption implemented through route re-optimization when priority orders arrive.

Attention-VRP: Extended attention mechanism to include priority embeddings (dimension 32). Added priority mask in decoder to ensure priority orders are served first.

DRL-Stochastic: Modified reward function to include priority success bonus ($+10$ for priority, $+2$ for regular). State space extended with priority indicators.

Hybrid RL-VNS: Priority handling added to both RL component (modified Q-values) and VNS local search (priority-preserving neighborhoods).

All baselines use identical priority order generation (20% priority probability) and preemption penalty structure for consistency.

3.7. Theoretical Framework

We employ piecewise-stationary approximation with explicit operational limits:

Theorem 1

(Approximation Quality with Operational Constraints). For arrival and service rates with bounded derivatives $L_{\lambda }$ and $L_{\mu }$, the piecewise-stationary approximation over windows $\Delta t$ satisfies

$$|\rho \left(t\right)-{\overline{\rho }}_k|\le C(L_{\lambda }+L_{\mu })\Delta t$$

where C depends on system configuration. The approximation becomes unreliable when $L_{\lambda }+L_{\mu }>0.5/\mathit{min}$, limiting applicability during rapid demand fluctuations.

Proof. 

Consider the utilization at time t within window k: $\rho \left(t\right)=\lambda \left(t\right)/\mu \left(t\right)$. Using Taylor expansion around the window midpoint $t_k$,

$$\begin{array}{cc}\hfill \lambda \left(t\right)& ={\overline{\lambda }}_k+{\lambda }^{\prime }\left({\tau }_1\right)(t-t_k),{\tau }_1\in [t_k,t]\hfill \end{array}$$

(9)

$$\begin{array}{cc}\hfill \mu \left(t\right)& ={\overline{\mu }}_k+{\mu }^{\prime }\left({\tau }_2\right)(t-t_k),{\tau }_2\in [t_k,t]\hfill \end{array}$$

(10)

where $|{\lambda }^{\prime }\left({\tau }_1\right)|\le L_{\lambda }$ and $|{\mu }^{\prime }\left({\tau }_2\right)|\le L_{\mu }$.

The utilization error is bounded by

$$\begin{array}{cc}\hfill |\rho \left(t\right)-{\overline{\rho }}_k|& =\left|{\displaystyle \frac{\lambda \left(t\right)}{\mu \left(t\right)}}-{\displaystyle \frac{{\overline{\lambda }}_k}{{\overline{\mu }}_k}}\right|\hfill \end{array}$$

(11)

$$\begin{array}{cc}\hfill & \le {\displaystyle \frac{|\lambda \left(t\right)-{\overline{\lambda }}_k|}{|{\mu }_{min}|}}+{\displaystyle \frac{|{\overline{\lambda }}_k||\mu \left(t\right)-{\overline{\mu }}_k|}{|{\mu }_{min}{|}^2}}\hfill \end{array}$$

(12)

$$\begin{array}{cc}\hfill & \le {\displaystyle \frac{L_{\lambda }\Delta t/2}{{\mu }_{min}}}+{\displaystyle \frac{{\lambda }_{max}{L}_{\mu }\Delta t/2}{{\mu }_{min}^2}}\hfill \end{array}$$

(13)

$$\begin{array}{cc}\hfill & =C(L_{\lambda }+L_{\mu })\Delta t\hfill \end{array}$$

(14)

where $C=max\{1/\left(2{\mu }_{min}\right),{\lambda }_{max}/\left(2{\mu }_{min}^2\right)\}$ and $|t-t_k|\le \Delta t/2$ within each window.    □

Within each 15 min window, we assume quasi-steady state with parameters ${\overline{\lambda }}_k$ and ${\overline{\mu }}_k$. State transitions follow standard birth–death processes with rates determined by averaged parameters.

Lemma 1

(Stability Preservation). If predicted utilization satisfies ${\widehat{\rho }}_i\left(t\right)<1-ϵ$ and prediction errors are bounded by ${\delta }_{\lambda }$ and ${\delta }_{\mu }$, then stability is preserved, provided that

$$ϵ>{\displaystyle \frac{{\delta }_{\lambda }}{{\mu }_{min}}}+{\displaystyle \frac{{\lambda }_{max}{\delta }_{\mu }}{{\mu }_{min}^2}}$$

The true utilization can be expressed as

$${\rho }_i\left(t\right)={\displaystyle \frac{{\lambda }_i\left(t\right)}{{\mu }_i\left(t\right)}}={\displaystyle \frac{{\widehat{\lambda }}_i\left(t\right)+e_{\lambda }}{{\widehat{\mu }}_i\left(t\right)+e_{\mu }}}$$

where $|e_{\lambda }|\le {\delta }_{\lambda }$ and $|e_{\mu }|\le {\delta }_{\mu }$ are prediction errors.

For stability, we require ${\rho }_i\left(t\right)<1$. Given ${\widehat{\rho }}_i\left(t\right)<1-ϵ$,

$$\begin{array}{cc}\hfill {\rho }_i\left(t\right)& ={\displaystyle \frac{{\widehat{\lambda }}_i\left(t\right)+e_{\lambda }}{{\widehat{\mu }}_i\left(t\right)+e_{\mu }}}\hfill \end{array}$$

(15)

$$\begin{array}{cc}\hfill & \le {\displaystyle \frac{{\widehat{\lambda }}_i\left(t\right)+{\delta }_{\lambda }}{{\widehat{\mu }}_i\left(t\right)-{\delta }_{\mu }}}\hfill \end{array}$$

(16)

$$\begin{array}{cc}\hfill & ={\displaystyle \frac{{\widehat{\rho }}_i\left(t\right){\widehat{\mu }}_i\left(t\right)+{\delta }_{\lambda }}{{\widehat{\mu }}_i\left(t\right)-{\delta }_{\mu }}}\hfill \end{array}$$

(17)

$$\begin{array}{cc}\hfill & <{\displaystyle \frac{(1-ϵ){\widehat{\mu }}_i\left(t\right)+{\delta }_{\lambda }}{{\widehat{\mu }}_i\left(t\right)-{\delta }_{\mu }}}\hfill \end{array}$$

(18)

For ${\rho }_i\left(t\right)<1$, we need

$$(1-ϵ){\widehat{\mu }}_i\left(t\right)+{\delta }_{\lambda }<{\widehat{\mu }}_i\left(t\right)-{\delta }_{\mu }$$

$$ϵ{\widehat{\mu }}_i\left(t\right)>{\delta }_{\lambda }+{\delta }_{\mu }$$

Since ${\widehat{\mu }}_i\left(t\right)\ge {\mu }_{min}-{\delta }_{\mu }$, the condition becomes

$$ϵ>{\displaystyle \frac{{\delta }_{\lambda }+{\delta }_{\mu }}{{\mu }_{min}-{\delta }_{\mu }}}\approx {\displaystyle \frac{{\delta }_{\lambda }}{{\mu }_{min}}}+{\displaystyle \frac{{\lambda }_{max}{\delta }_{\mu }}{{\mu }_{min}^2}}$$

This guides the selection of stability margin $ϵ=0.05$ used in our implementation.

3.8. Implementation and Testing Environment

The ML-CALMO framework was implemented using PyTorch 1.13 with CUDA 11.7 support for GPU acceleration. The software architecture consists of four main modules:

• Data Processing Module: Handles real-time data ingestion, preprocessing, and feature extraction from traffic sensors and order management systems.
• ML Prediction Module: Implements LSTM networks (2 layers, 128 hidden units) for demand forecasting and CNN (3 convolutional layers, 64-128-256 filters) for traffic prediction.
• Routing Optimization Module: Contains the DQN implementation (3 hidden layers, 512-256-128 neurons) with experience replay buffer (capacity 10,000).
• Queueing Control Module: Implements stability monitoring, admission control, and dispatcher logic using NumPy for efficient matrix operations.

Module interfaces use REST APIs for loose coupling, enabling independent scaling and updates. The system deploys on Kubernetes for production environments, with each module containerized using Docker.

Hardware Specifications: All experiments were conducted on a compute cluster with the following specifications:

• GPU: NVIDIA RTX 4090 (24 GB VRAM);
• CPU: AMD Ryzen 9 5950X (32 cores, 3.4 GHz base);
• RAM: 32 GB DDR4-3600;
• Storage: 2 TB NVMe SSD;
• Operating System: Ubuntu 20.04 LTS;
• Software Stack: PyTorch 1.13, CUDA 11.7, Python 3.9.

3.9. Test Scenarios and Datasets

Our experimental evaluation uses three categories of test data:

• Training Data: Historical delivery records from 3 months (January–March 2024) including 15,000 orders with timestamps, locations, priorities, and actual delivery times. Data augmentation techniques generate additional 10,000 synthetic training samples.
• Validation Data: One month of held-out data (April 2024) with 5000 orders for hyperparameter tuning and model selection.
• Test Data:
  • MDDVRPSRC benchmark: A total of 10 dynamic networks with 14–49 nodes.
  • Real-world spatial data: Singapore CBD (25 nodes), Madrid suburbs (40 nodes), São Paulo logistics district (35 nodes).
  • Synthetic scenarios: A total of 50 generated instances with varying demand patterns and priority distributions.

Each test scenario runs 50 times with different random seeds (0–49) for statistical validity. Complete details on dynamic benchmark generation, including instance generation procedures and stochastic process parameters, are provided in Appendix A.

4. Results

4.1. Dataset Selection and Experimental Design

Full dataset specifications and data-generation procedures are documented in Section 3.9. Below, we describe the specific datasets actually used in our experiments.

Primary Dataset: Multi-Depot Dynamic VRP with Stochastic Road Capacity (MDDVRPSRC) 2022 [8]—10 dynamic networks (14–49 nodes) with realistic damage scenarios representing urban disaster conditions.

Secondary Dataset: Real-world spatial data generated using OSMnx [9] from three metropolitan areas: Singapore CBD (25 nodes), Madrid suburbs (40 nodes), and São Paulo logistics district (35 nodes).

Validation Dataset: Asymmetric Clustered VRP instances [19] adapted with dynamic arrivals for consistency checking.

Baseline Selection: Methods represent current state-of-the-practice rather than theoretical benchmarks:

• OR-Tools VRP Solver (Google, 2024 version)—industry standard;
• Attention-VRP [20]—leading academic method;
• DRL-Stochastic [21]—recent RL approach (2023);
• Hybrid RL-VNS [22]—2023 hybrid method.

4.2. Hardware Configuration

For reproducibility, we report the complete hardware specifications (CPU, GPU, RAM, OS, and software versions) in Section 3.8. To avoid duplication, the details are not repeated here.

4.3. Performance Comparison Results

Machine learning component evaluation uses Mean Absolute Percentage Error (MAPE) for LSTM demand predictions and CNN traffic analysis:

$$\begin{array}{cc}\hfill {\mathrm{MAPE}}_{\mathrm{LSTM}}& ={\displaystyle \frac{1}{n}}\sum _{k=1}^n{\displaystyle \frac{|{\widehat{\lambda }}_k\left(t\right)-{\lambda }_k\left(t\right)|}{|{\lambda }_k\left(t\right)|}}\hfill \end{array}$$

(19)

$$\begin{array}{cc}\hfill {\mathrm{MAPE}}_{\mathrm{CNN}}& ={\displaystyle \frac{1}{n}}\sum _{k=1}^n{\displaystyle \frac{|{\widehat{T}}_k^{\mathrm{travel}}\left(t\right)-T_k^{\mathrm{travel}}\left(t\right)|}{|T_k^{\mathrm{travel}}\left(t\right)|}}\hfill \end{array}$$

(20)

Table 1. Performance comparison with clear separation of absolute values and improvements ($n=50$ runs, paired t-test).

Method	Service	Success	Delivery	Priority	Total
	Efficiency (%)	Rate (%)	Time (min)	Success (%)	Cost (USD)
OR-Tools VRP	$64.2\pm 3.1$	$82.1\pm 1.9$	$52.8\pm 4.2$	$85.2\pm 2.1$	$1284.5\pm 67.3$
Attention-VRP	$68.7\pm 2.8$	$84.6\pm 1.7$	$49.1\pm 3.8$	$87.4\pm 1.8$	$1205.2\pm 52.1$
DRL-Stochastic	$71.3\pm 2.5$	$86.2\pm 1.5$	$46.8\pm 3.5$	$89.1\pm 1.7$	$1156.7\pm 48.9$
Hybrid RL-VNS	$72.9\pm 2.3$	$87.3\pm 1.4$	$45.2\pm 3.2$	$90.2\pm 1.6$	$1134.2\pm 45.6$
ML-CALMO	$\mathbf{81.8}\pm \mathbf{2.1}$	$\mathbf{89.7}\pm \mathbf{1.4}$	$\mathbf{36.8}\pm \mathbf{2.9}$	$\mathbf{92.1}\pm \mathbf{1.6}$	$\mathbf{985.3}\pm \mathbf{41.2}$
Improvement vs. Best Baseline (Hybrid RL-VNS)	+8.9 pp	+2.8 pp	−18.5%	+2.1 pp	−13.1%

presents our main experimental results comparing ML-CALMO against four state-of-the-art baseline methods. The table shows absolute performance values with standard deviations from 50 independent runs, along with percentage improvements relative to the best baseline (Hybrid RL-VNS).

4.4. Comparative Analysis with Latest Methods

Table 2. Detailed comparison with recent state-of-the-art methods including computational requirements.

Aspect	DRL-Stochastic	Hybrid RL-VNS	ML-CALMO	Improvement
	[21]	[22]	(Ours)	(%)
Approach	Deep RL with	RL + Variable	ML + Queueing	—
	stochastic demand	neighborhood search	Theory
Scalability	Up to 100 orders	Up to 150 orders	Up to 200 orders	33%
Priority Handling	Limited	Basic	Advanced with	—
			preemption
Theoretical Guarantee	No	No	Yes (stability bounds)	—
Real-time Adaptation	Moderate	Limited	High	—
Computational Time (s)	$8.3\pm 0.4$	$12.7\pm 0.6$	$3.2\pm 0.2$	61.4%
Memory Usage (GB)	2.8	3.4	1.8	35.7%
Hardware Requirement	GPU recommended	CPU sufficient	GPU required	—

provides a detailed comparison of ML-CALMO with the two most recent state-of-the-art methods published in 2023. This comparison highlights the architectural differences, scalability improvements, and computational efficiency of our approach.

4.5. Machine Learning Component Performance

Table 3. Machine learning component performance analysis with statistical significance.

ML Component	MAPE (%)	R-Squared	Training Time (min)	p-Value vs.
				Persistence Model
LSTM Demand (${\widehat{\lambda }}_p,{\widehat{\lambda }}_r$)	$9.2\pm 0.4$	$0.873\pm 0.021$	$45.3\pm 2.1$	<0.001
CNN Traffic (${\widehat{T}}_{ij}^{\mathrm{travel}}$)	$12.4\pm 0.6$	$0.804\pm 0.032$	$38.7\pm 1.8$	<0.001
DQN Route Optimization	N/A	N/A	$72.5\pm 3.2$	<0.001
Queueing Controller	N/A	N/A	$12.3\pm 0.8$	<0.001
Integrated ML-CALMO	$\mathbf{7.1}\pm \mathbf{0.2}$	$\mathbf{0.924}\pm \mathbf{0.014}$	$\mathbf{168.8}\pm \mathbf{4.7}$	<0.001

demonstrates the effectiveness of each algorithmic element.

4.6. Theoretical Model Validation

Table 4. Theoretical model validation: predicted vs. actual performance.

Performance Metric	Analytical	Simulation	Deviation	95% CI
	Prediction	Result	(%)
System Utilization (${\widehat{\rho }}_{\mathrm{system}}$ vs. ${\rho }_{\mathrm{system}}$)	0.784	0.791	0.9	[0.782, 0.800]
Average Response Time (min)	38.7	36.8	4.9	[35.2, 38.4]
Priority Queue Length (orders)	2.14	2.23	4.2	[2.11, 2.35]
System Throughput (orders/h)	47.3	48.1	1.7	[47.2, 49.0]
Regular Queue Length (orders)	3.92	4.07	3.8	[3.89, 4.25]
Preemption Probability	0.089	0.093	4.5	[0.087, 0.099]

validates our theoretical predictions against simulation results, demonstrating that our analytical queueing model accurately predicts system behavior with deviations below 5% for all metrics.

4.7. Statistical Significance Analysis

Table 5. Statistical significance tests (paired t-test with Bonferroni correction, $\alpha =0.01$).

Metric	vs. Hybrid	vs. DRL-	vs. Attention-	Cohen’s d	Significant?
	RL-VNS	Stochastic	VRP
Service Efficiency	$p<0.001$	$p<0.001$	$p<0.001$	2.84	Yes
Service Success Rate	$p=0.003$	$p<0.001$	$p<0.001$	1.73	Yes
Avg. Delivery Time	$p<0.001$	$p<0.001$	$p<0.001$	3.21	Yes
Priority Success	$p=0.002$	$p<0.001$	$p<0.001$	1.92	Yes
Total Cost	$p<0.001$	$p<0.001$	$p<0.001$	2.56	Yes

presents comprehensive statistical significance testing results using paired t-tests with Bonferroni correction to account for multiple comparisons. All comparisons show statistical significance with large effect sizes (Cohen’s d > 0.8), confirming the practical importance of our improvements.

4.8. Robustness Analysis Under Assumption Violations

Table 6. Performance degradation under varying assumption violation levels.

Parameter Drift Rate (/min)	Efficiency Loss (%)	Stability Maintained	Operational Viability
<0.1 (normal traffic)	0.0	Yes	Full
$0.1$–$0.3$ (moderate variation)	$2.8\pm 0.8$	Yes	Full
$0.3$–$0.5$ (rush hour)	$7.2\pm 1.4$	Yes	Reduced
$0.5$–$0.8$ (traffic incidents)	$18.6\pm 2.9$	Marginal	Limited
>0.8 (emergency conditions)	>35.0	No	Not viable

examines system performance when our theoretical assumptions are violated, showing graceful degradation up to a parameter drift rate of 0.5/min. This analysis establishes clear operational boundaries for practical deployment.

4.9. Stability Analysis Under Varying Conditions

Table 7. System stability under stress testing scenarios.

Scenario	Load Factor	Max Utilization	Stability Maintained	Performance Degradation
Normal Operations	$1.0\times$	0.784	Yes	Baseline
Peak Hour Surge	$1.5\times$	0.891	Yes	8.3%
Extreme Demand	$2.0\times$	0.943	Yes	18.7%
System Limit	$2.2\times$	0.949	Yes (marginal)	24.2%
Overload	$2.5\times$	0.971	No (rejected)	N/A

demonstrates the system’s ability to maintain stability under increasing load conditions. The queueing controller successfully prevents overload by rejecting orders that would violate stability constraints, maintaining system integrity up to 2.2× normal load.

5. Discussion

The experimental results demonstrate measurable improvements in delivery optimization, though with important caveats regarding deployment constraints and validation limitations. The 18.5% delivery time improvement and 8.9 percentage point service efficiency gain represent meaningful operational benefits. It is important to note that the 8.9 percentage point service efficiency improvement is calculated relative to the best baseline method (Hybrid RL-VNS at 72.9%), resulting in an absolute efficiency of 81.8%. This represents a relative improvement of approximately 12.2% over the baseline’s efficiency. We have clarified this distinction in Table 1 to avoid confusion.

5.1. Theoretical Contributions

Our work makes several important theoretical contributions to the intersection of machine learning and queueing theory:

• Piecewise-Stationary Framework: The introduction of piecewise-stationary approximation with proven error bounds provides a principled approach to handling time-varying parameters in queueing systems augmented with machine learning predictions.
• Stability Preservation: Lemma 1 establishes conditions under which prediction errors from machine learning components do not compromise system stability, providing crucial safety guarantees for practical deployment.
• Convergence Analysis: The integration of DQN with queueing constraints represents a novel contribution to constrained reinforcement learning, with guaranteed convergence under mild assumptions.

5.2. Limitations and Validity Boundaries

While our results are promising, several limitations must be acknowledged:

• Simulation-Based Evaluation: The framework evaluation relies on synthetic and adapted benchmarks, limiting real-world applicability claims. Field validation with actual logistics partners remains necessary to establish operational effectiveness.
• Computational Requirements: The system requires mid-range GPU hardware (RTX 3080 minimum) for real-time operation. CPU-only deployment results in 24.1% performance degradation and limits problem size to 50 orders.
• Parameter Drift Sensitivity: Performance degrades significantly when parameter drift exceeds 0.5/min, restricting applicability during rapid changes like traffic incidents or emergency scenarios.
• Training Data Requirements: The ML components require substantial historical data (minimum 3 months) for effective training, limiting applicability in new markets or rapidly changing operational contexts.
• Preemption Cost Modeling: While we integrate preemption costs explicitly, real-world switching overhead may vary significantly across operational contexts and driver experience levels.

5.3. Failure Mode Analysis

Understanding when and why the method fails is crucial for practical deployment:

• Prediction Error Cascade: When LSTM demand predictions exceed 15% MAPE, the admission control mechanism becomes overly conservative, rejecting viable orders and reducing system throughput by up to 30%.
• Traffic Anomalies: Unexpected events (accidents, road closures) not captured in historical data cause CNN predictions to fail catastrophically, with travel time errors exceeding 40%, leading to infeasible routing decisions.
• Priority Storms: Sudden surges in priority orders (>40% of total) can cause system thrashing, where vehicles constantly switch between orders without completing deliveries, reducing effective capacity by 50%.

These failure modes inform deployment strategies and highlight areas for future research. We recommend implementing fallback mechanisms for each failure mode, including manual override capabilities for extreme scenarios.

5.4. Future Work

Future research directions include

• Extension to multi-depot scenarios with transfer points;
• Integration of environmental objectives for sustainable delivery;
• Development of distributed learning mechanisms for privacy-preserving optimization;
• Adaptation to autonomous vehicle fleets with different operational characteristics.

6. Conclusions

This research presents ML-CALMO, a comprehensive framework that successfully integrates machine learning techniques with established queueing theory principles to address fundamental challenges in last-mile delivery optimization. The framework achieves statistically significant improvements (18.5% delivery time reduction, 8.9 percentage point service efficiency gain versus recent methods) while acknowledging critical deployment constraints including GPU hardware requirements, parameter drift sensitivity, and validation limitations to simulation environments.

The key contributions of this work include the following:

• A novel integration architecture that seamlessly combines machine learning predictions with queueing theory constraints, providing both optimization capability and theoretical guarantees.
• Rigorous theoretical foundations including Theorem 1 for piecewise-stationary approximation and Lemma 1 for stability preservation under prediction errors.
• Comprehensive experimental validation demonstrating significant improvements over state-of-the-art methods, including recent 2023 approaches, with rigorous statistical analysis.
• Honest assessment of computational requirements, operational boundaries, and the gap between simulation performance and field deployment needs.

The theoretical foundations established in this work—particularly the piecewise-stationary approximation framework and stability preservation under prediction errors—provide a principled approach for integrating machine learning with classical operational research methods. This integration represents a promising direction for addressing complex real-world optimization problems that require both predictive capability and performance guarantees.

Most importantly, our work demonstrates both the potential and the practical constraints of integrating ML with operational research. While we achieve meaningful improvements, the gap between simulation results and operational deployment remains a challenge across the entire field. Success requires careful attention to computational feasibility, failure mode planning, and honest assessment of performance boundaries.

The success of ML-CALMO suggests that future research in logistics optimization should focus on principled integration of learning and optimization, rather than treating them as separate paradigms. As delivery systems become increasingly complex and dynamic, such integrated approaches will be essential for maintaining both efficiency and reliability, though always within clearly understood operational constraints.