Topology-Aware Deep Reinforcement Learning for Dynamic Multicast Routing in Software-Defined Networks

Abstract

Dynamic multicast routing in software-defined networks is challenging due to continuously changing network states, multicast branch coupling, and the dependency between local forwarding decisions and global multicast tree construction. Existing multicast routing approaches mainly rely on static heuristics or snapshot-based optimization, which makes them difficult to maintain routing adaptability and decision stability under dynamic network conditions. To address these limitations, this paper proposes a topology-aware deep reinforcement learning multicast routing algorithm, named Graph-structured Hierarchical Actor–Critic for Multicast Routing (GHAC-MR). Specifically, the multicast routing process is formulated as a sequential tree construction problem, where each forwarding action incrementally affects the subsequent multicast tree evolution. A graph-structured state representation mechanism is designed to encode network topology information, link resource states, and multicast branch dependencies, enabling the routing agent to capture structural correlations among multicast forwarding nodes. Furthermore, a hierarchical actor–critic learning architecture is introduced to jointly optimize multicast forwarding policies and long-term routing rewards, thereby improving routing adaptability and convergence stability in dynamic network environments. Experimental results on multiple representative network topologies demonstrate that the proposed GHAC-MR algorithm achieves superior performance in multicast acceptance ratio, resource utilization efficiency, and routing adaptability compared with representative heuristic, evolutionary, and reinforcement learning-based multicast routing schemes.

1. Introduction

Future intelligent networks are expected to support large-scale service diversity, adaptive traffic management, and dynamic resource coordination across heterogeneous communication environments [1]. In addition to ultra-reliable low-latency communication and massive machine-type access, emerging network services increasingly require efficient content dissemination [2], collaborative perception, and distributed intelligence among geographically distributed devices and infrastructures. In this context, multicast communication plays a fundamental role by enabling a single data source to simultaneously serve multiple receivers, thereby reducing redundant transmissions and improving overall spectrum and bandwidth efficiency. As multicast services become increasingly prevalent in dynamic network scenarios, designing efficient and adaptive multicast routing mechanisms has become an important research challenge [3].

Unlike unicast routing, multicast routing must jointly consider the selection of branching points and the construction of a shared delivery structure [4]. The objective is not only to reach all destination nodes but also to minimize resource consumption while satisfying Quality-of-Service (QoS) constraints such as delay, bandwidth, and reliability. This problem becomes increasingly challenging in dynamic networks, where traffic demands evolve continuously, link conditions fluctuate over time, and service requirements become more heterogeneous. Under such conditions, multicast routing policies should continuously adapt to periodically updated network observations while avoiding excessive control overhead and computational complexity.

From an optimization perspective, multicast routing is commonly formulated as a Steiner Tree problem, which seeks a minimum-cost subgraph connecting the source and a set of destination nodes. Although this abstraction provides a theoretical foundation, its direct application in practical networks is limited [5]. The Steiner Tree problem is NP-complete, and the complexity of finding high-quality solutions increases rapidly with network size and dynamics [6]. Classical approximation algorithms can achieve acceptable performance under static or slowly varying conditions [7], but they are inherently snapshot-based and lack the ability to adapt to rapid changes in link states or traffic patterns. As a result, their effectiveness degrades in highly dynamic network environments.

To overcome these limitations, Software-Defined Networking (SDN) has been widely recognized as a promising architectural paradigm for multicast routing [8]. By decoupling the control plane from the data plane, SDN enables centralized network management, global state visibility, and flexible control over forwarding behaviors. In practical SDN environments, the controller periodically collects network state information from distributed forwarding devices, which may introduce synchronization overhead and partially delayed network observations under highly dynamic traffic conditions. Through continuous monitoring of link metrics and traffic statistics, an SDN controller can dynamically update multicast forwarding rules and respond to network variations in a timely manner. These capabilities provide a practical foundation for implementing adaptive multicast routing strategies that go beyond static optimization models.

However, existing SDN-based multicast solutions still face fundamental challenges. Many approaches rely on deterministic optimization or heuristic-based algorithms that operate on instantaneous network states. While such methods can exploit the global view provided by SDN, they typically lack the ability to capture temporal dependencies and long-term impacts of routing decisions [9]. In dynamic multicast scenarios, decisions made based solely on current network conditions may lead to suboptimal resource utilization or future performance degradation. Moreover, as the scale of the network and the number of multicast group members increase, the computational overhead of centralized optimization becomes a bottleneck, limiting real-time applicability.

Recently, graph-based learning methods and multi-agent reinforcement learning approaches have attracted increasing attention in intelligent routing research due to their capability to model topology-aware interactions and adaptive decision-making processes. Graph neural network-based approaches can effectively characterize structural correlations among network nodes, while reinforcement learning frameworks provide adaptive policy optimization under dynamic environments. Nevertheless, most existing graph-based routing methods are primarily designed for unicast communication or independent path optimization [10], making it difficult to explicitly capture multicast-specific branch dependencies and shared forwarding structures. In addition, many multi-agent reinforcement learning solutions rely on local optimization objectives or independent agent coordination, which may result in inconsistent multicast tree construction and unstable routing performance under dynamic network conditions.

Meta-heuristic algorithms [11] and learning-based approaches [10] have been investigated to improve the adaptability of multicast routing. While these methods exhibit enhanced exploration capability or benefit from data-driven decision-making, they typically rely on handcrafted objective aggregation or implicitly assume independent routing decisions. Such assumptions make it difficult to explicitly capture the coupling between multiple destinations, where path sharing and branching structures play a critical role. Moreover, many learning-based solutions are primarily developed for unicast routing or focus on optimizing fixed performance objectives, limiting their ability to dynamically balance heterogeneous QoS requirements across multicast receivers. As illustrated in Figure 1, directly applying existing routing intelligence methods to multicast scenarios often results in suboptimal multicast tree construction and inefficient resource utilization, thereby making it challenging to satisfy the stringent and dynamic requirements of future intelligent networks.

Motivated by these observations, this work investigates dynamic multicast routing from a learning-driven and topology-aware perspective under an SDN framework. Instead of relying on rigid centralized optimization, the proposed approach exploits the global observability of SDN to support adaptive multicast tree construction under continuously changing network conditions. The main objective is to dynamically construct near-optimal multicast distribution trees while jointly considering multicast branch dependencies, heterogeneous link conditions, and long-term routing efficiency. By integrating graph-structured topology representation with sequential reinforcement learning-based routing decisions, the proposed method enables multicast forwarding policies to continuously evolve with network dynamics while maintaining routing stability and scalability under periodically updated network observations.

The main contributions of this work are summarized as follows. First, we formulate dynamic multicast routing as a topology-aware sequential tree construction problem by explicitly incorporating multicast branch dependency and shared forwarding behaviors into the reward design. The proposed reward function jointly captures local forwarding decisions and global tree-level performance, enabling the learning process to account for shared paths and branching behaviors among multiple receivers. This design provides a principled mechanism for guiding agents toward multicast trees that balance individual routing choices with overall multicast efficiency under dynamic network conditions.

Second, we develop an actor–critic based multicast routing algorithm in which routing decisions are generated sequentially to construct multicast distribution trees. A graph-structured state representation mechanism is introduced to encode network topology information, link resource states, and multicast tree construction context, enabling the routing agent to capture structural correlations among multicast forwarding nodes. The actor network is responsible for selecting forwarding actions based on current network observations and tree construction context, while the critic network evaluates these actions using the combined reward signal. Through this coordinated architecture, the algorithm effectively integrates local decision-making with global performance feedback, allowing multicast routing policies to be learned in an adaptive and structurally consistent manner.

Extensive experimental results on multiple representative network topologies demonstrate that the proposed approach consistently outperforms existing multicast routing methods in terms of multicast acceptance ratio, resource utilization efficiency, and routing adaptability, validating its effectiveness for dynamic multicast services in SDN-enabled wireless networks.

2. Related Work

Multicast routing has received renewed attention in recent years due to the increasing demand for efficient group communication in intelligent and dynamic wireless networks. With the continuous evolution of intelligent network services, multicast communication is expected to operate under highly dynamic conditions, where link states, traffic patterns, and service requirements change continuously. These characteristics challenge traditional multicast routing schemes and have motivated extensive research on intelligent and adaptive multicast routing methods. Against this backdrop, this section presents the latest research advances in multicast routing approaches, exploring how to address the aforementioned routing challenges in dynamic environments.

Vera-Aragón et al. encoded light forests as chromosomes for multicast routing and applied genetic operators to minimize blocked requests, maximum frequency slot usage, and transponder count [12]. Jacob et al. proposed a hybrid metaheuristic that integrates the exploitation capability of Ant Colony Optimization with the exploration rate of Artificial Bee Colony, where pheromone-based path construction is combined with employed bee and onlooker bee phases to balance global search and local refinement for multicast routing optimization [13]. Wang et al. designed a multilayer obstacle avoidance spanning graph and a rectilinear Steiner tree construction algorithm utilizing multilayer link replacement to optimize tree length while mitigating link congestion in LEO satellite networks [14]. Hu et al. proposed software-defined multicasting with scalable segment routing protocols featuring source routing, equal cost multipath routing, and efficient tree recovery mechanisms [15]. Chen et al. proposed a multicast-efficient interconnect network with a DNN-tailored routing algorithm optimizing multicast tree structure and a lightweight router microarchitecture simplifying multicast ports [16]. Gao et al. proposed an ordered update scheme configuring inport match fields to avoid duplicate entries, implementing a dependency graph for dynamic scheduling and a greedy solution to the Replace Operation Tree Migration Problem for consistent multicast routing updates [17].

Despite these advances, existing methods predominantly rely on centralized computation, exhaustive search, or offline training, which inherently limits their adaptability to highly dynamic network conditions. These limitations stem from the fundamental trade-offs between solution quality and computational efficiency, as well as the difficulty of capturing time-varying network states within static optimization frameworks or pre-trained models that cannot generalize across evolving traffic patterns and topology changes.

Recently, graph-based deep learning and intelligent SDN routing approaches have attracted increasing attention due to their capability to model topology-aware dependencies and adaptive routing behaviors in dynamic network environments. Simon et al. proposed a Deep Graph Neural Network (GNN) based intelligent traffic routing framework for SDN, where network topology and dynamic link attributes are represented as graph structures to support adaptive routing decisions under changing traffic conditions [18]. Their work demonstrated the effectiveness of graph-based representations for capturing spatial correlations and improving routing adaptability in large-scale SDN environments. However, the proposed framework mainly focuses on unicast traffic routing and does not explicitly consider multicast tree construction or multicast branch dependency optimization.

In addition to intelligent routing optimization, machine learning techniques have also been applied to enhance the security and deployment capability of SDN systems. Song et al. investigated machine learning-based Distributed Denial of Service (DDoS) detection in SDN environments and proposed an optimized Voting Classifier combined with resampling mechanisms to improve minority attack traffic identification accuracy [19]. Their work demonstrated the feasibility of integrating intelligent learning mechanisms into practical SDN controllers and highlighted the importance of adaptive learning frameworks in dynamic SDN environments. Nevertheless, the study mainly focuses on SDN security management rather than multicast routing optimization and dynamic multicast decision coordination.

Li et al. proposed a stateless multicast mechanism using deep reinforcement learning with temporal convolutional networks to proactively build multicast trees and a source aggregation mechanism for high request volumes [20]. Li et al. proposed a deep reinforcement learning-assisted multi-objective algorithm with joint routing, modulation and spectrum assignment schemes for bandwidth-resource and service-quality optimization [21]. Liu et al. proposed a deep reinforcement learning-based multipath cooperative routing scheme combining Deep Reinforcement Learning (DRL)-based routing discovery with water filling algorithm-based multipath scheduling to optimize maximum path cost under dynamic satellite network conditions [22]. Wang et al. proposed an intelligent edge routing framework named SAG, which integrates GNN, Attention Mechanisms (AM), and Soft Actor–Critic (SAC) to dynamically capture network topology features and optimize routing decisions for Consumer Internet of Things (CIoT) edge environments [23]. By combining graph representation learning with reinforcement learning-based routing optimization, their method achieved improved QoS performance under heterogeneous and dynamic network conditions. However, the proposed framework mainly focuses on edge unicast routing and QoS-aware path optimization, while multicast-specific routing characteristics such as branch coupling, shared forwarding structures, and multicast tree evolution are not explicitly addressed.

Overall, recent advances in multicast routing reveal a clear shift toward learning-driven and intelligent approaches that leverage deep reinforcement learning, hierarchical decision-making, multi-agent cooperation, and graph-based representations. Despite their potential, existing methods often address only part of the multicast routing problem, such as path selection or local optimization, without fully capturing the dynamic evolution of multicast trees under realistic network conditions. These limitations motivate the development of adaptive multicast routing frameworks that jointly consider dynamic network observations, tree-level optimization, and stable sequential decision-making under evolving network conditions, which is the focus of this work. To further highlight the differences between existing multicast routing methods and the proposed GHAC-MR algorithm, a comparative summary of representative approaches and their limitations is presented in

Table 1. Comparison of representative multicast routing approaches and the proposed GHAC-MR method.

Representative Methods	Limitations	Advantages of GHAC-MR
Meta-heuristic multicast routing [12,13]	High computational overhead and weak dynamic adaptability.	Adaptive multicast tree optimization through sequential policy learning.
Steiner tree-based optimization [14]	Mainly designed for static multicast tree construction.	Topology-aware multicast routing under dynamic network conditions.
SDN multicast update methods [15,17]	Depend on centralized scheduling and fixed update strategies.	Sequential actor–critic-based multicast routing optimization.
DNN-assisted multicast architectures [16]	Focus on fixed multicast structures rather than dynamic routing evolution.	Adaptive multicast tree construction with topology-aware learning.
Graph-based SDN routing [18]	Mainly designed for unicast routing without multicast branch modeling.	Explicit multicast branch dependency and tree-level optimization.
Machine learning-based SDN management [19]	Focus on SDN security rather than multicast routing optimization.	Learning-driven multicast routing under SDN environments.
Learning-based multicast routing [20]	Limited global multicast tree awareness and structural modeling.	Joint optimization of multicast forwarding and multicast tree evolution.
DRL-assisted routing methods [21,22,23]	Primarily optimize unicast or multipath routing scenarios.	Topology-aware actor–critic multicast routing framework.

.

3. System Model

This section introduces the system model for dynamic multicast routing in SDN-enabled wireless networks. We consider a programmable network environment in which multicast services must be supported under time-varying link conditions and dynamic traffic demands. The system model aims to capture the essential characteristics of multicast communication, including shared transmission paths, branching structures, and evolving network states, while providing a clear abstraction for the design of adaptive multicast routing algorithms.

The underlying network is modeled as a graph whose link attributes, such as delay, bandwidth availability, and transmission cost, may vary over time. Multicast services are characterized by a source node and a set of destination nodes, where multicast distribution trees are constructed to efficiently deliver data from the source to all receivers. Within the SDN framework, a centralized controller maintains a global view of the network and is responsible for monitoring network dynamics and updating multicast forwarding decisions accordingly. In practical network environments, network state collection and synchronization between forwarding devices and the SDN controller may introduce observation delay and control overhead, especially under highly dynamic traffic conditions. Therefore, multicast routing decisions are generated based on periodically updated network observations rather than ideal instantaneous global states. Based on this model, multicast routing is treated as a dynamic sequential decision-making process, laying the groundwork for the problem formulation and topology-aware learning-based solution presented in the subsequent sections.

3.1. Problem Formulation

We consider a dynamic multicast routing problem in an SDN-enabled network, which is modeled as a directed graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$, where $\mathcal{V}$ denotes the set of network nodes and $\mathcal{E}$ represents the set of links. Each link $e\in \mathcal{E}$ is associated with time-varying attributes, such as transmission cost, delay, or available bandwidth, which collectively reflect the current network condition. These link attributes may change over time due to traffic fluctuations and network dynamics.

A multicast service request is defined by a source node $s\in \mathcal{V}$ and a destination set $\mathcal{D}\subset \mathcal{V}\setminus \left\{s\right\}$. To support the multicast service, a multicast distribution tree $\mathcal{T}=({\mathcal{V}}_{\mathcal{T}},{\mathcal{E}}_{\mathcal{T}})$ is constructed, where ${\mathcal{V}}_{\mathcal{T}}\subseteq \mathcal{V}$ and ${\mathcal{E}}_{\mathcal{T}}\subseteq \mathcal{E}$. The tree connects the source node to all destination nodes through shared paths and branching nodes, enabling efficient data delivery to multiple receivers.

The cost of a multicast tree is determined by the aggregate cost of its constituent links. At time slot t, the cost of a multicast tree ${\mathcal{T}}_t$ can be expressed as

$$C\left({\mathcal{T}}_t\right)=\sum _{e\in {\mathcal{E}}_{{\mathcal{T}}_t}}{c}_e\left(t\right),$$

(1)

where $c_e\left(t\right)$ denotes the cost associated with link e at time t. Depending on the system objective, $c_e\left(t\right)$ may represent a weighted combination of delay, bandwidth consumption, or other performance metrics.

At each time slot, the SDN controller observes the current network state and determines a multicast distribution tree that satisfies connectivity constraints while optimizing long-term performance. The objective is to construct a sequence of multicast trees $\left\{{\mathcal{T}}_t\right\}$ that minimizes the cumulative routing cost over time, which can be formulated as

$$\underset{\left\{{\mathcal{T}}_t\right\}}{\min }{\displaystyle \sum _{t=1}^T}C\left({\mathcal{T}}_t\right),$$

(2)

subject to the constraints that each destination node in $\mathcal{D}$ is reachable from the source node s through the multicast tree at every time slot.

This problem is inherently challenging due to two key factors. First, multicast routing requires joint optimization of shared paths and branching structures, rather than independent path selection. Second, the time-varying nature of link states makes static or snapshot-based optimization insufficient. As a result, the dynamic multicast routing problem can be viewed as a sequential decision-making process, where the routing decision at each time slot influences both the immediate routing cost and future network states. This formulation provides a foundation for developing learning-based multicast routing strategies that adapt to network dynamics while maintaining efficient multicast tree structures.

3.2. Multicast Tree Evaluation Function

In the proposed multi-agent multicast routing framework, multiple agents collaboratively construct a multicast distribution tree, where each agent corresponds to a destination node and determines how it is integrated into the evolving tree. The evaluation function quantifies the impact of each routing decision on both local expansion efficiency and the global structure of the multicast tree. The construction process is inherently sequential: starting from the source node, agents incrementally establish forwarding paths to connect their respective destinations. Each decision not only affects the individual transmission cost but also reshapes the subsequent tree topology and influences path-sharing opportunities among other destinations, thereby requiring the evaluation mechanism to jointly consider local routing efficiency and global multicast optimization.

At time slot t, let ${\mathcal{T}}_t$ denote the current multicast distribution tree rooted at the source node, and let ${\mathcal{E}}_{{\mathcal{T}}_t}$ represent the set of links already included in the tree. For a destination node $d_i$, the corresponding agent selects a candidate path $P_i\left(t\right)$ that connects $d_i$ to the current multicast tree through a selected attachment node belonging to ${\mathcal{T}}_t$. Since multicast routing allows multiple destinations to share common transmission paths, only the links newly introduced into the tree incur additional cost.

The incremental cost induced by agent i is defined as

$$\Delta C_i\left(t\right)=\sum _{e\in P_i\left(t\right)\setminus {\mathcal{E}}_{{\mathcal{T}}_t}}{c}_e\left(t\right),$$

(3)

where $c_e\left(t\right)$ denotes the time-varying cost of link e. This formulation explicitly captures the benefit of path sharing in multicast routing, as links that already belong to the multicast tree are not penalized multiple times.

Beyond link-level cost, the structural efficiency of a multicast tree is also influenced by how routing decisions reuse existing branching structures. Let ${\mathcal{B}}_t$ denote the set of branching nodes in ${\mathcal{T}}_t$, where a branching node is defined as a tree node whose degree is greater than one. To encourage agents to reuse existing branching structures, a branching reuse indicator is introduced. Specifically, the indicator function equals one if the attachment node of path $P_i\left(t\right)$ belongs to ${\mathcal{B}}_t$, and equals zero otherwise. Based on this definition, the local evaluation value for agent i is given by

$$r_i^{\mathrm{local}}\left(t\right)=-\Delta C_i\left(t\right)+\beta \mathbb{I}\left(\mathrm{attachment}\mathrm{node}\mathrm{of}{P}_i\left(t\right)\in {\mathcal{B}}_t\right),$$

(4)

where $\beta$ is a positive weighting parameter. This local evaluation function encourages agents to minimize incremental routing cost while preferentially attaching destinations through existing branching nodes, leading to more compact and resource-efficient multicast tree structures.

The weighting parameter $\beta$ controls the trade-off between minimizing incremental routing cost and encouraging multicast branch reuse. A moderate positive value of $\beta$ allows the routing process to balance transmission efficiency and multicast structural compactness, thereby avoiding excessive branch expansion caused by purely shortest-path oriented routing decisions.

While the local evaluation reflects the immediate structural impact of individual routing decisions, cooperative multicast routing further requires consistency with the overall evolution of the multicast tree. Let the total cost of the multicast tree at time slot t be defined as

$$C\left({\mathcal{T}}_t\right)=\sum _{e\in {\mathcal{E}}_{{\mathcal{T}}_t}}{c}_e\left(t\right).$$

(5)

Instead of using the absolute tree cost as a global signal, a global incremental evaluation term is introduced to reflect the net change in tree cost resulting from agents’ routing decisions at time slot t, which is defined as

$$r^{\mathrm{global}}\left(t\right)=-\left(C\left({\mathcal{T}}_t\right)-C\left({\mathcal{T}}_{t-1}\right)\right).$$

(6)

This global evaluation signal provides a clearer credit assignment by directly measuring how the multicast tree evolves due to collective agent actions.

Compared with directly optimizing the absolute multicast tree cost, the incremental global evaluation mechanism provides more stable learning feedback by focusing on the relative change introduced by current routing actions. This design helps the learning agent better capture the long-term influence of sequential multicast tree construction decisions under dynamic network conditions.

The final evaluation value for agent i is obtained by combining the local and global terms as

$$r_i\left(t\right)=r_i^{\mathrm{local}}\left(t\right)+\lambda r^{\mathrm{global}}\left(t\right),$$

(7)

where $\lambda$ is a weighting factor that balances local routing efficiency and global multicast tree optimization. Through this evaluation function, agents are guided to minimize incremental cost, reuse existing multicast structures, and cooperatively construct an efficient multicast distribution tree under dynamic network conditions.

Specifically, a relatively small value of $\lambda$ makes the routing policy focus more on local path optimization, while a larger value strengthens the influence of global multicast tree evolution during policy learning. Therefore, the parameter $\lambda$ provides a flexible mechanism for balancing local forwarding behavior and global multicast tree coordination during sequential multicast routing optimization.

3.3. Actor–Critic Network Design

To address the combinatorial complexity of multicast routing, the proposed framework adopts an incremental multicast tree construction strategy, where destination nodes are progressively attached to the existing partial tree through sequential routing decisions. Starting from the source node, each attachment action is determined according to the current network state and evolving multicast topology. Since each newly established attachment path influences subsequent tree expansion and future routing choices, the multicast routing process exhibits strong sequential dependency and multicast branch coupling characteristics.

We model this distributed decision process using a multi-agent reinforcement learning framework, in which each destination-specific attachment step is treated as an individual decision-making agent. Let the multicast session involve a destination set $\mathcal{D}=\{d_1,d_2,\dots ,d_m\}$. At step i, the agent responsible for destination $d_i$ selects an attachment action that connects $d_i$ to the current multicast tree.

This multi-agent formulation enables different destination nodes to cooperatively participate in multicast tree construction while sharing a common optimization objective. Compared with directly generating a complete multicast tree in a single optimization stage, the sequential attachment mechanism reduces the search complexity of multicast routing and improves the stability of policy learning under dynamic network environments.

Formally, the multicast routing process is described as a Multi-Agent Markov Decision Process (MAMDP) defined by the tuple

$$(\mathcal{X},{\left\{{\mathcal{U}}^i\right\}}_{i\in \mathcal{D}},\mathcal{T},\mathcal{G}),$$

where $\mathcal{X}$ denotes the global network state, ${\mathcal{U}}^i$ is the action space for agent i, $\mathcal{T}$ defines the state transition dynamics, and $\mathcal{G}$ represents the global reward associated with multicast tree quality. Each agent has access to the shared state representation and receives common reward feedback reflecting the overall multicast performance.

Let ${\mathcal{T}}_{i-1}$ denote the multicast tree constructed after attaching the first $i-1$ destinations. The action ${\pi }_i\in {\mathcal{U}}^i$ selected by agent i corresponds to a feasible attachment path that connects destination $d_i$ to any node already included in ${\mathcal{T}}_{i-1}$. The complete multicast tree is thus obtained as a result of a sequence of attachment actions

$$\mathsf{\Pi }=\{{\pi }_1,{\pi }_2,\dots ,{\pi }_m\}.$$

Due to the sequential dependency among attachment decisions, the joint probability of selecting $\mathsf{\Pi }$ under state $x\in \mathcal{X}$ is factorized as

$$\begin{array}{cc}\hfill P(\mathsf{\Pi }\mid x)& =P({\pi }_1\mid x)P({\pi }_2\mid x,{\mathcal{T}}_1)\cdots \hfill \\ \hfill & P({\pi }_m\mid x,{\mathcal{T}}_{m-1}),\hfill \end{array}$$

(8)

where each conditional probability explicitly depends on the current partial multicast tree.

This factorized formulation allows future attachment decisions to dynamically adapt according to the evolving multicast tree structure, thereby enabling the learning process to capture multicast branch reuse relationships and structural dependencies during multicast tree construction.

To encode the influence of previously selected attachments, we introduce a context state $h^i$ for each agent. The context state captures the structural information of ${\mathcal{T}}_{i-1}$, including node membership and utilized links, and is represented as a binary or embedded vector over the network topology. When making an attachment decision, agent i conditions its policy on the pair $(x,h^i)$, which jointly reflects the global network conditions and the current multicast tree configuration.

The context state mechanism enables the routing agent to maintain awareness of the current multicast tree evolution process instead of generating routing decisions solely based on instantaneous network states. Consequently, the learned routing policy can better coordinate multicast branch sharing and tree-level structural consistency during sequential multicast tree construction.

Given the high-dimensional state and action spaces induced by large-scale networks, deep neural networks are employed to parameterize the agents’ policies. The actor for agent i is defined as

$${\mu }_{{\varphi }^i}({\pi }_i\mid x,h^i),$$

(9)

where ${\varphi }^i$ denotes the trainable parameters. This policy estimates the conditional distribution

$$\mathbb{P}({\pi }_i\mid x,{\mathcal{T}}_{i-1}),$$

thereby enabling tree-aware attachment decisions that adapt to both network dynamics and multicast structure evolution.

Specifically, the actor network is responsible for generating routing attachment decisions according to the current network state and multicast tree context, while the critic network evaluates the long-term contribution of these attachment decisions to multicast tree quality. Through this actor–critic interaction mechanism, the routing policy gradually learns multicast routing behaviors that balance local forwarding efficiency and global multicast tree optimization.

The objective of the multi-agent system is to maximize the expected long-term return associated with multicast routing performance, denoted by $F\left(\varphi \right)$, where $\varphi$ aggregates the parameters of all actors. A policy gradient estimator is used to optimize this objective:

$${\nabla }_{\varphi }F\left(\varphi \right)={\mathbb{E}}_{\rho \sim q_{\varphi }\left(\rho \right)}\left[\Delta ({\mathcal{T}}_{i-1},{\pi }_i){\nabla }_{\varphi }\log {\mu }_{\varphi }(\pi \mid x)\right],$$

(10)

where $\rho$ denotes a sampled trajectory of attachment actions and $\Delta ({\mathcal{T}}_{i-1},{\pi }_i)$ is an advantage estimator measuring the incremental contribution of the attachment ${\pi }_i$ to the overall multicast tree quality.

The agent-specific policy gradient for agent i is given by

$${\nabla }_{{\varphi }^i}F\left(\varphi \right)=\mathbb{E}\left[{\delta }^i·\Delta ({\mathcal{T}}_{i-1},{\pi }_i){\nabla }_{{\varphi }^i}\log {\mu }_{{\varphi }^i}({\pi }_i\mid x,h^i)\right],$$

(11)

where ${\delta }^i$ is an indicator variable that equals 1 if destination $d_i$ is attached at the current step and 0 otherwise. This gating mechanism ensures that only agents actively participating in the current tree expansion are updated.

This selective update mechanism avoids unnecessary parameter updates for inactive destination agents and improves the training efficiency of sequential multicast tree construction.

To improve training stability and maintain consistency between the optimization procedure and the implemented routing framework, GHAC-MR adopts a Proximal Policy Optimization (PPO)-based actor–critic optimization strategy. Unlike off-policy actor–critic methods such as DDPG, the proposed method does not rely on replay buffers, target actor networks, or deterministic Q-learning updates. Instead, multicast routing policies are directly optimized using clipped policy gradient updates based on on-policy sampled multicast routing trajectories.

To enhance training stability, PPO is adopted. Let ${\varphi }_{\mathrm{old}}^i$ denote the historical policy parameters. The probability ratio is defined as

$$\rho \left({\varphi }^i\right)={\displaystyle \frac{{\mu }_{{\varphi }^i}({\pi }_i\mid x,h^i)}{{\mu }_{{\varphi }_{\mathrm{old}}^i}({\pi }_i\mid x,h^i)}}.$$

The clipped surrogate objective is then expressed as

$${\mathcal{L}}^{\mathrm{CLIP}}=\mathbb{E}\left[\min \left(\rho \left({\varphi }^i\right)\Delta ,\mathrm{clip}(\rho \left({\varphi }^i\right),1-ϵ,1+ϵ)\Delta \right)\right],$$

(12)

which constrains policy updates to prevent excessive deviations while preserving beneficial attachment behaviors.

In addition, the reward weighting parameters, including the global reward factor $\lambda$ and the branch reuse factor $\beta$, were determined empirically to balance multicast tree optimization and routing adaptability during policy learning. To maintain scalability in large-scale networks, both the actor and critic operate on localized observations augmented with multicast tree context information, including neighborhood-level metrics such as residual link capacity and local path cost. This decentralized design improves computational efficiency while preserving coordinated multicast tree optimization capability.

3.4. Graph-Structured Policy Network

In multicast routing, forwarding decisions jointly affect the construction and transmission efficiency of the multicast distribution tree, making the routing process highly dependent on global tree evolution rather than a single end-to-end path. To capture the coupling between sequential forwarding decisions and dynamic multicast topology under partial observability, a graph-structured actor–critic architecture is designed for multicast routing. The relationships among the equations of different modules are illustrated in Figure 2.

At each decision step, the network is modeled as a directed graph $\mathcal{G}=(\mathcal{V},\mathcal{E})$, where each node $i\in \mathcal{V}$ represents a forwarding device participating in multicast delivery. Each node observes only locally available information and multicast-related context. The observation of node i is encoded as

$$x_i=\left[s_i,m_i,t_i\right],$$

(13)

where ${\mathbf{s}}_i$ denotes local link states of outgoing interfaces (e.g., residual bandwidth, queueing delay, and packet loss), ${\mathbf{m}}_i$ captures multicast-specific context such as the number of uncovered receivers reachable via node i, and ${\mathbf{t}}_i$ characterizes the current multicast tree status, including whether node i has been included and its current branching degree. This formulation ensures that routing decisions rely solely on local observations without assuming global topology knowledge.

To extract topology-aware representations, we employ a graph-based encoder with structured neighborhood message passing. For each node i, messages from neighboring nodes are computed as

$$m_{ij}=W_m{x}_j,$$

(14)

where $W_m$ is a shared learnable projection. To account for the multicast-specific relevance of each neighbor, an attention score is computed as

$$e_{ij}=a^{\top }\sigma \left(W_a[x_i\parallel x_j\parallel t_j]\right),$$

(15)

followed by normalization

$${\alpha }_{ij}={\displaystyle \frac{\exp \left(e_{ij}\right)}{{\sum }_{k\in \mathcal{N}\left(i\right)}\exp \left(e_{ik}\right)}},$$

(16)

where $W_a$ and a are trainable parameters, $\parallel$ denotes concatenation, and ${\alpha }_{ij}$ reflects the forwarding relevance of neighbor j. By explicitly incorporating the tree-related state $t_j$, the attention mechanism emphasizes neighbors that can potentially reuse existing multicast branches or serve additional receivers.

The node-level latent representation is obtained via weighted aggregation,

$$h_i=\sigma \left(W_h{x}_i+\sum _{j\in \mathcal{N}\left(i\right)}{\alpha }_{ij}{m}_{ij}\right),$$

(17)

where $W_h$ is a learnable self-projection. This representation jointly encodes local link conditions, multicast context, and neighborhood structural information.

Based on the embedding $h_i$, the actor network produces link-wise forwarding decisions. For each outgoing link $e=(i,j)\in {\mathcal{E}}_i$, a forwarding score is computed as

$$z_{i,e}=w^{\top }\left[h_i\parallel h_j\parallel l_{i,e}\right],$$

(18)

where $l_{i,e}$ denotes link-level attributes associated with e, and w is a shared parameter vector. The forwarding action is modeled as an independent Bernoulli decision,

$${\pi }_{\theta }(a_{i,e}=1\mid h_i)=\sigma \left(z_{i,e}\right),$$

(19)

allowing node i to activate multiple outgoing links simultaneously. This formulation explicitly supports multicast branching, enabling local forwarding decisions to collectively form a distribution tree.

To evaluate the long-term impact of decentralized forwarding actions, the critic network estimates the value of the evolving multicast tree. Let ${\mathcal{T}}_t\subseteq \mathcal{V}$ denote the set of nodes included in the multicast tree at time t. A tree-level representation is constructed via structure-preserving aggregation,

$$g_t=\sum _{i\in {\mathcal{T}}_t}{U}_1{h}_i,$$

(20)

where $U_1$ is a learnable projection. To capture higher-order interactions among tree nodes, the aggregated representation is further transformed as

$$u_t=\sigma \left(U_2{g}_t\right),$$

(21)

and the global value estimate is obtained by

$$V_{\omega }\left({\mathcal{T}}_t\right)=U_3{u}_t,$$

(22)

with $U_2$ and $U_3$ denoting trainable parameters. This hierarchical value estimation enables the critic to assess how local branching decisions propagate through the multicast tree, reflecting accumulated delay, loss propagation, and resource utilization across multiple delivery paths.

Through joint optimization of the graph-structured actor and the tree-aware critic, the proposed architecture coordinates decentralized multicast forwarding decisions under partial observability. The actor focuses on topology-aware local branching actions, while the critic provides a structured global learning signal aligned with multicast tree evolution, ensuring that local optimizations collectively improve end-to-end multicast performance. The detailed workflow of this learning and decision-making process is summarized in Algorithm 1, which outlines the initialization of the actor–critic networks, the interaction with the network environment, the generation of multicast forwarding actions based on graph-structured states, and the iterative policy optimization process based on on-policy PPO updates.

During training, multicast routing trajectories generated by the current policy are temporarily stored in an on-policy rollout buffer and cleared immediately after PPO updates, thereby ensuring fully on-policy optimization throughout the routing process. To improve training stability and reduce variance during multicast tree construction, generalized advantage estimation (GAE) is employed to estimate the advantage function:

$$A_t={\displaystyle \sum _{l=0}^{\infty }}{\left(\gamma {\lambda }_{\mathrm{GAE}}\right)}^l{\delta }_{t+l},$$

(23)

where the temporal-difference residual is defined as

$${\delta }_t=r_t+\gamma V_{\omega }\left(x_{t+1}\right)-V_{\omega }\left(x_t\right).$$

(24)

The actor network is optimized using the PPO clipped surrogate objective

$${\mathcal{L}}^{\mathrm{CLIP}}=\mathbb{E}\left[\min \left({\rho }_t{A}_t,\mathrm{clip}({\rho }_t,1-ϵ,1+ϵ)A_t\right)\right],$$

(25)

where ${\rho }_t$ denotes the probability ratio between the current policy and the old policy. Meanwhile, the critic network is updated by minimizing the mean squared error between predicted state values and empirical returns. Multiple PPO epochs are performed for each rollout trajectory segment to improve policy convergence and routing stability under dynamic multicast environments.

Entropy regularization is not incorporated in the current implementation in order to maintain stable multicast tree evolution during sequential routing optimization. The actor and critic networks are jointly optimized using the Adam optimizer with a learning rate of $3\times {10}^{-4}$, a clipping parameter of $0.2$, and a mini-batch size of 256, while the discount factor is set to $0.99$ to preserve long-term multicast tree rewards during sequential routing optimization.

Algorithm 1 GHAC-MR: Graph-Aware PPO-Based Actor–Critic for Dynamic Multicast Routing.

Input: Network graph $G(V,E)$, source node s, destination set D, learning rate $\alpha$, discount factor $\gamma$, clipping parameter $ϵ$, rollout length H, PPO epochs K, training episodes M. Output: Optimal multicast tree $\mathcal{T}$ connecting $(s,D)$.   1: Initialize actor parameters $\varphi$   2: Initialize critic parameters $\omega$   3: Initialize old policy parameters ${\varphi }_{\mathrm{old}}\leftarrow \varphi$   4: Initialize on-policy rollout buffer $\mathcal{D}$   5: for episode $=1$ to M do   6:      Reset environment with multicast request $(s,D)$   7:      Initialize multicast tree $\mathcal{T}=\left\{s\right\}$   8:      Observe initial network state $x_0$   9:      while not all destinations in D are connected do 10:        Extract graph-structured state representation $x_t$ via GNN message passing 11:        Sample forwarding action: 12:           $a_t\sim {\mu }_{\varphi }(a_t\mid x_t,{\mathcal{T}}_{t-1})$ 13:        Execute action $a_t$ and expand multicast tree $\mathcal{T}$ 14:        Observe next state $x_{t+1}$ 15:        Compute reward: 16:           $r_t=r_t^{\mathrm{local}}+\lambda r_t^{\mathrm{global}}$ 17:        Store transition $(x_t,a_t,r_t,x_{t+1},{\mathcal{T}}_{t-1})$ into rollout buffer $\mathcal{D}$ 18:        if rollout buffer is full or multicast session terminates then 19:            Estimate advantages $A_t$ using GAE 20:            for PPO epoch $=1$ to K do 21:                Compute probability ratio: 22:                    ${\rho }_t\left(\varphi \right)={\displaystyle \frac{{\mu }_{\varphi }(a_t\mid x_t,{\mathcal{T}}_{t-1})}{{\mu }_{{\varphi }_{\mathrm{old}}}(a_t\mid x_t,{\mathcal{T}}_{t-1})}}$ 23:                Update actor $\varphi$ using clipped PPO objective ${\mathcal{L}}^{\mathrm{CLIP}}$ 24:                Update critic $\omega$ by minimizing value loss 25:            end for 26:            Synchronize old policy: 27:               ${\varphi }_{\mathrm{old}}\leftarrow \varphi$ 28:            Clear rollout buffer $\mathcal{D}$ 29:        end if 30:      end while 31:      Output multicast tree $\mathcal{T}$ 32: end for

4. Experimental Setup and Results

4.1. Experimental Environment and Parameter Settings

The experiments were conducted on a local computing platform running a 64-bit Ubuntu 20.04 operating system. The hardware environment was equipped with an Intel Core i7-10710U processor, 16 GB of RAM, and a solid-state drive for data storage. Graphics acceleration was provided by an NVIDIA GeForce MX350 GPU. An SDN-based network environment was emulated using Mininet, while the control logic was implemented on an SDN controller based on the Ryu 4.34 framework. All experiments were executed under the same hardware and software configuration to ensure fairness and consistency in performance evaluation.

To comprehensively evaluate the performance of the proposed multicast routing algorithm, comparative experiments were conducted on three representative network topologies with different structural characteristics, namely the Fat-Tree topology, the Node14Net topology, and the ARPA-Net topology.

The Fat-Tree topology [24] represents a hierarchical network structure commonly used to model multi-rooted routing scenarios. It consists of multiple layers of switches with redundant interconnections, which makes it suitable for evaluating multicast routing behaviors involving path sharing and branching decisions. As illustrated in Figure 3, this topology provides a structured environment for analyzing multicast routing strategies under hierarchical network conditions.

The Node14Net topology [25] is a medium-scale network with an irregular connection pattern. Compared with the structured Fat-Tree topology, it exhibits more heterogeneous link connectivity, providing a complementary test environment for assessing the robustness and adaptability of the proposed algorithm under diverse network conditions. As illustrated in Figure 4, this topology reflects more realistic network connectivity characteristics with uneven link distributions.

The ARPA-Net topology represents a classical backbone network structure that has been widely adopted in networking research for evaluating routing and traffic engineering algorithms [24,26]. It consists of multiple interconnected nodes with relatively sparse yet strategically distributed links, forming a moderately complex network environment. Compared with hierarchical topologies such as Fat-Tree, ARPA-Net exhibits a more irregular connectivity pattern and longer path diversity among nodes. As illustrated in Figure 5, this topology provides a representative testbed for analyzing multicast routing performance under realistic wide-area network conditions.

In the Fat-Tree topology scenario, the source node of the multicast tree is node S10, and the destination nodes are nodes S1, S6, and S8. In the Node14Net topology scenario, the source node of the multicast tree is node S8, and the destination nodes are nodes S2, S3, and S14. In the ARPA-Net topology scenario, the source node of the multicast tree is node S3, and the destination nodes are nodes S5, S12, and S15.

For each topology, multicast sessions were generated by randomly selecting one source node and multiple destination nodes from the network. The destination nodes were uniformly sampled from the set of all switches, excluding the source node. To control the multicast group size while maintaining sufficient branching complexity, the number of destination nodes was set to range from 15% to 30% of the total nodes in the topology. This configuration ensures that the multicast tree involves multiple branches and shared forwarding paths, while avoiding degenerate cases with overly small or excessively dense receiver sets. Such a randomized and uniformly distributed destination selection strategy allows the evaluation to cover diverse multicast patterns and reduces bias toward specific node placements.

Traffic loads were generated using Iperf3 to emulate time-varying network conditions over a 24-h period. The average traffic rate of each node evolves smoothly with time, exhibiting clear daily fluctuations rather than remaining constant. Specifically, the per-node traffic volume ranges approximately from 9000 Mbps during low-load periods to around 18,000 Mbps at peak hours. The traffic gradually increases from off-peak intervals, reaches its maximum during high-activity periods, and then decreases before stabilizing again, forming a non-uniform yet continuous temporal pattern. Such traffic dynamics are consistent with typical diurnal load variations in operational networks and avoid unrealistic abrupt changes. During the simulation, network state information was sampled at regular intervals. For each topology, traffic and network metrics were collected at 100 distinct time instants and used for performance evaluation. These measurements were then grouped into 10 segments, each containing 10 consecutive time instants, to compute the average bottleneck bandwidth, delay, and packet loss rate for each segment. Consequently, a total of 10 group-wise averaged results were obtained for each topology.

The actor network computes link-wise forwarding scores using a shared linear scoring function over concatenated node and link embeddings, and outputs independent Bernoulli actions for each outgoing link. This design is motivated by the multicast tree construction process described in the system model, where a forwarding node may simultaneously connect multiple downstream receivers through different outgoing links. Compared with single-path action selection, Bernoulli-based forwarding decisions enable the routing policy to naturally model multicast branch expansion and shared forwarding behaviors under dynamic network conditions.

The graph-structured state representation consists of node connectivity, residual link bandwidth, transmission delay, packet loss rate, and multicast tree membership information. These state features directly correspond to the dynamic link attributes and multicast tree evolution characteristics defined in Section 3. To preserve structural correlations among multicast forwarding nodes, graph embeddings with dimension 128 are adopted for both node and link representations. This embedding size provides sufficient expressive capability for topology-aware routing decisions while maintaining computational efficiency during training.

The critic network aggregates node embeddings belonging to the current multicast tree via summation operations, followed by two fully connected layers with hidden dimensions of 256 and 128, respectively, to estimate the long-term utility of the multicast tree. This aggregation mechanism is designed in accordance with the multicast tree evaluation model presented in Section 3, where routing quality depends on the cumulative contribution of distributed forwarding decisions and multicast branch reuse behaviors. ReLU activation functions are employed in both the actor and critic networks to enhance nonlinear representation capability and stabilize gradient propagation during training.

To validate the robustness of the proposed GHAC-MR algorithm and verify the rationality of hyperparameter selection, we conduct comprehensive sensitivity analysis on five key hyperparameters: branch reuse factor, discount factor, global reward weight, learning rate, and PPO clip parameter. The convergence performance under different hyperparameter settings is illustrated in Figure 6, Figure 7, Figure 8, Figure 9 and Figure 10. The results demonstrate that the selected optimal hyperparameter configuration achieves the highest stable average reward and the fastest convergence rate, while inappropriate settings lead to lower final performance, slower convergence, and more severe training fluctuations.

The discount factor is set to 0.99 because multicast routing in the proposed framework is formulated as a sequential multicast tree construction process. Each attachment action may influence subsequent routing decisions, multicast branch structures, and overall tree-level resource utilization. Therefore, a relatively large discount factor is adopted to preserve long-term routing rewards and encourage globally efficient multicast tree evolution rather than short-term local forwarding optimization.

The clipping parameter is set to 0.2 to constrain abrupt policy updates during multicast routing optimization. Since the multicast tree evolves incrementally under dynamic network states, excessively large policy variations may lead to unstable multicast tree construction behaviors and inconsistent forwarding decisions. The selected clipping range was empirically observed to provide stable convergence while preserving sufficient policy exploration capability across different network topologies.

The actor and critic networks are jointly optimized using the Adam optimizer with a learning rate of $3\times {10}^{-4}$. This learning rate was selected to balance convergence speed and optimization stability for high-dimensional graph-structured routing states. A mini-batch size of 256 is adopted to improve gradient estimation stability while maintaining acceptable computational overhead during policy training. In addition, the training process employs 2000 routing episodes for each topology to ensure sufficient exploration of multicast routing behaviors under different dynamic traffic conditions.

The reward weighting parameters introduced in the multicast tree evaluation function are selected according to the relative importance of multicast tree cost minimization and multicast branch reuse optimization. Specifically, the branching reuse weighting parameter $\beta$ is set to 0.3 to moderately encourage shared multicast forwarding structures without excessively constraining routing flexibility. The global reward weighting parameter $\lambda$ is set to 0.5 to balance local forwarding efficiency and global multicast tree quality during sequential routing optimization. Empirically, these parameter settings were observed to provide stable multicast tree evolution and consistent routing performance across Fat-Tree, Node14Net, and ARPA-Net topologies.

All reported results represent the mean and standard deviation obtained from 10 independent experimental runs under different random initializations and multicast traffic distributions. In each run, multicast source–destination groups and traffic arrival patterns were randomly regenerated while maintaining identical network topologies and simulation configurations. The reported variances therefore reflect repeated full-experiment fluctuations under dynamic multicast routing scenarios rather than per-episode variations. To further evaluate the reliability of the observed performance trends, additional statistical comparisons were conducted across repeated experiments for major routing metrics, including delay, bandwidth utilization, and packet loss. The results consistently demonstrated stable performance improvements of GHAC-MR under different randomized multicast scenarios. Therefore, the comparative analysis primarily emphasizes the consistency, robustness, and repeatability of the observed performance improvements rather than claiming strict statistical superiority solely based on mean differences.

Overall, the above parameter configurations are closely correlated with the system model and multicast routing formulation presented in Section 3. The Bernoulli forwarding mechanism corresponds to multicast branching behaviors, the graph embeddings reflect topology-aware network states, the large discount factor captures sequential multicast tree dependency, and the reward weighting parameters align with the joint optimization objective of local forwarding efficiency and global multicast tree quality. These parameter settings collectively support stable and adaptive multicast routing optimization under dynamic network environments.

4.2. Performance Analysis

To comprehensively evaluate the performance of the proposed GHAC-MR algorithm, comparative experiments are conducted against four representative multicast routing approaches that are widely adopted or recently developed in SDN-based multicast studies. These comparison methods span classical graph-theoretic heuristics, evolutionary optimization techniques, and learning-driven multicast routing schemes, thereby providing a broad and balanced evaluation baseline.

Specifically, the KMB [27] algorithm is selected as a classical near-optimal multicast routing method, which constructs multicast trees by iteratively connecting destination nodes through minimum-cost paths derived from a spanning structure, and is commonly used as a benchmark for cost-aware multicast routing. In addition, a genetic algorithm-based multicast routing approach is adopted as a representative evolutionary method, where multicast trees are optimized through population-based search with specialized path encoding mechanisms, enabling effective exploration of the global solution space under complex routing constraints. Following this, DRL-M4MR [28] is included as a deep reinforcement learning-based multicast routing method, which formulates multicast routing as a sequential decision-making problem and leverages a deep Q-network to learn forwarding policies based on network state observations. Furthermore, a recently proposed deep hierarchical reinforcement learning-based multicast routing approach [26] is considered as a learning-driven baseline, which employs multi-level decision structures to coordinate multicast forwarding behaviors and reflects recent advances in reinforcement learning-based multicast optimization.

By comparing GHAC-MR with these heterogeneous routing approaches, the performance analysis systematically highlights the relative strengths and limitations of different multicast routing paradigms, and demonstrates the effectiveness of the proposed graph-structured hierarchical actor–critic framework in addressing multicast routing challenges under dynamic network conditions.

4.3. Performance Evaluation Under Fat-Tree Topology

As shown in Figure 11, Figure 12 and Figure 13, the average end-to-end delay of different multicast routing algorithms under the Fat-Tree topology exhibits distinct variation trends as the traffic load increases. For the KMB algorithm, the delay rises rapidly with increasing traffic and reaches a pronounced peak in the medium-load region, indicating that its tree construction based on minimum-cost paths becomes increasingly congested when multiple multicast flows share common links. The GA-based method exhibits a smoother delay curve than KMB, but still shows a clear upward trend under moderate traffic due to its reliance on static evolutionary optimization, which lacks timely adaptation to instantaneous network conditions. Both DRL-M4MR and Deep Hierarchical Reinforcement Learning (DHRL) demonstrate improved delay control compared with traditional heuristics, benefiting from learning-based decision mechanisms; however, noticeable delay fluctuations are observed around medium traffic levels, suggesting limited coordination among forwarding decisions. In contrast, GHAC-MR consistently achieves the lowest delay across almost all traffic levels, with a notably flatter curve under medium and high loads. This indicates that the proposed graph-structured hierarchical actor–critic framework effectively captures multicast tree dependencies and dynamically balances forwarding decisions, thereby alleviating congestion and reducing end-to-end latency.

The bandwidth utilization trends reveal clear performance differences among the compared multicast routing algorithms as traffic intensity increases. KMB exhibits relatively low and unstable bandwidth utilization, particularly under high load, due to its tendency to construct suboptimal trees with redundant link usage. The GA method achieves higher bandwidth utilization than KMB in several traffic regions, but its performance fluctuates noticeably, reflecting the limited responsiveness of offline evolutionary optimization to dynamic traffic variations. DRL-M4MR and DHRL exhibit more stable bandwidth curves, indicating that learning-based approaches can better exploit available link capacity through adaptive routing decisions. Among all methods, GHAC-MR consistently maintains the highest average bandwidth utilization across most traffic levels. This improvement stems from its ability to jointly consider local forwarding actions and global tree-level rewards, enabling efficient sharing of links and balanced traffic distribution throughout the multicast tree.

The packet loss behavior further highlights the robustness differences among the evaluated multicast routing algorithms under increasing traffic load. For KMB, packet loss rises sharply under medium traffic, highlighting its vulnerability to congestion caused by overlapping paths in the multicast tree. The GA-based approach reduces packet loss compared with KMB, but still experiences noticeable spikes under moderate loads, indicating insufficient robustness to traffic bursts. DRL-M4MR exhibits improved loss performance at low and high traffic levels, yet suffers from intermittent loss peaks, suggesting that its learning process may not fully account for tree-wide congestion propagation. DHRL further mitigates packet loss but still shows variability in certain traffic regions. In contrast, GHAC-MR achieves the lowest and most stable packet loss across all traffic levels, with loss ratios remaining close to zero under high load. This demonstrates that GHAC-MR effectively coordinates multicast forwarding decisions by leveraging graph-structured state representations and hierarchical learning, thereby reducing congestion-induced packet drops and enhancing overall transmission reliability.

To further provide a quantitative comparison of the overall performance,

Table 2. Performance comparison of different algorithms across various topologies (mean ± standard deviation over 10 independent experiments).

Topology	Indicator	GHAC-MR	DHRL	DRL-M4MR	GA	KMB
Fat-Tree	Average Delay	$24.12\pm 1.12$	$31.47\pm 2.36$	$28.36\pm 1.95$	$33.18\pm 2.74$	$36.52\pm 3.12$
	Average Bandwidth	$14.63\pm 0.82$	$16.74\pm 1.21$	$15.28\pm 1.03$	$12.94\pm 1.35$	$11.86\pm 1.42$
	Average Loss	$0.00102\pm 0.00008$	$0.00143\pm 0.00017$	$0.00118\pm 0.00012$	$0.00127\pm 0.00015$	$0.00151\pm 0.00019$
Node14Net	Average Delay	$16.84\pm 0.94$	$17.63\pm 1.21$	$17.05\pm 1.08$	$19.42\pm 1.65$	$21.78\pm 1.98$
	Average Bandwidth	$13.37\pm 0.73$	$11.84\pm 0.96$	$11.12\pm 0.88$	$9.76\pm 1.05$	$9.18\pm 1.12$
	Average Loss	$0.00048\pm 0.00004$	$0.00063\pm 0.00007$	$0.00057\pm 0.00006$	$0.00072\pm 0.00009$	$0.00069\pm 0.00008$
ARPA-Net	Average Delay	$10.37\pm 0.62$	$11.64\pm 0.88$	$11.08\pm 0.79$	$11.72\pm 0.93$	$12.34\pm 1.05$
	Average Bandwidth	$15.12\pm 0.91$	$13.24\pm 1.12$	$12.91\pm 1.05$	$11.76\pm 1.18$	$12.03\pm 1.21$
	Average Loss	$0.00009\pm 0.00001$	$0.00011\pm 0.00002$	$0.00012\pm 0.00002$	$0.00016\pm 0.00003$	$0.00025\pm 0.00004$

summarizes the average delay, bandwidth utilization, and packet loss achieved by each multicast routing algorithm under the Fat-Tree topology. It can be observed that GHAC-MR obtains the lowest average delay and packet loss while maintaining the highest bandwidth utilization among all compared methods. Specifically, the average delay achieved by GHAC-MR is consistently lower than that of KMB, GA, DRL-M4MR, and DHRL, indicating its stronger capability in mitigating congestion and reducing end-to-end transmission latency. Meanwhile, GHAC-MR also achieves the highest bandwidth utilization, demonstrating its effectiveness in exploiting available network resources and distributing multicast traffic more evenly across the network. In addition, the packet loss ratio of GHAC-MR remains the lowest among all algorithms, which further verifies the robustness of the proposed graph-structured hierarchical actor–critic framework under varying traffic conditions.

4.4. Performance Evaluation Under Node14Net Topology

As shown in Figure 14, Figure 15 and Figure 16, the average end-to-end delay of different multicast routing algorithms under the Node14Net topology exhibits noticeable variation patterns as the traffic load increases. The KMB algorithm shows the highest delay in most traffic conditions, with a pronounced peak around the medium-load region, indicating that its Steiner tree-based construction tends to concentrate traffic on shared links and thus easily causes congestion. The GA-based approach performs better than KMB but still shows delay fluctuations under varying loads due to the limited adaptability of evolutionary search to dynamic network conditions.

Compared with traditional heuristic algorithms, the reinforcement learning-based approaches DRL-M4MR and DHRL achieve lower delay across most traffic levels, demonstrating the advantage of adaptive routing decisions. However, both methods still exhibit noticeable oscillations in the medium-load region, suggesting limited coordination among multicast forwarding paths. In contrast, GHAC-MR consistently maintains the lowest or near-lowest delay, especially under moderate and high traffic conditions. Its delay curve remains relatively smooth, indicating that the proposed graph-structured hierarchical actor–critic framework effectively captures multicast tree dependencies and dynamically distributes traffic across available paths, thereby alleviating congestion and reducing end-to-end latency.

The bandwidth utilization trends under the Node14Net topology further highlight the differences among the compared algorithms. KMB achieves the lowest bandwidth utilization and exhibits considerable fluctuations due to its static minimum-cost tree construction, which may lead to inefficient link usage. The GA method shows moderate improvement but still varies noticeably under dynamic traffic conditions. In contrast, the learning-based approaches DRL-M4MR and DHRL achieve more stable bandwidth utilization, indicating that adaptive learning helps exploit available network resources more effectively. Among all methods, GHAC-MR consistently achieves the highest bandwidth utilization, particularly under medium traffic loads where resource competition becomes increasingly pronounced. This improvement stems from its hierarchical actor–critic design and graph-based state representation, which jointly optimize local forwarding actions and global multicast tree efficiency.

The packet loss behavior further reflects the robustness differences among the evaluated algorithms as traffic load increases. KMB exhibits relatively higher packet loss under moderate traffic conditions due to congestion caused by overlapping multicast paths. Although the GA-based approach reduces packet loss compared with KMB, it still exhibits noticeable fluctuations under dynamic traffic conditions. The learning-based methods DRL-M4MR and DHRL further mitigate packet loss but still experience moderate loss peaks around the medium-load region, suggesting incomplete coordination among routing decisions. In contrast, GHAC-MR maintains the lowest and most stable packet loss across nearly all traffic levels, demonstrating its ability to effectively coordinate multicast forwarding through graph-structured state representations and hierarchical learning.

Table 2 summarizes the average delay, bandwidth utilization, and packet loss achieved by each algorithm under the Node14Net topology. It can be observed that GHAC-MR achieves the lowest delay and packet loss while maintaining the highest bandwidth utilization among all compared methods.

4.5. Performance Evaluation Under ARPA-Net Topology

As shown in Figure 17, Figure 18 and Figure 19, the average end-to-end delay of different multicast routing algorithms under the ARPA-Net topology shows noticeable variation trends as the traffic load increases. Compared with the previous topologies, ARPA-Net contains a larger number of nodes and more complex interconnections, which increases the difficulty of multicast path coordination and resource scheduling. The KMB algorithm generally produces the highest delay under most traffic conditions, particularly around the medium-load region, where multiple multicast flows tend to share limited links, leading to increased congestion and longer transmission paths. The GA-based method achieves lower delay than KMB in several traffic scenarios, but its delay curve still exhibits noticeable fluctuations due to the limited adaptability of evolutionary search to rapidly changing network conditions.

In contrast, the reinforcement learning-based approaches DRL-M4MR and DHRL demonstrate improved delay performance, benefiting from their ability to adapt routing decisions according to dynamic network states. Nevertheless, both methods still show certain delay oscillations when traffic becomes relatively dense, indicating that the coordination among multicast branches is not fully optimized in large-scale topologies. The proposed GHAC-MR algorithm consistently achieves lower delay across most traffic levels and maintains a relatively stable delay curve. This behavior indicates that the graph-based hierarchical actor–critic framework can effectively capture the structural characteristics of large-scale networks and coordinate multicast forwarding decisions more efficiently.

The bandwidth utilization results under the ARPA-Net topology further reveal the performance differences among the compared algorithms. Due to its static Steiner tree-based construction strategy, KMB often fails to fully utilize available network resources, resulting in relatively low bandwidth utilization in most traffic scenarios. The GA algorithm provides moderate improvement but still shows considerable fluctuations as traffic varies. The learning-based methods DRL-M4MR and DHRL demonstrate more stable bandwidth utilization because they can dynamically adjust routing paths based on network feedback. Among all the algorithms, GHAC-MR consistently achieves the highest bandwidth utilization, especially when the traffic load increases. This improvement can be attributed to its graph-structured state representation and hierarchical learning mechanism, which jointly optimize local branching decisions and global multicast tree efficiency in larger network environments.

The packet loss trends further demonstrate the robustness of different algorithms under increasing traffic demand. In the ARPA-Net topology, the KMB algorithm exhibits relatively higher packet loss under medium traffic conditions, mainly due to congestion caused by overlapping multicast paths. Although the GA algorithm reduces packet loss compared with KMB, it still experiences noticeable fluctuations under dynamic traffic conditions. The reinforcement learning-based methods DRL-M4MR and DHRL further mitigate packet loss, but moderate variations can still be observed in certain traffic scenarios. In comparison, GHAC-MR maintains the lowest and most stable packet loss across nearly all traffic levels. This result indicates that the proposed method can better coordinate multicast forwarding decisions and distribute traffic more evenly across the large-scale network.

Table 2 summarizes the average delay, bandwidth utilization, and packet loss of the compared algorithms under the ARPA-Net topology. It can be observed that GHAC-MR achieves the lowest delay and packet loss while simultaneously obtaining the highest bandwidth utilization.

4.6. Statistical Performance Analysis and Model Validation

The statistical analysis results in

Table 3. Statistical significance analysis of GHAC-MR against representative reinforcement learning-based baseline methods.

Topology	Metric	Compared Method	t-Value	p-Value
Fat-Tree	Average Delay	DHRL	$-7.732$	$2.9\times {10}^{-5}$
	Average Delay	DRL-M4MR	$-5.234$	$5.4\times {10}^{-4}$
	Average Bandwidth	DHRL	$-3.803$	$4.2\times {10}^{-3}$
	Average Bandwidth	DRL-M4MR	$-2.771$	$2.2\times {10}^{-2}$
	Average Packet Loss	DHRL	$-5.001$	$7.4\times {10}^{-4}$
	Average Packet Loss	DRL-M4MR	$-4.054$	$2.9\times {10}^{-3}$
Node14Net	Average Delay	DHRL	$1.552$	$1.55\times {10}^{-1}$
	Average Delay	DRL-M4MR	$1.122$	$2.91\times {10}^{-1}$
	Average Bandwidth	DHRL	$3.412$	$7.7\times {10}^{-3}$
	Average Bandwidth	DRL-M4MR	$5.265$	$5.2\times {10}^{-4}$
	Average Packet Loss	DHRL	$-4.312$	$2.0\times {10}^{-3}$
	Average Packet Loss	DRL-M4MR	$-4.522$	$1.4\times {10}^{-3}$
ARPA-Net	Average Delay	DHRL	$-4.822$	$9.5\times {10}^{-4}$
	Average Delay	DRL-M4MR	$-2.582$	$3.0\times {10}^{-2}$
	Average Bandwidth	DHRL	$4.993$	$7.5\times {10}^{-4}$
	Average Bandwidth	DRL-M4MR	$5.079$	$6.6\times {10}^{-4}$
	Average Packet Loss	DHRL	$-3.858$	$3.9\times {10}^{-3}$
	Average Packet Loss	DRL-M4MR	$-5.118$	$6.3\times {10}^{-4}$

demonstrate that GHAC-MR achieves statistically significant improvements over the representative reinforcement learning-based baseline methods in most experimental scenarios. In the Fat-Tree and ARPA-Net topologies, the proposed method shows significant advantages in average delay, bandwidth utilization, and packet loss, with most p-values below 0.01. For the Node14Net topology, statistically significant improvements are still observed in bandwidth utilization and packet loss. However, the delay improvements over DHRL and DRL-M4MR do not reach statistical significance, which indicates that the compared methods exhibit relatively similar delay performance under small-scale network conditions.

To further evaluate the effectiveness of the proposed GHAC-MR framework, an ablation study is conducted under the Fat-Tree, Node14Net, and ARPA-Net topologies. The ablation results are summarized in

Table 4. Ablation study of GHAC-MR under different topologies (mean ± standard deviation over 10 independent experiments).

Topology	Indicator	GHAC-MR	w/o Graph	w/o Global	w/o Actor–Critic
Fat-Tree	Average Delay	$24.12\pm 1.12$	$28.47\pm 1.85$	$26.93\pm 1.54$	$27.85\pm 1.67$
	Average Bandwidth	$14.63\pm 0.82$	$13.12\pm 0.96$	$13.88\pm 0.89$	$13.45\pm 0.92$
	Average Loss	$0.00102\pm 0.00008$	$0.00131\pm 0.00014$	$0.00118\pm 0.00011$	$0.00122\pm 0.00012$
Node14Net	Average Delay	$16.84\pm 0.94$	$18.92\pm 1.32$	$17.96\pm 1.08$	$18.37\pm 1.15$
	Average Bandwidth	$13.37\pm 0.73$	$12.11\pm 0.88$	$12.76\pm 0.81$	$12.43\pm 0.85$
	Average Loss	$0.00048\pm 0.00004$	$0.00061\pm 0.00007$	$0.00055\pm 0.00006$	$0.00057\pm 0.00006$
ARPA-Net	Average Delay	$10.37\pm 0.62$	$11.68\pm 0.84$	$11.02\pm 0.73$	$11.29\pm 0.78$
	Average Bandwidth	$15.12\pm 0.91$	$13.86\pm 1.02$	$14.32\pm 0.96$	$14.05\pm 0.99$
	Average Loss	$0.00009\pm 0.00001$	$0.00013\pm 0.00002$	$0.00011\pm 0.00002$	$0.00012\pm 0.00002$

. Specifically, several simplified variants of the proposed framework are constructed by removing different functional components, including the graph-structured topology representation mechanism, the global tree-level reward design, and the actor–critic optimization architecture. Through these comparisons, the contribution of each component to multicast routing performance can be further analyzed.

From Table 4, it can be observed that the complete GHAC-MR framework consistently achieves the best overall performance across different network topologies and evaluation indicators. When the graph-structured representation mechanism is removed, the routing performance degrades noticeably in terms of delay, bandwidth utilization, and packet loss. This result indicates that topology-aware structural modeling plays an important role in capturing multicast branch dependencies and network state correlations during multicast tree construction. Similarly, removing the global tree-level reward weakens the coordination between local forwarding decisions and overall multicast tree optimization, resulting in reduced routing efficiency and less stable multicast forwarding behaviors. In addition, removing the actor–critic optimization mechanism also leads to observable performance degradation, demonstrating the importance of coordinated policy learning and value estimation for sequential multicast routing optimization.

Furthermore, the computational complexity of the proposed GHAC-MR framework is analyzed to evaluate its scalability under different network sizes. The overall computational overhead mainly originates from three components: graph-structured state encoding, multicast forwarding policy inference, and tree-aware value evaluation during PPO-based optimization.

For graph representation learning, the proposed framework employs GNN-based message passing to aggregate topology-aware node and link features. Assuming that each message-passing layer processes all network links once, the computational complexity of graph embedding generation can be approximated as

$$\mathcal{O}\left(L_g\right|E\left|d\right),$$

where $L_g$ denotes the number of GNN propagation layers, $\left|E\right|$ represents the number of network links, and d is the hidden feature dimension.

During multicast routing decision generation, the actor network evaluates candidate forwarding actions according to the current multicast tree state. Assuming that the average number of candidate forwarding neighbors is bounded by node degree, the action inference complexity can be expressed as

$$\mathcal{O}\left(\right|V|d+|A\left|\right),$$

where $\left|V\right|$ is the number of network nodes and $\left|A\right|$ denotes the candidate action space size.

For the tree-aware critic network, multicast tree-level state evaluation mainly depends on graph embedding aggregation and value estimation, resulting in a complexity of approximately

$$\mathcal{O}\left(\right|V\left|d\right).$$

Therefore, the overall per-step computational complexity of the proposed GHAC-MR framework can be summarized as

$$\mathcal{O}\left(L_g\right|E|d+|V|d+|A\left|\right).$$

Since the hidden feature dimension d and the number of GNN layers $L_g$ remain relatively small and fixed during training and inference, the overall computational overhead increases approximately linearly with network topology size. This indicates that the proposed framework remains computationally feasible and scalable for dynamic multicast routing optimization in SDN-enabled next-generation wireless network environments.

Figure 20 compares the training time overhead of different multicast routing algorithms under the Node14Net topology. The horizontal axis represents the index of Network Link Status Information batches. Each batch contains network state observations sampled over consecutive time intervals, including link bandwidth, delay and packet loss rate. As the index increases, the agent conducts sequential training under dynamic network conditions. The proposed GHAC-MR algorithm maintains the lowest training overhead across all batches and shows a near-linear growth trend, which is consistent with the computational complexity analysis in Section 4.6. DHRL and DRL-M4MR achieve moderate performance but still incur higher overhead than GHAC-MR. These results verify the good scalability of the proposed graph-structured hierarchical actor–critic framework in the Node14Net topology.

5. Conclusions

This paper investigated the problem of dynamic multicast routing in SDN-enabled wireless networks and proposed a learning-driven routing framework named GHAC-MR. By modeling the multicast routing process as a sequential decision-making problem, the proposed method integrates graph-structured network representations with a hierarchical actor–critic architecture to construct multicast distribution trees adaptively. The reward design explicitly captures both local forwarding decisions and global tree-level performance, enabling the learning process to account for multicast-specific characteristics such as path sharing and branching structures. Extensive simulation results demonstrate that the proposed approach consistently achieves better resource efficiency and adaptability compared with representative heuristic and learning-based routing methods. These results demonstrate that the proposed framework can effectively support adaptive multicast routing under dynamic network conditions through topology-aware learning and sequential routing optimization. Meanwhile, the impact of network state synchronization delay and control overhead in SDN environments deserves further investigation for large-scale and highly dynamic multicast scenarios. In future work, we plan to further explore scalable multi-agent learning mechanisms and incorporate additional QoS constraints to enhance the robustness and applicability of learning-based multicast routing in large-scale and highly dynamic network environments.