A3C-R: A QoS-Oriented Energy-Saving Routing Algorithm for Software-Defined Networks

Abstract

With the rapid growth of Internet applications and network traffic, existing routing algorithms are usually difficult to guarantee the quality of service (QoS) indicators such as delay, bandwidth, and packet loss rate as well as network energy consumption for various data flows with business characteristics. They have problems such as unbalanced traffic scheduling and unreasonable network resource allocation. Aiming at the above problems, this paper proposes a QoS-oriented energy-saving routing algorithm A3C-R in the software-defined network (SDN) environment. Based on the asynchronous update advantages of the asynchronous advantage Actor-Critic (A3C) algorithm and the advantages of independent interaction between multiple agents and the environment, the A3C-R algorithm can effectively improve the convergence of the routing algorithm. The process of the A3C-R algorithm first takes QoS indicators such as delay, bandwidth, and packet loss rate and the network energy consumption of the link as input. Then, it creates multiple agents to start asynchronous training, through the continuous updating of Actors and Critics in each agent and periodically synchronizes the model parameters to the global model. After the algorithm training converges, it can output the link weights of the network topology to facilitate the calculation of intelligent routing strategies that meet QoS requirements and lower network energy consumption. The experimental results indicate that the A3C-R algorithm, compared to the baseline algorithms ECMP, I-DQN, and DDPG-EEFS, reduces delay by approximately 9.4%, increases throughput by approximately 7.0%, decreases the packet loss rate by approximately 9.5%, and improves energy-saving percentage by approximately 10.8%.

1. Introduction

In recent years, with the explosive growth of network traffic of various new network applications and services, data flows with a large number of business characteristics have been generated. For example, online teaching and video conferencing have high requirements on bandwidth, and online games and instant messaging have high requirements on delay. Therefore, in order to ensure the real-time nature of network service interaction, network services in different scenarios put forward higher requirements for quality of service (QoS) such as network bandwidth, delay, packet loss rate, and throughput [1]. At the same time, with the continuous popularization of data-intensive applications and the continuous growth of the number of network devices, the proportion of network infrastructure energy consumption is gradually increasing, and energy-saving flow scheduling has become a common trend [2]. In the face of huge network QoS and energy-saving demands, the disadvantages of the traditional “best effort” network architecture and design concept are gradually exposed, such as difficulty in controlling the network scale, implementing network management, guaranteeing network services, and updating network technologies [3]. Therefore, how to achieve more efficient QoS differentiated services and rational allocation of network energy resources under the existing available network resources is of great significance.

In order to improve the openness and innovation of the traditional network architecture and promote the optimal selection of traffic scheduling strategies, software defined network (SDN) [4] technology has been well developed. SDN decouples the control plane and data forwarding plane of network equipment, which can make the utilization of network resources more fully and effectively [5]. Compared with traditional networks, SDN simplifies the complexity of network equipment, reduces network management configuration and operation requirements, effectively improves the flexibility of network control and management, and enhances the rapid support capabilities of new network technologies, new requirements, and new protocols. SDN-oriented routing planning makes the path selection flexible and changeable, which can realize the rational utilization of network resources, effectively avoid network congestion, and better improve the overall network service quality [6].

With the rise in artificial intelligence, machine learning technology has developed rapidly, and significant progress has been made in data processing and combinatorial optimization. Utilizing the convenience provided by SDN, SDN-oriented intelligent routing optimization and network resource scheduling can be realized, making network operation, maintenance, and management continue to develop in the direction of intelligence. Based on this, this paper proposes a QoS-oriented energy-saving routing optimization algorithm A3C-R in the SDN environment. A3C-R divides different types of service traffic, uses delay, bandwidth, and packet loss rate as QoS measurement indicators, and considers the energy consumption of the network. Based on the advantage of multi-threaded asynchronous update of the A3C algorithm [7], multiple agents can be executed simultaneously. It can reduce the correlation between samples in the training process through a variety of different state samples, thereby reducing the computing power demand in the process of agent and environment exploration, so as to choose the energy-saving path that meets the QoS requirements as much as possible. The main contributions of this paper are as follows:

• We investigated the research value and significance of SDN-oriented QoS and energy-saving routing schemes, analyzed the research status of existing routing algorithms, and proposed an A3C-R intelligent routing algorithm that takes into account both QoS and energy saving.
• A routing optimization goal that takes into account both QoS and energy saving is designed, and the A3C-R intelligent routing algorithm training framework is built based on A3C’s asynchronous training and advantage functions. And we design the state, action, and reward of the intelligent routing and network environment interaction process, carry out multi-agent model training and global parameter sharing, and improve algorithm efficiency and convergence.
• A simulation experiment environment was built, and we verified the convergence of the A3C-R routing algorithm, as well as the QoS and energy-saving optimization effects of flows under different network traffic loads.

2. Related Research

With the continuous diversification of business traffic requirements and the increasing network energy consumption, routing traffic scheduling for QoS and energy saving is facing severe challenges. Due to the continuous development of SDN and artificial intelligence technology, many domestic and foreign researchers continue to propose intelligent traffic scheduling methods based on machine learning in the SDN environment.

• Research Status of QoS Routing Technology

QoS routing aims to provide differentiated services under different business traffic requirements, network operating status and link available resources, and dynamically select transmission paths that meet network load and resource configuration for different business flows. With the rise in SDN, SDN-oriented QoS routing calculations continue to develop. Li S H [8] proposed a routing optimization technology solution based on traffic classification in the SDN environment. This solution can identify and classify the service types of data flows entering the network and perform routing optimization scheduling according to the QoS requirements corresponding to the service types of data flows, so as to provide a QoS guarantee for network applications of different service types. Compared with the traditional scheduling mechanism, it can effectively reduce network congestion and improve network service quality. However, the calculation efficiency of QoS of this algorithm still needs to be improved under complex network traffic. In order to further improve the computational efficiency of the QoS routing algorithm, some researchers have introduced the classic heuristic algorithm into SDN for research. Fei H et al. [9] proposed a novel energy efficient QoS secure routing algorithm based on the Levy Chaos Adaptive Snake Optimization-based Multi-Trust Routing Method (LCASO-MTRM). The method combines chaotic and adaptive operators and aims to reduce energy consumption, latency, and packet loss while increasing bandwidth. By introducing the link trust mechanism, LCASO-MTRM is able to more accurately assess the link trust level and reflect the current link status. Although the above QoS routing algorithms have better QoS performance than the classical Dijkstra routing algorithm, it is difficult to consider the balance of QoS parameters more comprehensively in the case of more complex business traffic. With the rise in artificial intelligence, Y Wang et al. [10] proposed a Dynamic Fuzzy Routing Algorithm based on Deep Reinforcement Learning (DFRDRL). This involved applying deep reinforcement learning for path routing decisions with the help of Target and Online Neural Networks (TONN). It also introduces a delay mechanism to detect congestion based on real-time traffic forecasts and prioritize low resource demand requests. The method achieves a significant improvement in optimizing network QoS performance such as delay, bandwidth, and link utilization. Sanchez L P A et al. [11] proposed a QoS-driven routing optimization method (DQS) based on Deep Reinforcement Learning (DRL) for routing optimization in Software Defined Network (SDN). The approach dynamically adapts routing decisions through a multi-objective function-driven DRL agent that combines link and queuing metrics to reduce end-to-end delay and improve network performance. However, when facing multiple traffic types, it is difficult to meet the QoS requirements of different applications.

2.

Research status of energy-saving routing technology

Energy-saving routing is a key means to solve the problem of network energy consumption. It is of great significance to reduce network operation costs and improve the efficiency of network resource utilization. The exploration of its optimization strategy has always been a research hotspot in the field of network research. Wang Z et al. [12] proposed an unequally clustered routing protocol based on a multi-hop threshold distance (UCRTD). The method determines the energy-saving threshold distance for multi-hop communication by analyzing the energy consumption model of WSNs and proposes an optimal relay node selection strategy based on distance and residual energy. However, the existing non-uniform clustering protocols lack the theoretical basis of multi-hop threshold distance, have insufficient load balancing during data transmission, and perform poorly in the face of complex and variable networks. Tang Q et al. [13] proposed a chaos particle swarm optimization (CPSO)-based cluster routing algorithm for a wireless sensor network (WSN). The method optimizes network performance through clustering, cluster head selection, and energy management. However, the performance of the algorithm is greatly affected by the parameters, and the adaptability and scalability of the algorithm still need to be improved in large-scale and dynamically changing network environments. Shu X et al. [14] proposed a multi-intelligent deep reinforcement learning algorithm (EMADRL). By defining the multi-agent reinforcement learning form of MDVRP, a policy network model based on the encoder–decoder structure is designed, and it is trained using the policy gradient algorithm, and local search strategies such as 2-opt search and sampling search are introduced to optimize the solution. However, the EMADRL algorithm has a low accuracy of Q-value estimation at the early stage of training, resulting in large fluctuations in the reward curve. Its convergence time is longer than some of the compared algorithms, and the convergence speed still needs to be improved, even though it can produce high-quality solutions quickly in practical high-computing power platforms. Niranjana M I et al. [15] proposed a Grid Based Reliable Routing (GBRR) algorithm for Wireless Sensor Networks (WSN) that combines virtual grid cluster construction and energy efficient next hop selection strategy. The main approach is to improve the reliability and energy efficiency of data transmission through mesh partitioning and intra/inter-cluster communication optimization. Akyıldız O et al. [16] proposed a task offloading scheme (TOS-P4) based on the P4 programming language for task allocation and resource management in fog computing networks. This method is able to improve efficiency and reduce the latency of task processing by performing task type identification and dynamic adjustment of load state in the data plane through P4 switches. However, in traditional fog computing network task offloading, it is difficult for tasks to be accurately assigned to the right server based on type, and SDN controllers have limitations in network traffic management and programmability.

To sum up, although the existing routing algorithms for QoS and energy saving can achieve the optimization goals set by themselves, they still have a single consideration factor, and it is difficult to take into account multiple optimization goals such as QoS and energy saving at the same time. Moreover, the convergence efficiency of existing routing algorithms and the effectiveness of algorithm models still need to be improved, which makes QoS-oriented energy-saving routing optimization of great research significance. Therefore, based on the efficient routing scheduling mechanism of SDN, this paper further studies how to provide reasonable allocation of network resources through intelligent routing scheduling, realize differentiated services for QoS requirements, and take into account network energy consumption optimization.

3. Modeling of QoS and Energy-Efficient Routing

3.1. Modeling of QoS Routing

QoS is the guarantee of network transmission quality, which can provide better service quality according to different business requirements [17]. The process of defining and measuring QoS parameters mainly involves the collection of network link state information, through which the port and flow information of the switch can be obtained and processed as the input of the intelligent routing module. This paper mainly designs QoS parameter indicators including: delay, bandwidth and packet loss rate. The parameters of QoS are mainly calculated through the message transmission process between the controller and the switch. The message transmission process between the controller and the switch is shown in Figure 1. The specific calculation method of each QoS parameter is as follows:

• Delay

The SDN controller can generate a Packet-out message and send a data packet including LLDP to the switch S_2, through the switch S₁, and the time stamp is t₁. When the controller receives the Packet-in message received by the switch S₂, the time stamp is t₂, and the time of the message from “controller-S₁-S₂-controller” is $T_1=t_1-t_2$. Similarly, the “controller-S₂-S₁-controller” time can be measured as $T_2$. Then, the controller can measure the round-trip delays $t_{c-s_1}$ and $t_{c-s_2}$ of the switches S₁ and S₂, respectively, through the echo data packets, so the calculation formula of the link delay $d_{\left(S_1,S_2\right)}$ of the switches S₁ and S₂ is shown in (1).

$$d_{\left(S_1,S_2\right)}={\displaystyle \frac{1}{2}}\left(T_1+T_2-t_{c-s_1}-t_{c-s_2}\right)$$

(1)

Therefore, the delay function of path P can be defined as Formula (2).

$$Delay\left(P\right)={\displaystyle {\sum }_{\left(i,j\right)\in P}{d}_{\left(i,j\right)}}$$

(2)

where $d_{\left(i,j\right)}$ is the delay of link $\left(i,j\right)$.

2.

Bandwidth

Bandwidth is a measure of the link’s ability to transmit data per unit of time. We can use rx_Bytes to measure the number of bytes received by the port and use tx_Bytes to measure the number of bytes sent by the port. Suppose we use tx_Bytes to measure the number of bytes sent by switch S₁ to S₂ at time t₁ to t₂ as $W_t$ and use rx_Bytes to measure the number of data packets received by switch S₂ from S₁ at time t₁ to t₂ as $W_r$. Then, the average bandwidth and remaining bandwidth of the switches S₁ and S₂ used to transmit data in the $\left[t_1,t_2\right]$ time interval are shown in Formula (3).

$$\left\{\begin{array}{l}{b}_{average}={\displaystyle \frac{\min \left\{W_t,W_r\right\}}{t_2-t_1}}\\ b_{remain}=b_{total}-b_{average}\end{array}\right.$$

(3)

Since the minimum bandwidth requirement must be considered when transmitting data flows, the bandwidth of each link must not be less than the minimum bandwidth when selecting a path for routing. The bandwidth of path P is defined as shown in Formula (4).

$$B\left(P\right)=\underset{\left(i,j\right)\in P}{min}\left\{b\left(i,j\right)\right\}$$

(4)

where $b_{\left(i,j\right)}$ is the available bandwidth on the link $\left(i,j\right)$.

3.

Packet loss rate

The packet loss rate refers to the ratio of lost packets to the total number of packets within a period of time. We can use rx_packets to measure the number of packets received by the port and tx_packets to measure the number of packets sent by the port. Suppose we use tx_packets to measure the number of data packets sent by switch S₁ to S₂ at time t₁ to t₂ as $\Delta p_1$ and use rx_packets to measure the number of data packets received by switch S₂ from S₁ at time t₁ to t₂ as $\Delta p_2$. The packet loss rate of switches S₁ and S₂ in the $\left[t_1,t_2\right]$ time interval is shown in Formula (5).

$$l_{\left(S_1,S_2\right)}={\displaystyle \frac{\Delta p_2}{\Delta p_1}}$$

(5)

Therefore, we can define the packet loss rate function of path P as Formula (6).

$$Loss\left(P\right)=1-{\Pi }_{\left(i,j\right)\in P}\left(1-l_{\left(i,j\right)}\right)$$

(6)

Since the packet rate calculation cannot be multiplied, it needs to be converted into a transmission success rate before processing, where $l_{\left(i,j\right)}$ is the packet loss rate on link $\left(i,j\right)$.

3.2. Modeling of Energy-Efficient Routing

The energy consumption of switches in the network topology is mainly composed of two parts: fixed energy consumption and dynamic energy consumption. The fixed energy consumption is a constant related to the switch hardware. When the switch is turned on, there is energy consumption, and the dynamic energy consumption is a variable related to the forwarding rate of the switch port. This paper mainly conducts research on network energy consumption from the aspect of flow routing scheduling, aiming to transmit traffic evenly in the network topology, and reduce dynamic energy consumption related to port forwarding rate by improving link utilization. In this paper, the network topology, G, is defined as a collection of forwarding nodes and links, which can be expressed as $G=\left(V,E,C\right)$. Among them, V represents the collection of all switches and hosts, including all transmission nodes, E represents the collection of all links, and C represents the maximum bandwidth capacity of the link. In order to simplify the optimization idea of energy consumption, this paper abstracts the energy consumption of network ports as link energy consumption and adopts the energy consumption function of the more commonly used rate scaling model [18]. The calculation process of the energy consumption function $p_e\left(x_e\right)$ of the links in the network topology is shown in Formula (7).

$$p_e\left(x_e\right)=\left\{\begin{array}{l} 0, x_e=0 \\ \sigma +\mu {x_e}^{\alpha }, 0\le x_e\le C\end{array}\right.$$

(7)

In Formula (7), $x_e$ is the transmission rate of the link, $\sigma$ represents the inherent energy consumption when the link is turned on, $\mu {x_e}^{\alpha }$ represents the dynamic energy consumption of the link, where $\mu$ represents the link rate correlation coefficient, and $\alpha$ represents the link rate correlation index and $\alpha >1$.

We assume that each flow is independent and indivisible, and the set of flows is denoted as $Flow=\left\{f_1,f_2,\cdots f_i,\cdots ,f_n|n\in N^{+}\right\}$ and $f_i=\left(r_i,d_i,p_i,q_i,w_i\right)$, where $r_i$ represents the start time of the flow $f_i$, $d_i$ represents the end time of the flow $f_i$, $p_i$ represents the starting node of the transmission path of the flow $f_i$, $q_i$ represents the termination node of the transmission path of the flow $f_i$, and $w_i$ represents the data volume of the flow $f_i$. Assume that the transmission rate of $f_i$ at time, t, is $s_i\left(t\right)$, and Formula (8) needs to be satisfied.

$${\displaystyle {\int }_{r_i}^{d_i}{s}_i\left(t\right)dt}=w_i, \forall f_i\in Flow$$

(8)

If the link set of the flow transmission path is denoted as $P_i$, our process of optimizing routing scheduling can be regarded as the solution to Formula (9).

$$S=\left\{s_i\left(t\right),P_i|\forall f_i\in Flow,\forall t\in \left[r_i,d_i\right]\right\}$$

(9)

Define the active link in time $T=\left[t_1,t_2\right]$ as $E_a$, and the total number of active links is denoted as $L_a=\left|E_a\right|$. The total energy consumption of the link during the transmission of the flow on the active link is recorded as ${\varphi }_f\left(S\right)$, and the calculation process is shown in Formula (10).

$$\left\{\begin{array}{l}{E}_a=\left\{e\in E|\exists t\in \left[t_1,t_2\right],x_e\left(t\right)>0\right\}\\ {\varphi }_f\left(S\right)=\left(t_2-t_1\right)·\left|L_a\right|·\sigma +{\displaystyle {\int }_{t_1}^{t_2}{\displaystyle {\sum }_{e\in E_a}\mu }{\left(x_e\left(t\right)\right)}^{\alpha }dt}\end{array}\right.$$

(10)

We assume that the working status of network devices and links in the current network topology remains unchanged, which is equivalent to the fixed network energy consumption of the network topology. We only need to pay attention to the dynamic energy consumption during the flow transmission process. Therefore, reducing dynamic energy consumption requires keeping the flow uniform in time and space, which is equivalent to solving the minimum value of Formula (11).

$${\varphi }_f={\displaystyle {\int }_{t_1}^{t_2}{\displaystyle {\sum }_{e\in E_a}\mu }{\left(x_e\left(t\right)\right)}^{\alpha }dt}$$

(11)

We use the delay, bandwidth, and packet loss rate in QoS as the main optimization goals and reduce network energy consumption. In order to facilitate the optimization training, we need to normalize the energy consumption and QoS data and use the method of Z-score [19] for processing. The Z-score method is a widely used statistical technique for data standardization. It transforms the original data into a standard normal distribution, which is beneficial for comparing data from different sources and detecting outliers. In our study, we used the Z-score method for two main purposes. First, to pre-process the experimental data. Given that the network performance data (such as delay, bandwidth utilization) collected from various sources and under different measurement conditions may have significant differences in scale and distribution, Z-score standardization helps to unify these data onto a common scale, ensuring more stable and accurate subsequent data analysis and model training. Second, we utilized the Z-score to identify outliers in the dataset. Data points with an absolute Z-value greater than 3 (or less than −3) are often considered as outliers. For such outliers, we conducted further investigations to determine whether they were caused by measurement errors or special network events, thus ensuring the quality of the data. The calculation process is shown in Formula (12).

$$y^{*}={\displaystyle \frac{y-u}{\sigma }}$$

(12)

where $y^{*}$ represents the target value, $y$ represents the original data, $u$ is the mean, and $\sigma$ is the standard deviation. Therefore, we can use the Formula (13) to express the optimization process.

$$\min (\delta ·{\varphi }_f\prime +\beta ·Delay\left(P\right)\prime +\lambda ·{\displaystyle \frac{1}{D\left(P\right)\prime }}+\gamma ·Loss\left(P\right)\prime )$$

(13)

Among them, ${\varphi }_f\prime$, $Delay\left(P\right)\prime$, $D\left(P\right)\prime$, and $Loss\left(P\right)\prime$ are normalized energy consumption, delay, bandwidth, and packet loss rate, respectively, and $\delta$, $\beta$, $\lambda$, and $\gamma$ are parameter weights.

In order to ensure that the above traffic scheduling process is not affected by the environment, we stipulate that the traffic transmission process must be completed before the latest deadline, the bandwidth occupation of each link does not exceed the link capacity C, and each flow is received by the destination node.

4. QoS and Energy-Saving Routing Based on A3C

4.1. A3C-R Routing Framework

A3C-R uses the asynchronous multi-thread training mechanism of the A3C algorithm to execute multiple agents, including a Global network and multiple Worker threads, and their network structures are exactly the same. Each Worker thread includes the Actor network and the Critic network to interact independently with the environment. The Actor network can realize the mapping from state to action, and the Critic network is to evaluate whether the action is good or bad. During the independent exploration process of multiple Workers and the environment of the A3C algorithm, it not only breaks the correlation between training data but also saves storage space, reduces computing power requirements, and shows obvious advantages in training speed and convergence. The A3C algorithm asynchronous update framework is shown in Figure 2.

In the traditional Actor network and the Critic network, the Actor network refers to the policy function ${\pi }_{\theta }\left(a|s\right)$, and the Critic network refers to the value function $V_{\varphi }\left(s\right)$. The policy function ${\pi }_{\theta }\left(a\left|s\right.\right)$ defines the probability distribution of selecting action a given the current state $s$. It is used to guide the agent’s decision-making process. The value function, $V_{\varphi }(s)$, represents the expected cumulative reward that the agent can obtain starting from state, $s$. It evaluates the quality of the current state and helps the agent optimize its policy. The strategy function and the value function are continuously learned during the training process, and the reward starting at time, t, can be recorded as $R\left({\tau }_{t:T}\right)$, and the calculation process of the approximate value, $\widehat{R}\left({\tau }_{t:T}\right)$, is shown in Formula (14).

$$\widehat{R}\left({\tau }_{t:T}\right)=r_{t+1}+\gamma V_{\varphi }\left(s_{t+1}\right)$$

(14)

Among them, $s_{t+1}$ and $r_{t+1}$ are the state and reward at time $t+1$, respectively, and the strategy function ${\pi }_{\theta }\left(a|s\right)$ and function $V_{\varphi }\left(s\right)$ are learned during the update process. On the one hand, the parameter $\varphi$ can be updated to make the value function, $V_{\varphi }\left(s_t\right)$, close to the real return value, $\widehat{R}\left({\tau }_{t:T}\right)$. The calculation process Formula (15) is shown.

$$\underset{\varphi }{\min }{\left(\widehat{R}\left({\tau }_{t:T}\right)-V_{\varphi }\left(s_t\right)\right)}^2$$

(15)

On the other hand, the value function is used as a baseline update to reduce the variance of the policy gradient, and the update process is shown in Formula (16).

$$\theta \leftarrow \theta +\alpha {\gamma }^t\left(\widehat{R}\left({\tau }_{t:T}\right)-V_{\varphi }\left(s_t\right)\right){\nabla }_{\theta }\log {\pi }_{\theta }\left(a_t|s_t\right)$$

(16)

The traditional Actor network and Critic network have the phenomenon that no matter which action is used in a certain state, they can obtain better action value, ignoring the differences between actions, making the training process unstable. Therefore, the A3C algorithm introduces an advantage function. The process is shown in Formula (17).

$$A\left(s,a\right)=Q\left(s,a\right)-V\left(s,a\right)$$

(17)

Among them, $Q\left(s,a\right)$ is the action value function, $V\left(s\right)$ is the estimated cumulative state value function in the states, and the training sequence $\left(s_t,a_t,r_t,s_{t+1}\right)$ is given. The calculation process of $Q\left(s,a\right)$ is shown in Formula (18).

$$Q\left(s,a\right)=r_t+\gamma V\left(s_{t+1}\right)$$

(18)

The policy update of the Actor network will adopt the policy gradient training method. The calculation process is shown in Formula (19).

$$\theta \leftarrow \theta +\alpha {\displaystyle \sum _t{\nabla }_{\theta }\log {\pi }_{\theta }\left(s_t|a_t\right)}+\beta {\nabla }_{\theta }H\left({\pi }_{\theta }\left(·|s_t\right)\right)$$

(19)

Among them, $\alpha$ is the learning rate, and the entropy value of strategy $\pi$ is introduced to optimize the exploration process by organizing premature convergence to the suboptimal determination strategy. H is the entropy of the strategy. The entropy function, $H\left({\pi }_{\theta }\left(s_i\right)\right)$, measures the randomness of the policy, ${\pi }_{\theta }$. It is introduced to prevent premature convergence to suboptimal solutions and encourages exploration during the training process. The parameter $\beta$ controls the influence of the entropy regularization term.

The Critic network will update the parameters using the temporal difference method. The calculation process is shown in Formula (20).

$$\varphi \leftarrow \varphi +\alpha {\displaystyle \sum _t{\nabla }_{\theta }{\left(r_t+\gamma V\left(s_{t+1};\varphi \right)-V\left(s_t;\varphi \right)\right)}^2}$$

(20)

4.2. Design of A3C-R Routing and Environment Interaction

We realize the interaction process between A3C-R routing and the environment in the SDN environment. The SDN mainly includes the data plane, the control plane, and the intelligent plane, as well as the southbound interface and the northbound interface. The training process of the A3C algorithm is mainly located in the intelligent plane. First, the data plane collects network status information and then transmits it to the control plane through the southbound interface to extract valid status information. Finally, the effective state information is input to the A3C-R routing algorithm of the intelligent plane through the northbound interface for asynchronous training. After the A3C-R routing algorithm training converges, it can output the link weights of the network topology, and then the control plane generates a flow table after receiving the link weight information and sends it to the data plane. In this way, the interaction between A3C-R routing and the environment is completed. The interaction process between A3C-R routing and the environment is shown in Figure 3. The design of the specific state, action, and reward during the interaction between A3C routing and the environment is as follows:

• State

The state design in this paper mainly includes QoS information such as network delay, bandwidth, and packet loss rate, as well as network link energy consumption. We collected various network performance indicators according to the network information collection module, and then calculated the normalized network link energy consumption, delay, bandwidth, and packet loss rate through ${\varphi }_f\prime$, $Delay\left(P\right)\prime$, $D\left(P\right)\prime$, and $Loss\left(P\right)\prime$ respectively. Finally, the above state information is recorded as a state set $s=\left\{{\varphi }_f\prime ,Delay\left(P\right)\prime ,D\left(P\right)\prime ,Loss\left(P\right)\prime \right\}$.

2.

Action

The design of the action is the link weights of the network topology. Through the continuous interaction between the A3C-R routing algorithm and the environment, we can gradually train the optimal link weights suitable for the current network environment. We can record the action output by the A3C-R routing algorithm as $a=\left\{w_{1,}{w}_{2,}\cdots ,w_n\right\}$, where w is the weight value of each link in the network topology, and n is the total number of links in the network topology. The A3C-R algorithm, through the interaction with the network environment and continuous learning, generates link weights that comprehensively consider QoS and energy-saving requirements. To calculate the optimal routing path, the A3C-R algorithm utilizes the Dijkstra algorithm, which takes the optimized link weights generated by A3C-R as input and computes the optimal path from the source node to the destination node in the network topology. This path is not only the shortest in terms of link weights but also ensures that it meets the QoS and energy-saving requirements, as determined by the A3C-R algorithm. Finally, based on the link weights optimized by the A3C-R algorithm and the path calculated by the Dijkstra algorithm, the optimal routing strategy is implemented in the network.

3.

Reward

The design of the reward mainly takes into account the energy consumption of the current network link, as well as network performance indicators such as network delay, bandwidth, and packet loss rate. The design of the reward is shown in Formula (21).

$$r=\delta {\varphi }_f\prime +\beta Delay\left(P\right)\prime +\lambda {\displaystyle \frac{1}{D\left(P\right)\prime }}+\gamma Loss\left(P\right)\prime ,\left(\delta ,\beta ,\lambda ,\gamma \right)\in \left[0,1\right]$$

(21)

Among them, $\delta$, $\beta$, $\lambda$, and $\gamma$ are the weight parameters of link energy consumption, delay, bandwidth, and packet loss rate, respectively. In order to ensure the optimal balance of various performance indicators, we set the above parameters to 0.25, and their sum is 1.

4.3. A3C-R Routing Algorithm Process

During the training process of the QoS-oriented energy-saving routing algorithm A3C-R, the network status, including network energy consumption, and QoS indicators such as delay, bandwidth, and packet loss rate, are the main inputs. Through the asynchronous training mechanism of the A3C algorithm, multiple agents are executed to complete the independent interaction with the environment, and the experience value obtained during the interaction process is sampled to complete the update of the entire network. Finally, the weights of various network links that meet the requirements of QoS and network energy consumption are output, so as to calculate the optimal routing strategy. The process of A3C-R routing algorithm oriented to QoS and energy saving is shown in Algorithm 1.

Algorithm 1 A3C-R Routing Algorithm Process for QoS and Energy Saving

Input: The set of network states $\left\{{\varphi }_f\prime ,Delay\left(P\right)\prime ,D\left(P\right)\prime ,Loss\left(P\right)\prime \right\}$

Output: The set of network link weights $\left\{w_{1,}{w}_{2,}\cdots ,w_n\right\}$

Initialize: Critic parameter $\varphi$ and Actor parameter $\theta$ of the global network. Critic parameter $\varphi \prime$ and Actor parameter $\varphi \prime$ of Worker thread network.

Initialize: Global shared counter $T$, global maximum shared count $T_{max}$, thread step size $t=1$.

(1) Repeat:

(2) Reset $\varphi =\varphi \prime$, $\theta =\theta \prime$

(3) $t\leftarrow t+1$, $s_t=\left\{{\varphi }_f\prime ,Delay\left(P\right)\prime ,D\left(P\right)\prime ,Loss\left(P\right)\prime \right\}$

(4) Repeat

(5) The Worker thread executes $a_t\leftarrow \pi \left(a_t|s_t;\theta \prime \right)$

(6) Obtain $r_t,s_{t+1}\leftarrow a_t$

(7) $t\leftarrow t+1$, $T\leftarrow T+1$

(8) Until $s_t$ is the terminal state, or $t-t_{start}=t_{\max }$

(9) $R=\left\{\begin{array}{cc}0,& s_{terminal}\\ V(s_t;\varphi \prime ),& others\end{array}\right.$

(10) For $i\in \{t-1,t-2,\cdots ,t_{start}\}$ do

(11) Calculate $R\leftarrow r_i+\gamma R$ at each moment

(12) $d\theta \leftarrow d\theta +{\nabla }_{\theta \prime }\log \pi \left(a_i\right|s_i;\theta \prime \left)\right(R-V(s_i;\varphi \prime \left)\right)+c{\nabla }_{\theta \prime }H(\pi (s_i,\theta \prime ))$

(13) $d\varphi \leftarrow d\varphi +\partial (R-V{\left(s_i;\varphi \prime \right)}^2)/\partial \phi \prime$

(14) End For

(15) Global network asynchronous update $\theta \leftarrow d\theta$, $\varphi \leftarrow \varphi \prime$

(16) Until $T>T_{max}$

The A3C-R algorithm, designed for QoS-oriented and energy-efficient intelligent routing, begins with a series of rigorous initialization steps. First, the Critic and Actor parameters of the global network are initialized, where the Critic evaluates state values, and the Actor generates decision actions. Simultaneously, the Critic and Actor parameters for each Worker thread are also initialized, establishing a solid foundation for multi-agent asynchronous training. In addition to parameter initialization, key global variables such as the global shared counter, maximum shared update steps, and thread step size are configured. These settings play a crucial role in defining the iteration process and termination conditions, ultimately influencing the algorithm’s convergence efficiency.

The core training process is detailed in Algorithm 1, lines 1–15, which describes the parameter update mechanism during training. Specifically, in lines 2–3, the agent retrieves the initial network state, including key indicators such as energy consumption, latency, bandwidth, and packet loss rate, which serve as the basis for decision making. In lines 4–9, the Worker thread executes actions based on the current state, then obtains the corresponding reward value to assess the effectiveness of the action, along with the next state for further evaluation. Finally, in lines 10–15, the agent updates global network parameters based on reward values, state information, and policy gradients, continuously refining the routing strategy. This parameter update mechanism enables A3C-R to dynamically adjust routing decisions, balancing QoS requirements and energy efficiency. As training progresses, the algorithm iteratively optimizes the routing policy, ultimately converging toward an optimal solution that enhances overall network performance.

5. Experimental Evaluation

5.1. Experimental Environment and Comparison Algorithms

In this paper, the network simulation software Mininet is used to realize the QoS and energy-saving test of the A3C-R routing algorithm. The experiment process adopts the RYU controller (version 4.34) and Open vSwitch switch (version 2.9.8). The RYU controller was selected for its support of OpenFlow 1.3, which allows for dynamic flow table management and fine-grained control of network flows. Its flexible, scalable, and modular architecture facilitates seamless integration with the A3C-R algorithm, enabling QoS-aware and energy-efficient routing. We use OS3E as the network topology. OS3E network topology consists of 38 nodes, which simulate various network devices such as switches, routers, and hosts. These nodes are interconnected through 48 links. The bandwidth of each link is uniformly set to 200 Mbps, which serves as a standardized parameter for testing the performance of different routing algorithms. This setting enables a fair comparison of how algorithms affect network performance metrics like delay, throughput, packet loss rate, and energy-saving efficiency under the same bandwidth condition. The experimental hardware implementation platform is the Linux operating system Ubuntu 18.04, tensorflow 1.8.0, and Python 3.5.0. The hardware is an i5-10600KF-CPU, with 16 GB-DDR4 memory, and two GTX-1080 8 G graphics cards. During the training process of A3C-R, the neural network uses the Relu activation function. In the pre-experiments, we tried a number of different combinations of learning rates. When the learning rate of the Actor network is too large, the model training is unstable and easy to skip the optimal solution; when it is too small, the training speed is slow and falls into the local optimum. It has been tested that a learning rate of 0.002 enables the Actor network to efficiently explore and learn the optimal policy while maintaining the training speed. For Critic networks, a learning rate that is too large leads to large fluctuations in the value assessment, while a learning rate that is too small is too slow to guide the Actor network in a timely manner. It was ultimately determined that a learning rate of 0.001 would enable the Critic network to stabilize the learning value function and provide reliable feedback to the Actor network. The algorithm trains for 100,000 steps, with the learning rates for the Actor and Critic set to 0.002 and 0.001, respectively. The batch size for training is 128, and the discount factor is 0.9. Specifically, the discount factor (0.9) is chosen based on related deep reinforcement learning (DRL) research in SDN routing optimization [20], which suggests that values between 0.8 and 0.99 balance long-term and short-term rewards effectively. We also conducted initial experiments with discount factors of 0.7, 0.8, 0.9, and 0.99, and found that 0.9 led to better convergence stability and long-term performance. Similarly, for the batch size, we tested values of 32, 64, 128, and 256, and found that a batch size of 128 achieved a good trade-off between training stability and computational efficiency.

This experiment is compared with traditional routing algorithm, intelligent routing algorithm, and energy-saving routing algorithm. The specific comparison content includes (1) the traditional multi-path routing algorithm ECMP. ECMP is a widely used routing algorithm that distributes traffic across multiple equal-cost paths, serving as a baseline for traditional routing performance. Comparing with ECMP allows us to demonstrate how A3C-R improves QoS and energy efficiency beyond conventional approaches. (2) I-DQN, a fast-converging DQN routing algorithm considering multi-type QoS requirements [21]. I-DQN is a deep reinforcement learning-based routing algorithm designed to optimize multiple QoS metrics such as delay and throughput. Since A3C-R also targets QoS improvements, I-DQN serves as a strong benchmark for evaluating QoS optimization effectiveness. (3) DDPG-EEFS, an energy-saving flow scheduling mechanism based on DRL algorithm [22]. This algorithm optimizes network energy consumption using deep reinforcement learning. Given that A3C-R also aims to reduce energy consumption, comparing it with DDPG-EEFS allows us to assess A3C-R’s energy-saving performance relative to other energy-efficient routing strategies. The comparison content includes QoS performance indicators such as network delay, throughput, and packet loss rate, as well as network energy-saving effects. All comparison experiments were conducted under the same experimental conditions, including network topology, traffic patterns, and configuration settings, to ensure a fair and consistent evaluation across all algorithms.

5.2. Convergence of Routing Algorithms

This experiment verified the convergence effect of the A3C-R algorithm through the reward value of the training process. The experimental process is based on the Formula (21) reward formula as the standard. In the energy consumption function, parameter $\alpha$ is set to 2, and parameter $\mu$ is set to 1. The rewards of the A3C-R routing algorithm training process are shown in Figure 4. The whole training process includes a total of 100,000 steps of interaction, and the reward of the A3C-R routing algorithm tends to be stable at about 40,000 steps, showing a good convergence effect. The reason is that the A3C-R routing algorithm is based on A3C’s asynchronous training method, which can interact with the environment and update model parameters through multiple agents at the same time. This method can speed up training and improve the utilization efficiency of samples. In addition, the A3C algorithm introduces the Advantage function, which is used to measure the advantage of performing a certain action relative to the average level, which can help the agent to better improve the strategy, thereby improving the convergence.

5.3. Performance Analysis of Routing Algorithms

This experiment sets up the comparison of routing QoS and energy-saving indicators under different network loads. Since the link bandwidth of the experimental topology is 200 Mbps by default, we increase the traffic load from 20 Mbps to 180 Mbps. We used a combination of different types of traffic workloads to mimic real-world network traffic. Specifically, we employed three main types of traffic: delay-sensitive traffic, which is characteristic of real-time applications like online gaming, video conferencing, and instant messaging; bandwidth-sensitive traffic, representing services such as high-definition video streaming and large-file transfers; and packet loss-sensitive traffic, corresponding to applications such as financial transaction data transmission and medical data transfer. These traffic types were randomly generated and mixed in certain proportions to create a complex and realistic traffic load. During the experiment, we tested QoS indicators such as delay, throughput, and packet loss rate under different network traffic loads, as well as network energy-saving effects. We use the percentage of energy saving (PES) of the network link to evaluate the energy efficiency of the routing algorithm. PES is defined as the percentage reduction in energy consumption achieved by the proposed algorithm compared to a baseline energy consumption, typically the total network energy consumption under full link load. The specific calculation process is shown in the Formula (22) [23]. Where ${\varphi }_{f-routing algorithm}$ represents the link energy consumption of the current routing algorithm, while ${\varphi }_{f-full load}$ represents the energy consumption under full link load conditions. A higher PES indicates better energy-saving performance. After many experiments, we found that parameter $\alpha$ affects the dynamic energy consumption part of the link. When the $\alpha$ value is too small, the algorithm is not sensitive enough to the link traffic changes to optimize the energy consumption effectively; when it is too large, it will pay excessive attention to the energy consumption at the expense of the QoS performance. It has been tested that setting $\alpha$ to 2 optimizes network power consumption to the maximum extent while ensuring QoS. The parameter $\mu$ affects the link intrinsic energy consumption part, its value is too small, and the algorithm does not take into account the link intrinsic energy consumption enough; when it is too large, it will lead to the unreasonable allocation of network resources and affect the QoS. Setting $\mu$ to 1 enables a better balance between considering the inherent energy consumption of the link and the dynamic energy consumption, ensuring that the algorithm optimizes energy consumption while maintaining good network performance. Therefore, In the energy consumption function, parameter $\alpha$ is set to 2, and parameter $\mu$ is set to 1. The routing algorithm performance test results are shown in Figure 5.

$$PES=1-{\displaystyle \frac{{\varphi }_{f-routing algorithm}}{{\varphi }_{f-full load}}}*\%$$

(22)

It can be seen from Figure 5 that as the traffic load continues to increase, the A3C-R routing algorithm is superior to other routing algorithms in terms of QoS performance indicators such as delay, throughput, and packet loss rate, as well as energy saving effects. The reason is that the traditional routing algorithm ECMP mainly uses equal-cost multi-path transmission, and does not comprehensively consider the delay, throughput, packet loss rate and energy performance indicators of each path in the current network. It is easy to cause link congestion when the network traffic increases. The I-DQN algorithm considers multiple types of QoS requirements in the routing algorithm modeling process, so the algorithm has better optimization effects such as delay, throughput, and packet loss rate compared with the ECMP algorithm. Due to the uniform transmission of traffic, its energy saving advantage is still better than that of ECMP algorithm. The DDPG-EEFS algorithm takes the minimum network energy consumption and the average completion time of network flow as the joint optimization goal in modeling. Therefore, compared with the I-DQN algorithm, it has a better energy-saving advantage and a delay optimization effect, but the throughput and packet loss rate effects are slightly weaker than the I-DQN algorithm. Our A3C-R algorithm takes QoS and energy consumption as the joint optimization goal and uses the asynchronous update advantage of the A3C algorithm to fully perceive the network status and select an energy-saving path that meets the QoS requirements. Therefore, the A3C-R algorithm has better QoS performance indicators and energy-saving advantages than the ECMP algorithm, I-DQN algorithm, and DDPG-EEFS algorithm.

6. Conclusions

The experimental results show that, compared with the ECMP algorithm, I-DQN algorithm, and DDPG-EEFS algorithm, the A3C-R algorithm reduces the delay by approximately 9.42%, increases the throughput by approximately 7.04%, reduces the packet loss rate by approximately 9.51%, and increases the energy saving percentage by approximately 10.78%. The A3C-R algorithm in this paper only considers the QoS and energy-saving optimization of the SDN data plane. With the continuous complexity of network devices and traffic, the performance and energy-saving optimization of the SDN control plane still has important research significance.

In the future, we will further investigate the joint optimization of both the SDN control and data planes to achieve a more comprehensive balance between QoS and energy efficiency. Furthermore, incorporating real-world deployment and validation in large-scale SDN environments will be a crucial step to assess the practicality and scalability of our approach.