Multi-Controller Deployment in SDN-Enabled 6G Space–Air–Ground Integrated Network

Abstract

The space–air–ground Integrated Network (SAGIN) is considered to be a significant framework for realizing the vision of “6G intelligent connection of all things”. The birth of 6G SAGIN also brings many problems, such as ultra-dense dense networks, leading to a decrease in the efficiency of traditional flat network management, and traditional satellite networking solidified network functions, etc. Therefore, combining the 6G SAGIN network with the software-defined network (SDN) is an excellent solution. However, the satellite network topology changes dynamically and the ground user unbalanced distribution leads to the unbalanced load of the SDN controller, which further leads to the increased communication delay and throughput drop, etc. For these problems, a hierarchical multi-controller deployment strategy of an SDN-based 6G SAGIN is proposed. Firstly, the delay model of the network, the load model of the SDN controller, and a loss value as a measure of whether the network delay and controller load are optimal are defined. Then, using the distribution relationship between the SDN controller and the switch node as the solution space, and taking the loss value as the optimization goal, a multi-controller deployment strategy based on the simulated annealing algorithm is used to search for the optimal solution space. Lastly, considering the network topology changes dynamically and the SDN controller imbalance, a switch migration strategy oriented toward load balancing is proposed. We aimed to determine the controller deployment plan through the above two points, balance the controller load, and then improve the network performance. The simulation results show that the controller load is increased by about 7.71% compared to OCLDS, and the running time is increased by 17.7% compared to n-k-means.

1. Introduction

With the development of science and technology, 5G has entered the stage of commercial application, and more studies have begun to focus on the development of 6G. The vision of 6G communication in the future is the intelligent connection of all things, breaking through the limitations of terrain and surface, expanding to natural space such as space, sky, land, and sea, and truly realizing the universal ubiquitous connection [1,2]. The space–air–ground integrated network (SAGIN), which combines the airspace network, sky domain network, and ground network, has become a trend to realize the vision of “6G intelligent connection with everything” [3,4]. The SAGIN architecture is considered as an important framework for the future communication network because it has the characteristics of a space-based network and satellite-based network, such as wide coverage, high flexibility, strong cloud-edge cooperation [5], and the high transmission rate of a ground network [3]. SAGIN is a layered network architecture consisting of satellite networks, space-based networks, and ground-based networks. As shown in Figure 1, a geosynchronous orbit (GEO) satellite, medium-earth orbit (MEO) satellite and low-earth orbit (LEO) satellite constitute the satellite network communication system. The space-based network communication system is composed of a high-altitude platform (HAP) and low-altitude platform (LAP), such as unmanned aerial vehicle (UAV) formation, space aerostats, airships and hydrogen balloons. The ground-based network consists of a cellular wireless network, satellite ground station, ground data processing center, ground stations, ground Internet of Vehicles, etc. [6,7,8,9,10,11]. So, a large-scale network composition model will inevitably face many problems, such as the solidification of the functions of the traditional satellite networking model, and extremely complex upgrades and maintenance. The traditional flat management-intensive network nodes lead to a sharp drop in network delay and throughput, and load imbalances between nodes are prone to appearing. The network is highly heterogeneous, and different network dimensions require different communication protocols and communication links, resulting in increased network management complexity and increased operating network costs [12,13].

Figure 1. Hierarchical 6G space–air–ground integrated network architecture.

To realize the global optimization of resources and the cooperation of heterogeneous networks in such a large-scale network, a current trend is to introduce the idea of SDN centralized perception and data/control decoupled network design into the design of SGAIN [14,15,16,17,18]. SDN uses the layering to separate the data plane of the network from the control plane, and the control plane controls the network equipment of the data plane centrally through the unified and open south interface; it also uses network function virtualization (NFV) technology to abstract the physical network and device components, decouples the physical device layer and the business logic layer, collects the state information of the whole network from the control plane, and creates an efficient resource allocation strategy. The data plane is only responsible for forwarding the resource according to the command of the control plane. The centralized control feature of SDN enhances the resource utilization of space-based networks, and supports the global information acquisition of network resources and the global allocation and optimization of resources according to service requirements in [19,20]. The SDN open programming interface, namely the north and south interface, makes an application layer and control layer; the control layer and network layer are seamlessly connected through a custom application to make SAGIN more efficient in [21]. An SDN-enabled SAGIN can simplify the complexity of the network and improve the network performance, but it needs an efficient network design scheme, that is, the SDN-enabled multi-controller deployment strategy.

Therefore, this paper proposes a multi-controller deployment strategy based on a simulated annealing algorithm and a switch migration strategy oriented toward load balancing for the SDN-based 6G space–air–ground integrated network. Unlike previous work, satellite gateways and controllers are jointly deployed, or all controllers are deployed on satellites. The optimization goal is singular, and the deployment strategy does not consider network topology dynamics. This paper hierarchically deploys controllers on the ground network: the main controller is deployed at the network operation controller center, and the domain controller is deployed near the ground station to minimize the time delay loss from the ground station to the controller. The deployment strategy jointly optimizes the network propagation delay and controller load. The switch migration strategy oriented toward load balancing solves the problem of load imbalance caused by the network topology changes dynamically. In short, the main contributions of this article are as follows:

• Firstly, based on the hierarchical SDN-enabled 6G SAGIN model, the controller hierarchical network design scheme is adopted, and the delay model of the network and the load model of the controller are constructed. The loss value, as an important basis for network division, is defined by combining the delay model and the load model.
• Then, in order to obtain the global optimal solution and reduce the time complexity of the algorithm, a multi-controller deployment strategy based on simulated annealing is proposed, and a concrete deployment scheme is given according to different delay and load requirements.
• Finally, considering the dynamic nature of the network topology, a switch migration strategy is proposed, in order to achieve load balance among the controllers, and some switches within the jurisdiction of the high-load controller are selected to be transferred to the low-load controller, improving the utilization of network resources. Due to the need to implement this strategy at certain intervals to avoid load imbalance between the controllers, resulting in a sharp decline in network performance, the time complexity of the switch migration algorithm cannot be too high.

The rest of this paper is organized as follows: Section 1.1 and Section 1.2 introduce the related work of the SDN-enabled 6G SAGIN and multi-controller deployment strategy. Section 2 introduces the hierarchical and domain SDN-based 6G SAGIN architecture. The multi-controller deployment problem SDN-based 6G SAGIN is described as a mathematical model and a hierarchical and domain multi-controller deployment strategy. Section 3 compares and analyzes the performance of this strategy and gives a feasible deployment plan. Section 4 summarizes this article.

With the proposal of the SDN-enabled SAGIN, more and more studies are focused on how to design the hierarchical structure of the network and how to deploy the SDN controller reasonably in order to solve the problem of network delay, link throughput and other network performance degradation problems caused dynamic changes in SAGIN, the unbalanced distribution of large-scale nodes and strong heterogeneity.

1.1. In SDN-Enabled SAGIN Architecture Design

The SDN multi-controller deployment architecture can be divided into two types: a flat design and layered design. In [22,23], a cross-domain SDN network architecture for a multi-layer SAGIN is proposed. The data plane includes satellites, aircraft, and ground networks [24]. The control plane is divided into two layers, consisting of the main controller layer and the secondary controller layer. The secondary controller layer is deployed in the space-based network, air-based network, and ground-based network, respectively, and cross-network communication is realized through the main controller. It is a layered design. N. Zhang in [21] and Wu, Hua Qing, et al. in [25] also used a multi-controller hierarchical design method to propose a software-defined space–air–ground integrated vehicle network to achieve flexible, reliable and scalable network resource management. D Wei, N Wei, et al. in [26] use a flat multi-controller deployment design in the software-defined satellite network. The control plane is composed of geostationary satellites, and the data plane includes low-orbit satellites, space-based networks and ground networks. This solution avoids dynamic changes in the network topology, but brings tremendous pressure to the geostationary satellite.

1.2. In Terms of Multi-Controller Deployment Strategies

Researchers focused on optimizing network and controller loads in [27,28,29]. Mostafaei H, Menth M, et al. in [30] abstract the controller deployment problem as an NP-hard problem, and propose a learning automata-based heuristic with the goal of minimizing the propagation delay between switches and controllers in the network. This is a static deployment, with no consideration for topo dynamics. Qu in [31] and Qi et al. in [32] aimed to minimize the propagation delay of switches and controllers. The former propose an SDN-enabled SAGIN architecture, and optimize the K-means algorithm to solve the controller deployment problem of the dynamic topology network composed of UAV in the air-based domain, hereinafter referred to as n-k-means. The latter proposed a clustering algorithm based on improved density peaks to solve the multi-controller deployment problem. The algorithm has achieved significant results in optimizing the delay, but the load of a single controller cannot be controlled, which is a disaster for the scarcity of satellite resources. D Wei, N Wei, et al. in [26] proposed a strategy to migrate switch nodes under the administration of an idle controller and high-load controller to an adjacent low-load controller (hereinafter referred to as OCLDS) to reduce network delay and improve network throughput, considering the large load difference between SDN-based satellite network controllers. The load balancing effect of the controller is significant, but the disconnection and connection process between the switch node and the controller is time consuming, resulting in a delay that is not well controlled. P. Tao, C. Ying, et al. in [33] considered the routing times of a switch controller to the switch of the routing number and the number of routes between controllers, based on the load balancing factor of the controller, by minimizing the cost factors and traffic request of the linear function, which determines the location of the controller and the change in the load factor. However, the disadvantage of this is, in the case of given a network topology, he could not determine the number of controllers. To sum up the previous research on multi-controller deployment strategies based on SDN, there are mainly several problems: whether to consider the dynamics of network topology, how to consider both network delay and controller load, and whether to have a low-time-complexity algorithm. In response to these issues, specific solutions are given in the following chapters of this paper.

2. Problem Formulation and Solution

2.1. Problem Formulation

Figure 2 shows the hierarchical domain-based SDN-enabled space–air–ground integration network model adopted in this paper; the controller plane mainly consists of a multi-domain controller and central controller deployed near the gateway station. The data plane mainly consists of LEO satellites (Walker constellation Iridium constellation et al.), MEO satellites, GEO satellites and ground network nodes. Each group of domain controllers is responsible for the communication tasks of its own domain. Only when there is a cross-domain communication task will it send a request to the central controller, and the hierarchical and domain-divided network design pattern reduces the time of synchronous information consumption between controllers and increases the flexibility of controller expansion. The next problems to be solved are how to divide the network nodes into domains, how to deploy the controller, and how to make the satellite switch migration strategy due to the satellite network dynamics.

Figure 2. Hierarchical SDN-enabled 6G SAGIN architecture.

2.1.1. Problem Description

In SDN-based satellite networks, due to the dynamic changes of network traffic and network topology and unreasonable deployment of SDN controllers seriously affecting the performance of SDN controllers, a load imbalance is caused in the controllers. The overload or idle state of the controller leads to a decrease in server resource utilization. Furthermore, the unreasonable network partition also leads to an increase in network delay and a decrease in network throughput. The satellite always moves along a particular orbit, and its topology changes periodically and highly dynamic during the system cycle. In the study of the optimal deployment of multi-controllers, a highly dynamic topology is planned as a system cycle by using the idea of time slice partition, and it is decomposed into several small time slices. It is assumed that the satellite network topology does not change in the time slice. With the change in time slice, the corresponding switch node migration strategy is made.

In the research of SDN-based multi-controller deployment for satellite networks, the problem that should be solved is how to divide the network reasonably and to determine the relationship between the controller and switch node. That is, the multi-controller deployment problem can be transformed into a task assignment problem by assigning a task to a target object in order to achieve the overall optimal goal.

2.1.2. Hierarchical Multi-Controller Deployment Modeling

The SDN-based satellite network model is abstracted into an undirected graph with three dimensions, expressed by $G=(S,E,C)$, where S is the set of switches $S=\left\{s_1,s_2,\dots ,s_i,\dots ,s_n\right\}$, and there are n switch nodes in total. C is the set of controllers $C=\left\{c_1,c_2,\dots ,c_j,\dots ,c_m\right\}$, there are n controller nodes in total, E is the communication link between the switch node and the controller $E=\left\{e_{ij}|i=1,2,\dots ,n;j=1,2,\dots ,m\right\}$, where $e_{ij}=1 \mathrm{or} 0$, one is the communication link available, and zero is the broken link. In this paper, the delay and the load of the controller are taken as the indicators of the multi-controller deployment strategy, in which the delay refers to the propagation delay from the switch node to the controller, since the switch sends packet-IN packets to the controller in real time over the SDN network, and the load of the controller is measured by the number of packet-IN packet requests per unit time.

Network Delay Model

The propagation delay between the switch and the controller is the distance between two points in space divided by the speed of light. The propagation delay unit is ms. Let the known data be the longitude, latitude and altitude information of the satellite node and controller, as shown in Figure 3.

Figure 3. The 3D coordinate diagram of the satellite operation.

Assuming that point A is the latitude and longitude of a controller $c_j=({\theta }_{cj},{\phi }_{cj})$, and point B is the latitude and longitude of a satellite node $s_i=({\theta }_{si},{\phi }_{si})$, first convert the geographical coordinates into three-dimensional Cartesian coordinates by using (1) and (2):

$$(x_{cj},y_{cj},z_{cj})=r\ast (cos{\theta }_{cj}cos{\phi }_{cj},sin{\theta }_{cj}cos{\phi }_{cj},sin{\phi }_{cj})$$

(1)

$$\begin{array}{c}\hfill (x_{si},y_{si},z_{si})=R\ast (cos{\theta }_{si}cos{\phi }_{si},sin{\theta }_{si}cos{\phi }_{si},sin{\phi }_{si})\end{array}$$

(2)

where R is the radius of the orbit and r is the radius of the Earth; theoretically, $R=r+h$, and h is the altitude of the satellite. Then, the distance between A and B is (Equation (3)):

$$d_{ij}=\parallel \overrightarrow{AB}\parallel =\sqrt{{\left(x_{cj}-x_{si}\right)}^2+{\left(y_{cj}-y_{si}\right)}^2+{\left(z_{cj}-z_{si}\right)}^2}$$

(3)

Finally, the propagation delay from the switch $s_i$ to the controller $c_j$ in t time chip is expressed by using (4):

$$L_{ij}\left(t\right)=\frac{1}{\mathbb{C}}{d}_{ij}\left(t\right)$$

(4)

where $\mathbb{C}$ represents the speed of light $3\times {10}^8$ m/s. In addition, because the satellite node is dynamic, set a maximum delay $L_{max}$; when $L_{ij}>L_{max}$ the satellite switch node is not in the jurisdiction of the current controller.

The above analysis leads to the optimization of sub-goal 1 as follows: (Equation (5)):

$$f_1=min{L}_{ij},\{i=1,2,\dots ,n.j=1,2,\dots ,m\}$$

(5)

Controller Load Model

There are three main considerations for controller load: number of the packet-IN packet requests to the controller, memory load $B_m$, and CPU load $B_c$. If the packet-IN packet rate of the switch node $s_i$ is $q_i\left(t\right)$, the controller load for time slice t is expressed by using (6):

$$B_j\left(t\right)=\alpha \times \frac{{\sum }_{s_i\in S_{cj}}{q}_i\left(t\right)\ast e_{ij}}{Q_{c_j}}+\beta \times B_{\mathrm{m}}\left(t\right)+\gamma \times B_C\left(t\right)$$

(6)

where $s_{cj}$ is the set of switch nodes governed by the controller $c_j$, $Q_{c_j}$ is the resource capacity of the controller $c_j$, and $\left\{\alpha +\beta +\gamma =1|\alpha ,\beta ,\gamma \in (0,1)\right\}$ is the variable parameter [34]. In addition, the maximum resource capacity of the controller is $Q_{max}$, when $B_j>\rho \ast Q_{max}$, which means that the current controller cannot move into the switch node, and $\rho \in (0,1)$ is a variable parameter.

In order to achieve load balance among the controllers, the optimization sub-goal is defined as minimizing the variance of the controller load. The physical meaning of variance is to measure the deviation between the random variable and its mathematical expectation (i.e., the mean); thus, the variance can be used to measure whether the controllers are in equilibrium with each other. The above analysis leads to the optimization of sub-goal 2 as follows (Equation (7)):

$$f_2=min\frac{{\sum }_{j=1}^m{\left[B_j-\frac{1}{m}\sum B_j\left(t\right)\right]}^2}{m}$$

(7)

Loss Function

The existence of the loss function is mainly used to determine whether to update the position coordinates of the controller and the relationship between the switch and the controller. The loss function is defined as the weighted sum of the maximum delay of the network and the total average load of the controller. The formula is as follows (Equation (8)):

$$\delta \left(t\right)=\mu \times \frac{max\left(\frac{1}{\mathbb{C}}{d}_{ij}\ast e_{ij}\right)}{L_{max}}+\lambda \times \frac{{\sum }_{j=1}^m{\left[B_j-\frac{1}{m}\sum B_j\left(t\right)\right]}^2}{m}$$

(8)

where $\left\{\mu +\lambda =1|\mu ,\lambda \in (0,1)\right\}$ is the weight of the delay and load, variable parameters, and $B_j\left(t\right)=max\left\{B_j|j=1,2,\dots ,m\right\}$ is the maximum load.

To sum up, the packet-IN packet rate $q_i\left(t\right)$ and transmission delay $L_{ij}\left(t\right)$ of switch nodes $s_i$ are optimized by multi-objective optimization. The resulting multi-objective optimization function is described as follows (Equation (9)):

$$\begin{array}{c}\hfill F\left(X\right)=\left[f_1\left(X\right),f_2\left(X\right)\right]\end{array}$$

(9)

The solution space X is the controller position matrix and the corresponding relation matrix between the switch and the controller.

The objective function condition is described as follows:

• Condition 1: Communication link status of switch and controller: $e_{ij}\in \left\{0,1\right\}$.
• Condition 2: The number of controllers is greater than 2.
• Condition 3: The controller to the jurisdiction of the switch node delay is less than the maximum delay: $L_{ij}\left(t\right)<L_{max}$.
• Condition 4: Controller resource capacity greater than 0: $Q_{c_i}>0$.
• The solution space, i.e., the latitude and longitude of the controller is limited in China (latitude: 3.86–53.55°; longitude: 73.66–135.05°).

2.2. Problem Solution

Multi-Controller Deployment Algorithm Based on Simulated Annealing

In order to find the global optimal solution of the optimization model, avoid falling into the local optimal solution, flexibly control the iteration speed, and quickly obtain the global optimal solution, this strategy draws on the idea of a simulated annealing algorithm, accepts the poor solution space with a certain probability, and presents the local optimal solution. The next solution space is randomly generated and has good robustness. The cooling rate is controllable, and the search results can be obtained faster.

The pseudo-code of Algorithm 1 outlines the framework of the multi-controller deployment strategy. The inputs include the controller (ground station) longitude and latitude matrix C, satellite longitude and latitude matrix S, height matrix $S_H$, traffic of switch nodes $SF$, and the maximum controller capacity $CF$. The initialization parameters include the relation matrix of the satellite corresponding controller R, initial temperature t, and descending speed step and final temperature e. First, the data plane, that is, the satellite node latitude and longitude data, is transformed into the Cartesian coordinate data by Formula (1). Then, after a certain number of iterations, the temperature is increased. In the next iteration, the following is carried out:

• First, a new solution space is randomly generated, which is the controller’s new latitude and longitude, and is converted to Cartesian coordinates according to Formula (2).
• Second, the propagation delay from the satellite node to the controller is solved according to Formula (3).
• In the third step, the satellite node is assigned to the target controller with the goal of minimum time delay, and then the load of the controller is calculated; if a controller load is greater than $\rho \in (0,1)$ times the maximum load, the part of the controller controlled by the current controller is transferred to the controller with the least load, and the relation matrix R between the satellite node and the controller is obtained.
• The fourth step is to calculate the loss function value according to Formula (8). if the loss value is less than the previous loss value, the new solution space generated in the first step is accepted, or the new solution space can be accepted with some probability calculated by Formula (10).

Finally, the relation matrix R, and the solution space, i.e., the final latitude and longitude of the controller, the propagation delay and the load of the controller are saved.

$$p\left(i\right)=e^{-\frac{\delta \left(i\right)-\delta (i-1)}{t}}$$

(10)

where $p\left(i\right)$ is the probability of accepting the current solution space, $\theta \left(i\right)$ is the loss value of the current iteration, $\theta (i-1)$ is the loss value of the previous iteration, and t is the rate at which the temperature decreases.

Algorithm 1 Multi-controller deployment algorithm based on simulated annealing.

Input: Controller(Ground station) longitude and latitude matrix C, Satellite longitude and latitude matrix S, height matrix $S_H$, Traffic of switch nodes $SF$, and the maximum controller capacity $CF$.

Output: Number of controllers and longitude and latitude matrix $C_{best}$, relation matrix of satellite corresponding controller R, propagation delay L and controller load B.

1: Initialize The relation matrix of the satellite corresponding controller is R, initial temperature t, descending speed $step$ and final temperature e.

2: $pos\_xyz\leftarrow smiller\_to\_xyz\left(S\right)$

3: while $t>e$ do

4:       for $iterinrange\left(iteration\right)$ do

5:            $C\_temp\leftarrow product\_new\_point\left(C\right)$

6:            $controller\_xyz\leftarrow miller\_to\_xyz(S,S_H)$

7:            $L\leftarrow calculate\_delay(controller\_xyz,pos\_xyz)$

8:            $R,CF\leftarrow relation\_satellite\_controller(L,SF)$

9:            $loss\_temp\leftarrow calculate\_loss(R,L,SF)$

10:          $probability\leftarrow np.random.rand\left(\right)$

11:          $pro\_new\leftarrow calculate\_pro(loss\_temp,loss,t)$

12:          if $loss\_temp<loss$ or

13:              $pro\_new>probability$ then

14:                $loss\leftarrow loss\_temp$

15:                $C\leftarrow C\_temp$

16:          end if

17:       end for

18:       $t\leftarrow stepastt$

19:       $save(R\_best,C\_best,L,B)$

20: end while

2.3. Switch Node Migration Strategy for Controller Load Balancing

Because the satellite constellation moves periodically, the load of the controller will be out of balance with the change in time slice, so it is necessary to trigger the node migration of the switch in order to achieve load balance among the controllers. A pseudo-code description of the migration strategy is given below as Algorithm 2.

When the relation matrix is updated in the seventh step, the propagation delay between the switch and the controller is less than the maximum propagation delay, and the switch node is not migrated; then, the load of the controller is considered, and a migration strategy is made. Figure 4 shows the flowchart of the switch migration policy.

Algorithm 2 Switch node migration strategy for controller load balancing.

Input: The longitude and latitude matrix C of the controller(ground station) is obtained by Algorithm 1, The satellite longitude and latitude matrix S and height matrix $S_H$ of 100 time slices, $SF$ packet-IN flow of switch nodes and the maximum capacity $CF$ of the controller increasing linearly with time slices.

Output: The time slice T, the average propagation delay L, the relation matrix R of the satellite corresponding to the controller, and $CF$ the load of the controller.

1: $controller\_xyz\leftarrow miller\_to\_xyz\left(C\right)$

2: for $iinrange(1:timeslice+1)$ do

3:      $S=readswitchinformation\left(i\right)$

4:      $position\_xyz\leftarrow miller\_to\_xyz\left(S\right)$

5:      $L\leftarrow calculate\_delay(controller\_xyz,pos\_xyz)$

6:      $Linearlyincreasesswitchtraffic$

7:      $\mathbf{R},\mathbf{CF}\leftarrow \mathbf{UpdateRelationMatrix}(\mathbf{L},\mathbf{SF},\mathbf{R})$

8:      $save(R,T,CF,L)$

9: end for

Figure 4. Flow chart of switch migration policy.

First, the process begins to input the delay from all switches to all controllers, and determines whether there is a switch node that is larger than the maximum propagation delay, and if there is a target controller that is reallocated to minimize the propagation delay; if it does not exist, all switches are judged to be bad. In the next step, according to the relation matrix obtained in the previous step, the load of the controller is counted, and then it is judged whether the controller achieves load balance: one is all the controller load less than 85% of the maximum load, and the other is to find out the variance of all the controller load, to determine whether the variance is less than a parameter (according to the specific experimental settings). If the condition that the controller satisfies the load balance is satisfied, the calculation of the next time slice is finished. If the load balance condition is not satisfied, the switching strategy is started. Firstly, the controller with the largest load is selected as the outgoing controller, and the switch with the largest load is selected as the outgoing switch; then, the minimum load controller is selected as the immigration controller, and then the propagation delay between the switch to be migrated and the immigration controller is determined to be less than the maximum propagation delay, and the switch migration is completed by the cooperation of the out-and-in controller and the in–out controller. Otherwise, the switch is cycled until the load balance is achieved. Finally, the program ends by returning the relational matrix and the controller load.

2.4. Algorithm Complexity Analysis

Firstly, considering the time complexity analysis of Algorithm 1 (multi-controller deployment strategy based on the simulated annealing algorithm), the algorithm is the outermost cycle of the temperature drop T, each temperature K iteration, each iteration process N switch, and M controllers; the time complexity is therefore $O\left(TKMN\right)$. For Algorithm 2, each time slice handles N switches and M controllers at a time, so the algorithm complexity is $O\left(MN\right)$.

Secondly, for the space complexity, Algorithm 1 needs to update the control deployment information N, the delay matrix $M\ast N$, the relation matrix $M\ast N$ and the loss function value, so the space complexity is $O(N+2\ast M\ast N)$. For Algorithm 2, the relational matrix and the controller load need to be stored so the controller complexity is also $O(N+2\ast M\ast N)$.

3. Results

3.1. Experimental Scene Description

The iridium constellation network is used as the data surface of SDN in the simulation scene, and the controller is deployed at the ground gateway station. The iridium constellation consists of 66 working satellites in six polar circular orbits, that is, 11 in-orbit satellites in each orbit plane, separated by an interval of about 4500 km. The satellites are located in low-earth polar orbit 780 km above the Earth and orbit the Earth once every 100 min at a speed of 27,070 km per hour. The constellations are shown in Table 1.

Table 1. Iridium system parameters.

Parameters	Value	Parameters	Value
Orbital altitude	780 km	Satellite planes	6
Orbital type	Polar orbit	Satellites per plane	11
Inter-satellite link bandwidth	10 Mbps	inclination	86.4°
Angle difference between adjacent planes	31.6°	Phase factor of the adjacent plane	16.36°

The data source is from STK (Systems Tool Kit), which collects the latitude and longitude of satellite nodes and the altitude, latitude and longitude of the ground station. The iridium satellite system’s satellite cycle is about 100 min, the length of each time slice is set to 1 min, and the satellite constellation’s operating cycle will be divided into 100 time slices. The iridium constellation network topology generated by STK software is shown in Figure 5.

Figure 5. Iridium system network.

The other parameters are as follows: a single controller can handle up to 60,000 packet-IN request packets, that is, $Q_{max}=60,000(packet-IN)$, and $\rho$ is 0.85. The switch switchover occurs when the load reaches 85% of the server load to achieve load balancing. The weight ratio of load factor is $(\alpha ,\beta ,\gamma )=(0.8,0.1,0.1)$. The maximum propagation delay from the satellite node to the controller is $L_{max}=45$ ms. The weight ratio of propagation delay to load is $(\mu ,\gamma )=(0.6,0.4)$.

3.2. Result Analysis

3.2.1. In the Case of Different Number of Controllers, the Load of Each Controller and the Maximum Propagation Delay from the Switch Node to the Controller Are Calculated

As shown in Figure 6 (to visualize how the controller handles packet-in, where the load indicates the packet-in packet received by the controller), each subgraph represents the current number of controllers, changes in load and maximum propagation delay for each controller. With the increase in the number of controllers, the maximum propagation delay from the switch to the controller decreases, and the load difference between the controllers decreases, in order to better analyze the relationship between the load and the propagation delay and the number of controllers, the graph of the relationship between the number of controllers and the variance and average delay of the controller load is given in Figure 7.

Figure 6. Diagram of delay and load variation for each controller.

Figure 7. Diagram of the number of controllers and the variation of load variance and delay.

Through the analysis of the graph above, the average delay of the whole network shows a downward trend with the increase in the number of controllers, which is in line with our theoretical formula, but combined with the analysis of the load value, the load of the six controllers is the best at the time of load; however, the propagation delay is not optimal at this time. When the transmission delay changes from seven controllers to eight controllers, the load decreases, but the delay increases. This is also in line with our theoretical derivation, and the number of controllers can be determined according to the minimum required delay and the optimal load. If the minimum propagation delay is less than 20 (ms), the number of controllers can be determined according to the minimum required delay and load; if the controller load variance is less than 6000, you can choose seven controllers.

3.2.2. The Controller Load Varies with the Packet-In Request Packet of the Switch

To better analyze the switching process, the packet-IN request packets of the switch node are linearly increased by 100 packets per time slice in a simulation. On the basis of the first experiment, after reaching the optimal number and position of controller deployment, Algorithm 2 should be run once per time slice. The simulation results show that the controller load changes linearly with packet-IN request packets (time slice). The results of the simulation are shown in Figure 8.

Figure 8. Diagram of controller load versus time slice.

Due to the linear growth of switch traffic with time slice, the controller load also shows a linear growth trend. The position of the curvilinear punctuation marks the time slice in which the switch node migration occurs, where the time slice {5–12} is due to the variance of all controller loads greater than 6000 (packet-IN), leading to the switch node migration, and the time slice {17,19,20} is migrated by 85% of the switch nodes with C2 and C7 loads greater than the maximum load. The specific information is shown in Table 2.

Table 2. Switch migration information.

Time Slice	Switch No.	Src Controller No.	Des Controller No.
5	S1	C2	C7
	S5	C3	C1
6	S8	C2	C6
7	S14	C2	C7
8	S15	C2	C1
9	S16	C2	C6
10	S19	C2	C1
11	S7	C2	C1
12	S11	C2	C5
17	S4	C2	C6
19	S1	C7	C1
20	S26	C2	C5
	S12	C7	C6

Figure 9 shows the change in controller load variance before and after switch migration. In order to show the simulation effect better, the results show the change of the first 14 time slices. As can be seen from the diagram, the variance of the controller load increases with the time slice from 1 to 4. 5 to 12 time slices, controller load variance over 6000, triggered switch migration, and with controller load variance that decreased significantly. After the 13th time slice, the controller load variance decreases gradually, and no switch migration occurs.

Figure 9. Variation of controller load variance before and after switch migration.

3.2.3. Compare the Load Variances of Different Algorithms with Time

In this contrast experiment, OCLDS [26] and n-k-means [31] were selected as the contrast experiments; the former divides the load into four states, with a switch migration strategy, and the latter has no migration strategy. Each time slice requires an algorithm to be run to redistribute the relationship between the switch and the controller.

As shown in Figure 10, this strategy is compared with the other two algorithms in terms of controller load variance. First, the size of the comparison is obviously the best result obtained by the n-k-means strategy, and the load variance of the first 14 time-slice controllers does not change much. The main reason for this is that each time slice needs to re-execute the algorithm and re-distribute the relationship between the switch and the controller. Although this result is the best, it takes so long that there is not enough time to complete the switch migration before the next time slice starts. Secondly, as the analysis above, the switch migration occurs from the fifth time slice, and the controller load variance is equal to n-k-means, which is better than the OCLDS algorithm, and the controller load variance obtained by this strategy and OCLDS is increased first and then decreased, but the latter is delayed by several time slices. The reason for this is that the switch migration is restricted by the controller load variance and 85% of the maximum controller load, while OCLDS only controls the switch migration by the controller load segmentation; therefore, the performance of this strategy is improved by about 7.71% compared to OCLDS (calculated here by averaging the difference between the two time slices). Finally, because this strategy and OCLDS have developed a strategy to trigger the switch migration, it takes less time to complete the migration before the next time slice than the redistribution of n-k-means.

Figure 10. Variation of controller load variance with time slice for different algorithms.

3.2.4. Compared with Different Algorithms, the Running Time of the Migration Algorithm Changes with the Increase in Switch Nodes

In addition to the 66 polar-orbiting satellites in the iridium constellation, the switch node uses the satellite data from the Starlink program and in turn uses STK to simulate the four Walker constellations (inclined orbits), a hybrid orbital constellation model (inclined and polar orbits) with the parameters in Table 3.

Table 3. Detailed parameters for different constellations.

Constellation Name	Altitude (km)	Orbital Planes	Satellites per Plane	Total Number of Satellites
Walker 1	580	4	43	172
Walker 2	550	11	22	242
Walker 3	580	4	43	172	414
	550	11	22	242
Hybrid orbit constellation	570	36	20	720	786
	780	6	11	66
Walker 4	550	72	22	1584

As shown in Figure 11, the scattered points represent the number of nodes corresponding to the algorithm running time and running time of the calculation: the information of a cycle of each time slice of the running time is given as the average value. The curve represents the polynomial fitting of scattered points, and the running time of the three algorithms increases with the number of nodes. According to the time complexity analysis, the migration strategy of this paper and OCLDS is O(m), so the fitting curve shows a linear increasing trend. The time complexity of n-k-means is approximately O(n * m), where m is the number of controllers and n is the node of switches. In addition, when the number of nodes exceeds 1000, n-k-means uses more than half of a time slice to figure out the migration strategy, and it is highly likely that the controller fails to complete the migration within a time slice. The other two strategies are significantly faster than n-k-means.

Figure 11. The running times of different algorithms vary with the number of switch nodes.

4. Conclusions

Aiming to solve the problems of a large number of nodes and dynamic changes in the network topology of the SDN-based hierarchical space–air–ground integrated network scenario, this paper first constructs a delay model, a load model, and a loss function model. The loss function model is defined by two factors, namely the propagation delay from the controller to the switch node and the load of the controller. Then, a multi-control deployment strategy based on simulated annealing is proposed, and a strategy for dynamic switch migration with the change in time slice is proposed. In the multi-control deployment strategy, the loss function is used as a measure of algorithm convergence. In the switch migration strategy, the introduction of the controller load variance is used as a condition for judging whether the controller load satisfies the balance. Finally, the experimental part of this paper compares and analyzes the changes in controller load and average delay under different numbers of controllers, and a specific deployment plan is given under the condition of specifying the maximum delay and maximum controller load. In addition, to verify the reliability of the switch migration strategy, a comparison experiment before and after the switch migration load was also conducted. To verify the superiority of the algorithm, a comparison experiment was carried out with the two algorithms of n-k-means and OCLDS in terms of the load performance and running time of the controller, and the experimental data were all derived from reliable data simulated by STK. The results show that the algorithm in this paper improves the load performance of the controller by about 7.71% compared with the other two algorithms.

Our future work will introduce the vehicle and UAV into our SAGIN infrastructure [35], and discuss the mobile controller equipped on the vehicle and/or UAV to fulfill the service requirements from ground and space area.