Software-Defined Virtual Private Network for SD-WAN

Abstract

Software-Defined Wide Area Networks (SD-WANs) are an emerging Software-Defined Network (SDN) technology to reinvent Wide Area Networks (WANs) for ubiquitous network interconnections in cloud computing, edge computing, and the Internet of Everything. The state-of-the-art overlay-based SD-WANs are simply conjunctions of Virtual Private Network (VPN) and SDN architecture to leverage the controllability and programmability of SDN, which are only applicable for specific platforms and do not comply with the extensibility of SDN. This paper motivates us to refactor traditional VPNs with SDN architecture by proposing an overlay-based SD-WAN solution named Software-Defined Virtual Private Network (SD-VPN). An SDN-based auto-constructed VPN model and its evaluating metrics are put forward to automatically construct overlay WANs by node placement and service orchestration of SD-VPN. Therefore, a joint placement algorithm of VPN nodes and algorithms for overlay WAN service loading and offloading are proposed for SD-VPN controllers. Finally, a three-layer SD-VPN system is implemented and deployed in actual network environments. Simulation experiments and system tests are conducted to prove the high-efficiency controllability, real-time programmability, and auto-constructed deployability of the proposed SD-VPN. Performance trade-off between SD-VPN control channels and data channels is evaluated, and SD-VPN controllers are proven to be extensible for other VPN protocols and advanced services.

1. Introduction

SD-WAN is now widely applied for network interconnection in various application scenarios, ranging from the heterogeneous connectivity of Internet Service Providers (ISP) [1] to interconnected data centers [2], security solutions in cloud computing [3,4], network interconnection of enterprise branches [5], municipal systems connection [6], and the Internet of Everything in the Internet of Things [7]. Compared to the traditional network interconnection technologies of WAN, with dedicated lines or devices, SD-WAN is superior in its low cost, rapid deployment, convenient management, remote maintenance, and agile updating [8].

From the perspective of service objects, SD-WAN can be further classified into ISP-level SD-WAN and enterprise-level SD-WAN in the form of underlay-based and overlay-based implementation, respectively [9]. Underlay-based SD-WANs are commonly constructed by commercial SDN devices and deployed for restrictive network environments only. In contrast, overlay-based SD-WANs are suitable for deployment in almost all kinds of clouds, edges, and endpoints due to their flexibility for asymmetric network environments and extensibility for heterogeneous network devices.

This paper focuses on a mainstream implementation of overlay-based SD-WANs by utilizing a VPN for network interconnections across WANs and applying SDN for controllability, programmability, and extensibility. The state-of-the-art research on SDN-based VPNs is simply conjunctions of VPN and SDN architecture to leverage the controllability of SDNs [10,11,12]. The SDN-based VPN solutions of these studies are basically transplanted on open-source VPN projects or implemented by commercial VPN service components of ISPs, which are only applicable for specific platforms and not extensible for supplementary VPN services.

Considering the limitations of the existing SDN-based VPNs mentioned above, this paper motivates us to refactor traditional VPNs with SDN architecture for the automated construction of overlay WANs and to design a basic VPN controller enabling node placement, service orchestration, and extensibility for other VPN protocols and advanced services. Therefore, this paper proposes SD-VPN, an overlay-based SD-WAN solution to construct overlay WANs by the on-demand placement of VPN nodes and automated orchestration of VPN services, routing rules, and QoS strategies. An SDN-based auto-constructed VPN model of SD-VPN is put forward with its evaluating metrics. Then, an SD-VPN controller is put forward by introducing its modules and interfaces, algorithms of node placement, and service orchestration. A joint placement algorithm of SD-VPN controllers and gateways is proposed for high-efficiency controllability and low-latency transmission. Meanwhile, algorithms of overlay WAN service loading and offloading are proposed to orchestrate VPN services, routing rules, and Quality of Service (QoS) strategies. Finally, a three-layer SD-VPN system is implemented and deployed in actual network environments. Simulation experiments and system tests are conducted to prove the high-efficiency controllability, real-time programmability, and auto-constructed deployability of the proposed SD-VPN. Performance of node placement, service orchestration, and data transmission are evaluated following metrics of SD-VPN. In addition, performance trade-offs between SD-VPN control channels and data channels are analyzed and SD-VPN controllers are proven to be extensible for other VPN protocols and advanced services.

The main contributions of this paper are as follows:

• Refactoring traditional VPNs with SDN architecture by proposing an overlay-based SD-WAN solution named SD-VPN and including its evaluating metrics;
• Proposing a joint placement algorithm of SD-VPN controllers and gateways for high-efficiency controllability and low-latency transmission;
• Proposing algorithms of overlay WAN service loading and offloading to orchestrate VPN services, routing rules, and QoS strategies;
• Implementing an SD-VPN system and evaluating it in both simulation environments and actual network systems.

The rest of this paper is organized as follows: Section 2 provides the related work. Section 3 proposes SD-VPN and its evaluating metrics. Section 4 introduces the design of SD-VPN controllers, including modules and interfaces, algorithms of node placement, service loading, and offloading. Section 5 represents an SD-VPN system and then Section 6 introduces experiments, system tests, evaluations, and discussions of the proposed SD-VPN. Section 7 concludes this paper.

2. Related Work

This section introduces state-of-the-art investigations on VPN and SD-WAN, respectively, summarizes the limitations of existing SDN-based VPN solutions, and puts forward the motivations of this paper.

2.1. Research on VPNs

Research on VPNs can be divided into four aspects: improvements of VPN protocols [13,14], optimizations of VPN gateways [15,16,17], innovations of VPN systems [18,19,20,21,22,23,24,25], and evaluations of VPN performance [15,26,27,28,29]. The last two aspects are related to our research.

In the aspect of innovations of VPN systems, applying SDN architecture is a priority for the low cost and complexity of VPN configuration, deployment, and maintenance. To enhance centralized management of enterprise branches, Elizabeth et al. [18] established a dynamic multipoint VPN solution for SD-WANs by applying open-source protocols: multipoint generic routing encapsulation, IPsec (Internet Protocol Security) encryption, and the next hop resolution protocol. To reduce the running time of federated VPNs for specific ISPs, Mostafaei et al. [19] proposed an SDN-based VPN framework to create federated networks, set up VPN services, and allow customers to join or leave the service. To lower the complexity of VPN services’ definition and management, Mirkhanzadeh et al. [20] proposed a software-defined networking solution for service providers offering MPLS VPN and VPLS services. To leverage the programmability of SDNs, Lospoto et al. [21] improved the realization of MPLS VPNs based on SDNs to bring about a smooth provision, setup, and management experience for ISPs. In addition, other innovations include reducing the complexity of configuration [22] and enhancing the security [23], mobility [24], and stability [25] of VPN systems.

In regards to evaluations of VPN performance, the evaluating metrics of VPN connections refer to establishment time, Round-Trip Time (RTT), TCP and UDP throughput, jitter, and packet loss rate. Fu et al. [15] designed a user space software VPN gateway and then evaluated the performance of TCP throughput, UDP throughput, UDP packet forwarding rate, and jitter for both the relay mode and agent mode of VPN gateways, respectively. Kjorveziroski et al. [26] evaluated full-mesh VPN solutions by testing the TCP throughput, UDP throughput, and response time of applications in different full-mesh network topologies. Chua an Ng [27] investigated the performance of VPN remote access by conducting experiments on three popular open-source VPNs and confirmed that the implementation of a VPN is an important influence factor. Pudelko et al. [28] evaluated three open-source software VPN gateways, analyzed the effects of software architecture, and determined bottlenecks of data structures and multi-core synchronization. Wu and Xiao [29] explored the impacts of three basic topologies on the performance of VPNs, including chain cascade, star, and tree topologies.

2.2. Research on SD-WANs

Research on SD-WANs can be classified into five categories: traffic engineering [30,31,32], network security [33,34,35], network systems [36,37,38], node placement [39,40,41,42,43,44,45], and service orchestration [46,47,48,49,50]. The last two categories are related to our research.

In terms of node placement, control latency is the primary optimization objective. Dou et al. [39] defined a controller placement with a switch-controller mapping solution and developed a programmability explorer by calculating the programmability of critical flows at switches to optimize the control latency of SD-WANs. Qi et al. [40] optimized SD-WAN control latency by rationally placing controllers, establishing switch-controller mapping, and developing a heuristic algorithm to achieve a trade-off between network performance and time complexity. Adekoya and Aneiba [41] applied and improved an evolutionary algorithm, called Non-dominated Sorting Genetic Algorithm III, from mechanical engineering to achieve high convergence and diversification of controller placement. Chakraborty et al. [42] designed a distributed scheme for a coalition formation game and social choice theory, which is able to optimally place controllers and periodically assess the placement of controllers on real-time network traffic. Other complementary optimization objectives include the capacity of controllers [43], failure of nodes [44], and the transmission latency of data planes [45].

In terms of service orchestration, refs. [46,47,48] the focus is on Service Function Chaining (SFC). Jiang et al. [46] formulated an SFC problem considering the heterogeneity of geographically distributed SD-WANs and designed heuristic algorithms to deploy SFC requests in batches. Leivadeas et al. [47] established collaboration and information exchange between enterprise branches and networks by configuring the security, data privacy, and routing services of Amazon SD-WAN services, examining and evaluating overall performance. Zhang et al. [48] proposed a service offloading method for jointly allocating communication and computation resources based on cloud–edge collaboration in SD-WANs. In addition, refs. [49,50] are related to service orchestration platforms. Perez et al. [49] set up a flexible, resilient, and Cloud-native SD-WAN orchestration solution for enterprise and academic networks purely based on open-source tools. Kone and Kora [50] put forward a practical approach for management and orchestration based on open-source platforms and evaluated their proposed testbed by orchestrating the services of the Voice over Internet Protocol.

2.3. Summary

To summarize, the most relevant pieces of research for SDN-based VPNs are [18,19,20,21]. The limitations of these studies are that they are simply conjunctions of VPN and SDN architecture to leverage the controllability and programmability of SDNs. In addition, the implemented software-defined VPNs are basically transplanted on open-source VPN projects or improvements of commercial VPN components of ISPs, which are only applicable for specific platforms and do not comply with the extensibility of SDNs.

Therefore, the motivations of this paper are to refactor traditional VPNs with SDN architecture for the automated construction of overlay WANs and to design a basic VPN controller enabling node placement, service orchestration, and extensibility for other VPN protocols and advanced services.

3. Software-Defined Virtual Private Network (SD-VPN)

In this section, an SDN-based auto-constructed VPN model named SD-VPN is proposed and the construction of overlay WANs by SD-VPN is formulated. Finally, the evaluating metrics of the SD-VPN are put forward.

3.1. Network Model

Figure 1 illustrates the network model of the SD-VPN, with the remote control of VPN nodes through VPN control channels by VPN controllers and the on-demand creation of overlay networks over underlay networks with VPN data channels. The overlay networks are divided into the control plane and data plane, which are composed of VPN controllers, gateways, and clients, respectively. An overlay WAN is constructed through VPN data channels by a VPN controller on the demand of users.

3.2. Problem Statement

As Figure 1 depicts an overlay-based SD-WAN solution, network interconnection across WANs can be formulated as the construction of overlay WANs by SD-VPN, mainly referring to SD-VPN node placement and SD-VPN service orchestration.

SD-VPN node placement refers to the placement of VPN controllers, gateways, and clients. VPN controller placement, similar to the controller placement problem in SD-WAN [37], directly affects the remote controllability and programmability of overlay WANs. VPN gateway placement determines the transmission performance of VPN data channels and the efficiency of VPN routing and forwarding, resembling optimization of transmission latency by dynamically selecting relay nodes in [45]. VPN client placement is relatively fixed on the demands of users. Therefore, VPN node placement is a joint placement of VPN controllers and gateways to balance the control latency of VPN control channels and the transmission latency of VPN data channels simultaneously.

SD-VPN service orchestration refers to the orchestration of VPN services, routing rules, and QoS strategies to load or offload the services of overlay WANs. The procedures of orchestration, guided by VPN controllers and involved with other VPN nodes, refer to the preparation, distribution, and execution of SFCs [46,47,48]. Detailed SFCs of VPN services include attributes of VPN nodes, information of VPN users, executable programs and configuration files of VPN connections, and cryptography of VPN data channels. When an existing topology updates, the SFCs mentioned above are re-prepared, re-distributed, and re-executed.

To conclude, an SD-VPN is an overlay-based SD-WAN solution to the construction of overlay WANs by the on-demand placement of VPN clients, rational placement of VPN controllers and gateways, and automated orchestration of VPN services, routing rules, and QoS strategies.

3.3. Evaluating Metrics

The proposed SD-VPN can be evaluated from three aspects: performance of node placement, service orchestration, and data transmission.

SD-VPN Node placement, defined as a joint placement of VPN controllers and gateways to balance control latency and transmission latency, is essentially similar to the controller placement problem in [39,40,41,42] and the relay node selection in [45]. The performance metrics are time and the computational complexity of node placement, which are mainly affected by network size and link delay.

The performance of SD-VPN service orchestration depends on the time consumption of service loading and offloading according to a comprehensive survey of network service orchestration [51]. The detailed metrics are the time consumption of overlay WAN service loading and service offloading, which rely on network size, topological structure, and link delay.

The performance metrics of SD-VPN data transmission focus on RTT, TCP throughput, UDP throughput, and jitter, which are almost equivalent to those of traditional VPN connections [15,28]. It is worth mentioning that SD-VPN controllers influence the data transmission of SD-VPN data channels differently under different work states. The influences should be discussed separately when controllers are at leisure with a tunnel probing protocol [52] only and when controllers are busy with transactions.

4. SD-VPN Controller

This section introduces the design of the system and algorithms for SD-VPN controllers. Modules and interfaces are detailed and illustrated. Then, a joint placement for VPN controllers and gateways is proposed. Finally, algorithms of overlay WAN service loading and offloading are put forward.

4.1. Modules and Interfaces

As shown in Figure 2, an SD-VPN controller is implemented by three modules and four interfaces, including a node placement module, VPN service orchestrator, topology monitor, interface of management plane and control plane, management message middleware, control message middleware, and interface of control plane and data plane.

The node placement module records attributes of underlay nodes, places VPN nodes, and monitors the state of nodes after the deployment of VPN services. The VPN service orchestrator is the core module of an SD-VPN controller managing VPN-related SFCs to load, offload, and update overlay WAN service. VPN-related SFCs are composed of user-defined SFCs and automatically generated SFCs. User-defined SFCs subscribe to messages from management message middleware to fill in information about VPN users, category of VPN protocols, cryptography of VPN services, and attributes of VPN nodes. Automatically generated SFCs including VPN configuration files, executable programs, routing rules, and QoS strategies are prepared for runtime VPN services. The topology monitor is activated after overlay WAN service loading to constantly monitor the state of nodes, connections, and traffic.

Other interfaces can be divided into the communication interface and message interface. The interface of the management plane and control plane is a management channel of SD-VPN. The interface of the data plane and control plane is a control channel of SD-VPN. Management message middleware is a broker for message registration from users and a message subscription from the VPN service orchestrator. Control message middleware is a classifier for control message encoding and decoding.

4.2. SD-VPN Node Placement

As introduced in Section 3.2, VPN clients are placed to interconnect networks on the demand of users so that VPN client placement is relatively fixed. Hence, a joint placement of VPN controllers and gateways is proposed in Algorithm 1. The inputs are graph $G\left(N,E\right)$, fixed VPN clients ${ N}_{cl}$, required control latency ${ L}_c,$ and required transmission latency $L_t$. The outputs are placement of VPN gateways $N_g$ and VPN controllers $N_C$. VPN gateway placement is a minimum k-median problem to achieve average-case transmission latency between VPN gateways and clients less than requirement $L_t$. VPN controller placement is a minimum k-center problem to guarantee that worst-case control latency is acceptable under ${ L}_c$. The time complexity of Algorithm 1 is $O\left(\left|N_C\right|·\left(\left|N\right|-\left|N_C\right|\right)\right)$ so that the best time complexity is $O\left(\left|N\right|\right)$ when the number of VPN controllers equals one.

Algorithm 1 Algorithm of a joint placement of VPN controllers and gateways
Input: The graph of a VPN topology, $G\left(N,E\right);$ The VPN nodes $N$ include controllers $N_C$, gateways $N_g,$ and clients ${ N}_{cl}$, $N=N_C\cup N_g\cup { N}_{cl};$ The number of VPN nodes of set $N$$, \left|N\right|$; The latency between nodes $E$ includes control latency $E_c$ and transmission latency $E_t;$ The requirement of control latency ${ L}_c$ and transmission latency $L_t;$ Output: $N_g,N_C$
1	set $N_C\leftarrow \varnothing , N_g\leftarrow \varnothing {,N}_{cl}\leftarrow \varnothing$
2	place $N_{cl}$ on demands of users
3	for each ${n\in N}_{cl}$ do
4	find $\frac{1}{\left|N_g\right|}\sum _{{n\in N}_{cl}}\mathit{min}d(n,m)$ $<=R_t$
5	add the gateway node $m$ into set $N_g$
6	for each $p \in N_g\cup N_{cl}$ do
7	find $\frac{1}{\left|N_C\right|}\sum _{p \in N_g\cup N_{cl}}max\mathit{min}d(p,q)$ $<=R_{tc}$
8	add the controller node $q$ into set $N_C$
9	return$N_g,N_C$

4.3. SD-VPN Service Orchestration

As depicted in Figure 2 of Section 4.1, VPN-related SFCs including information on VPN users, category of VPN protocols, cryptography of VPN services, attributes of VPN nodes, VPN configuration files, VPN executable programs, VPN routing rules, and VPN QoS strategies are symbolized in

Table 1. Symbols and descriptions for VPN-related SFCs.

Categories	Symbols	Descriptions
user-defined SFCs	$A$	Attributes of VPN nodes
	$U$	Information on VPN users and requirements for QoS
	$\mathbb{C}$	Category of VPN protocols
	C	Cryptography of VPN services
automate-generated SFCs	F	VPN configuration files
	$P$	VPN executable programs
	$R$	VPN routing rules, including Generic Routing Encapsulation (GRE) tunnels, static routing rules, and Network Address Translation (NAT) forwarding rules
	$Q$	VPN QoS strategies

. Algorithms of overlay WAN service loading and offloading are described in Algorithm 2 and Algorithm 3, respectively. The time complexity of Algorithm 2 and Algorithm 3 are $O\left(\left|N_g\right|+\left|{ N}_{cl}\right|\right)$.

Overlay WAN service loading is conducted after node placement so that the inputs of Algorithm 2 are VPN nodes, an under topology, and demands of users in the format of user-defined SFCs. The attributes of VPN nodes are first checked and VPN executable programs and VPN configuration files are generated according to user-defined SFCs. Then, VPN services of gateways and clients are started in order to achieve the parameters of communication addresses, ports, and virtual addresses, which are used for subsequent configuration of VPN routing rules and VPN QoS strategies. Finally, the attributes of VPN nodes are updated and the overlay WAN service has been loaded.

Compared to overlay WAN service loading, the service offloading algorithm is relatively simple, as described in Algorithm 3. The attributes of VPN nodes are first checked and then VPN routing rules are configured and QoS strategies are removed. Finally, VPN clients and gateways stop VPN services in order and the attributes of VPN nodes are updated.

Algorithm 2 Algorithm of overlay WAN service loading
Input: Graph of an underlay topology, $G\left(N,E\right);$ The VPN nodes $N$ include controllers $N_C$, gateways $N_g,$and clients${ N}_{cl}$, $N=N_C\cup N_g\cup N_{cl};$ User-defined SFCs, $A$, $U$, $\mathbb{C}$, $C$ Output: Graph of an overlay WAN topology, $\overline{G}\left(\overline{V},\overline{E}\right);$
1	for each ${n\in N}_g\cup N_{cl}$ do
2	check $A$ of $n$
3	if $n$ is invalid or offline then
4	return;
5	$\mathrm{generate} \left(P,F\right) \mathrm{w}\mathrm{i}\mathrm{t}\mathrm{h}\left\{A,U,\mathbb{C},C\right\}$;
6	for each ${p\in N}_g$ do
7	$p$ $\mathrm{execute} (P,F)$;
8	for each ${q\in N}_{cl}$ do
9	$q$ $\mathrm{execute} (P,F)$;
10	for each ${m\in N}_g\cup N_{cl}$ do
11	$\mathrm{generate} (R,Q)$ with results of executing $(P,F)$;
12	$m$ add $(R,Q)$;
13	update $A$ of $m$;
14	return$\overline{G};$

Algorithm 3 Algorithm of overlay WAN service offloading
Input: Graph of an underlay topology, $G\left(N,E\right);$ Graph of an overlay WAN topology, $\overline{G}\left(\overline{V},\overline{E}\right);$ The VPN nodes $N$ include controllers $N_C$, gateways $N_g,$ and clients${ N}_{cl}$, $N=N_C\cup N_g\cup N_{cl};$ User-defined SFCs, $A$, $U$, $\mathbb{C}$, $C$
1	for each ${n\in N}_g\cup N_{cl}$ do
2	check $A$ of $n$
3	if $n$ is invalid or offline then
4	return;
5	for each ${p\in N}_{cl}$ do
6	$p$ $\mathrm{delete} (R,Q)$;
7	$p$ $\mathrm{stop} P$;
8	update $A$ of $p$;
9	for each ${q\in N}_g$ do
10	$q$ $\mathrm{delete} (R,Q)$;
11	$q$ $\mathrm{stop} P$;
12	update $A$ of $q$;

5. SD-VPN System

In this section, a three-layer SD-VPN system shown in Figure 3 is implemented with web services for the management plane, controllers for the control plane, and gateways and clients for the data plane.

SD-VPN web services are designed to implement graphical management and configuration for users to define topologies, protocols, routing, and QoS for an overlay-based SD-WAN. SD-VPN controllers are designed according to Section 4 in this paper. The implementation of VPN gateways refers to our previous work in [15], including detailed design of VPN protocols, VPN sessions, VPN routing, and VPN NAT. The implementation of VPN clients refers to another of our work in [25].

In addition, it should be noted that the SD-VPN system is extensible for optional VPN protocols. Public VPN protocols including the Point-to-Point Tunneling Protocol (PPTP), Internet Protocol Security (IPsec), Secure Sockets Layer (SSL), and private VPN protocols of [15] have been integrated and made available in our proposed system as SFCs of SD-VPN controllers.

6. Evaluations

In this section, experiments and system tests are conducted to evaluate the performance of node placement, service orchestration, and data transmission of the SD-VPN. Both advantages and disadvantages are discussed and future work is clarified.

6.1. Experiments and System Tests

Experiments are carried out on the platform MATLAB_R2018a using a MacBook Pro to simulate four typical SD-WAN topologies with different sizes of networks and numbers of VPN nodes to test the performance of SD-VPN node placement. The system configuration of the simulation environment is shown in

Table 2. Details system configuration of the simulation environment.

Names	Parameters	Descriptions
MacBook Pro	CPU	2.3 GHz 8 cores Intel Core i9
	GPU	Radeon Pro 560X 4 GB, Intel UHD Graphics 630 1536 MB
	RAM	16 GB 2400 MHz DDR4
	OS	macOS 14.4.1 (23E224)

. Network parameters mentioned in Section 4.3 are set in

Table 3. Network Parameters of small, medium, large, and extra-large network topologies.

Parameters	Small	Medium	Large	Extra-Large
$\left|N\right|$	30	100	500	1000
$\left|N_g\right|$	3	10	20	50
$\left|N_C\right|$	1	3	5	10
${ L}_c$	20 ms	30 ms	30 ms	50 ms
$L_t$	20 ms	30 ms	30 ms	50 ms

. Detailed experimental results are evaluated in Section 6.2.

System tests are conducted on our SD-VPN online system, geographically deployed on Qingdao Alibaba Cloud, Weihai Tianzhiwei Cyberspace Security Technology Co., Ltd. (Weihai, China), and Cyberspace Security Laboratory of Harbin Institute of Technology, Weihai. The VPN controller and VPN gateway are pre-deployed on the cloud server. Two VPN clients are deployed on gateway devices in the enterprise network and campus network, respectively. The performance of SD-VPN service orchestration and SD-VPN data transmission are evaluated on this online system. The system configuration and network topology are shown in

Table 4. Detailed system configuration of SD-VPN online system.

Names	Parameters	Descriptions
Cloud server	CPU	Intel (R) Xeon(R) CPU E5-2680 v3 @ 2.50 GHz
	RAM	1G
	NIC	Red Hat, Inc Virtio network device
	OS	Linux version 3.10.0-514.16.1.el7.x86_64(Red Hat 4.8.5-11)
Gateway device	CPU	Intel(R) Atom (TM) CPU D525 @1.80 GHz
	RAM	1G
	NIC	Intel Corporation 82583 V Gigabit Network Connection
	OS	Linux version 3.10.0-514.16.1.el7.x86_64(Red Hat 4.8.5-11)

and Figure 4. The test results and evaluations are available in Section 6.3 and Section 6.4.

6.2. Evaluations of SD-VPN Node Placement

The proposed joint placement algorithm of VPN controllers and gateways is evaluated on different sizes of network topologies with the parameters set in Table 3. Experimental results and statistics are shown in Figure 5.

As shown in Figure 5a, the time consumption of placement for small networks is at the millisecond level while others are at the second level. Therefore, Algorithm 1 is available for both the online and offline calculations of node placement for small-size networks. When encountering medium networks, large networks, or even extra-large networks, this algorithm is more suitable for offline calculation.

In addition, we find the time consumption of node placement increases slowly from the medium network to the large network, while surging rapidly from the large network to the extra-large network. According to the means and variances calculated in Figure 5b, the mean value of time consumption of the extra-large network is 3.68 times that of the large network, while the mean value of time consumption of the large network is 1.83 times that of the medium network. Referring to network parameters set in Table 3, the number of nodes increases 2.5 times and 2 times from a large network to an extra-large network and from a medium network to a large network, respectively. In addition, the number of controllers increases 2 times and 1.67 times, respectively, under the same circumstances. Therefore, the placement of VPN controllers, including the number of controllers and their locations, is more complex compared to the placement of VPN gateways and should be emphasized in this joint placement problem.

6.3. Evaluations of SD-VPN Service Orchestration

To evaluate the deployability and extensibility of SD-VPN controllers, PPTP VPN, IPsec VPN, SSL VPN, and SD-VPN are tested to construct overlay WANs in our actual network environment described in Figure 4. Tests of overlay WAN service loading and offloading were carried out for ten rounds for each group, respectively.

The overall time consumptions of the test results are shown in Figure 6. All service loading tests are accomplished at the second level while all service offloading tests are completed at the millisecond level. It seems that the performance of service loading is relevant to VPN services while the performance of service offloading is irrelevant. Therefore, quantitative analysis for service loading and offloading is performed using SD-VPN, and detailed time consumption results are depicted in Figure 7. It is calculated that the time of control channel occupied, data channel occupied, and program execution in service loading is 34.58%, 63.27%, and 1.90%, respectively, while corresponding times are 89.65%, 8.40%, and 1.30% in service offloading, respectively. It is clear that the data channel uses more time in service loading than in service offloading. Therefore, the relevance between service loading and VPN services is related to the establishment time of the VPN service, which can be attributed to different designs of protocols and differences in cryptography.

6.4. Evaluations of SD-VPN Data Transmission

The performance of data transmission is tested in our online system to determine how controllers influence data channels under different work states. The tested SD-VPN systems are represented in

Table 5. Symbols and descriptions for tested SD-VPN systems in different work states.

Symbols	Descriptions
T-VPN	The SD-VPN is deployed as a traditional VPN without controllers
SD-VPN(L)	The SD-VPN controller is at leisure.
SD-VPN(R)	The SD-VPN controller is busy with routing-related transactions.
SD-VPN(Q)	The SD-VPN controller is busy with QoS-related transactions.
SD-VPN(V)	The SD-VPN controller is busy with VPN-related transactions.

to distinguish if a controller is at leisure or not. The open-source network tools Nping and Iperf are utilized to obtain the RTT, TCP throughput, UDP throughput, and jitter of data channels. For detailed test processes, refer to our previous work in [15]. Test results are shown in Figure 8.

It is observed that RTT is almost not impacted in Figure 8a. The other three metrics are impacted to different degrees. When the controller is at leisure, the TCP throughput of SD-VPN is 96.76% of that of T-VPN so the impact of controllers can be negligible. A similar situation occurs in the metrics of UDP throughput and jitter. When the controller deals with routing-related and QoS-related transactions, the TCP throughput of SD-VPN is 91.73% and 91.01%, respectively, of that of T-VPN. The UDP throughput and jitter of SD-VPN are obviously impacted according to Figure 8c,d. The worst situation occurs when the controller deals with VPN-related transactions, where the TCP throughput of the SD-VPN is 71.23% of that of the T-VPN. The UDP throughput of SD-VPN is 60.87% and 82.75% of that of T-VPN in the worst case and best case, respectively.

To conclude, SD-VPN controllers have impacts on the performance of SD-VPN data transmission to some extent, especially when controllers deal with VPN-related transactions. When controllers are at leisure or deal with other simple transactions, the impacts are basically acceptable considering their high-efficiency controllability, real-time programmability, and auto-constructed deployability.

6.5. Discussion and Future Work

Through experiments, systems tests, and evaluations of the node placement, service orchestration, and data transmission of SD-VPN, three points are worth discussing and exploring in future work.

In regards to node placement, controller placement is vital to network performance compared to the placement of other nodes. Due to the experimental results obtained and evaluated in Section 6.2, the proposed joint placement algorithm, referring to Algorithm 1, is only applicable for online calculations in small-size networks. Heuristic methods and deep learning algorithms are to be applied to achieve approximate optimal solutions for medium networks, large networks, and even extra-large networks. The other limitation of Algorithm 1 ignores the finite network resources except latency so that future work on the controller placement problem can be extended to multi-objective optimization for different scenarios when considering metrics of control latency, transmission latency, capacity of controllers, failure of controllers, load balancing, and cost.

In the aspect of service orchestration, the automated construction of overlay WANs for network interconnections by SD-VPN is fundamental. More importantly, SD-VPN is a ubiquitous model for software-defined security, privacy, and perimeter so that corresponding service orchestrators and SFCs are to be defined and designed.

In the aspect of data transmission, performance trade-off between control channels and data channels is challenging but interesting for diverse application scenarios, abundant network environments, various topology structures, and even heterogeneous smart devices.

7. Conclusions

The state-of-the-art overlay-based SD-WANs are simply conjunctions of VPN and SDN architecture to leverage the controllability and programmability of SDNs, which are only applicable for specific platforms and do not comply with the extensibility of SDNs. This paper refactors traditional VPNs with SDN architecture by proposing an overlay-based SD-WAN solution named SD-VPN, formulating an SDN-based auto-constructed VPN model and its evaluating metrics. Then, an SD-VPN controller is put forward with algorithms for node placement and service orchestration. A joint placement algorithm of SD-VPN controllers and gateways is proposed for high-efficiency controllability and low-latency transmission. Meanwhile, algorithms of overlay WAN service loading and offloading are proposed to orchestrate VPN services, routing rules, and QoS strategies. Finally, a three-layer SD-VPN system is implemented and deployed in actual network environments. Simulation experiments and system tests are conducted to prove the high-efficiency controllability, real-time programmability, and auto-constructed deployability of the proposed SD-VPN. The performance trade-off between SD-VPN control channels and data channels is evaluated and SD-VPN controllers are proven to be extensible for other VPN protocols and advanced services. Future work will focus on multi-objective optimized controller placement problems, and service orchestration for security, privacy, and perimeter.