Routing Performance Evaluation of a Multi-Domain Hybrid SDN for Its Implementation in Carrier Grade ISP Networks

: Legacy IPv4 networks are strenuous to manage and operate. Network operators are in need of minimizing the capital and operational expenditure of running network infrastructure. The implementation of software-deﬁned networking (SDN) addresses those issues by minimizing the expenditures in the long run. Legacy networks need to integrate with the SDN networks for smooth migration towards the fully functional SDN environment. In this paper, we compare the network performance of the legacy network with the SDN network for IP routing in order to determine the feasibility of the SDN deployment in the Internet Service provider (ISP) network. The simulation of the network is performed in the Mininet test-bed and the network trafﬁc is generated using a distributed Internet trafﬁc generator. An open network operating system is used as a controller for the SDN network, in which the SDN-IP application is used for IP routing. Round trip time, bandwidth, and packet transmission rate from both SDN and legacy networks are ﬁrst collected and then the comparison is made. We found that SDN-IP performs better in terms of bandwidth and latency as compared to legacy routing. The experimental analysis of interoperability between SDN and legacy networks shows that SDN implementation in a production level carrier-grade ISP network is viable and progressive.


Introduction
Networks are organized groups of devices or nodes with communication links among them. Traditional IP networks are termed legacy networks. Despite the world-wide use, the traditional network is complex and tedious to manage. Adding a new device or changing the network configuration is complex in traditional networking, that is, it should be implemented using low-level languages and rigid commands thus taking days and months for large networks. Thus, the expansion of the existing legacy network is fairly expensive. The demand for network based services is increasing rapidly. To meet those growing demands, the numbers of running hardware should be increased and changes should be made in software leading to a higher cost in system upgrades. Manual configuration, inconsistent policies and inability to scale are the major problems of the traditional IP networks. Vertical integration has made traditional IP networks more complex in their operation and management. Legacy networking devices have the control plane and data plane integrated into the same physical device. The data plane, also called the forwarding plane, is a hardware unit whose job is to collect the network packets and forward them to the destination based on the entries provided in the routing table. Network policies are enforced in the control plane. The control plane determines how the packets in the data plane should be handled, for example, drop, reject, forward and so forth.
In SDN, network devices are controlled by an SDN controller and the applications are installed on the top of the controller. When the control and data plane functionalities are segregated, the network switch becomes simply a data plane forwarding device. The control logic is implemented in a logically centralized controller, which runs a network operating system (NOS) that runs on commodity server hardware to provide necessary resources to facilitate the proper control and operation of data plane devices based on a logically centralized, abstract network view [1]. The logical centralization of the control logic makes it flexible and provides error-free policy deployment through high-level programming languages as an application programming interface (API), compared to the low-level device-specific configurations frequently used in legacy networks. The controller provides the global view of the overall network, which simplifies the development of more robust network functions, services and applications. The resulting network is also programmable through software applications running on top of the controller OS, which interact with the underlying data plane devices. The APIs at the controller can automatically react to spurious changes of the network state and keep high-level policies intact [1].
Though SDN implementation in datacenter networks is popular, its proper implementation with the migration of a legacy network to carrier grade ISP (CG-ISP) and telecommunication (Telcos) networks is becoming a central challenge for service providers due to the need for real time migration as well as the higher cost of investment [2,3]. Additionally, real time implementation of SDN in production networks is the subject of ongoing research. Hybrid SDN is the only solution for the smooth migration of legacy ISP/Telcos networks into SDN [4,5]. In this paper, an emulated network environment is created in Mininet for hybrid SDN implementation and routing performance evaluation by comparing legacy routing and SDN routing with their interoperability through an experimental analysis using an open network operating system (ONOS)/SDN-IP [6]. The viability of the SDN implementation in ISP/Telcos networks is then evaluated. The major contributions of this paper are as follows. • A multi-domain hybrid SDN environment is created and implemented in the routing between SDN and legacy networking domains. • Routing performance is evaluated, comparing legacy routing and SDN routing using ONOS/SDN-IP. Routing using SDN-IP has a better performance than legacy routing. • This experimental analysis evaluates the viability of SDN implementation in ISPs and Telcos networks. This encourages the service providers to smoothly migrate their legacy networks into SDN.
The rest of this paper is organized as follows. Section 2 presents the background and related work in the field of SDN and legacy network integration. Section 3 describes the research problem and the proposed method. We present the details of the experimental setup, analysis, and evaluations in Section 4. Section 5 provides a discussion and suggests future work, and the paper is concluded in Section 6.

Background and Related Work
The cost of new technology implementation is generally higher in terms of capital and operational expenditure (CapEX and OpEX) investment and the development of technical human resources (HR) to manage and operate those newer technologies for ISPs and Telcos service providers. There are also certain implementation challenges with respect to management, availability of technological standards and user interface provisioning when providing new technology-based services to customers during and after network migration [7,8].
A legacy network is less flexible to customized programming and is more vendor specific, leading to higher dependency on support, management and operation. A better solution would be the implementation of SDN. In SDN, networks are controlled by software applications running on the top of an SDN controller. The separation of control and data plane operations in SDN simply transforms the network switches into simple forwarding devices. NOS provides the essential resources and abstractions to facilitate the programming of forwarding devices based on a logically centralized, abstract network view. The resulting network is also programmable through software applications running on top of the NOS, which interact with the underlying data plane devices. ONOS [9] is the robust and distributed controller OS, which provides a better solution for building next-generation SDN/NFV solutions. The need for ISPs and Telcos service providers can simply be fulfilled by the introduction of ONOS to carrier-grade solutions that leverage the economics of white box merchant silicon hardware while offering the flexibility to create and deploy new dynamic network services with simplified programmatic interfaces [9]. ONOS is developed with a set of several other applications that make it flexible, modular, and scalable in terms of both the architecture and the cluster, and the distributed SDN controller. SDN-IP is one of the ONOS applications developed to implement routing with external legacy networks using the standard border gateway protocol (BGP) to enable interoperability with legacy routing. In ONOS/SDN-IP, the SDN can be treated as a single autonomous system (AS) and communicates simply with external AS, as with traditional routing. SDN-IP integrated BGP and ONOS enable communication with external AS in the hybrid SDN so that SDN-IP behaves like a regular BGP speaker and uses its services to install and update the appropriate forwarding state in the SDN data plane devices.
Dawadi et al. [3,[10][11][12] presented approaches for legacy network transformation to SDN and IPv6 networking so that service providers can smoothly plan for network transformation with optimised migration cost. The authors implemented IPv4/IPv6 routing in a multi-domain SoDIP6 network using ONOS/SDN-IP.
Friyanto [13] presented the use of multiple ONOS controllers in a cluster for high available services using ONOS/SDN-IP reactive routing. Efficient routing implementations in the different aspects of wired and wireless SDN [14][15][16][17] are the primary concerns, but their implementations in production level networks have to be evaluated considering the smooth transition from legacy to hybrid SDN to pure SDN.
There are many studies on hybrid SDN implementations. For example, HARM-LESS [18], Panopticon [19], RouteFlow [20], Fibbing [21] and OSHI [22] are some of the attempts, but most of this research has not been implemented at the production level. OSHI considers the hybrid switch, while our implementation only considers the hybrid SDN/legacy network as a mix network and routs implementations on that mixed network. Tomovic et al. [23] presented a modified version of three dynamic routing algorithms viz. MIRA, DORA and SWP, to be implemented with the OpenFlow controller. These routing modules are implementable only in an SDN environment.
Similarly, few studies have dealt with the implementation of SDN in CG-ISP networks using ONOS/SDN-IP [6,[24][25][26], even though ONOS is the best controller in terms of its features and dedication towards transport SDN over a wide area network [27]. Kong et al. (2013) [28] evaluated the performance of an OpenFlow switch in SDN by comparing it with legacy networking and found that OpenFLow based communication did not perform well. During that period, SDN was just at the emerging stage but now it is well matured [29][30][31] and robust. Research on implementing SDN in an ISP network is in progress [24,32]. SDN is popularly implemented in data center networks [33,34] and its implementation in ISP networks is part of ongoing research. Because we are at the early stage of SDN implementation in service provider networks, we found limited or no research related to routing performance comparison in terms of BGP implementation in legacy networks and SDN in the service provider networks. Our approach in this study is not scoped towards migration techniques, instead we focused on the production level implementation of routing in hybrid SDN and its performance in comparison with legacy network to measure the viability of legacy network transition towards SDN by implementing ONOS/SDN-IP.

Problem Description and Proposed Method
The overall structure of our experimental evaluation is based on the generic architecture of ONOS/SDN-IP implementation [35] as shown in Figure 1 and generated the experimental output of our research use case using Mininet as shown in Figure 2. All the external networks, that is, external ASs, are supposed to be the legacy networks, directly connected with backbone transit AS, which is SDN based. For routing purposes, the Quagga routing suite [36] was used, in which BGP was implemented. For SDN integration with a legacy network, the ONOS controller was configured to implement BGP and SDN-IP with reactive routing enabled. A complete legacy network was created as shown in Figure 3, where BGP and OSPF were used for routing purposes. Each router is a virtual machine loaded with BGP and OSPF configurations. In Figure 3, r5 is a route reflector router [37] and r4, r3, r6, r7, and r8 are its clients, whereas r4-r2 and r3-r1 are BGP peers. The SDN integrated legacy network and pure legacy network were chosen in the present study to analyze network performance. For comparison between them (hybrid SDN and legacy network), a distributed internet traffic generator (D-ITG) [38], a Packet InterNet Groper (PING), Iperf, and statistical rules were applied. A varying number and size of packets were created and transmitted from one host to another in each network, and the corresponding delay, latency and total packet transmission were observed. The PING utility was used to check the network connectivity between hosts. In both networks, first, PING tool was used to ensure total communication in the network; then it was used for finding out RTT. We used IPERF tool to determine the maximum bandwidths offered by the network. The compatibility and performance of SDN integration with the legacy network for multi-domain routing purposes were clearly depicted by the statistical values and comparison techniques required for SDN and the legacy network. Mininet emulated the hosts as a Linux machine. ONOS, running on the same machine, was used as an SDN controller.
The experimental setup of this study is depicted in Figure 4. The experiment was executed in a virtual environment for which we used Oracle VirtualBox as the hyper-visor. The guest operating system was allocated with 4 GB of base memory and two processor cores. The experiment was carried out on the Ubuntu 14.10 (64 bit) operating system. The network topology file used Mininet to create routers, hosts and OpenFlow switches. Quagga version 0.99.23 was configured in the routers for BGP routing in SDN, while OSPF and BGP were configured for the legacy network. Routers and hosts received data routed through the OpenFlow switches. The ONOS (version 1.2.1 user) interface provided the visualization of the network. DITG, PING, and IPERF tools were used for the data collection. The collected data were analyzed and visualized using the Python Matplotlib module. A brief descriptions of the APIs enabled over ONOS are follows.
• OpenFlow: It is the southbound protocol in the SDN control plane. It enables communications between the data plane and the control plane devices so that the control plane deploys flow rules in the OpenFlow table of the data plane and provides a decision for every packet incoming to the data plane devices. • Config: This application is activated for loading the network configuration. • ProxyARP: It is the process of resolving the address of a host. It responds to address resolution protocol (ARP) or neighbor discovery protocol requests on behalf of a target host if the target host is known to ONOS (i.e., it is present and active in the host store). ProxyARP considers the network to be a single IP network, therefore, if it receives a request for which the target host is not known, it will flood the request to every edge port in the network (apart from the input port) in the hope that the host exists in the network and will reply. • SDN-IP: It is an ONOS application for routing implementation that allows an SDN to connect to external networks on the internet using the standard BGP. After running those APIs, we can see the routes learned by ONOS, which can be seen using the 'routes' command on the ONOS console. In the legacy ASs, an eBGP router is connected with a single host. The SDN network in our proposed use case consisted of a BGP border router, which acts as iBGP to SDN-IP and eBGP speaker to the external legacy networks. The BGP policies advertised by external network border routers will be received by the ONOS/BGP speakers in the transit SDN and re-advertised to other external legacy networks by SDN-IP as an iBGP peer. SDN-IP translates the best route policies into the ONOS application intent request (AIR) and deploys them into data plane devices as forwarding rules to route the transit traffic into appropriate external domains. Internal BGP speakers were configured to use the TCP port at 179 to communicate with other BGP speakers and the TCP port at 2000 to send the route information to ONOS/SDN-IP instances, and the external BGP speakers were connected to OpenFlow enabled switches.
The BGP routers were emulated using the Linux hosts in which the Quagga routing suite was running. The BGP daemon of the Quagga routing suite was used for the BGP routers. Figure 2  Using D-ITG, UDP flows were generated with varying packet size and packet rate for 15 s. Two log files were generated for both the sender and receiver side. The log file generated at the receiver side contained values for different performance indicators such as delay, jitter, bit-rate, bytes received, packets dropped and average loss-burst size. On decoding the log file, we received a stream of data containing these performance indicators. The values of these parameters were taken from the log file on receiver's side. Multiple unidirectional flows were sent for fifteen seconds between pairs of hosts in turns by varying transmission rate and packet size to study the performance of the network under low and high load. The packet size set were 512 and 1024 bytes and the packet transmission rates were varied to 100, 1000, and 3000 packets per second. These flows were sent in the traffic via UDP where each host acted both as a client and a server. Ultimately, the log files of flows generated from ten hosts (ref. Figure 2) in the SDN network were further processed to a CSV file using a Python script. The D-ITG data samples generation snapshot is shown in Figure 5. Moreover, 507 data samples were captured during the simulation. Figure 6a-d shows the pattern and distribution interval of the data samples for delay, jitter, bit rate and the packet rate, respectively.
The bi-variate distribution of numerical attributes can be observed in the pair-plot shown in Figure 7. A pair-plot visualizes given data to understand the best set of features to explain a relationship between two variables (attributes) or to form the most separated clusters. The plot is in matrix format where the x-axis is represented by row name and the y-axis is represented by column name. The main-diagonal subplots are the univariate histograms (distributions) for each attribute. For example, the plot in the fourth row and the fourth column shows the univariate histogram for average-jitter, where the minimum and maximum of the parameter can be observed. On the non-diagonal subplots, bi-variate distribution between attributes can be observed. Some plots, such as those in the first column and the first row, represent separate clusters, which show that no relationship exists between the attributes. Certain scatter plots that follow a clear linear pattern with an increasing or decreasing slope can be seen, which shows that some conclusions can be drawn from the distribution. Plots with a linear relationship between the two attributes make it possible to identify in which space the classes will be well separated from each other. For example, the plot in the second row and the fourth column shows a linear relationship between the two variables. Such linear patterns represent dependencies among the attributes, bringing us to the conclusion that average delay and average jitter are correlated.  The pair-plot is an efficient tool for exploratory data visualization and analysis but does not show the strength of correlation between the attributes with dependencies. This can be achieved through a correlation heat-map using the heatmap() function available in the Seaborn package. The correlation of all the numerical attributes as a heat-map using the Spearman correlation coefficient is shown in Figure 8. The scale of color in the heat-map varies from blue to yellow, where blue signifies a correlation of +1 and yellow signifies a correlation of −1. The Spearman correlation coefficient measures the extent to which two variables tend to change together. The coefficient describes both the strength and the direction of the relationship. The Spearman's rank-order correlation is the non-parametric version of the Pearson correlation. Spearman's correlation coefficient (r s ) measures the strength and direction of association or the monotonic relationship between two ranked variables. It requires two variables that are either ordinal, interval, or ratio. For a sample size of n, the attributes X i and Y i are converted to ranks rg x and rg y . Spearman correlation coefficient (r s ) is computed by Equation (1), r s = cov(rg x , rg y )/σ r g x .σ r g y , where cov(rg x , rg y ) is the co-variance of the rank variables. σ r g x and σ r g y are standard deviations of the rank variables. It can be observed from Figure 8 that average delay and average jitter are highly correlated with a Spearman coefficient of 0.92. Similarly, bytes received, average bit rate, and average packet rate are highly correlated as well. Parameters, for example, average delay and average packet rate, can be observed to be negatively correlated, with a Spearman coefficient of −0.75.

Analysis and Evaluations
This section aims to highlight the differences between the integrated SDN network and the legacy network by comparing them in terms of various performance indicators. Additionally, the performance of the integrated SDN network was further observed by taking note of the network quality of service (QOS) parameters defined.

Comparison between SDN and Legacy Network
In this section, we provide a comparative analysis of the use case network proposed as a hybrid SDN and legacy network in terms of the QoS parameters defined. The comparison is done by analysing the performance of both networks (legacy and SDN) on the basis of bandwidth capacity, packet transmission rate and time required to transmit the packet from the source node to the destination node.

Bandwidth
The results of bandwidth capacity in both the SDN and the legacy network were obtained by using IPERF tool in Mininet. Here, we show the maximum bandwidth between hosts in neighbouring ASs and within the same AS for both the SDN and the legacy network. In addition to that, the maximum bandwidth when the hosts use AS65000 or SDN as a transit network to connect is also shown. As shown in Figure 9, the maximum bandwidth seems to be higher in the case of the SDN network, which can highlight the ability of SDN to transfer more data per second compared to their legacy counterparts. It can also be seen that the maximum bandwidth is higher for nodes within AS65000 or within the SDN compared to the neighbouring AS or when AS65000 and SDN are used as a transit network.

Packet Transmission Rate (PTR)
We compared both networks based on the PTR obtained by executing the PING tool in Mininet. PTR for both topologies is shown in Figure 10 for source and destination hosts that use AS65000 or the SDN network as a transit network for a varying number of packets.
The results obtained for the PTR indicate that the total time taken by both networks for a given number of packets is almost equal. It is noted that both networks are active for the same number of time intervals to execute the command.

Round Trip Time (RTT)
Next, we compare the delay between nodes in each network. The PING tool was used to obtain the RTT between the source and the destination nodes. The networks are compared on the basis of the variance of RTT, and the minimum and maximum values of RTT. Here, the standard deviation of the values of RTT for both the networks are plotted in a graph. This is done for the source and destination nodes within an AS, in a neighbouring AS, and when AS65000 and the SDN network are used as transit networks.
Standard deviation is simply the average of how far each RTT value is from the mean RTT. This shows how the round trip time varies with time. A higher value of variation in the RTT value results in a more unstable network. As shown in the plots of Figures 11a,b, 12a,b, 13a,b and 14a,b, the standard deviation values with a varying number of packets for SDN is lesser than that of the legacy network. This signifies that SDN can be more reliable and stable than their legacy counterparts.
The values used in the plots are used as point estimates to provide a rough estimate of the population parameter (i.e., mean RTT variance of the population) within a confidence interval. Two different samples for the legacy (ref. Figures 11a, 12a, and 13a,b) and SDN networks (ref. Figures 11b, 12b, and 14a,b) were taken, respectively. The Tdistribution of the samples are determined at a confidence level of 95% as the standard deviation of the population is unknown to us. For the legacy network, a confidence interval of (0.02495, 0.06938) is obtained with a sample mean equal to 0.0472 and a t-critical value equal to 2.200985. Similarly, for the SDN network, a confidence interval of (0.0304, 0.0551) is obtained with a sample mean equal to 0.042767 and a t-critical value equal to 2.000995.  Some of the observations presented in the graphs have spikes at the beginning because of the fact that, in both the networks, the pinging host first needs to discover the target to be communicated, which takes additional time. Initial readings of the packets are avoided in the plots to show the actual deviation in RTT. From the observations, the variability in the standard deviation of RTT is small and smooth. We run multiple rounds of executions to see the pattern of variation in the standard deviation, while no significant pattern changes have been found. Additionally, the experiments were executed in virtual environments of higher memory and processing capacity host. The results of the additional round of tests are provided in our GitHub page mentioned in the supplementary section of this article. Furthermore, we run the experiments in the virtual machine hosted on the azure and DNS servers of Google and Cloud-flare to demonstrate the presence of variability in RTT values over time and the results are plotted in Figure 15. This shows that the RTT variance of the PING results between machines in a real scenario. We observed the values of RTT with a deviation from 0.1 to 0.8. There are other communications and the protocol's internal signaling (e.g., OpenFlow communication, Host_Location_service_Provider etc.), which affect the RTT values. The minimum and maximum values of RTT for both networks are plotted in the graph in Figures 16a-d, 17a-d, and 18a-d. This is done for source and destination nodes within an AS, in a neighbouring AS, and when AS65000 and the SDN network are used as transit networks. The traffic were captured at multiple rounds and more than one plot is provided in Figures 16-18 respectively.
From the plots of Figures 16a-d, 17a-d, 18a-d and 19a-d, it can be observed that, for a varying number of packets, the minimum and maximum values of RTT are lesser in SDN compared to that of the legacy network. A higher RTT value signifies more time taken for the transmission of packets that can affect the speed and reliability of the network. Thus, it can be concluded that routing in SDN using ONOS/SDN-IP performs much better than the legacy network in terms of speed and reliability. Figures 20 and 21 show the routes learnt by the ONOS controller in SDN and our BGP router in legacy topology, respectively. It is noted that the controller took about ten to fifteen seconds to learn the complete routes, whereas the BGP routers in the legacy network took around one minute. This shows that the learning rate of SDN is faster than that of the legacy network. In the legacy network, every router needs to learn about the changes in the topology, whereas in SDN, only the controller needs to know about the changes and it will install corresponding flow entries in the forwarding devices if required. This improves network performance as no routing advertisement messages are advertised in the network.

Discussion and Future Work
Virtualization capability of SDN enables powerful APIs, which can be used to control many network functionalities with intelligence. The lack of backward compatibility of SDN with the legacy network enforces research communities to develop a robust system for interoperability between SDN and legacy networks for smooth and low cost migration approaches. Hybrid SDN is the only solution for having a smooth transition to pure SDN for service providers so that customers can receive uninterruptible services during the migration period. SDN-IP is an ONOS application that is used to peer SDN networks with external networks on the Internet using the standard BGP so that service providers can run hybrid SDN in their existing network during the transition period of their network migration. The SDN-IP controlled network acts as a transit AS that interconnects different legacy IP networks, considering each external network as a different AS domain, and interfaces with the SDN network through its BGP-speaking border routers.
In this study, an SDN integrated network was created with Mininet and experimental tests were carried out to ensure successful communication between the hosts in different ASs. The output of the tests showed a smooth transmission of data between hosts, thus providing testament to the possibilities of interoperability between two different networking paradigms. A similar legacy network was also created and observed to note the difference in the characteristics of an integrated SDN and legacy network.
To give statistical significance to this experiment, correlation coefficients of the QoS parameters were determined. These values can be further utilized with artificial intelligence technologies or to develop machine learning models or rules about the network. These insights can be useful in wide areas of applications such as traffic classification, routing optimization, quality of service and quality of experience prediction, resource management, security enhancement and many other purposes.
The comparison between legacy and SDN integrated networks aimed at verifying the speculations about SDN performing better than legacy networks. By studying both the networks based on key performance indicators, such as bandwidth, packet transmission rate, and round trip time, the performance of the SDN network was observed to be much better than its legacy counterpart. Similarly, the exploratory data analysis of the values of the QoS parameters collected from the network provided us with insights about the network.
The scope of this experimental analysis was limited to an IPv4 addressing network only. A routing performance analysis with the networks enabled with IPv6 addresses could further help to establish the significance and prospects of SDN implementation with future networking technologies. Additionally, this study was limited to comparing the performance of hybrid SDN and a legacy network to determine the feasibility of the migration of the legacy networks into SDN. We read some of the literature related to SDN routing approaches [22,23], already mentioned in Section 2, since we are particularly dealing with routing in hybrid networks. Reference [23] deals with dynamic routing algorithm implementation in a pure SDN environment, while the article OSHI [22] considers hybrid switch networks. However, our scenario of routing implementation does not particularly match with the concepts provided in [22,23]. OSHI [22] considers hybrid or dual stack switch, that is, legacy routing and OpenFlow implementation, in a single switch. The dual stack switch consumes a higher load as compared with a single stack switch because it has to handle dual packets routing/forwarding at once as well as a separate access control list needing to be maintained. Hence, we will consider the comparison of performance evaluation of routing in an SDN only environment using ONOS/SDN-IP with other SDN routing approaches for future work.
The next generation wireless networking viz. 5G/6G network is considered to be fully SDN based [39,40]. The paradigm shifts in mobile communications by the conceptualization and implementation of 5G/6G added a strong requirement for SDN in the modern network environment. The wired/wireless network and server virtualization, high speed communications at highly dense smart devices with ultra low latency for real time mission critical applications, energy efficient smart network deployment with efficient radio access network (RAN) design, smart spectrum management and implementation are the part of 5G and beyond networks that directly concern people's modern living standards. Hence, from core network to end-access network service provisioning, cloud computing to fog computing to edge computing-they are mostly related to the availability of 5G and SDN, and the 4G to 5G migration and the performance evaluation of SDN based 5G network as compared with 4G networks are considered to be future work. Though network congestion optimization is not within the scope of this study, queue length analysis of the network device provides the trade-off between cost and performance of the network to understand the impact of congestion in SDN. Queue length analysis is also considered as future work.
Routing implementations over hybrid SDN with multiple controllers and their placement, a large and varying number of data plane devices, and so forth, could be the subject of future work to reflect the situations of CG-ISP networks when implemented into reality. A single instance of an ONOS controller is used in the present study. Multiple instances can be run at the same time for high available services. Multiple instances help in the load balancing of the whole network and add reliability to the network. Additionally, other available network controllers can be used to replicate the results for comparison in addition to ONOS.

Conclusions
The implementation of SDN is regarded as the best solution to meet modern communication requirements and to avoid the issues of legacy networks. The data plane devices that support the SDN network generally use the OpenFlow protocol to establish communication with the controller. However, the legacy devices generally already in operation these days do not support the OpenFlow protocol. The replacement of the existing networking devices at once by the OpenFlow supporting devices is not feasible and hence, a phase-wise migration is the only tangible solution. The features of SDN are compelling enough to encourage the migration of larger networks of ISPs from legacy to SDN. This study has demonstrated the successful integration of SDN networks with legacy networks to show the possibility of smooth migration and interoperability with legacy networks during migration. Multi-domain routing using ONOS/SDN-IP in the SDN environment has a better performance as compared with the legacy routing. The experimental analysis presented contributes to verifying the seamless interoperability of legacy and SDN networks so that service providers can be confident in the SDN implementation in their ISP and Telcos networks. This study is also a testament to SDN-integrated networks performing better than traditional legacy networks based on QOS parameters viz. bandwidth, PTR, and RTT.