Emulating Software-Deﬁned Disaggregated Optical Networks in a Containerized Framework

: Telecom operators’ infrastructure is undergoing high pressure to keep the pace with the trafﬁc demand generated by the societal need of remote communications, bandwidth-hungry applications, and the fulﬁlment of 5G requirements. Software-deﬁned networking (SDN) entered in scene decoupling the data-plane forwarding actions from the control-plane decisions, hence boosting network programmability and innovation. Optical networks are also capitalizing on SDN beneﬁts jointly with a disaggregation trend that holds the promise of overcoming traditional vendor-locked island limitations. In this work, we present our framework for disaggregated optical networks that leverages on SDN and container-based management for a realistic emulation of deployment scenarios. Our proposal relies on Kubernetes for the containers’ control and management, while employing the NETCONF protocol for the interaction with the light-weight software entities, i.e., agents , which govern the emulated optical devices. Remarkably, our agents’ structure relies on components that offer high versatility for accommodating the wide variety of components and systems in the optical domain. We showcase our proposal with the emulation of an 18-node European topology employing Cassini-compliant optical models, i.e., a state-of-the-art optical transponder proposed in the Telecom Infrastructure Project. The combination of our versatile framework based on containerized entities, the automatic creation of agents and the optical-layer characteristics represents a novel approach suitable for operationally complex carrier-grade transport infrastructure with SDN-based disaggregated optical systems.


Introduction
The rapid growth of Internet traffic that we are currently experiencing [1,2] is based on the expansion of cloud services and the huge amount of traffic supported by the content delivery networks (CDNs) [1]. This clearly increases congestion issues in communication networks, with particular emphasis in the core and backbone segments [3]. Furthermore, the global crisis of COVID-19 has revealed the need for remote communications in both the social and business spheres [2,4]. This crisis has become an incipient increase in video-callrelated traffic through the use of applications such as Zoom [5], Microsoft Teams/Skype [6] or Cisco Webex [7]. Finally, these challenges are compounded by the need to meet the performance specifications for the fifth generation Internet (5G), where it is planned to obtain (a) a scalable management environment that enables the dynamic deployment of cloud-based applications, (b) a reduction of approximately 20% in operational costs and (c) total latency to the user of less than 1 ms [8].
The combination of all these challenges requires drastic changes in the way transport network resources are controlled and managed, moving away from traditional network management techniques. Under this umbrella, software defined networking (SDN) and communicate with the optical devices. In [22], this idea is expanded to not only be applied to the controller but also to a high level of abstraction that comprises the control plane in multi-domain disaggregated optical networks. Furthermore, a monitoring and data analytics architecture is proposed in [23] to monitor optical disaggregation.
On the other hand, several studies focus on the data plane, precisely on modelling devices in the context of DON. For instance, YANG models and their implementation, via agents, of (SDN-enabled) sliceable bandwidth/bitrate variable transceivers (S-BVTs) have been presented for partially [24] and fully disaggregated optical networks [25]. One of the most important actors in optical networks is the reconfigurable optical add/drop multiplexer (ROADM) that implements the add and drop capabilities in optical nodes. They are addressed within the DON scope in [26,27]. Additionally, focusing on the data plane, [28] reports a gap analysis on physical-layer parameters of YANG models available in the OpenConfig, OpenROADM and OpenDevice projects for estimating the quality of transmission (QoT). In addition, this work is complemented by a proof of concept focusing on the reception of the relevant parameters from an OpenConfig muxponder in the application layer for planning purposes.
Relevant in the context of data-plane initiatives, it is worthwhile mentioning GNPY [29]: the open-source planning tool for the data plane in optical networks. GNPY is based on the Gaussian Noise (GN) model [30], and reported validation results in large-scale optical transmission testbeds [31]. In particular, [31] reports the GNPY validation in a mixed-fibre test-bed at Microsoft labs in which transponders of eight different manufacturers operated in the C-band reaching a propagation distance of 1945 km. Relevantly, GNPY has also been considered as an assistance tool for the computation of impairments in optical-layer paths on top of SDN controllers [32]. Nonetheless, these efforts are scarce in the literature because optical-layer awareness is commonly considered as stand-alone planning tools separated from operational aspects addressed by SDN controllers.
The next stage towards emulating a DON environment is device virtualisation that exploits the benefits provided by YANG models of devices, which is also explored in the community. A network virtualisation architecture for open (partially) disaggregated network uses the concept of device hypervisor in virtual networks by using OpenROADM data models [33]. Furthermore, another common approach for adding virtualization to DON-based devices is to implement their related agent in a virtual machine (VM)/container that can emulate a realistic behaviour of the optical device. This eases the instantiation and the management of the virtual entities with a computer or a server. Those VMs/containers must be configured to enable the SBI communication implementing the interaction with the SDN controller [13].
Finally, it is also worthwhile mentioning comparable initiatives to our proposal such as the software-defined packet-optical network emulator [34], which is based on an optical extension of the popular packed-oriented network emulator Mininet [35]. Relevantly, ref. [34] proposes a software suite capable of emulating optical-layer QoT performances such as OSNR, gOSNR and BER, and report its impact to the SDN controller. This emulation is based on augmenting packet in traditional Mininet instances with wavelength/channel information, which is used in modified instances of Open vSwitch. In fact, the entire Mininet-Optical emulation relies on a software structure that invokes the Open vSwitch instances as Python processes. Consequently, this approach diverges from realistic deployment/production scenarios in which optical systems are managed/controlled by its own light-weight software agent in dedicated physical or virtual resources.

Motivation and Previous Work
The state of the art in SDN DON and optical devices' emulation reveals that the community is pushing considerably in this direction. YANG-based models in accordance with the NETCONF protocol are efficient approaches towards minimizing or even removing the drawbacks of the so-called vendor islands. Thus, there are ongoing projects to provide unified models for optical devices. However, to the best of our knowledge, optical device emulation is in its early stages. Only some of the analysed works assume a scenario where SDN DON emulation tackles from the highest application layer down to the data plane. Handing over control and management of emulation to the application layer will permit a more precise optical network design and an automatic deployment of software agents related from device catalogue at the request of the user. Moreover, as pointed out in the data-plane initiatives in the previous subsection, interactions between realistic agentbased SDN implementations and data-plane models are scarce. In fact, the challenge of these initiatives relies on the combination of physical-layer modelling commonly devoted to off-line planning tools with operation-based systems such as SDN controllers. The complementary nature of those systems makes the development of a DON emulator a difficult and challenging task. Such context is the starting point that motivates this paper with the aim at contributing to achieve a such challenging goal.
This work is a significant follow-up of an open line started in [36], in which we reported our preliminary progress towards an optical emulation framework built on automatic agent creation. In particular, ref. [36] focused on a specific use case of an 18node European topology with nodes based on open packet transponder (Cassini [37]) in which we were able to modify parameters such as optical launch power, wavelength, and modulation format. Here, we extend that work and present a substantial evolution of our Containerized Framework for emulating SDN-DONs with special attention to the coexistence of the Kubernetes-based [38] master nodes server-client architecture and a new automatic agent creation framework using NETCONF in the SDN controller-optical agent interaction irrespective of the YANG model used. Additionally, to test the practical feasibility of our proposal, a proof-of-concept is also presented.

Structure of the Paper
The rest of the paper is structure as follows. In Section 2, all the software tools needed, and the methodology followed are exposed. Section 3 is aimed at showing the proof-ofconcept proposed to evaluate the proposal. Section 4 reports the results obtained with our framework accompanied with their discussion. Concluding remarks and future directions of our work are highlighted in Section 5.

Proposed Architecture for SDN-DON Emulation
In this section, we present all the details required to define and configure our proposal for the SDN DON emulated framework. First, we expose a summary of the previous work that feeds this proposal. Then, we present the general architecture that permits the automatic deployment of emulated SDN DON, the core of this work. This section concludes with configuration considerations that are required to set up the framework.

Legacy Work
In the previous work presented in [36], we reported our preliminary advancements towards an optical emulation framework for automatic creation of agents in SDONs.
As seen in Figure 1a, this architecture comprises three main elements: the ONOS SDN controller [39]; a pool of software agents that emulate the interaction with optical devices, in this case, Cassini transponders; and the API gateway.
ONOS is chosen to play the role of the SDN controller, and is configured with the required modules and YANG drivers to interact with the optical devices or agents in the data plane via NETCONF protocol. A pool of agents emulates a realistic environment of a DON data plane; those agents are implemented in docker containers and they are created automatically by the API gateway REST-based interface according to the topology specification given by ONOS. ONOS is chosen to play the role of the SDN controller, and is configured with the required modules and YANG drivers to interact with the optical devices or agents in the data plane via NETCONF protocol. A pool of agents emulates a realistic environment of a DON data plane; those agents are implemented in docker containers and they are created automatically by the API gateway REST-based interface according to the topology specification given by ONOS.
The other key point of this work is the structure of the optical network agent, depicted in Figure 1b. The software agents in the pool are virtualised entities within their respective Docker container. In addition, each agent includes four components: (i) the OpenConfig YANG model of Cassini transponders, (ii) the connection manager (i.e., the Netopeer2 NETCONF server) that enables connectivity with the controller, (iii) a database to store the configuration files and (iv) a software engine to detect and interact with changes in the configuration files. It is relevant to mention that the last three elements belong to the same subsysem, a Sysrepo framework aimed at automatic creation of YANG-based agent configuration. The Netopeer2 server is built in the Sysrepo framework and it provides the adequate NETCONF-based server for interacting with the SDN client. The Netopeer2-Sysrepo symbiosis provides benefits in the automatic creation of YANG agents but they have a closed and short catalogue of YANG-based optical device models that limits the scalabilty of a SDN DON emulation framework.

Advanced Automatic Deployment of Emulated SDN DON
The main target of this work is to enhance the dynamicity and the scalability of the automatic deployment of an emulated SDN DON environment. To achieve this challenging objective, we leverage the previous work focusing on two major and significative changes: (i) automatic container management by Kubernetes and (ii) a novel scalable agent definition. Figure 2a showcases the general architecture of our proposal for an advanced automatic deployment of an emulated SDN DON framework. As it can be seen in the figure, the main difference with the legacy work is to leverage Kubernetes functionalities to manage and automatize the creation of agents, thus, minimizing the dependency of the API gateway. The other key contribution is the novel structure of the optical agent, shown in Figure 2b, facilitating the scalability and the emulation of a wide catalogue of YANG- The other key point of this work is the structure of the optical network agent, depicted in Figure 1b. The software agents in the pool are virtualised entities within their respective Docker container. In addition, each agent includes four components: (i) the OpenConfig YANG model of Cassini transponders, (ii) the connection manager (i.e., the Netopeer2 NETCONF server) that enables connectivity with the controller, (iii) a database to store the configuration files and (iv) a software engine to detect and interact with changes in the configuration files. It is relevant to mention that the last three elements belong to the same subsysem, a Sysrepo framework aimed at automatic creation of YANG-based agent configuration. The Netopeer2 server is built in the Sysrepo framework and it provides the adequate NETCONF-based server for interacting with the SDN client. The Netopeer2-Sysrepo symbiosis provides benefits in the automatic creation of YANG agents but they have a closed and short catalogue of YANG-based optical device models that limits the scalabilty of a SDN DON emulation framework.

Advanced Automatic Deployment of Emulated SDN DON
The main target of this work is to enhance the dynamicity and the scalability of the automatic deployment of an emulated SDN DON environment. To achieve this challenging objective, we leverage the previous work focusing on two major and significative changes: (i) automatic container management by Kubernetes and (ii) a novel scalable agent definition. Figure 2a showcases the general architecture of our proposal for an advanced automatic deployment of an emulated SDN DON framework. As it can be seen in the figure, the main difference with the legacy work is to leverage Kubernetes functionalities to manage and automatize the creation of agents, thus, minimizing the dependency of the API gateway. The other key contribution is the novel structure of the optical agent, shown in Figure 2b, facilitating the scalability and the emulation of a wide catalogue of YANG-based optical models. Further details of the structure of the agent are elaborated in next subsections.

General Architecture
This architecture is targeted to maximize the automation and the scalability of the emulation framework trying to exploit the benefits of a dynamic management and orchestration of the containerized optical agents. Moreover, two architectural decisions in our implementation deserve further description. First, it is worthwhile mentioning that we choose to employ docker containers because of they are more lightweight in terms of resources than VMs. We leverage on the fact that containers share the operating systems whereas VMs aim at emulating virtual hardware. Given that docker containers share the operating systems, docker applications consume a portion of the resources compared to a VM. Second, our architecture considers one docker container per agent for a realistic emulation of a carrier-grade deployed scenario. In particular, telecom operators commonly deploy control architectures that handle distinct geo-located optical nodes with dedicated computing resources for service deployment (e.g., in the edge computing paradigm) while also devoting part of those resources for the control of the network infrastructure. In this context, optical resources are managed and operated with a per-node computing element that receives and sends commands from/to the SDN controller while interfacing the hardware at the data plane. Hence, our proposal aims at emulating such conditions with an independent and isolated entity, i.e., docker container, per software agent as in the case of distinct geo-located optical nodes. This architecture is targeted to maximize the automation and the scalability of the emulation framework trying to exploit the benefits of a dynamic management and orchestration of the containerized optical agents. Moreover, two architectural decisions in our implementation deserve further description. First, it is worthwhile mentioning that we choose to employ docker containers because of they are more lightweight in terms of resources than VMs. We leverage on the fact that containers share the operating systems whereas VMs aim at emulating virtual hardware. Given that docker containers share the operating systems, docker applications consume a portion of the resources compared to a VM. Second, our architecture considers one docker container per agent for a realistic emulation of a carrier-grade deployed scenario. In particular, telecom operators commonly deploy control architectures that handle distinct geo-located optical nodes with dedicated computing resources for service deployment (e.g., in the edge computing paradigm) while also devoting part of those resources for the control of the network infrastructure. In this context, optical resources are managed and operated with a per-node computing element that receives and sends commands from/to the SDN controller while interfacing the hardware at the data plane. Hence, our proposal aims at emulating such conditions with an independent and isolated entity, i.e., docker container, per software agent as in the case of distinct geo-located optical nodes.

Kubernetes-Based Agent Management
In this work we propose to manage the agents' docker containers through Kubernetes. The choice of Kubernetes is based on the necessity of providing scalability and intelligent computational management in order to support large scale SDN DON emulated scenarios. Kubernetes has a wide range of benefits for virtualised systems that could help

Kubernetes-Based Agent Management
In this work we propose to manage the agents' docker containers through Kubernetes. The choice of Kubernetes is based on the necessity of providing scalability and intelligent computational management in order to support large scale SDN DON emulated scenarios. Kubernetes has a wide range of benefits for virtualised systems that could help to accomplish the expected targets, e.g., service discovery and load balancing, storage orchestration, automated rollouts and rollbacks or self-healing just to mention a few. Moreover, Kubernetes is currently one of the most popular DevOps tools for container management, hence with a large support in the community [38]. Additionally, Kubernetes and Docker are complementary technologies, Kubernetes orchestrates one or more hosts that run containers, and Docker is the technology that starts, stops, and also manages those containers. In our model, Docker is a low-level technology orchestrated and managed by Kubernetes. To complement the justification about why Kubernetes is key in our proposal, we present a present a representative use case. Consider a need to emulate a full SDN-based DON scenario assuming realistic functionalities, with multiple optical nodes and at each node multiple optical devices such as, transponders, wavelength selective switches, EDFA amplifiers or optical channel monitor. Just considering a 25-node optical network and for these four types of devices, 25 × 4 = 100 agents are needed, in the most favourable case assuming only 1 device per type and node (in reality this number is much higher). For this case alone, managing 100 agents instantiated in virtual machines or virtual containers without intelligent management, the scenario is completely unaffordable for one single personal computer, computationally speaking. This is where Kubernetes comes in; this tool not only provides intelligent management of containers (agents), but its automatic load balancing and the minimization of the usage of computational resources, both features related to scalability, among others, are basic to allow the emulation of realistic and largescale SDN DON scenarios. Thus, one of the major benefits that Kubernetes brings to our work is that it allows the emulation of complex SDN DON scenarios running on a single laptop, this way exploiting the potential usability and exploitation of our framework (e.g., teaching, academia, SDN DON pre-deployment phases, etc.).
From an implementation perspective, we follow a traditional master nodes approach. A Kubernetes cluster is made up of one master and one or more nodes, as can be seen in Figure 3. Both master and nodes are Linux hosts that run on anything from VMs, bare metal servers, to private and public cloud instances.

Kubernetes.
To complement the justification about why Kubernetes is key in our proposal, we present a present a representative use case. Consider a need to emulate a full SDN-based DON scenario assuming realistic functionalities, with multiple optical nodes and at each node multiple optical devices such as, transponders, wavelength selective switches, EDFA amplifiers or optical channel monitor. Just considering a 25-node optical network and for these four types of devices, 25 × 4 = 100 agents are needed, in the most favourable case assuming only 1 device per type and node (in reality this number is much higher). For this case alone, managing 100 agents instantiated in virtual machines or virtual containers without intelligent management, the scenario is completely unaffordable for one single personal computer, computationally speaking. This is where Kubernetes comes in; this tool not only provides intelligent management of containers (agents), but its automatic load balancing and the minimization of the usage of computational resources, both features related to scalability, among others, are basic to allow the emulation of realistic and large-scale SDN DON scenarios. Thus, one of the major benefits that Kubernetes brings to our work is that it allows the emulation of complex SDN DON scenarios running on a single laptop, this way exploiting the potential usability and exploitation of our framework (e.g., teaching, academia, SDN DON pre-deployment phases, etc.).
From an implementation perspective, we follow a traditional master nodes approach. A Kubernetes cluster is made up of one master and one or more nodes, as can be seen in Figure 3. Both master and nodes are Linux hosts that run on anything from VMs, bare metal servers, to private and public cloud instances. The Kubernetes master is composed of a set of small services that conform the control plane of the cluster and it can be externally accessed by the kubectl service, a RESTful API or a graphical dashboard, all managed by the kube API-server. The configuration and status of the cluster gets persistently stored in the etcd, while the controller implements the functionalities for controlling the Kubernetes cluster, such as, the node controller or the endpoints controller. Finally, the Kubernetes master offers the possibility to monitor new workloads and assign them to the nodes. Still regarding Figure 3, the nodes can be seen as clients in a server-client architecture, with the master as server. The nodes are where the containers are hosted in correspondence to the agents in our framework. Within the nodes, there are independent spaces to run the containers, called pods. One node can have several pods and in each pod several containers. This approach is totally in line with network slicing, essential for 5Gaware deployments [40], being this fact another solid reason to choose Kubernetes as The Kubernetes master is composed of a set of small services that conform the control plane of the cluster and it can be externally accessed by the kubectl service, a RESTful API or a graphical dashboard, all managed by the kube API-server. The configuration and status of the cluster gets persistently stored in the etcd, while the controller implements the functionalities for controlling the Kubernetes cluster, such as, the node controller or the endpoints controller. Finally, the Kubernetes master offers the possibility to monitor new workloads and assign them to the nodes. Still regarding Figure 3, the nodes can be seen as clients in a server-client architecture, with the master as server. The nodes are where the containers are hosted in correspondence to the agents in our framework. Within the nodes, there are independent spaces to run the containers, called pods. One node can have several pods and in each pod several containers. This approach is totally in line with network slicing, essential for 5G-aware deployments [40], being this fact another solid reason to choose Kubernetes as agent manager in a SDN DON emulation framework. Note that the server-client architecture, in correspondence with the Kubernetes master nodes (also illustrated with blue dashed arrows in Figure 2a), coexists with the client-server structure used in NETCONF protocol in correspondence with SDN controller-optical agent container (illustrated in solid red arrows in Figure 2). Concerning Kubernetes integration as part of the Advanced Automatic Deployment of Emulated SDN DON oversees agents' management. First, the configuration for creating containers (e.g., agents) can be performed by three different ways: via script by using the kubectl service, manually from dashboard or remotely through the Kubernetes API.
Once the containers are up and running in one or multiple pods, the Kubernetes Master is in charge of managing the computational resources according to the workload and the specifications stated in the configuration phase.

Versatile Agent Definition
The second major contribution of this work is a novel definition of versatile optical agents that can be automatically instantiated. We present a generic definition of agents suitable not only for specific YANG models belonging to specific projects (e.g., Telecom Infra Project [37]) but also for all optical models that can be defined using the YANG language. This novel structure is decisive in the sense that a user can evaluate scenarios with a potentially infinite number of optical devices under the same emulation framework irrespective of their vendor, technology, or type. Therefore, this new functionality is critical not only to emulate DON scenarios with realistic current data models, but also to be compatible with all future data models using the YANG language. Figure 2b depicts the structure proposed for a generic agent that emulates the role of a software entity in charge of governing optical devices. It is composed of four major components, a NETCONF server, a YANG validator, an XML (eXtensible Markup Language) /JSON (JavaScript Object Notation) config database and the files of the YANG model necessary to emulate an optical device, all coming from open-source initiatives and they are implemented in one single Docker container. Further details follow: • NETCONF server: it provides NETCONF connectivity with the SDN controller with the aim at avoiding dependency of existing agent automation framework, similarly to the Sysrepo in the legacy work. In this work, the NETCONF skeleton is given by the Choppsv1 library [41] that supports both the creation of NETCONF clients and servers, however it does not provide the full implementation of the server. The pyangbind module complements the server, because it permits the generation of automatic code from YANG files, the validation of XML calls, essential in NETCONF-based communications and XML/JSON conversion amplifying the potential for connectivity [42]. • YANG validator: pyang is Python-based library playing the role of a validator, transformer, and generator of YANG code. It can be used for validating YANG modules, YANG modules transformation into different formats, and code generation from modules. This module is key to provide potential scalability and generalization [43]. • XML/JSON config database: it stores the configuration files related to the configuration of the agent for the different functional states, such as boot or running. • YANG model files: finally, a set of YANG files that define the behaviour required to emulate an optical device. They are validated by the pyang module.
From a global point of view, it is worthwhile to indicate the interaction among the components described above. First, YANG models are provided, e.g., such as Cassini transponder models. Then, agents leverage on the YANG validator based on pyang for the creation of the NETCONF skeleton within the NETCONF server. Notably, it is worth recalling that SDN controllers commonly include a series of drivers that implement NETCONF client functionalities for interacting with the agents via NETCONF protocol. This interaction, on the agent side, will leverage on the pyangbind module for XML/JSON validation. Then, also in the NETCONF server side, custom instructions are implemented for interacting with the data plane. Finally, we report the interaction between control and data signals flow among the components in the next subsection. For instance, Figure 8 will showcase the interaction between agents (data plane) when connectivity is requested from the SDN controller (control plane).

Proof-of-Concept Configuration
In order to test and evaluate the emulation framework of this work, in this section we propose a proof-of-concept that integrates the key elements of the entire architecture. Here we detail the configuration and settings for performing such proof-of-concept.
The proof-of-concept follows the design exposed in Figure 2a, with three main actors: (i) ONOS playing the role of SDN controller that interacts and configure the emulated optical data plane (set of agents in this case), (ii) Kubernetes, in charge of the management of the Docker containers where the optical agents are implemented, and finally (iii) the data plane, represented by the set of agents emulating the software that governs the optical devices. For the sake of simplicity, we only use an agent of Cassini transponder according to the OpenConfig project. All the evaluation system has been deployed and executed in a high-performance laptop (Intel 8th generation i7 with 8 logical cores, 32 GB of 2400 MHz DDR4 RAM and 1 TB of SSD). Further details follow: • Agent definition and integration: a specific docker container is built to integrate the architecture proposed and described in Section 2.2.3 for an automatic agent deployment. The set of YANG files that replicate the behaviour of a Cassini transponder are extracted from the GitHub repository of the OpenConfig project [44]. Precisely, the files openconfig-terminal-device.yang, openconfig-if-ethernet.yang and openconfig-types.yang are stored in the YANG file catalogue of the agent container.

•
Kubernetes configuration: following the structure presented in Figure 3 Kubernetes Cluster is installed in Linux, an Ubuntu distribution, and it is composed by one master and one worker node and one pod in this worker node. Agents' creation and configuration are driven by the deploy.yaml file, see Figure 4, where it commands the deployment of the Docker containers. The service kubectl is the one to execute the deployment of the agents where results of such deployment are presented in the next section.
showcase the interaction between agents (data plane) when connectivity is requested from the SDN controller (control plane).

Proof-of-Concept Configuration
In order to test and evaluate the emulation framework of this work, in this section we propose a proof-of-concept that integrates the key elements of the entire architecture. Here we detail the configuration and settings for performing such proof-of-concept.
The proof-of-concept follows the design exposed in Figure 2a, with three main actors: (i) ONOS playing the role of SDN controller that interacts and configure the emulated optical data plane (set of agents in this case), (ii) Kubernetes, in charge of the management of the Docker containers where the optical agents are implemented, and finally (iii) the data plane, represented by the set of agents emulating the software that governs the optical devices. For the sake of simplicity, we only use an agent of Cassini transponder according to the OpenConfig project. All the evaluation system has been deployed and executed in a high-performance laptop (Intel 8th generation i7 with 8 logical cores, 32 GB of 2400 MHz DDR4 RAM and 1 TB of SSD). Further details follow: • Agent definition and integration: a specific docker container is built to integrate the architecture proposed and described in Section 2.2.3 for an automatic agent deployment. The set of YANG files that replicate the behaviour of a Cassini transponder are extracted from the GitHub repository of the OpenConfig project [43]. Precisely, the files openconfig-terminal-device.yang, openconfig-if-ethernet.yang and openconfigtypes.yang are stored in the YANG file catalogue of the agent container.

•
Kubernetes configuration: following the structure presented in Figure 3 Kubernetes Cluster is installed in Linux, an Ubuntu distribution, and it is composed by one master and one worker node and one pod in this worker node. Agents' creation and configuration are driven by the deploy.yaml file, see Figure 4, where it commands the deployment of the Docker containers. The service kubectl is the one to execute the deployment of the agents where results of such deployment are presented in the next section.  • ONOS: we chose a virtualized distribution of ONOS (version 2.3.0) in a Docker container. The installation using Docker also permits to open the necessary ports for this proof-of-concept in one single command as presented below. Port 8181 is used for external communication with ONOS via GUI, port 5005 is a Java debugger port, port 8101 permits command line interface (CLI) communication via CLI and the port 830 permits NETCONF-based connectivity. Finally, once the container is up and running, pertinent "apps" must be activated to enable the expected functionalities within the SDN controller framework: "odtn-service", "roadm" and "optical-rest".

•
Topology definition: the topology is defined in a JSON file so that it can be interpreted by ONOS. Seven metrics can be set "Devices", "links", "hosts", "Apps", "ports", "regions" and "layouts". In this work we only use the objects "device" and "link". Each device object is linked to each container previously deployed by indicating some specific fields, such as IP of the container and port to enable NETCONF connectivity, ONOS driver or map location, as can be seen in the exemplary definition if Figure 5a. Moreover, the links define the interconnection between two agents by setting their IP and ports, type of connection or bidirectionality, among others as depicted in Figure 5b.
"ports", "regions" and "layouts". In this work we only use the objects "device" and "link". Each device object is linked to each container previously deployed by indicating some specific fields, such as IP of the container and port to enable NETCONF connectivity, ONOS driver or map location, as can be seen in the exemplary definition if Figure 5a. Moreover, the links define the interconnection between two agents by setting their IP and ports, type of connection or bidirectionality, among others as depicted in Figure 5b.
Once the file is ready, it is sent via ONOS NBI API to inform the details of the topology. Topology definition and JSON file creation are performed manually in this proof-ofconcept emulating the 18-nodes European Optical Network. Future works will target the automatization of this procedure.

Results
This section collects and analyses the results obtained in the deployment of the proofof-concept focusing on the main points of our proposal for an Advanced Automatic Deployment of Emulated SDN DON. The execution of this experiment is performed following the statements proposed in Section 2 and configured according to Section 3. Given the lack of complex SDN DON emulators in the literature (see Section 1.1) that can not only emulate such scenarios but also interact with it in a realistic way, the objective of this proof of concept is not aimed at evaluating the performance of our framework, but to highlight the feasibility and functionality of our proposal. All the results exposed in this section are obtained after the configuration stage is properly completed.
The first stage of the running workflow in this proof-of-concept is to show the performance of Kubernetes in the agent/container management. Figure 6 illustrates the 18 containers of the Cassini agents properly running according to the configuration file (deploy.yaml) defined under the Kubernetes umbrella. Once the file is ready, it is sent via ONOS NBI API to inform the details of the topology. Topology definition and JSON file creation are performed manually in this proof-of-concept emulating the 18-nodes European Optical Network. Future works will target the automatization of this procedure.

Results
This section collects and analyses the results obtained in the deployment of the proofof-concept focusing on the main points of our proposal for an Advanced Automatic Deployment of Emulated SDN DON. The execution of this experiment is performed following the statements proposed in Section 2 and configured according to Section 3. Given the lack of complex SDN DON emulators in the literature (see Section 1.1) that can not only emulate such scenarios but also interact with it in a realistic way, the objective of this proof of concept is not aimed at evaluating the performance of our framework, but to highlight the feasibility and functionality of our proposal. All the results exposed in this section are obtained after the configuration stage is properly completed.
The first stage of the running workflow in this proof-of-concept is to show the performance of Kubernetes in the agent/container management. Figure 6 illustrates the 18 containers of the Cassini agents properly running according to the configuration file (deploy.yaml) defined under the Kubernetes umbrella.
One of the main benefits of managing Docker containers with Kubernetes is that the computational resources of the host are efficiently used to balance the workload of the containers according to their computational needs and performance. Once all the containers are up and running, the next stage is to bind the containers to the proposed optical topology by sending the JSON configuration file via ONOS API. At this point, ONOS recognizes the query and the topology is tied to the containers by assigning the existing agents to the nodes defined in the topology file and links are created to interconnect the nodes. As can be seen in Figure 7, ONOS shows the emulated 18-node European optical network topology in which each node corresponds to a Cassini agent. Appl. Sci. 2021, 11, x FOR PEER REVIEW 11 of 14 One of the main benefits of managing Docker containers with Kubernetes is that the computational resources of the host are efficiently used to balance the workload of the containers according to their computational needs and performance. Once all the containers are up and running, the next stage is to bind the containers to the proposed optical topology by sending the JSON configuration file via ONOS API. At this point, ONOS recognizes the query and the topology is tied to the containers by assigning the existing agents to the nodes defined in the topology file and links are created to interconnect the nodes. As can be seen in Figure 7, ONOS shows the emulated 18-node European optical network topology in which each node corresponds to a Cassini agent. The functionality of the agents deployed in this emulated framework is not only related to the optical topology discovery from the SDN controller side, but also they can perform other common network tasks. Figure 8 illustrates how two containers establish real TCMP-based package traffic flows leveraging the deployment of the Emulated SDN DON framework.  One of the main benefits of managing Docker containers with Kubernetes is that the computational resources of the host are efficiently used to balance the workload of the containers according to their computational needs and performance. Once all the containers are up and running, the next stage is to bind the containers to the proposed optical topology by sending the JSON configuration file via ONOS API. At this point, ONOS recognizes the query and the topology is tied to the containers by assigning the existing agents to the nodes defined in the topology file and links are created to interconnect the nodes. As can be seen in Figure 7, ONOS shows the emulated 18-node European optical network topology in which each node corresponds to a Cassini agent. The functionality of the agents deployed in this emulated framework is not only related to the optical topology discovery from the SDN controller side, but also they can perform other common network tasks. Figure 8 illustrates how two containers establish real TCMP-based package traffic flows leveraging the deployment of the Emulated SDN DON framework. The functionality of the agents deployed in this emulated framework is not only related to the optical topology discovery from the SDN controller side, but also they can perform other common network tasks. In a similar way than a real optical device, our emulation framework permits to retrieve the status of the emulated devices in real time regarding their current configuration or other functional metrics. Figure 9 showcases an exemplary query requested by the SDN controller via NETCONF protocol about the status of the configuration of an optical agent In a similar way than a real optical device, our emulation framework permits to retrieve the status of the emulated devices in real time regarding their current configuration or other functional metrics. Figure 9 showcases an exemplary query requested by the SDN controller via NETCONF protocol about the status of the configuration of an optical agent (namely, emulated Cassini transponder) in a specific instant of time. In particular, the agent is queried about its current target-output, current-output, and current-input optical powers. In a similar way than a real optical device, our emulation framework permits to retrieve the status of the emulated devices in real time regarding their current configuration or other functional metrics. Figure 9 showcases an exemplary query requested by the SDN controller via NETCONF protocol about the status of the configuration of an optical agent (namely, emulated Cassini transponder) in a specific instant of time. In particular, the agent is queried about its current target-output, current-output, and current-input optical powers. The functionality of the deployed agents is not only limited to configuration and status queries, the SDN controller can also change physical parameters of emulated devices in the same way as for a real optical device. Figure 10 is a proof of the performance of this procedure by using the CLI of ONOS to change the output power in an optical port for a specific Cassini transponder.

Conclusions
In this work, we presented an emulation framework that leverages on a combination of SDN and container management with Kubernetes for the emulation of both controland data-planes of disaggregated optical networks. Our legacy proposal was reviewed including the architectural details and the classical optical network agent structure that leveraged on the Netopeer2-Sysrepo framework. Then, we presented the Containerized Framework for emulating SDN-DONs paying special attention to the coexistence of the The functionality of the deployed agents is not only limited to configuration and status queries, the SDN controller can also change physical parameters of emulated devices in the same way as for a real optical device. Figure 10 is a proof of the performance of this procedure by using the CLI of ONOS to change the output power in an optical port for a specific Cassini transponder. In a similar way than a real optical device, our emulation framework permits to retrieve the status of the emulated devices in real time regarding their current configuration or other functional metrics. Figure 9 showcases an exemplary query requested by the SDN controller via NETCONF protocol about the status of the configuration of an optical agent (namely, emulated Cassini transponder) in a specific instant of time. In particular, the agent is queried about its current target-output, current-output, and current-input optical powers. The functionality of the deployed agents is not only limited to configuration and status queries, the SDN controller can also change physical parameters of emulated devices in the same way as for a real optical device. Figure 10 is a proof of the performance of this procedure by using the CLI of ONOS to change the output power in an optical port for a specific Cassini transponder.

Conclusions
In this work, we presented an emulation framework that leverages on a combination of SDN and container management with Kubernetes for the emulation of both controland data-planes of disaggregated optical networks. Our legacy proposal was reviewed including the architectural details and the classical optical network agent structure that leveraged on the Netopeer2-Sysrepo framework. Then, we presented the Containerized Framework for emulating SDN-DONs paying special attention to the coexistence of the

Conclusions
In this work, we presented an emulation framework that leverages on a combination of SDN and container management with Kubernetes for the emulation of both controland data-planes of disaggregated optical networks. Our legacy proposal was reviewed including the architectural details and the classical optical network agent structure that leveraged on the Netopeer2-Sysrepo framework. Then, we presented the Containerized Framework for emulating SDN-DONs paying special attention to the coexistence of the server-client architecture based on Kubernetes master nodes and the client-server structure that used NETCONF in the SDN controller-optical agent interaction. We further detailed the software agent structure which includes NETCONF server, YANG validator, XML/JSON config database and YANG models, whose combination offers versatility for accommodating the wide variety of optical components and systems. The implementation of the agents in this emulating system was deployed in Docker containers allowing advanced and efficient management by Kubernetes. We described all the configuration and settings for performing the proof-of-concept that consisted in an 18-node European topology with Cassini transponders. We reported the successful status request and configuration settings of optical-layer parameters such as transmitted and received optical power. Given the successful interaction between the control and data plane positions, our proposal demonstrates an adequate approach for managing an operationally complex carrier-grade transport infrastructure with SDN-based disaggregated optical systems.