#
A Multi-Layer Multi-Timescale Network Utility Maximization Framework for the SDN-Based LayBack Architecture Enabling Wireless Backhaul Resource Sharing^{ †}

^{1}

^{2}

^{*}

^{†}

## Abstract

**:**

## 1. Introduction

#### 1.1. Motivation

#### 1.2. Contributions

## 2. Background and Related Work

#### 2.1. SDN-Based Backhaul Architectures

#### 2.2. Network Optimization

#### 2.3. Wireless Backhaul Network Optimization

## 3. Overview of Layered Backhaul (LayBack) Network Architecture

#### 3.1. Layers in LayBack

#### 3.2. Management in LayBack

## 4. Layered SDN-Based Optimization Framework

#### 4.1. Overview

#### 4.2. Model Definitions

#### 4.3. Centralized Queue Length Minimization

#### 4.4. Operator Resource Constraints

#### 4.5. Iterative Solution via Gradient Descent

Algorithm 1: Solution of (9) (at GW g). |

Algorithm 2: At the SDN orchestrator. |

Algorithm 3: Iterates for ${x}_{o}$ (at operator o). |

Algorithm 4: Iterates for ${\lambda}_{{x}_{o}}$ (at operator o). |

Algorithm 5: Iterates for ${y}_{g}$ (at GW g). |

#### 4.6. Stochastic Optimization and Temporal Decomposition

## 5. Numerical Evaluation Results

#### 5.1. Evaluation Setup

#### 5.1.1. LayBack Architecture

#### 5.1.2. Optimization Parameters

#### 5.1.3. Comparison Benchmark

#### 5.1.4. Traffic Model

#### 5.2. Results

#### 5.2.1. Temporal Spacing of Operator Peak Demands

#### Overlapping Peak Demands

#### Separated Peak Demands

#### 5.2.2. Impact of Flexibility Parameter V

#### 5.2.3. Impact of Spacing between Operator Traffic Bursts

#### 5.2.4. Impact of Random Traffic Bursts at Operators

#### 5.2.5. Impact of Random eNB Traffic Bursts

## 6. Conclusions

## Author Contributions

## Funding

## Conflicts of Interest

## References

- Ferrari, L.; Karakoc, N.; Scaglione, A.; Reisslein, M.; Thyagaturu, A. Layered Cooperative Resource Sharing at a Wireless SDN Backhaul. In Proceedings of the IEEE International Conference on Communications Workshops (ICC Workshops), International Workshop on 5G Architecture (5GARCH), Kansas City, MO, USA, 20–24 May 2018; pp. 1–6. [Google Scholar]
- Andrews, J.; Singh, S.; Ye, Q.; Lin, X.; Dhillon, H. An overview of load balancing in HetNets: Old myths and open problems. IEEE Wirel. Commun.
**2014**, 21, 18–25. [Google Scholar] [CrossRef] - Lopez Rodriguez, F.; Silva Dias, U.; Campelo, D.R.; Oliveira Albuquerque, R.D.; Lim, S.J.; Garcia Villalba, L.J. QoS Management and Flexible Traffic Detection Architecture for 5G Mobile Networks. Sensors
**2019**, 19, 1335. [Google Scholar] [CrossRef] [PubMed] - Hassan, T.U.; Gao, F. An Active Power Control Technique for Downlink Interference Management in a Two-Tier Macro–Femto Network. Sensors
**2019**, 19, 2015. [Google Scholar] [CrossRef] [PubMed] - Mikaeil, A.M.; Hu, W.; Hussain, S.B.; Sultan, A. Traffic-Estimation-Based Low-Latency XGS-PON Mobile Front-Haul for Small-Cell C-RAN Based on an Adaptive Learning Neural Network. Appl. Sci.
**2018**, 8, 1097. [Google Scholar] [CrossRef] - Wang, N.; Hossain, E.; Bhargava, V. Backhauling 5G small cells: A radio resource management perspective. IEEE Wirel. Commun.
**2015**, 22, 41–49. [Google Scholar] [CrossRef] - Yang, W. Conceptual Verification of Integrated Heterogeneous Network Based on 5G Millimeter Wave Use in Gymnasium. Symmetry
**2019**, 11, 376. [Google Scholar] [CrossRef] - Kelly, F.P.; Maulloo, A.K.; Tan, D.K.H. Rate Control for Communication Networks: Shadow Prices, Proportional Fairness and Stability. J. Oper. Res. Soc.
**1998**, 49, 237–252. [Google Scholar] [CrossRef] - Lin, X.; Shroff, N.B.; Srikant, R. A tutorial on cross-layer optimization in wireless networks. IEEE J. Sel. Area Comm.
**2006**, 24, 1452–1463. [Google Scholar] - Chiang, M.; Low, S.H.; Calderbank, R.; Doyle, J.C. Layering as optimization decomposition. Proc. IEEE
**2007**, 95, 255–312. [Google Scholar] [CrossRef] - Chiang, M. Stochastic network utility maximization. Eur. Trans. Telecommun.
**2008**, 22, 1–22. [Google Scholar] - Shantharama, P.; Thyagaturu, A.S.; Karakoc, N.; Ferrari, L.; Reisslein, M.; Scaglione, A. LayBack: SDN Management of Multi-Access Edge Computing (MEC) for Network Access Services and Radio Resource Sharing. IEEE Access
**2018**, 6, 57545–57561. [Google Scholar] [CrossRef] - Amin, R.; Reisslein, M.; Shah, N. Hybrid SDN networks: A survey of existing approaches. IEEE Commun. Surv. Tutor.
**2018**, 20, 3259–3306. [Google Scholar] [CrossRef] - Haque, I.T.; Abu-Ghazaleh, N. Wireless software defined networking: A survey and taxonomy. IEEE Commun. Surv. Tutor.
**2016**, 18, 2713–2737. [Google Scholar] [CrossRef] - Jagadeesan, N.A.; Krishnamachari, B. Software-Defined Networking Paradigms in Wireless Networks: A Survey. ACM Comput. Surv.
**2014**, 47, 27:1–27:11. [Google Scholar] [CrossRef] - Marabissi, D.; Fantacci, R.; Simoncini, L. SDN-Based Routing for Backhauling in Ultra-Dense Networks. J. Sens. Actuator Netw.
**2019**, 8, 23. [Google Scholar] [CrossRef] - Niephaus, C.; Aliu, O.G.; Kretschmer, M.; Hadzic, S.; Ghinea, G. Wireless Back-haul: A software defined network enabled wireless Back-haul network architecture for future 5G networks. IET Netw.
**2015**, 4, 287–295. [Google Scholar] [CrossRef] - Tayyaba, S.K.; Shah, M.A. Resource allocation in SDN based 5G cellular networks. Peer Netw. Appl.
**2019**, 12, 514–538. [Google Scholar] [CrossRef] - Cavaliere, F.; Iovanna, P.; Mangues-Bafalluy, J.; Baranda, J.; Núñez-Martínez, J.; Lin, K.Y.; Chang, H.W.; Chanclou, P.; Farkas, P.; Gomes, J.; et al. Towards a unified fronthaul-backhaul data plane for 5G The 5G-Crosshaul project approach. Comp. Stand. Interfaces
**2017**, 51, 56–62. [Google Scholar] [CrossRef] - Costa-Perez, X.; Garcia-Saavedra, A.; Li, X.; Deiss, T.; de la Oliva, A.; di Giglio, A.; Iovanna, P.; Moored, A. 5G-Crosshaul: An SDN/NFV Integrated Fronthaul/Backhaul Transport Network Architecture. IEEE Wirel. Commun.
**2017**, 24, 38–45. [Google Scholar] [CrossRef] [Green Version] - Elgendi, I.; Munasinghe, K.S.; Sharma, D.; Jamalipour, A. Traffic offloading techniques for 5G cellular: A three-tiered SDN architecture. Ann. Telecommun.
**2016**, 71, 583–593. [Google Scholar] [CrossRef] - González, S.; Oliva, A.; Costa-Pérez, X.; Di Giglio, A.; Cavaliere, F.; Deiß, T.; Li, X.; Mourad, A. 5G-Crosshaul: An SDN/NFV control and data plane architecture for the 5G integrated Fronthaul/Backhaul. Trans. Emerg. Telecom. Technol.
**2016**, 27, 1196–1205. [Google Scholar] [CrossRef] [Green Version] - Gutiérrez, J.; Maletic, N.; Camps-Mur, D.; García, E.; Berberana, I.; Anastasopoulos, M.; Tzanakaki, A.; Kalokidou, V.; Flegkas, P.; Syrivelis, D.; et al. 5G-XHaul: A converged optical and wireless solution for 5G transport networks. Trans. Emerg. Telecommun. Technol.
**2016**, 27, 1187–1195. [Google Scholar] [CrossRef] - Oliva, L.; De, A.; Perez, X.C.; Azcorra, A.; Giglio, A.D.; Cavaliere, F.; Tiegelbekkers, D.; Lessmann, J.; Haustein, T.; Mourad, A.; et al. Xhaul: Toward an integrated fronthaul/backhaul architecture in 5G networks. IEEE Wirel. Commun.
**2015**, 22, 32–40. [Google Scholar] [CrossRef] - Mayoral, A.; Munoz, R.; Vilalta, R.; Casellas, R.; Martinez, R.; Lopez, V. Need for a transport API in 5G for global orchestration of cloud and networks through a virtualized infrastructure manager and planner [invited]. IEEE/OSA J. Opt. Commun. Netw.
**2017**, 9, A55–A62. [Google Scholar] [CrossRef] - Chih-Lin, I.; Li, H.; Korhonen, J.; Huang, J.; Han, L. RAN Revolution With NGFI (xhaul) for 5G. J. Lightwave Technol.
**2018**, 36, 541–550. [Google Scholar] - Thyagaturu, A.; Dashti, Y.; Reisslein, M. SDN Based Smart Gateways (Sm-GWs) for Multi-Operator Small Cell Network Management. IEEE Trans. Netw. Serv. Manag.
**2016**, 13, 740–753. [Google Scholar] [CrossRef] - Tonini, F.; Khorsandi, B.M.; Bjornstad, S.; Veisllari, R.; Raffaelli, C. C-RAN Traffic Aggregation on Latency-Controlled Ethernet Links. Appl. Sci.
**2018**, 8, 2279. [Google Scholar] [CrossRef] - Cilfone, A.; Davoli, L.; Belli, L.; Ferrari, G. Wireless Mesh Networking: An IoT-Oriented Perspective Survey on Relevant Technologies. Future Internet
**2019**, 11, 99. [Google Scholar] [CrossRef] - Silva, J.d.C.; Rodrigues, J.J.P.C.; Al-Muhtadi, J.; Rabelo, R.A.L.; Furtado, V. Management Platforms and Protocols for Internet of Things: A Survey. Sensors
**2019**, 19, 676. [Google Scholar] [CrossRef] - Kostal, K.; Bencel, R.; Ries, M.; Truchly, P.; Kotuliak, I. High Performance SDN WLAN Architecture. Sensors
**2019**, 19, 1880. [Google Scholar] [CrossRef] - King, D.; Farrel, A.; King, E.N.; Casellas, R.; Velasco, L.; Nejabati, R.; Lord, A. The dichotomy of distributed and centralized control: METRO-HAUL, when control planes collide for 5G networks. Opt. Switch. Netw.
**2019**, 33, 49–55. [Google Scholar] [CrossRef] - Tzanakaki, A.; Anastasopoulos, M.; Berberana, I.; Syrivelis, D.; Flegkas, P.; Korakis, T.; Mur, D.C.; Demirkol, I.; Gutierrez, J.; Grass, E.; et al. Wireless-Optical Network Convergence: Enabling the 5G Architecture to Support Operational and End-User Services. IEEE Commun. Mag.
**2017**, 55, 184–192. [Google Scholar] [CrossRef] - Tassiulas, L.; Ephremides, A. Stability properties of constrained queueing systems and scheduling policies for maximum throughput in multihop radio networks. IEEE Trans. Automat. Contr.
**1992**, 37, 1936–1948. [Google Scholar] [CrossRef] [Green Version] - Tassiulas, L.; Ephremides, A. Dynamic server allocation to parallel queues with randomly varying connectivity. IEEE Trans. Inf. Theory
**1993**, 39, 466–478. [Google Scholar] [CrossRef] - Kar, K.; Luo, X.; Sarkar, S. Throughput-optimal scheduling in multichannel access point networks under infrequent channel measurements. IEEE Trans. Wirel. Commun.
**2008**, 7, 2619–2629. [Google Scholar] [CrossRef] [Green Version] - Ji, B.; Joo, C.; Shroff, N. Throughput-optimal scheduling in multihop wireless networks without per-flow information. IEEE/ACM Trans. Netw.
**2013**, 21, 634–647. [Google Scholar] [CrossRef] - Cui, Y.; Yeh, E.M. Delay optimal control and its connection to the dynamic backpressure algorithm. In Proceedings of the 2014 IEEE International Symposium on Information Theory, Honolulu, HI, USA, 29 June–4 July 2014; pp. 451–455. [Google Scholar]
- Cui, Y.; Yeh, E.M.; Liu, R. Enhancing the delay performance of dynamic backpressure algorithms. IEEE/ACM Trans. Netw.
**2016**, 24, 954–967. [Google Scholar] [CrossRef] - Kar, K.; Sarkar, S.; Ghavami, A.; Luo, X. Delay Guarantees for Throughput-Optimal Wireless Link Scheduling. IEEE Trans. Automat. Contr.
**2012**, 57, 2906–2911. [Google Scholar] [CrossRef] [Green Version] - Neely, M.J. Delay-based network utility maximization. IEEE/ACM Trans. Netw.
**2013**, 21, 41–54. [Google Scholar] [CrossRef] - Palomar, D.P.; Chiang, M. A tutorial on decomposition methods for network utility maximization. IEEE J. Sel. Areas Commun.
**2006**, 24, 1439–1451. [Google Scholar] [CrossRef] - Johansson, B.; Soldati, P.; Johansson, M. Mathematical decomposition techniques for distributed cross-layer optimization of data networks. IEEE J. Sel. Areas Commun.
**2006**, 24, 1535–1547. [Google Scholar] [CrossRef] - Gupta, A.; Lin, X.; Srikant, R. Low-complexity distributed scheduling algorithms for wireless networks. IEEE/ACM Trans. Netw.
**2009**, 17, 1846–1859. [Google Scholar] [CrossRef] - Bui, L.X.; Sanghavi, S.; Srikant, R. Distributed link scheduling with constant overhead. IEEE TON
**2009**, 17, 1467–1480. [Google Scholar] [CrossRef] - Teng, Y.; Song, M. Cross-layer Optimization and Protocol Analysis for Cognitive Ad Hoc Communications. IEEE Access
**2017**, 5, 18692–18706. [Google Scholar] [CrossRef] - Lin, X.; Shroff, N.B.; Srikant, R. On the connection-level stability of congestion-controlled communication networks. IEEE Trans. Inf. Theory
**2008**, 54, 2317–2338. [Google Scholar] [CrossRef] - Srikant, R. On the positive recurrence of a Markov chain describing file arrivals and departures in a congestion-controlled network. In Proceedings of the IEEE Computer Communications Workshop, Orlando, FL, USA, 14–17 March 2004. [Google Scholar]
- Altman, E.; Avrachenkov, K.; Ramanath, S. Multiscale fairness and its application to resource allocation in wireless networks. Comput. Commun.
**2012**, 35, 820–828. [Google Scholar] [CrossRef] [Green Version] - Pham, Q.V.; To, H.L.; Hwang, W.J. A multi-timescale cross-layer approach for wireless ad hoc networks. Comput. Netw.
**2015**, 91, 471–482. [Google Scholar] [CrossRef] - Georgiadis, L.; Neely, M.J.; Tassiulas, L. Resource Allocation and Cross-layer Control in Wireless Networks. Found. Trends Netw.
**2006**, 1, 1–144. [Google Scholar] [CrossRef] - Neely, M.J. Energy Optimal Control for Time-varying Wireless Networks. IEEE Trans. Inf. Theory
**2006**, 52, 2915–2934. [Google Scholar] [CrossRef] - Ge, X.; Tu, S.; Mao, G.; Lau, V.; Pan, L. Cost Efficiency Optimization of 5G Wireless Backhaul Networks. IEEE Trans. Mob. Comput.
**2018**. [Google Scholar] [CrossRef] - Luong, N.C.; Wang, P.; Niyato, D.; Liang, Y.C.; Han, Z.; Hou, F. Applications of economic and pricing models for resource management in 5G wireless networks: A survey. IEEE Commun. Surv. Tutor.
**2018**. [Google Scholar] [CrossRef] - Bernal-Mor, E.; Pla, V.; Martinez-Bauset, J.; Pacheco-Paramo, D. A model of resource management in small cells with dynamic traffic and backhaul constraints. In Proceedings of the IEEE 19th European Wireless Conference (EW), Guildford, UK, 16–18 April 2013; pp. 1–6. [Google Scholar]
- Biermann, T.; Scalia, L.; Choi, C.; Karl, H.; Kellerer, W. CoMP clustering and backhaul limitations in cooperative cellular mobile access networks. Pervasive Mob. Comp.
**2012**, 8, 662–681. [Google Scholar] [CrossRef] - De Domenico, A.; Savin, V.; Ktenas, D. A backhaul-aware cell selection algorithm for heterogeneous cellular networks. In Proceedings of the IEEE Personal Indoor and Mobile Radio Communications, London, UK, 8–11 September 2013; pp. 1688–1693. [Google Scholar]
- Lakshminarayana, S.; Assaad, M.; Debbah, M. H-infinity control based scheduler for the deployment of small cell networks. Perform. Eval.
**2013**, 70, 513–527. [Google Scholar] [CrossRef] - Li, W.; Zi, Y.; Feng, L.; Zhou, F.; Yu, P.; Qiu, X. Latency-Optimal Virtual Network Functions Resource Allocation for 5G Backhaul Transport Network Slicing. Appl. Sci.
**2019**, 9, 701. [Google Scholar] [CrossRef] - Liu, T.; Wang, K.; Ku, C.; Hsu, Y. QoS-aware resource management for multimedia traffic report systems over LTE-A. Comput. Netw.
**2016**, 94, 375–389. [Google Scholar] [CrossRef] - Liu, J.; Zhou, S.; Gong, J.; Niu, Z.; Xu, S. Statistical Multiplexing Gain Analysis of Heterogeneous Virtual Base Station Pools in Cloud Radio Access Networks. IEEE Trans. Wirel. Commun.
**2016**, 15, 5681–5694. [Google Scholar] [CrossRef] [Green Version] - Niu, B.; Zhou, Y.; Shah-Mansouri, H.; Wong, V.W.S. A Dynamic Resource Sharing Mechanism for Cloud Radio Access Networks. IEEE Trans. Wirel. Commun.
**2016**, 15, 8325–8338. [Google Scholar] [CrossRef] - Samdanis, K.; Shrivastava, R.; Prasad, A.; Grace, D.; Costa-Perez, X. TD-LTE Virtual Cells: An SDN Architecture for User-centric Multi-eNB Elastic Resource Management. Comput. Commun.
**2016**, 83, 1–15. [Google Scholar] [CrossRef] - Semiari, O.; Saad, W.; Valentin, S.; Bennis, M.; Vincent Poor, H. Context-Aware Small Cell Networks: How Social Metrics Improve Wireless Resource Allocation. IEEE Trans. Wirel. Commun.
**2015**, 14, 5927–5940. [Google Scholar] [CrossRef] - Taleb, T.; Hadjadj-Aoul, Y.; Samdanis, K. Efficient solutions for enhancing data traffic management in 3GPP networks. IEEE Syst. J.
**2015**, 9, 519–528. [Google Scholar] [CrossRef] - Ali, A.; Shah, G.A.; Arshad, J. Energy Efficient Resource Allocation for M2M Devices in 5G. Sensors
**2019**, 19, 1830. [Google Scholar] [CrossRef] - Cen, Y.; Cen, Y.; Wang, K.; Li, J. Energy-Efficient Nonuniform Content Edge Pre-Caching to Improve Quality of Service in Fog Radio Access Networks. Sensors
**2019**, 19, 1422. [Google Scholar] [CrossRef] - Scarpiniti, M.; Baccarelli, E.; Momenzadeh, A. VirtFogSim: A Parallel Toolbox for Dynamic Energy-Delay Performance Testing and Optimization of 5G Mobile-Fog-Cloud Virtualized Platforms. Appl. Sci.
**2019**, 9, 1160. [Google Scholar] [CrossRef] - Yang, J.; Luo, J.; Lin, F.; Wang, J. Content-Sensing Based Resource Allocation for Delay-Sensitive VR Video Uploading in 5G H-CRAN. Sensors
**2019**, 19, 697. [Google Scholar] [CrossRef] - Prasad, N.; Arslan, M.; Rangarajan, S. A two time scale approach for coordinated multi-point transmission and reception over practical backhaul. In Proceedings of the Sixth International Conference on Communication Systems and Networks (COMSNETS), Bangalore, India, 6–10 January 2014; pp. 1–8. [Google Scholar]
- Tang, J.; Teng, L.; Quek, T.Q.S.; Chang, T.; Shim, B. Exploring the interactions of communication, computing and caching in cloud RAN under two timescale. In Proceedings of the IEEE 18th IEEE International Workshop on Signal Processing Advances in Wireless Communications (SPAWC), Sapporo, Japan, 3–6 July 2017; pp. 1–6. [Google Scholar]
- Tang, J.; Shim, B.; Quek, T.Q.S. Service Multiplexing and Revenue Maximization in Sliced C-RAN Incorporated With URLLC and Multicast eMBB. IEEE J. Sel. Areas Commun.
**2019**, 37, 881–895. [Google Scholar] [CrossRef] - Lyu, X.; Ren, C.; Ni, W.; Tian, H.; Liu, R.P.; Guo, Y.J. Multi-Timescale Decentralized Online Orchestration of Software-Defined Networks. IEEE J. Sel. Areas Commun.
**2018**, 36, 2716–2730. [Google Scholar] [CrossRef] - Xia, W.; Quek, T.Q.S.; Zhang, J.; Jin, S.; Zhu, H. Programmable Hierarchical C-RAN: From Task Scheduling to Resource Allocation. IEEE Trans. Wirel. Commun.
**2019**, 18, 2003–2016. [Google Scholar] [CrossRef] - Chen, X.; Ni, W.; Chen, T.; Collings, I.; Wang, X.; Liu, R.P.; Giannakis, G.B. Multi-timescale online optimization of network function virtualization for service chaining. IEEE Trans. Mob. Comput.
**2018**. [Google Scholar] [CrossRef] - Yao, Y.; Huang, L.; Sharma, A.B.; Golubchik, L.; Neely, M.J. Power cost reduction in distributed data centers: A two-time-scale approach for delay tolerant workloads. IEEE Trans. Parallel Distrib. Syst.
**2014**, 25, 200–211. [Google Scholar] - Yu, L.; Jiang, T.; Cao, Y.; Qi, Q. Joint workload and battery scheduling with heterogeneous service delay guaranteesfor data center energy cost minimization. IEEE Trans. Parallel Distrib. Syst.
**2015**, 26, 1937–1947. [Google Scholar] [CrossRef] - Wang, X.; Chen, X.; Chen, T.; Huang, L.; Giannakis, G.B. Two-scale stochastic control for integrated multipoint communication systems with renewables. IEEE Trans. Smart Grid
**2018**, 9, 1822–1834. [Google Scholar] [CrossRef] - Karakoc, N.; Scaglione, A.; Nedic, A. Multi-layer Decomposition of Optimal Resource Sharing Problems. In Proceedings of the 2018 IEEE Conference on Decision and Control (CDC), Miami Beach, FL, USA, 17–19 December 2018; pp. 178–183. [Google Scholar]
- Gowdal, N.M.; Si, X.; Sabharwall, A. Full-duplex DOCSIS: A Modem Architecture for Wideband (>1 GHZ) Self-interference Cancellation for Cable Modem Termination Systems (CMTS). In Proceedings of the IEEE 52nd Asilomar Conference on Signals, Systems, and Computers, Pacific Grove, CA, USA, 28–31 October 2018; pp. 2202–2206. [Google Scholar]
- Granizo Arrabe, R.A.; Platero, C.; Alvarez Gomez, F.; Rebollo Lopez, E. New Differential Protection Method for Multiterminal HVDC Cable Networks. Energies
**2018**, 11, 3387. [Google Scholar] [CrossRef] - Thyagaturu, A.S.; Alharbi, Z.; Reisslein, M. R-FFT: Function split at IFFT/FFT in unified LTE CRAN and cable access network. IEEE Trans. Broadcast.
**2018**, 64, 648–665. [Google Scholar] [CrossRef] - Sheu, J.B.; Pan, C. Relief supply collaboration for emergency logistics responses to large-scale disasters. Transp. A Transp. Sci.
**2015**, 11, 210–242. [Google Scholar] [CrossRef] - Chen, T.; Alsafasfeh, Q.; Pourbabak, H.; Su, W. The next-generation US retail electricity market with customers and prosumers—A bibliographical survey. Energies
**2017**, 11, 8. [Google Scholar] [CrossRef] - Chen, S.; Liu, C.C. From demand response to transactive energy: State of the art. J. Mod. Power Syst. Clean Energy
**2017**, 5, 10–19. [Google Scholar] [CrossRef] - Ożadowicz, A. A new concept of active demand side management for energy efficient prosumer microgrids with smart building technologies. Energies
**2017**, 10, 1771. [Google Scholar] [CrossRef] - Rahimi, F.A.; Ipakchi, A. Transactive energy techniques: Closing the gap between wholesale and retail markets. Electr. J.
**2012**, 25, 29–35. [Google Scholar] [CrossRef] - Allen, A.O. Probability, Statistics, and Queuing Theory with Computer Science Applications, 2nd ed.; Academic Press: San Diego, CA, USA, 1990. [Google Scholar]
- Banirazi, R.; Jonckheere, E.; Krishnamachari, B. Heat diffusion algorithm for resource allocation and routing in multihop wireless networks. In Proceedings of the IEEE Global Communications Conference (GLOBECOM), Anaheim, CA, USA, 3–7 December 2012; pp. 5693–5698. [Google Scholar]
- Low, S.H.; Lapsley, D.E. Optimization flow control. I. Basic algorithm and convergence. IEEE/ACM Trans. Netw.
**1999**, 7, 861–874. [Google Scholar] [CrossRef] [Green Version] - Bikram Kumar, B.; Sharma, L.; Wu, S.L. Online Distributed User Association for Heterogeneous Radio Access Network. Sensors
**2019**, 19, 1412. [Google Scholar] [CrossRef] - Caballero, P.; Banchs, A.; De Veciana, G.; Costa-Pérez, X. Multi-tenant radio access network slicing: Statistical multiplexing of spatial loads. IEEE/ACM Trans. Netw.
**2017**, 25, 3044–3058. [Google Scholar] [CrossRef] - Eramo, V.; Listanti, M.; Lavacca, F.G.; Iovanna, P. Dimensioning Models of Optical WDM Rings in Xhaul Access Architectures for the Transport of Ethernet/CPRI Traffic. Appl. Sci.
**2018**, 8, 612. [Google Scholar] [CrossRef] - Smith, D.R.; Whitt, W. Resource sharing for efficiency in traffic systems. Bell Syst. Technol. J.
**1981**, 60, 39–55. [Google Scholar] [CrossRef] - Aldous, D. Probability Approximations via the Poisson Clumping Heuristic; Springer Science & Business Media: Berlin, Germany, 2013; Volume 77. [Google Scholar]
- Louchard, G.; Finch, S. Traffic Light Queues and the Poisson Clumping Heuristic. arXiv
**2018**, arXiv:1810.12058. [Google Scholar] - Liu, B.; Xu, H.; Zhou, X. Resource Allocation in Wireless-Powered Mobile Edge Computing Systems for Internet of Things Applications. Electronics
**2019**, 8, 206. [Google Scholar] [CrossRef] - Wang, S.; Zhang, X.; Zhang, Y.; Wang, L.; Yang, J.; Wang, W. A survey on mobile edge networks: Convergence of computing, caching and communications. IEEE Access
**2017**, 5, 6757–6779. [Google Scholar] [CrossRef] - Xiang, Z.; Gabriel, F.; Urbano, E.; Nguyen, G.T.; Reisslein, M.; Fitzek, F.H. Reducing latency in virtual machines: Enabling tactile Internet for human-machine co-working. IEEE J. Sel. Areas Commun.
**2019**, 37, 1098–1116. [Google Scholar] [CrossRef] - Ogudo, K.A.; Muwawa Jean Nestor, D.; Ibrahim Khalaf, O.; Daei Kasmaei, H. A Device Performance and Data Analytics Concept for Smartphones’ IoT Services and Machine-Type Communication in Cellular Networks. Symmetry
**2019**, 11, 593. [Google Scholar] [CrossRef] - Arena, F.; Pau, G. An Overview of Vehicular Communications. Future Internet
**2019**, 11, 27. [Google Scholar] [CrossRef] - Santa, J.; Fernandez, P.J.; Ortiz, J.; Sanchez-Iborra, R.; Skarmeta, A.F. SURROGATES: Virtual OBUs to Foster 5G Vehicular Services. Electronics
**2019**, 8, 117. [Google Scholar] [CrossRef] - Storck, C.R.; Duarte-Figueiredo, F. A 5G V2X Ecosystem Providing Internet of Vehicles. Sensors
**2019**, 19, 550. [Google Scholar] [CrossRef] - Frascolla, V.; Dominicini, C.K.; Paiva, M.H.M.; Caporossi, G.; Marotta, M.A.; Ribeiro, M.R.N.; Segatto, M.E.V.; Martinello, M.; Monteiro, M.E.; Both, C.B. Optimizing C-RAN Backhaul Topologies: A Resilience-Oriented Approach Using Graph Invariants. Appl. Sci.
**2019**, 9, 136. [Google Scholar] [CrossRef] - Gazit, L.; Messer, H. Advancements in the Statistical Study, Modeling, and Simulation of Microwave-Links in Cellular Backhaul Networks. Environments
**2018**, 5, 75. [Google Scholar] [CrossRef] - Tran, G.K.; Santos, R.; Ogawa, H.; Nakamura, M.; Sakaguchi, K.; Kassler, A. Context-Based Dynamic Meshed Backhaul Construction for 5G Heterogeneous Networks. J. Sens. Actuator Netw.
**2018**, 7, 43. [Google Scholar] [CrossRef] - Caria, M.; Jukan, A.; Hoffmann, M. SDN partitioning: A centralized control plane for distributed routing protocols. IEEE Trans. Netw. Serv. Manag.
**2016**, 13, 381–393. [Google Scholar] [CrossRef] - Guck, J.W.; Van Bemten, A.; Reisslein, M.; Kellerer, W. Unicast QoS routing algorithms for SDN: A comprehensive survey and performance evaluation. IEEE Commun. Surv. Tutor.
**2018**, 20, 388–415. [Google Scholar] [CrossRef] - Navarro-Ortiz, J.; Cervelló-Pastor, C.; Stea, G.; Costa-Perez, X.; Triay, J. Testbeds for Future Wireless Networks. Wirel. Commun. Mob. Comput.
**2019**, 2019, 1–2. [Google Scholar] [CrossRef] - Sivaramakrishnan, S.R.; Mikovic, J.; Kannan, P.G.; Mun Choon, C.; Sklower, K. Enabling SDN experimentation in network testbeds. In Proceedings of the ACM International Workshop on Security in Software Defined Networks & Network Function Virtualization, Scottsdale, AZ, USA, 24 March 2017; pp. 7–12. [Google Scholar]
- Siracusano, G.; Salsano, S.; Ventre, P.L.; Detti, A.; Rashed, O.; Blefari-Melazzi, N. A framework for experimenting ICN over SDN solutions using physical and virtual testbeds. Comput. Netw.
**2018**, 134, 245–259. [Google Scholar] [CrossRef]

**Figure 1.**Illustration of LayBack architecture and multi-timescale optimization decomposition in context of cellular networks: LayBack partitions the wireless backhaul infrastructure into radio node layer, gateway layer, SDN switching layer, and core network layer. The entire network is controlled by the central unifying SDN orchestrator. This case study decomposes the optimization of the sharing of the backhaul bitrate of multiple operator core networks into fast-timescale sub-problems at the radio nodes and progressively slower timescale sub-problems at the gateways and operator core networks; whereby all sub-problems are coordinated through a root problem at the SDN orchestrator.

**Figure 2.**Illustration of the dynamics of the multi-timescale optimization framework within context of LayBack infrastructure: the optimal policy to minimize end-to-end delay is decoupled into multiple layers of sub-problems, with faster timescales at the lower LayBack layers The eNBs $n,\phantom{\rule{4pt}{0ex}}n\in {\mathcal{N}}_{g}$, at a GW g pass their queue occupancies each eNB-GW round-trip time RTT ${\tau}_{N}^{G}$ to GW g. Based on the received vector of queue occupancies $\mathbf{Q}$, GW g evaluates the allocations ${\mathbf{z}}_{g}$ to its eNBs with Algorithm 1. Similarly, the SDN orchestrator evaluates the allocations $\mathbf{x}$ to the operators with Algorithms 2 and 3; while each operator o evaluates the allocations ${\mathbf{y}}_{o}$ to its GWs with Algorithms 4 and 5. (In order to reduce clutter, the eNB-to-GW RTT ${\tau}_{N}^{G}$ has been normalized to one in the illustration, i.e., ${K}_{1}$ in the illustration corresponds to ${K}_{1}{\tau}_{N}^{G}$ in actual time).

**Figure 3.**Upstream traffic demands and corresponding backhaul bitrate allocations to operators as well as aggregated queue length of eNBs associated with a given operator when peak demand periods of the two operators overlap or are separated (fixed parameter: Mean drift-plus-penalty parameter $V=1000$): For overlapping peak demands (

**a**,

**b**), both the SDN-based optimization and the benchmark without SDN allocate to each operator its maximum capacity of ${Z}_{o}=10$ Mbps to serve the peak demands; there is no sharing among operators. For separated peak demands (

**c**,

**d**), the SDN orchestrator dynamically shares the total aggregated backhaul capacity of $Z=20$ Mbps among the two operators, reducing eNB queue lengths compared to the benchmark without SDN-based resource sharing.

**Figure 4.**Upstream backhaul bitrate allocations and eNB queue lengths when demand peaks for operators 1 and 2 are spaced apart: Increasing the “flexibility parameter” V, see Equation (7), increases the sharing of backhaul capacity among the two operators and decreases the queue lengths compared to the benchmark without SDN orchestrated backhaul resource sharing; for $V=1000$, please refer to Figure 3c,d. The QMW case corresponds to the SDN-based optimization without a long-term constraint.

**Figure 6.**Mean and CDF of eNB queue length in kB for $O=2$ operators with random traffic burst as a function of steady-state probability ${p}_{\mathrm{on}}$ of burst state.

**Figure 7.**Cumulative distribution function (CDF) of eNB queue length for independent eNB traffic bursts with various load levels for long (10 s) bursts and medium load for short (0.5 s) bursts; fixed parameters: $O=20$ operators, each with two gateways (each with five eNBs).

Values | ||
---|---|---|

Parameter | Notation | (For Eval. in Section 5) |

Backhaul Netw. Architecture | ||

# of Operators (indexed $o=1,\dots ,O$) | O | 2 |

# of GWs per oper. o | $|{\mathcal{G}}_{o}|$ | 3 |

# of eNBs per GW g | $|{\mathcal{N}}_{g}|$ | 10 |

Total Backhaul Cap. (Mbps) | Z | 20 |

Operator Backhaul Cap. (Mbps) | ${Z}_{o}$ | 10 |

eNB-to-GW RTT (ms) | ${\tau}_{N}^{G}$ | 1 |

GW to Operator RTT (ms) | ${\tau}_{G}^{O}$ | 100 |

Operat. to SDN Orch. RTT (s) | ${\tau}_{O}^{S}$ | 1 |

Resource Allocations | ||

Cap. alloc. to Oper. o | ${x}_{o}$ | |

Vector of Oper. alloc. | $\mathbf{x}=\{{x}_{1},\dots ,{x}_{O}\}$ | |

Cap. alloc. to GW g | ${y}_{g}$ | |

Vector of alloc. to GWs at Op. o | ${\mathbf{y}}_{o}=\{{y}_{g}:g\in {\mathcal{G}}_{o}\}$ | |

Cap. alloc. to eNB n | ${z}_{n}$ | |

Vector of alloc. to eNBs at GW g | ${\mathbf{z}}_{g}=\{{z}_{n}:n\in {\mathcal{N}}_{g}\}$ |

© 2019 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (http://creativecommons.org/licenses/by/4.0/).

## Share and Cite

**MDPI and ACS Style**

Wang, M.; Karakoc, N.; Ferrari, L.; Shantharama, P.; Thyagaturu, A.S.; Reisslein, M.; Scaglione, A.
A Multi-Layer Multi-Timescale Network Utility Maximization Framework for the SDN-Based LayBack Architecture Enabling Wireless Backhaul Resource Sharing. *Electronics* **2019**, *8*, 937.
https://doi.org/10.3390/electronics8090937

**AMA Style**

Wang M, Karakoc N, Ferrari L, Shantharama P, Thyagaturu AS, Reisslein M, Scaglione A.
A Multi-Layer Multi-Timescale Network Utility Maximization Framework for the SDN-Based LayBack Architecture Enabling Wireless Backhaul Resource Sharing. *Electronics*. 2019; 8(9):937.
https://doi.org/10.3390/electronics8090937

**Chicago/Turabian Style**

Wang, Mu, Nurullah Karakoc, Lorenzo Ferrari, Prateek Shantharama, Akhilesh S. Thyagaturu, Martin Reisslein, and Anna Scaglione.
2019. "A Multi-Layer Multi-Timescale Network Utility Maximization Framework for the SDN-Based LayBack Architecture Enabling Wireless Backhaul Resource Sharing" *Electronics* 8, no. 9: 937.
https://doi.org/10.3390/electronics8090937