An Experimental Study on D2D Route Selection Mechanism in 5G Scenarios

Abstract

This paper demonstrates a route selection mechanism on a testbed with heterogeneous device-to-device (D2D) wireless communication for a 5G network scenario. The source node receives information about the primary users’ (PUs’) (or licensed users’) activities and available routes from the macrocell base station (or a central controller) and makes a decision to select a multihop route to the destination node. The source node from small cells can either choose: (a) a route with direct communication with the macrocell base station to improve the route performance; or (b) a route with D2D communication among nodes in the small cells to offload traffic from the macrocell to improve spectrum efficiency. The selected D2D route has the least PUs’ activities. The route selection mechanism is investigated on our testbed that helps to improve the accuracy of network performance measurement. In traditional testbeds, each node (e.g., Universal Software Radio Peripheral (USRP) that serves as the front-end communication block) is connected to a single processing unit (e.g., a personal computer) via a switch using cables. In our testbed, each USRP node is connected to a separate processing unit, i.e., raspberry Pi3 B+ (or RP3), which offers three main advantages: (a) control messages and data packets are exchanged via the wireless medium; (b) separate processing units make decisions in a distributed and heterogeneous manner; and (c) the nodes are placed further apart from one another. Therefore, in the investigation of our route selection scheme, the response delay of control message exchange and the packet loss caused by the operating environment (e.g., ambient noise) are implied in our end-to-end delay and packet delivery ratio measurement. Our results show an increase of end-to-end delay and a decrease of packet delivery ratio due to the transmission of control messages and data packets in the wireless medium in the presence of the dynamic PUs’ activities. Furthermore, D2D communication can offload 25% to 75% traffic from macrocell base station to small cells.

1. Introduction

Fifth generation (5G) is a promising next-generation cellular mobile network that aims to enhance network performance, including higher quality of service (QoS) (e.g., ultra-low latency, and high throughput), higher reliability (e.g., better network connectivity with lower network congestion), and larger coverage, to cater for future network requirements, particularly handling a massive amount of data transmission and heterogeneous devices. 5G is equipped with new features, including device-to-device (D2D) communication and small cell (SC) e.g., femtocell ($fc$) deployment that help to increase network capacity to cope with ultra-dense networks that consist of a massive number of users and amount of data. Nevertheless, the new network requirements and features have brought some new issues, including higher co-channel interference among SCs [1]. Hence, 5G consists of a control plane, which is managed by a central controller (CC) in the core network that manages and optimizes network resources [2,3]. For instance, the CC manages secondary users’ (or SUs’) (or unlicensed users’) access to white spaces, which are underused licensed channels (or cognitive channels) owned by the primary users (or PUs) (or licensed users) [4]. Each node can access both cellular channels (in which the node is the PU) and cognitive channels (in which the node is the SU). D2D enables direct communication between two nodes without traversing the base station (BS), particularly the macrocell (MC) BS (or a central controller) that has the largest coverage in 5G networks [5] using D2D communication, a node can access both cellular and cognitive channels; as a result, it improves spectrum efficiency and QoS [6,7].

Theoretical analysis has been conducted in [1,4,8,9]. As an example, in [8], the handover process is investigated to determine the optimal number of SCs in the network. Data streams can be transmitted through either SCs or macrocells (MCs). Although a higher number of SCs in the network may seem to offload traffic from MCs and reduce the congestion level of MCs, it increases interference among SCs and the need for data transmission through MCs. As another example, in [4], a routing algorithm is proposed to select routes based on the CC’s network-wide information, including intermediate nodes, the traffic of each link, and the number of hops of a route, to reduce interference among routes and achieve better load balancing. Meanwhile, simulation has been conducted in [3,4,9,10,11,12]. In [3,12], traffic offload from the CC and backbone routes [Backbone route is not defined yet prior to this] to SC nodes has shown to reduce access and end-to-end delays. In [9], a multihop D2D communication to deliver a high-quality video stream to a destination has shown to increase network capacity at the expense of a higher end-to-end delay. In [10], a virtual BS is proposed to determine the available channels, so that a source node can use this information to reduce the delay of a route. In [11], CC manages D2D communication among user devices in different channels, and it has shown to improve data transmission. Nevertheless, there is lack of experimental study using a testbed.

In this paper, an experimental study is conducted to investigate route selection in the presence of PUs in the network layer of a 5G scenario. The testbed consists of universal software radio peripheral with GNU radio (USRP/GNU radio) [13] connected to Raspberry Pi3 B+ (RP3) [14], which serves as the decision-making engine at a source node. Equipped with RP3, our testbed is first of its kind to remove some significant assumptions used in previous testbeds, such as the use of a single processing unit (i.e., a personal computer, PC) via a switch, in route selection investigations.

The CC receives PUs’ activities information from source nodes and ranks available routes based on this information (or the number of white spaces) which is then informed to the source node. The source node has its data stream information (i.e., the format and size of data) and receives network-wide information, including the available routes, as well as the hop count and the number of white spaces of each route, from CC through MC BS. The source node makes route selection, taking into account: (a) the number of white spaces, which are random in nature, along with given routes ranked by CC; and (b) the hop count of the route. A route with lower hop count and higher number of white spaces is ranked higher.

To improve the accuracy of network performance measurement, we have chosen to use our testbed. In traditional testbeds, all USRP/GNU radio nodes are connected to a single processing unit (e.g., a personal computer, PC) via a switch using cables, and so there are three assumptions made in the experiment: (a) control messages are exchanged via the wired cables and data packets are exchanged via the wireless medium; (b) a single processing unit makes decisions in a centralized and homogeneous manner [15]; and (c) the nodes are placed close to one another due to the constraint of the wired cable. The assumptions affect the measurement of network performance, particularly end-to-end delay and packet loss ratio. Our testbed is different from the traditional testbed, whereby each node is connected to separate processing units (i.e., RP3), and so it provides three main advantages: (a) control messages and data packets are exchanged via the wireless medium; (b) separate processing units make decisions in a distributed and heterogeneous manner (e.g., separate processing units with different processing capability); and (c) the nodes are placed further apart from one another. Therefore,  the response delay of control message exchange and the packet loss caused by the operating environment (e.g., ambient noise) are not ignored in our end-to-end delay and packet delivery ratio measurement. Network performance measures, including the amount of traffic offload from the MC to SCs, as well as the packet delivery ratio and end-to-end delay of D2D routes and the backbone route, provide insights on whether traffic offload using D2D is preferred in route selection.

1.1. Our Contributions

Our contribution is a route selection mechanism that enables a source node to select a route to a destination node independently in the presence of PUs’ activities via either: (a) direct communication with the MC BS to improve the route performance; or (b) D2D communication among nodes in the SCs to offload traffic from the MC to alleviate traffic congestion in MCs and improve the overall spectrum efficiency. Our experiment is performed on a testbed that consists of USRP/GNU radio units connected to RP3, which serves as the decision-making engine of the source node, rather than in simulation. This helps to improve the accuracy of network performance measurement compared to that in simulation, which generally applies some assumptions, whereby: (a) nodes in simulation may be homogeneous, rather than heterogeneous with different processing and transmission capabilities, resources, and so on; and (b) the absence of the ambient environment that affects the quality of communication due to the presence of ambient signals (or interference) and white noise. This paper is an extension to our previous paper [16]. Although our focus is on route selection in this paper, in [16], the focus is on the testbed and to investigate and compare network performance, such as software and hardware processing delays, of a single two-hop route in both traditional and our proposed testbeds with fewer nodes and routes.

1.2. Organization of this Paper

The rest of this paper is organized as follows. Section 2 presents background. Section 3 presents related work. Section 4 presents system model and route selection. Section 5 presents experiment setup parameters and metrics. Section 6 presents results and discussions. Section 7 presents conclusion and future work.

2. Background

In this section, an overview of 5G, USRP, GNU radio, and RP3 is presented.

2.1. 5G

5G networks consist of heterogeneous network cells (i.e., MC and $fc$) and nodes with different kinds of characteristics and capabilities, including transmission power and use of frequency bands.

In Figure 1, a 5G network with control and data planes is shown. The MC BS in the control plane receives network-wide information, including changes of the routes and node availability [4,17], from the CC. The MC BS communicates with a higher transmission power in lower frequency bands, and so it covers a larger area with more $fc$ BSs and nodes, including the source node in the data plane. On the contrary, the $fc$ BSs and nodes in the data plane communicates with a lower transmission power in higher frequency bands since they are in close proximity. Nodes can communicate via D2D [18], and so a $fc$ source node $f{c}_s$ can communicate with a $fc$ destination node $f{c}_d$, which is located out of its transmission range, in multiple hops.

Although there are similarities between our 5G-based testbed and traditional networks, such as wireless sensor networks (WSN), particularly in the use of D2D routes, there are four main differences. Firstly, a source node must switch its route when PUs’ activities reappear in the operating route in our testbed, while there is lack of PUs’ activities in WSNs [19]. Secondly, the source node sends a single video stream to a destination node in our testbed, while sensor nodes send many packets to the sink node whenever an event is detected in WSNs [20]. Thirdly, there are heterogeneous routes, including D2D routes and the backbone route, and the D2D routes offload traffic from the backbone route in our testbed. However, there are homogeneous routes (i.e., D2D routes) only in WSNs. In other words, a source node can choose either next-hop node or the MC BS in our testbed, but a source node must choose a next-hop sensor node in WSNs [21]. Fourthly, our investigation focuses network performance, such as end-to-end delay and packet delivery ratio measurements, with a minimal focus on energy efficiency, which is the key performance measure in WSNs [22].

2.2. Universal Software Radio Peripheral

USRP is an off-the-shelf communication device with configurable operating parameters, such as operating frequency bands and types of modulations. Our testbed consists of USRP N200 series units (see Figure 2) with high processing and computation capabilities. Each USRP unit of our testbed consists of two omni-directional VERT900 antennas that enable simultaneous transmission and reception, respectively, in different frequency bands within 850–960 MHz VHF bands. Digital signals are transferred from the RP3 processing unit to the front-end antenna for transmission in a transmit path, and analogue signals are transferred from the front-end antenna to the RP3 processing unit for reception in a receive path.

In this testbed, the USRP unit consists of three main sections as below:

• Field-programmable gate array (FPGA), which is the Xilinx Spartan 3A-DSP 1800 board, consists of: (a) interpolation filter that constructs data within a predefined range in the transmit path, and decimation filter that ensures signal achieves the required interface bandwidth in the receive path; (b) USRP hardware driver (UHD) block is a software interface that enables communication with GNU radio in RP3; (c) a processor that processes signals required for software defined radio (SDR), including encoding/decoding, modulation/demodulation, and time synchronization operations [23]. FPGA communicates with RP3 using power over Ethernet (POE), which has a maximum data rate of 1000 megabits per second (Mbps).
• Converter consists of; (a) analogue-to-digital converter in the receive path; (b) digital down converter that selects desired signals from an array of signals captured by the analogue-to-digital converter in the receive path; (c) digital-to-analogue converter in the transmit path; and (d) digital-up converter that increases the bandwidth of the baseband signal so that it is compatible for digital-to-analogue conversion in the transmit path.
• Wide bandwidth transceiver (WBX) is the RF front-end that provides access to different frequency bands, particularly 16 bit-sample (or 40 MHz RF bandwidth) and 8 bit-sample (or 25 MHz bandwidth). The maximum transmission power is 20 dBm (or 100 mW) with a noise figure of 5 dB.

2.3. GNU Radio

GNU radio is an open source SDR based on C++ language that enables communication among configurable blocks of different nodes, including the source, intermediate, and destination nodes. GNU radio companion (GRC) is an extended version of GNU radio that provides a user-friendly interface, and it has wider range of configurable blocks and functions [24]. GNU radio flow graphs, which consist of redconfigurable blocks, for the source, intermediate, and destination nodes of our testbed are shown in Figure 3. The functions and processes of the blocks that a data stream goes through from a source node to a destination node is shown in

Table 1. Blocks used in the flow graphs of the GNU radio of the source, intermediate, and destination nodes.

Node Type	Process	Description
Source node	UDP source	RP3 unit with an Internet protocol (IP) address 192.168.10.2 and a port number 1234 generates data stream with a predefined payload size of 1472 bytes in the form of frames encapsulated in user datagram protocol (UDP).
	Packet encoder	Converts frames into packets with two samples per symbol (or one bit per symbol).
	Gaussian minimum shift keying (GMSK) modulator	Converts packets, or non-zero byte stream, into modulated signals at baseband.
	USRP sink	Sets the signal parameters, including the center frequency 850 MHz, channel gain 1 dB, sample rate 1.1 MB, and the intermediate node address, and then broadcasts the signals.
Intermediate node	USRP source	Receives signals from a source node or an upstream intermediate node.
	GMSK demodulator	Converts complex modulated signals at baseband into packets with a rate of 2 samples per symbol.
	Packet decoder	Converts packets into frames in bytes.
Destination node	USRP sink	RP3 unit receives data stream and run it on application, such as a media player.

Please note that the packet encoder, GMSK modulator, and USRP sink of the source and intermediate nodes are similar. Also, the USRP source, GMSK demodulator, and packet decoder of the intermediate and destination nodes are similar.

.

2.4. Raspberry Pi3 B+

Our testbed consists of USRP/GNU radio nodes equipped with separate RP3 processing unit. Traditionally, a testbed connects BSs and nodes to a single processing unit (i.e., a PC) using gigabit ethernet cables via a switch [15]. Hence, BSs and nodes exchange control messages through wired medium, and data packets through wireless medium. On the contrary, BSs and nodes exchange control messages and data packets through wireless medium in our testbed. Our testbed provides two main advantages and convenience. The two advantages are: (a) flexible network topology because USRP/GNU radio nodes can be placed further apart from one another, therefore network performance is more realistic since more aspects, including the delays incurred by processing units, the delays incurred by control message and data packet transmissions, and the effects of PU interference, are taken into account; and (b) lower cost because of the simplicity of the platform that consists of nodes equipped with mini-computers (i.e., RP3), which have a lower cost and energy consumption, while traditional testbeds are comprised of a PC and a switch. Moreover, our separate independent processing unit (i.e., RP3) embedded in each USRP/GNU radio unit provides realistic settings and scenarios for experiment in the following aspects:

• Processing capability. RP3 offers a lower computation power using a 1.4 GHz 64-bit quad-core processor with 1 GB unexpendable on-board RAM, which is sufficient for simple applications, such as GRC in the Linux operating system and basic computational tasks. RP3 also incurs a higher access delay because RP3 reads from and writes to an embedded SD card with limited read and write speed. Specifically, a high-capacity HC-I class 10 SD card enables a maximum 10 Mbps of read rate, which is similar to that in conventional smart phones (or cell phones). The speed is lower compared to a maximum of 550 Mbps in a solid-state hard drive available in a PC commonly found in a traditional testbed.
• Communication mechanism. RP3 offers a lower data rate, which is approximately 324 Mbps, compared to 761 Mbps using a gigabit ethernet cable and a CORE i7 PC. Although RP3 is expected to increase end-to-end delay, it reflects a more realistic D2D communication between nodes, whereby both control messages and data packets are transferred through wireless medium.
• Storage space. RP3 offers a lower storage space (i.e., 32 GB), which is similar to that in the RAM of conventional smart phones. The read and write speed reduce with increasing amount of data stored in the storage space. Nevertheless, it provides a more feasible platform compared to a traditional testbed comprised of a PC and a switch.

3. Related Work

This section presents related work on D2D route selection on testbeds, the 5G testbeds based on USRP/GNU radio, and the investigation of software and hardware delay on testbeds.

3.1. D2D Route Selection on Testbeds

D2D route selection has been investigated in the literature. In [25], there are two types of users: (a) cellular users are PUs; and (b) D2D users are SUs that use available channels owned by PUs to communicate among themselves under observations from a MC BS. It is assumed that the number of D2D users is less than that of cellular users, and each cellular link can be shared with a single D2D user only at a time. The cellular users use a transmission power above a predefined threshold, while D2D users must ensure that its transmission power is less than a predefined threshold in D2D communication (unless there is no transmission from cellular users) to reduce interference with cellular communication. The proposed route selection scheme has shown to increase throughput and reduce interference between cellular and D2D users.

In [26], a source node selects a route to a destination node using either: (a) cellular communication that goes through the MC BS; or (b) D2D communication that goes through intermediate nodes (or relay nodes) in its proximity regardless of the presence of the MC BS. The source node is aware of the locations of other nodes in the network using global positioning system (GPS). Therefore, the source node selects a neighbor node as the intermediate node if it has: (a) a route leading to the destination; (b) the lowest number of intermediate nodes; and (c) the lowest interference. Investigation shows that network performance improves when a route with less interference is selected in the presence of MC BS’s activities, and a route with the lowest number of intermediate nodes is selected in the absence of the MC BS’s activities.

In [27], a source node selects a route to a destination node while garnering the benefits of traffic offloading from a MC BS to SCs. The source node uses either: (a) a route that goes through the MC BS; (b) a route that uses D2D communication that goes through intermediate nodes; or (c) a direct D2D source-to-destination communication if they are in proximity. The MC BS monitors the traffic among the nodes and provides available routes to the source nodes. The proposed scheme has shown to increase traffic offloading using D2D communication under observation and control from MC BS, leading to an increased throughput.

3.2. USRP/GNU Radio and RP3 Testbeds

In this section, related works on USRP/GNU radio with and without RP3 testbeds are presented. For the testbed without RP3, the data packet processing unit is a PC; while for the testbed with separate processing unit, RP3 is used. The different processing units can affect performance in terms of end-to-end delay and packet loss. In our experiment, user datagram protocol (UDP) is used. In general, multimedia applications have stringent quality of service (QoS) requirements (i.e., delay sensitive yet loss tolerant), while data applications (e.g., email) require a higher data transfer reliability (i.e., loss sensitive yet delay tolerance). UDP has been the preferred transport layer protocol for multimedia applications because it is connectionless and does not perform retransmission during packet loss, which reduces delay at the expense of acceptable packet loss. Due to the challenges of meeting the stringent QoS requirements of multimedia applications, most investigations using testbeds in the literature use UDP rather than transmission control protocol (TCP) [15,16,28,29,30]. In this experiment, we use multimedia applications, whereby the source host sends a video stream to the destination host that plays out the video using the VLC media player.

3.2.1. USRP/GNU Radio without RP3 Testbeds

In [15,28,31,32,33,34,35], traditional testbeds are used in their investigations. In general, a testbed consists of USRP/GNU radio units (i.e., BSs and nodes) that exchange control messages (e.g., channel sensing outcomes and available routes) with a single processing unit such as a PC (i.e., CC) via a Gigabit switch [15]. Control messages are transmitted in the wired medium; while data packets are transmitted in the wireless medium [36]. Hence, the effects and the incurred delay of control message exchange to network performance (e.g., end-to-end delay), which may affect the fulfilment of the network requirements (e.g., the delay constraint imposed by IEEE [37]), are not considered in the investigations.

As shown in

Table 2. Experimental testbed comparison between literature and this paper.

Experimental Testbed in the Literature	Use of USRP/GNU Radio with RP3 (Represents That This Type of Testbed Is Used)	Number of Nodes in the Testbed	Number of Links for D2D Communication	Network Scenario with Heterogeneous Nodes (Represents That the Network Scenario Has Heterogeneous Nodes)
Barrak et al. (2017) [14]	Yes	1	0	No
Raschell and Umbert (2016) [15]	No	6	0	No
Jiang et al. (2016) [25]	No	5	4	Yes
Yuan et al. (2014) [26]	No	6	5	Yes
Ma et al. (2014) [27]	No	9	8	Yes
Chen et al. (2016) [31]	Yes	3	2	Yes
Nagaraju et al. (2010) [35]	No	5	0	No
Shariatmadari et al. (2014) [37]	No	6	0	No
Martínez-Quintero et al. (2019) [38]	Yes	2	0	No
Zhang et al. (2018) [39]	Yes	2	0	No
Park et al. (2017) [29]	Yes	4	3	Yes
Shi et al. (2018) [40]	Yes	2	2	Yes
Syed et al. (2016) [30]	No	10	0	No
Our testbed	Yes	9	8	Yes

, our testbed extends existing testbeds to provide separate processing units (i.e., RP3), has a higher number of nodes (i.e., 9), has the highest number of D2D links (i.e., 8), and considers heterogeneous network entities. Although the characteristics of our testbed are similar to that in [27], the main difference between these two testbeds is that the source node receives and follows route selection made by the base station since a single processing unit is used in [27], while the source node makes route selection independently based on the available routes given by the base station in our testbed that applies USRP/GNU radio units with RP3. In addition, in our testbed, the effects of control message exchange over wireless medium to the end-to-end delay performance is considered in our proposed route selection mechanism.

3.2.2. USRP/GNU Radio with RP3 Testbeds

In [15,28,31,32,33,34,35], testbeds consist of USRP/GNU radio nodes equipped with separate RP3 processing unit, which serves as the decision-making engine. This enables both control messages and data packets to be transmitted over wireless medium. The effect of criteria, such as the ambient noise, as well as interference from other devices, are naturally affecting the end-to-end delay and packet delivery ratio. All experimental results were gathered in the same operating environment in the laboratory. The nodes use similar devices, specifically USRP with GNU radio units connected to RP3 to minimize the differences in signal volatility and inconsistency in communication. Experiment has been conducted under the similar operating environment in the literature [13,14,15,24,28,29,30,31,32,33]. In the literature, RP3 has been used to:

• generate frequency modulation (FM) signals within a range of frequency bands [38]. Measurement using a spectrum analyzer shows that the FM signals, despite being transmitted using a lower transmission power, have close quality to the signals transmitted by radio stations. Energy consumption has shown to reduce at the expense of a lower throughput.
• perform channel sensing to detect white spaces [29,39], and subsequently access the available channels to reduce interference to PUs’ activities [29]. Measurement shows that RP3 consumes energy in computation (e.g., using the available memory in kernel to run GNU radio libraries) and software initialization [39]. Energy consumption has shown to reduce at the expense of a higher delay [14,39], which is up to twice the amount of energy consumption in a PC [Unclear] [29].
• send location information from an unmanned aerial vehicle (UAV), which is a dynamic node, to a ground BS, which is a static PC, so that the ground station can monitor the location of the UAV [40]. Energy consumption has shown to reduce, while the communication and information processing capabilities of RP3 are sufficient [40].
• generate and exchange signals between a RP3 embedded in a register transfer level (RTL) dongle and a PC embedded in a USRP/GNU radio unit using the 915 MHz bands, the 40.68 MHz FM bands [41], and the 24–1850 MHz bands. In [42], pulse width modulation signals are generated. Investigations show that: (a) energy consumption reduces, and it is less than 3 watts [42], which can be sufficiently powered by batteries; (b) the quality of the received signals depends on the height of the antenna and the transmission power [40]; and (c) the communication capability of RP3 is sufficient [40].

In this paper, the testbed consists of: (a) nine USRP/GNU radio nodes equipped with RP3 to handle the computational process, whereby a node is either a source, intermediate, or destination node; and (b) a PC performs the functions of a MC BS, such as establishing a backbone route. PUs interferes with the routes established between a source node and a destination node in a random manner. As shown in Table 2, our testbed extends existing testbeds to provide separate processing units (i.e., RP3), has a higher number of nodes (i.e., 9), has the highest number of D2D links (i.e., 8), and considers heterogeneous network entities. Hence, a source node, being a SU, must select a route with more white spaces. Although the focus of [14,29,38,39,40,41,42] are mainly the capability of RP3 and their compatibility with USRP, this paper uses a testbed with separate processing units in the investigation of a route selection mechanism in the presence of PUs’ activities for achieving an improved measurement accuracy of network performance (i.e., packet delivery ratio and end-to-end delay). In short, compared to existing testbeds in the literature, our testbed provides communication among a larger number of heterogeneous nodes between MC BS and D2D nodes, contributing to a larger number of D2D links in the network.

Experiments using the testbed were conducted in the laboratory environment, and so there are two types of noise: (a) noise within the operating frequency range (or interference) from 850 MHz to 960 MHz covering the TV bands of very high frequency (VHF) and ultra-high frequency (UHF), and this noise has been taken into consideration; and (b) noise below or beyond the operating frequencies, including the ambient noise, which has minimal effect to packet transmission. For the preceding type of noise, the received signal at each node is passed through Gaussian minimum shift keying (GMSK) demodulator and converted into data packets. GMSK ensures the signal has a sharp cut-off with no overshoot impulse responses and is closest to the Gaussian shaped response. For the latter type of noise, GMSK implements phase frequency shifting, unlike most noise which is mainly amplitude-based. This helps the received signals to become immune to ambient noise. GMSK has been used to mitigate ambient noise in the literature [15,28,29,30,31,33,39,43]. To avoid PUs’ activities, it is assumed that the central controller has a record of PUs’ activities and it prioritizes the available routes for the source node. However, in case of PUs’ reappearance, the source node must select other available routes.

3.3. Communication Delay between Nodes

The delay performance between nodes has been investigated in the literature. The traditional testbed that connects USRP/GNU radio nodes to a single processing unit has been investigated in [44]. There are two nodes in [43,45], four nodes comprised of a SU node pair and a PU node pair in [44], and ten SU nodes in [30]. In [45], the delay incurred in USRP and communication is lower than that in the SDR processes (e.g., the modulation process) of GNU radio. In [43], the shortest delay is incurred in transmission (i.e., packets are transferred to FPGA to be interpolated and sent), and the longest delay is the initialization process i.e., sampling, modulation, encoding, and the transfer of packet from GNU radio to the operating system (or kernel). In [44], the end-to-end delay of SUs includes: (a) the waiting time of a packet in a queue; and (b) the channel sensing time, which contributes up to 30% of the end-to-end delay. Experimental results show that a balanced tradeoff between throughput and delay performance can be achieved. Higher channel sensing time reduces interference at the expense of lower throughput and higher delay, and vice-versa; hence, a balanced tradeoff must be achieved. In [30], the end-to-end delay is reduced by selecting routes prior to data transmission in a proactive manner.

An enhanced testbed that consists of USRP/GNU nodes equipped with separate RP3 processing unit, which serves as the decision-making engine, has been investigated in [46] for two nodes. In [46], the end-to-end delay includes the processing delay of a USRP (e.g., decimation in FPGA), the operating system processes in the kernel, and the GNU radio processes (e.g., the modulation process). Measurement is conducted when a node pings another node and the processes in GNU radio are time stamped. Experimental results show that the processes running in the kernel incurs the highest processing delay; therefore, USRP/GNU radio has a lower efficiency in network communication.

4. System Model and Route Selection

This section presents the system model of our testbed, our proposed route selection mechanism, and the end-to-end delay measurement.

4.1. System Model

The testbed consists of a MC BS and a set of $fc$ BSs and nodes $f{c}_f\in F=\{f{c}_1,f{c}_2,\dots ,f{c}_{\left|F\right|}\}$. The source node $f{c}_s$ transmits UDP packets along an available route $k_k\in K=\left\{k_1,k_2,\dots ,k_{\left|K\right|}\right\}$. The set of routes K is readily available, which can be collected and provided by MC BS. A route $k_k=\cup l_n\in L$ consists of a set of links $l_n\in L=\{l_1,l_2,\dots ,l_l\}$ with each link uses one of the available channels $c_c\in C=\left\{c_1,c_2,\dots ,c_{\left|C\right|}\right\}$.

In Figure 4a, a traditional testbed that connects each USRP node to a single processing unit (i.e., a PC), which makes decisions in a centralized manner, via a switch using cables is shown. The source node generates data packets and performs packet initialization using a single processing unit and selects a route. The PC exchanges control messages with USRP nodes via a switch using cables, and the USRP nodes exchange data packets among themselves using wireless medium. A $fc$ source node $f{c}_s$ sends data packets to a $fc$ destination node $f{c}_d$ along a D2D route $k_1=(f{c}_s,f{c}_1,f{c}_2,f{c}_3,f{c}_d)$ selected by the MC BS $mc$.

In Figure 4b, our proposed testbed consists of nodes connected to separate independent processing unit RP3, which enables the source node to make decisions in an independent manner. A $fc$ source node $f{c}_s$ receives a set of routes $k_k\in K=\{k_1,k_2,k_b\}$ from the MC BS, selects a D2D route (i.e., $k_1$), and sends packets to a $fc$ destination node $f{c}_d$ along the route. One of the routes in the set of routes $k_k\in K$ is the backbone route $k_b\in K=(f{c}_s,mc,f{c}_d)$, which connects the $fc$ source node $f{c}_s$ to the $fc$ destination node $f{c}_d$ through MC BS $mc$. The backbone route can be selected when all $fc$ nodes have PUs’ activities in their operating channels $c_c\in C$. The other routes (D2D routes) are selected whenever they have white spaces to offload traffic from MC BS to $f{c}_s$ and reduce network congestion [5].

4.2. Route Selection

The D2D routes are preferred over the backbone route. For simplicity, D2D routes are called primary routes and the backbone route is called a secondary route. The secondary route is only selected whenever primary routes are unavailable due to the appearance of PUs’ activities. The end-to-end delay of a primary route increases with the hop count of the route [47]. Based on the IEEE 802.15.4 standard [12,37], the end-to-end delay of a route increases with the hardware and software processing delay, the queue size, and the appearance of PUs on a route [48,49]. Hence, a threshold $\alpha$, which is given to the source node by the MC BS or CC, is used to evaluate the end-to-end delay of D2D routes and monitored by the CC. As shown in Figure 5, a source node requests for a route to its destination node. The CC provides information about available routes through MC BS. The given routes are ranked by CC based on: (a) the presence of PUs’ activities (or the number of white spaces) on each route; (b) the end-to-end delay of the route; and (c) the number of hops of a route. A route with a higher number of white spaces is ranked higher. Using this information, the source node ranks the routes again based on its own criteria. This is because the CC gives a higher priority to the backbone route since it does not have PUs’ activities and provides lower end-to-end delay and number of hops. Meanwhile, the source node gives a higher priority to D2D routes and keep the backbone route as an alternative. In other words, the highest-ranked route given by the CC may not be selected by the source node. The source node makes route selection based on the presence of the random PUs’ activities in those channels. For instance, the source node has selected route 1, which has the highest ranking; however, it switches to route 2 when the PUs’ activities reappear in route 1. Therefore, the source node switches routes dynamically due to the random reappearance of PUs’ activities.

4.3. Delay Measurement

The communication between a node pair incurs: (a) software processing delay $t_{l_n}^s$ incurred in processes (i.e., GNU radio initialization) running on RP3; (b) hardware processing delay $t_{l_n}^h$ incurred in data packet transmission (i.e., data packet transmission) performed by USRP; and (c) propagation delay $t_{l_n}^p$ incurred in wireless signal broadcast [16]. The end-to-end delay increases with the number of hops of a D2D route, and it is the accumulated link delay of the links $l_n\in L$ of a route as follows:

$$t^{k_k}=\sum _{l_n\in k_k}\sum t_{l_n}$$

(1)

where $t_{l_n}$ consists of software, hardware and propagation delays as follows:

$$t_{l_n}=t_{l_n}^s+t_{l_n}^h+t_{l_n}^p$$

(2)

However, the propagation delay can be neglected as it is significantly lower than software and hardware delays. The presence of PUs’ activities can cause a source node to change its operating channel and route, which can increase delay as follows:

$$t^{k_k}=\sum _{l_n\in k_k}(\sum t_{l_{n\left(PU\right)}}+\sum t_{l_n})$$

(3)

where $t_{l_{n\left(PU\right)}}$ is the link delay caused by interference with PUs’ activities [50].

The PUs can be either ON (active) or OFF (idle). The duration of being active or idle follows a Poisson model, and so the active time is exponentially distributed with ${\lambda }_{c_C,ON}^{PU}$, and the idle time is also exponentially distributed with ${\lambda }_{c_C,OFF}^{PU}$ [51]. The ${\mathsf{\Phi }}_{t,c_c,OFF}^{l_n}$ determines the average channel availability of the link $l_n$ at time instant t, and it is expressed as follow:

$$\begin{array}{cc}\hfill {\mathsf{\Phi }}_{t,c_n,OFF}^{l_n}=& \frac{{\lambda }_{c_C,ON}^{PU}}{{\lambda }_{c_C,ON}^{PU}+{\lambda }_{c_C,OFF}^{PU}}\hfill \\ & +\frac{{\lambda }_{c_C,OFF}^{PU}}{{\lambda }_{c_C,ON}^{PU}+{\lambda }_{c_C,OFF}^{PU}}\times e^{-{({\lambda }_{c_C,ON}^{PU}+{\lambda }_{c_C,OFF}^{PU})}^t}\hfill \end{array}$$

(4)

Algorithm 1 provides a general description of our route selection scheme. Algorithm 2, which is based on Algorithm 1, provides a detailed description of our route selection scheme as presented in Figure 5.

Algorithm 1 General description of the route selection scheme by the source node in the presence of PUs

1: procedureIndependent route selection 2:      MC BS send available D2D routes $k_k\in K$, where $k_1$ has the highest priority and $k_k$ has the lowest priority, to $f{c}_s$ proactively 3:     for ($k_k\in K$) do 4:          Select route $k_k$ if it has the lowest k and OFF PU 5:         if all $k_k$ have ON PU then 6:             Select backbone route $k_b$ 7:         end if 8:     end for 9: end procedure

Algorithm 2 Detailed description of the route selection scheme by the source node in the presence of PUs

1: procedureIndependent route selection 2:      MC BS send available routes ($k_1,k_2,k_3,k_4$) to $f{c}_s$ proactively: 3:     for ($k_1=(f{c}_s,f{c}_1,f{c}_4,f{c}_d)$, $k_2=(f{c}_s,f{c}_2,f{c}_5,f{c}_d)$, $k_3=(f{c}_s,f{c}_3,f{c}_6,f{c}_d)$ and $k_4=(f{c}_s,mc,f{c}_d)$) do 4:         if $P{U}_0$ and $P{U}_3$ are OFF then 5:             Select route $k_1$ 6:             $f{c}_s$ send packets to $f{c}_d$ through $k_1$ 7:             Update the MC BS about selected route 8:         end if 9:         if $P{U}_0$ and/or $P{U}_3$ are ON then 10:             Check $P{U}_1$ and $P{U}_4$ in route $k_2$ 11:            if $P{U}_1$ and $P{U}_4$ are OFF then 12:                 Select route $k_2$ 13:                 $f{c}_s$ send packets to $f{c}_d$ through $k_2$ 14:                 Update the MC BS about selected route 15:            end if 16:         end if 17:         if $P{U}_1$ and/or $P{U}_4$ are ON then 18:             Check $P{U}_2$ and $P{U}_5$ in route $k_3$ 19:            if $P{U}_2$ and $P{U}_5$ are OFF then 20:                 Select route $k_3$ 21:                 $f{c}_s$ send packets to $f{c}_d$ through $k_3$ 22:                 Update the MC BS about selected route 23:            else Select route $k_4$ 24:                 $f{c}_s$ send packets to $f{c}_d$ through $k_4$ which is a backbone route 25:            end if 26:         end if 27:     end for 28: end procedure

Algorithm 2 provides more description about Figure 5. Based on PUs’ activities and channel availability, the CC sends a list of available routes to the source node. Suppose the CC sends four available routes ($k_1,k_2,k_3,k_4$) to the source node $f{c}_s$. Then, the source node $f{c}_s$ prioritizes the routes by giving a higher priority to D2D routes with no or less PUs’ activities. interference. The least priority is given to the backbone route, which goes through the MC BS. The source node updates the CC and MC BS about the selected route. Please note that nodes receive route information from CC through MC BS, and so it is assumed that they do not perform channel sensing. This paper focuses on network performance analysis, rather than network performance comparison among different schemes and mechanisms. Nevertheless, the route selection scheme investigated in our experiment has incorporated various features of existing route selection scheme in 5G as follows:

• A central controller is the decision-maker in the core network [2,3] that manages and optimizes network resources based on network-wide information (i.e., the available routes, and the hop count, traffic, and number of white spaces of each route) [4]. The CC sends a list of available D2D routes to nodes in small cells (SCs) to reduce interference among routes [4] and increase traffic offload from the macrocell (MC) base station (BS) [2]. In [3,12], traffic offload from the backbone route to small cell nodes has shown to reduce end-to-end delays.
• D2D communication has shown to enable multihop communication for delivering a high-quality video stream, contributing to an increased network capacity at the expense of a higher end-to-end delay [4]. Each node can access both cellular channels (in which the node is the PU) and cognitive channels (in which the node is the SU) to improve spectrum efficiency and quality of service (QoS) [6,7].
• Route selection enables a source node to select a route to a destination node while garnering the benefits of traffic offloading from a MC BS to SCs [27]. The source node uses either: (a) a route that goes through the MC BS; or (b) a route that uses D2D communication that goes through intermediate nodes; or (c) a direct D2D source-to-destination communication if they are in proximity. The MC BS monitors traffic among nodes and provides available routes to the source nodes. The proposed scheme has shown to increase traffic offloading using D2D communication under observation and control from MC BS, leading to an increased throughput.

5. Experimental Setup Parameters and Performance Metrics

This section presents experimental setup parameters and performance metrics, including the end-to-end delay, and packet delivery rate.

5.1. Simulation Platform

The experimental setup parameters are summarized in

Table 3. Experimental setup parameters.

Category	Parameter	PCU	RPU
Experiment	Duration	600 s	600 s
USRP	Number of channels	2	9
	Number of nodes	1	8
	Transport layer	UDP	UDP
	Bandwidth	40 MHz	40 MHz
	Sample rate	1.1 MB	1.1 MB
	Transmission power	20 dBm	10 dBm
Antenna	Carrier frequency	850 MHz	850 MHz
Computer	Operating system	Ubuntu	-
Switch	Number of units	1	-
	Number of inputs	1	-
RP3	Operating system	-	Ubuntu-Mate
PoE	Number of units	None	8
	Number of inputs	None	1
PU activity	PU ON time	None	150 s
	PU OFF time	None	150 s, 300 s, 450 s

. The testbed consists of nine USRPs. One of the USRPs serves as the MC BS that uses a PC for computation, and it connects to the rest of the USRPs via a switch. The rest of the USRPs are connected to RP3. The MC BS provides a backbone route and uses two frequency bands: (a) one channel is used for direct communication with the $fc$ source node $f{c}_s$; and (b) one channel is used for direct communication with the $fc$ destination node $f{c}_d$. In our testbed, the larger numbers of nodes and D2D links provide three disjoint three-hop routes in the network, which makes it possible to measure cumulative end-to-end delay as shown in Figure 6. Our testbed helps to achieve the main contribution of our paper, which is to investigate a route selection mechanism that enables a source node to select one of these routes: (a) the backbone route that uses direct communication with the MC BS to improve the route performance (i.e., a higher packet delivery ratio and a lower end-to-end delay); and (b) multihop D2D routes that are formed among SCs to offload traffic from the MC. In our testbed, the backbone route has the minimum number of hops (i.e., two hops), while each D2D route has three hops. The D2D routes have a higher number of hops because they are formed among SCs, which have a smaller transmission range. The number of hops for D2D routes are set to three hops so that their packet delivery ratio and cumulative end-to-end delay are comparable to those of the backbone route. As shown in Figure 1, the $fc$ source node $f{c}_s$ can also form multihop route to the $fc$ destination node $f{c}_d$ via one of the three available routes.

5.2. Experimental Setup Parameters

Each D2D route consists of two intermediate nodes. The intermediate nodes use different channels with different amount of white spaces. There are three available routes, namely route 1 ($f{c}_s$–$f{c}_1$–$f{c}_4$–$f{c}_d$), route 2 ($f{c}_s$–$f{c}_2$–$f{c}_5$–$f{c}_d$), and route 3 ($f{c}_s$–$f{c}_3$–$f{c}_6$–$f{c}_d$). Each PU has a fixed ON time of 150 s, and different OFF time of either 150 s, 300 s, or 450 s. The nodes set the signal parameters, including the carrier frequency 850 MHz and the sample rate $1.1$ MB. The routes are prioritized based on the PUs’ activity levels. Specifically, the source node $f{c}_s$ prioritizes and selects the D2D route with the highest number of white spaces (e.g., 450 s and 150 s OFF time have the highest and the least priorities, respectively) at all times, and so it switches to another route with the second highest number of white spaces when PUs’ activities reappear in the current route. Hence, the backbone route is not selected if one of the D2D routes does not have PUs’ activities. The least priority is given to the backbone route to promote traffic offload. This means that the backbone route is selected when PUs’ activities appear in the three D2D routes. Further description about route selection is provided in Section 4.2.

Figure 6 shows the experimental setup between source and destination node through three D2D routes (i.e., route 1, route 2 and route 3) and backbone route through MC BS.

5.3. Performance Metrics

The performance metrics are as follows:

• Cumulative end-to-end delay refers to the cumulative time for a data stream (or traffic) to travel from a source node $f{c}_s$ to a destination node $f{c}_d$. The end-to-end delay reduces when: (a) the number of hops in a route reduces; and (b) the presence of PUs’ activities that causes a switch to another route reduces.
• Packet delivery ratio refers to the number of packets successfully delivered over the total number of packets delivered from a source node to a destination node. Packets are delivered either via D2D across SCs or through MC BS. The packet delivery ratio increases when the number of white spaces of each D2D route reduces.

6. Results and Discussion

This section investigates the performance metrics of route selection, namely cumulative end-to-end delay and packet delivery rate.

The first investigation in Section 6.1 aims to understand the effect of PUs’ activities to traffic offload by measuring the traffic traverses along each D2D route in the presence and absence of a backbone route. Next, the aim is to understand the D2D and backbone routes, which consist of nodes with different processing capabilities in addition to the presence of PUs’ activities in D2D routes and the absence of PUs’ activities in the backbone route. The second investigation in Section 6.2 compares the packet delivery ratio of the traffic traverses along a D2D route with PUs’ activities and the backbone route without PUs’ activities. The third investigation in Section 6.3 compares the cumulative end-to-end delay of the traffic traverses along the D2D routes with different PUs’ activity levels and the backbone route without PUs’ activities.

6.1. Investigation of Traffic Offload to D2D Routes with and without Backbone Route

This investigation measures the packet delivery ratio of each available route to understand the amount of traffic offloaded to D2D routes transmission in backbone routes. Traditionally, due to the lack of D2D routes, all traffic must be transmitted over the backbone route without traffic offload. The backbone route uses licensed channels without PUs’ activities at all times, and so packet delivery is not interrupted by PUs’. The D2D communication uses white spaces (or underused licensed channels without PUs’ activities). Figure 7 presents the packet delivery ratio of both D2D and backbone routes. The packet delivery ratio of routes 1, 2, and 3 reflects the amount of traffic offloaded from MC BS or the backbone route. Route 1 ($f{c}_s$–$f{c}_1$–$f{c}_4$–$f{c}_d$) has the highest priority since it has 450 s out of 600 s (or $75\%$ of PUs’ OFF time), route 2 ($f{c}_s$–$f{c}_2$–$f{c}_5$–$f{c}_d$) has 300 s (or $50\%$), and route 3 ($f{c}_s$–$f{c}_3$–$f{c}_6$–$f{c}_d$) has 150 s (or $25\%$).

There are two main investigations. First, packet delivery over each of the three available D2D routes that contributes to traffic offload in the presence of a backbone route ($f{c}_s$–$mcBS$–$f{c}_d$) through MC BS (shown as lighter shade bars in Figure 7). Since the D2D routes have different amounts of white spaces, they have different packet delivery ratios. Route 1 has delivered $44.57\%$ of the traffic, route 2 has delivered $23.4\%$, and route 3 has delivered $12.13\%$, and so the total amount of traffic delivered along the D2D routes is $80.10\%$. When PUs’ activities reappear in all the three D2D routes, traffic must be transmitted over the backbone route, which has delivered $9.26\%$ of the traffic. Hence, the overall packet delivery ratio is $89.36\%$. Secondly, packet delivery over each of the three available D2D routes in the absence of a backbone route (shown as darker shade bars in Figure 7).

When PUs’ activities reappear in all the three D2D routes, packet loss occurs since traffic cannot be transmitted over the backbone route, causing a lower overall packet delivery ratio. Route 1 has delivered $42.15\%$ of the traffic, route 2 has delivered $21.3\%$, and route 3 has delivered $11.6\%$. Hence, the overall packet delivery ratio is $75.07\%$. However, the source node can still transfer traffic over the backbone route in the presence of a backbone route. Despite the presence of the backbone route, some packet loss is observed due to the random reappearance of PUs’ activities and route switches.

6.2. Packet Delivery Ratio Comparison between D2D and MC

Figure 8 compares the packet delivery ratio of a D2D route (i.e., route 1 ($f{c}_s$–$f{c}_1$–$f{c}_4$–$f{c}_d$) with 450 s out of 600 s (or $75\%$) of PUs’ OFF time) and the backbone route (i.e., $f{c}_s$–$mc$–$f{c}_d$) in order to understand the intrinsic differences between these two types of routes and their effects to packet delivery ratio. In this scenario, there is a single D2D route with three hops and its packet delivery is affected by PUs’ activities. The backbone route has two hops and packet delivery is not affected by PUs’ activities. In the D2D route, each intermediate node, uses a separate RP3 processing unit. Each intermediate node, which is placed apart from other intermediate nodes, has different transmission range and uses different transmission frequencies within the VHF bands and has different transmission ranges. For instance, in route 1, the first intermediate node $f{c}_1$ transmits in 860 MHz and has a proximity of 5 meters to the source node $f{c}_s$ and to another intermediate node $f{c}_4$, respectively.

Comparing the two routes, the packet delivery ratio is $76.6\%$ in the D2D route and $89.8\%$ in the backbone route. Hence, the packet loss rate is $23.4\%$ in the D2D route and $10.2\%$ in the backbone route. The difference is caused by three main reasons: (a) the D2D route has a higher number of hops (i.e., three hops) compared to the backbone route, which can increase packet loss (b) intermediate nodes (i.e., RP3) in the D2D routes have a lower processing capability compared to MC BS (i.e., PC) in the backbone route causing a higher heterogeneity; and (c) the D2D route uses cognitive channels with a certain amount of white spaces, while the backbone route uses cellular channels without the presence of PUs’ activities, which can increase end-to-end delay and packet loss.

It is noteworthy that control messages are transmitted via the wireless medium in D2D routes, while they are transmitted via cables in the backbone route. Therefore, there is less interference from the operating environment to control message exchange in the backbone route. However, due to the small packet size of a control message, this effect is negligible.

6.3. Investigation of Cumulative End-to-End Delay of Routes in the Presence and Absence of PUs’ Activities

This investigation presents the cumulative end-to-end delay of both D2D and backbone routes. The cumulative end-to-end delay increases as packets are delivered from a source node to a destination node with increasing number of hops. There are two main investigations, and their results are presented in Figure 9. First, the cumulative end-to-end delay over each of the three available D2D routes in the absence of PUs’ activities. Route 1 has an end-to-end delay of $0.1063$ s, route 2 has $0.1176$ s, and route 3 has $0.1298$ s. Secondly, the cumulative end-to-end delay over each of the three available D2D routes in the presence of PUs’ activities. Each D2D route has different levels of PUs’ activities (see Section 6.1). Route 1 has an end-to-end delay of $0.1501$ s, route 2 has $0.1653$ s, and route 3 has $0.1792$ s. Meanwhile, the backbone route does not have PUs’ activities in all cases, and it has an end-to-end delay of $0.079$ s. The backbone route has the least cumulative end-to-end delay due to its lower number of hops and the higher processing capability of the MC BS. Comparing the scenarios with and without PUs’ activities is higher due to the random reappearance of PUs’ activities and route switches.

Table 4. A summary comparison of the results gathered in this investigation.

Testbed Metrics and Results	Cumulative End-to-End Delay		Percentage of Packet Delivery Ratio		Percentage of Traffic Offload
	Without PU	With PU	Without PU	With PU
Route 1 (highest priority)	0.1063 s	0.1501 s	44.57%	42.15%	∼75%
Route 2	0.1176 s	0.1653 s	23.4%	32.3%	∼50%
Route 3 (lowest Priority)	0.1298 s	0.1792 s	12.13%	11.6%	∼25%
Total (Routes 1, 2, and 3)	-	-	89.36%	75.07%	-
Backbone route (without PU)	0.079 s		89.8%		-

presents a summary comparison of the measurements made in both D2D and backbone routes with and without PUs’ activities.

7. Conclusions and Future Work

7.1. Conclusions

This paper investigates a route selection mechanism using device-to-device (D2D) communication over a testbed based on 5G with heterogeneous devices, which have different transmission power i.e., 20 dBm for macrocell (MC) base station (BS) and 10 dBm for D2D nodes, frequency range (i.e., from 850 MHz to 910 MHz), and processing capability i.e., 1.4 GHz for femtocell ($fc$) nodes. In addition to a femtocell $fc$ source node and a $fc$ destination node, the testbed consists of six $fc$ intermediate nodes that forms a D2D routes and a MC BS that forms a backbone route. The $fc$ source node transmits a data stream (or traffic) towards a $fc$ destination node through three disjoint D2D routes or the backbone route. The D2D routes have different priority levels given by the central controller based on the level of PUs’ activities. In general, the D2D routes have higher priority levels to improve traffic offload from MC to $fc$, and a D2D route with a higher priority has a lower level of PUs’ activities. Nevertheless, the random reappearance of PUs’ activities is inevitable, and so the backbone route must be selected when PUs’ activities reappear in all the D2D routes. Experimental results show that the overall packet delivery ratio increases from $75.07\%$ in the absence of a backbone route to $89.36\%$ in the presence of a backbone route, giving a $14.29\%$ improvement and a traffic offload of up to $75.05\%$ of the MC BS traffic. The reappearance of PUs’ activities increases the cumulative end-to-end delay of $0.048$ s due to route reassessment and switches. The cumulative end-to-end delay is expected to further increase with the number of intermediate nodes (or hops) in a route. Compared to packet delivery ratio, the PUs’ activities have a greater effect on the cumulative end-to-end delay.

7.2. Future Work

Nevertheless, our 5G-based testbed can still be further improved to implement a more realistic 5G network by: (a) increasing the number of nodes for achieving ultra-densification among small cells; and (b) increasing the number of available routes so that the source node has more options of routes with different levels of PUs’ activities and hop counts. In addition, investigation using simulation can be performed as results are useful for estimating network performance in different scenarios, such as the random reappearance of PUs’ activities in each route based on their respective ON/ OFF time.