Call Model and Test-Veriﬁcation Methods for PS-LTE Core Equipment

: Existing technologies in disaster and safety communication networks (DSCNs) face problems such as narrow bandwidth, low transmission rate, and difﬁculty in securing channels. Con-versely, public safety long-term evolution (PS-LTE) technology provides high-speed, low-latency, quick-setup, and high-security data access for multimedia-based mission-critical services. To perform tests on each unit of core equipment with PS-LTE technology, a new call model such as push-to-talk, which is a service representing PS-LTE, is required. Moreover, objective and consistent metrics and methods are required for administering tests on the unique functions of DSCNs newly constructed with PS-LTE technology. This paper proposes a call model for the core facilities of DSCNs established in Korea, and it also suggests test and veriﬁcation methods for all the areas of PS-LTE, including mission-critical push-to-talk, evolved multimedia broadcast and multicast service, and coverage and quality assessment for wireless services.


Introduction
Due to the success of long-term evolution (LTE) technology, public safety LTE (PS-LTE) technology is replacing existing technologies in disaster and safety communication networks (DSCNs). Examples of existing technologies for DSCNs include terrestrial trunked radio (TETRA), Association of Public Safety Communications Officials-Project 25 (APCO-P25), integrated digital enhanced network (iDEN), ultrahigh frequency (UHF), and very high frequency (VHF). However, these technologies face problems such as low transmission rate, narrow bandwidth, and difficulty in securing a channel. Conversely, PS-LTE technology can provide high-speed, low-latency, quick-setup, and high-security data access for multimedia-based mission-critical services. The ultimate goal of PS-LTE is to provide existing push-to-talk (PTT) voice, videos, and data services combined in a single LTE network based on all-IP packets.
In 2014, 3GPP established the Technical Specification Group Service and System Aspect Six Working Group (TSG SA6 WG) dedicated to disaster and safety communication for LTE-based DSCNs. SA6 is responsible for standardization procedures for determining the application layer structures and specification of functions for mission-critical push-to-talk (MCPTT), which is a core technology in DSCNs. Other working groups for various system aspects are also standardizing PS-LTE-related technologies, and those working on architecture (SA2) have performed standardizations for group communication system enabler, proximity-based service (ProSe), and isolated operation for public safety (IOPS). As stageby-stage standardization efforts progressed for PS-LTE-related technologies, requirements for Release 13-Stage 1 were compiled in September 2014, and Release 12 was confirmed

Call Model and Test Result for Core Equipment
A PS-LTE project is driven by introducing a goal-based approach, where business operators establish all environments necessary for services from the core equipment to the radio access network (RAN), command devices, and terminals. Business operators need objective test/verification methods to understand the progress toward achieving the goal for the core equipment after establishing DSCNs. However, as the performance test method for existing LTE services defines a call model that only includes voice over LTE (VoLTE) and data service, defining a new call model including MCPTT services is necessary. In the call model for MCPTT, the definition of a user-based call model should include group voices, group images, and message services to preemptively identify problems that may occur in the field. PS-LTE is established to provide VoLTE, PTT voice and videos, and data services combined in a single LTE network based on all-IP. To provide such services, the Electronics 2021, 10, 2513 3 of 22 core equipment utilizes virtual network function (VNF) technology to ensure a fast and stable establishment of services by installing software on standard hardware platforms.

Core Equipment Test Environment
Core equipment comprises MCPTT, eMBMS, evolved packet core (EPC), RAN, sharing internetworking equipment, information management system (IMS), L3 switch, and others. As the core services of PS-LTE among the components above, the MCPTT entity performing the function of group communication for mission-critical services comprises common services core (CSC) servers, MCPTT servers, command servers, and command devices. Moreover, the eMBMS entity, which is an LTE-based mobile broadcasting technology for resolving call congestions, comprises a broadcast/multicast service center (BM-SC), multimedia broadcast and multicast service gateway (MBMS GW), and multicall/multicast coordination entity (MCE). Among the components of EPC, mobility management equipment (MME) is one of the network entities (NEs) that constitute the LTE core. It processes the mobility management and session management of the terminal; authentication of the terminal; handover control; control plane, which is connected to evolved node B (eNB); system architecture evolution gateway (SAE-GW); and home subscriber server (HSS) [12,13]. Core equipment must be tested for the required capacity and performance before establishment to smoothly perform the planned services. This paper proposes a performance test call model of MME and SAE-GW systems among core equipment for MCPTT which is a representative service of PS-LTE. Figure 1 shows the composition of the core equipment, measuring instrument, and simulator environment for performance tests. The main VNF configuration as a component of core networks comprises (i) a virtual operation and management application (VOMA) for system operation maintenance, (ii) a virtual interface and protocol distribution application (VIPA) for stream control transmission protocol (SCTP)/diameter call distribution with connection with HSS/eNB, (iii) a virtually established call-processing application (VECA) for session-initiation protocol (SIP)/media gateway control (Megaco) call distribution and PTT call-processing functions, (iv) a virtual interface and distribution application (VIDA) for physical linking with external NE and Layer 4 (L4) load distribution, (v) a virtual interface of session application (VISA) for diameter/HTTP call distribution and subscriber/session information management, and (vi) a virtually established packet processing application (VEPA) for processing PTT media. The components of HSS comprise a UPN (user data convergence (UDC) proxy node) for connection with external NE and L4 load distribution, a UDC function node (UFN) for HSS call processing and connection with the MME/call session control function (CSCF)/MCPTT, and a UDC database node (UDN) for subscriber information database management. MCE is a nonvirtualized system and is installed as separate hardware and software configurations. MCE comprises MCE processing software for base station resource allocation, MCE access control block for session management, MCE transmission control block for SCTP connection management, MCE management block for eMBMS service area management, and MCE connection admission control for session admission load management. In PS-LTE systems, VECA, UDN, and UFN for important services operate in an active-active mode, and the rest of the configurations operate in an active-standby mode.

MCPTT Core System Test Model
The NE comprising MCPTT services consists of command devices, CSC servers, and MCPTT servers. Representative services are group voice/video calling and group character services, and for these services, the control plane is processed by linking with terminals, CSCF, and HSS. MCPTT is provided as a virtualization system, and the software structure comprises VOMA, VISA, VECA, VIDA, and VEPA. The requirements for the MCPTT system capacity in Korea are 480,000 busy hour register attempts (BHRA) for registering and 82.4 million busy hour call attempts (BHCA) for service.
Based on the above requirements, the load conditions of the simulator for MCPTT system testing approve full performance traffic according to the call model shown in Table  1. The conversion to Regi BHRA per subscriber is calculated by initial registration (INIT-REG) + refresh registration + deregistration (DE-REG). The conversion to service BHCA per subscriber is calculated as short message service (SMS) BHCA + MCPTT BHCA. As shown in Figure 2

MCPTT Core System Test Model
The NE comprising MCPTT services consists of command devices, CSC servers, and MCPTT servers. Representative services are group voice/video calling and group character services, and for these services, the control plane is processed by linking with terminals, CSCF, and HSS. MCPTT is provided as a virtualization system, and the software structure comprises VOMA, VISA, VECA, VIDA, and VEPA. The requirements for the MCPTT system capacity in Korea are 480,000 busy hour register attempts (BHRA) for registering and 82.4 million busy hour call attempts (BHCA) for service.
Based on the above requirements, the load conditions of the simulator for MCPTT system testing approve full performance traffic according to the call model shown in Table 1. The conversion to Regi BHRA per subscriber is calculated by initial registration (INIT-REG) + refresh registration + deregistration (DE-REG). The conversion to service BHCA per subscriber is calculated as short message service (SMS) BHCA + MCPTT BHCA. As shown in Figure 2 Table 1, it can be confirmed that an appropriate call model was calculated, which can test the MCPTT service requirements of 82.4 million BHCA.

MME Core System Test Model
The MME for DSCNs is designed with a capacity of 400,000 simultaneous subscribers, and the detailed requirements for services are 3200 calls per second (CPS), 1,200,000 sessions (average of three sessions per subscriber), 2,400,000 bearers (average of six bearers per subscriber), 30,000 eNB, and 44,300 transactions per second (TPS). MME is installed as a virtual machine (VM) configuration on the two units of HP DL380, and it is provided as a virtualization system. In the performance test, the environment of the simulator and measuring instrument for each subsystem is set up, and the capacity and performance are measured by applying call connection/disconnection and voice/video call and general traffic to each subsystem. Because commercial measuring instruments do not support both user equipment (UE) and eNB, performance is measured using a self-developed simulator and a commercial packet generator for testing. System statistics confirm the number of subscribers, traffic, load, fail/drop and load status of CPU, and memory for each VM. It is assumed that 1 (subs):3 (session):6 (bearer) is created per subscriber. For a scenario for generating both 3200 CPS and 44,300 TPS, 370,000 fixed call subscribers are attached, and for 30,000 floating call subscribers, a scenario is created by attach → public data network (PDN) connectivity → X2 handover → tracking area update (TAU) → S1 release (S1REL) → scheduling request (SR) → PDN disconnect (PDN DISCON) → detach. As 16 tracking area codes (TAC) are assigned per tracking area list, the number of eNBs within 1 TAC is not limited. Terminal emulation for services was used for MCPTT group voice and video calls by 10 units, respectively, and for VoLTE voice and video calls, by two units.
Initial attach CPS means as the average (AVG) CPS × the number of transactions per attach. PDN connectivity CPS refers to total AVG CPS × the number of transactions per PDN connection (CON). X2HO Connectivity CPS is confirmed as total attempt/60 ch (CPS) × the number of transactions per PDN CON. TAU CPS refers to AVG CPS × the number of transactions per TAU. S1REL CPS transmits radio access bearer (RAB) request (REQ) message when S1REL occurs, and it is confirmed that total RAB REQ/60 ch (CPS) × the number of transactions per S1REL. SR CPS refers to total AVG CPS × the number of transactions per SR. PDN disconnect CPS is calculated as PDN disconnect = bearer deactivation per node (BDN)-DTACH and PDN DISCON CPS = BDN (attempt-DTACH attempt)/60. PDN DISCON TPS is calculated as PDN DISCON CPS × the number of transactions per SR. Detach TPS is calculated as AVG_CPS × the number of transactions per DTACH. Figure 3 shows the performance test results applying the call model set in Table 2, and it confirms the processing capacity of 48,807 TPS, which is higher than the requirement of 44,300 TPS. Therefore, the performance test is possible using the call model defined in Table 2.

SAE-GW Core System Test Model
SAE-GW is one of the NEs constituting LTE core and is integrated equipment in a serving gateway (S-GW) and a packet data network gateway (P-GW). The main functions of S-GW include inter-eNB handover, packet buffering, packet routing and forwarding,

SAE-GW Core System Test Model
SAE-GW is one of the NEs constituting LTE core and is integrated equipment in a serving gateway (S-GW) and a packet data network gateway (P-GW). The main functions of S-GW include inter-eNB handover, packet buffering, packet routing and forwarding, and transport-level packet marking. The main functions of P-GW include IP routing/forwarding, per-service data flow (SDF)/per-user-based packet filtering, IP address allocation to UE, and policy and charging control (PCC) rule processing. SAE-GW has an active-standby structure, and the interface comprises a small form-factor pluggable (SFP) + 12-port supporting 10-gigabit ethernet (GE) and RJ-45 8-port supporting 1 GE. The requirements for DSCNs are based on 400,000 subscribers, 16,000 TPS, and 10 Gbps (PCC on, deep packet inspection (DPI) on, row manager on). The call model for calculating the capacity and performance of SAE-GW is shown in Table 3. Table 3. SAE-GW call model: assuming 400,000 subscriber base. The simulator load conditions for the SAE-GW test have used the fixed calls of 346,000 subscribers, floating calls of 10,000 subscribers, and traffic calls of 44,000 subscribers as inputs. Per subscriber, subscribers:session:bearer ratio (1:3:6) is calculated as 400,000 (subs): 1,200,000 (session):2,400,000 (bearer). As for fixed-call subscribers, attach/PDN conn./bearer activation is performed. The simulator setting for floating call bulk subscribers is approved as 2400 CPS as a scenario of attach and detach, and attach/detach transaction is calculated as CPS × 7 (CSR, DSR, MBR, CCR-I, CCR-T, ACR_START, ACR_STOP), which is 16,800 TPS. As for UE traffic, traffic of 8 Gbps is applied to 44,000 subscribers, which is approximately 10% of 400,000 subscribers, using a commercial traffic generator, and the packet size is set by mixing 128, 512, and 1024 bytes. The total number of bearers is calculated as the number of fixed call bearers + the number of traffic call bearers, and the numbers of sessions and subscribers must be checked equally. Figure 4 shows the test results applying the SAE-GW call model as shown in Table 3, and it can be seen that performance test is possible with the proposed call model by confirming the performance of 17,559 TPS and 16.9 Gbps (8.2 + 8.7 Gbps), which are higher than the requirements of 16,000 TPS and 10 Gbps.

Discussion
Since the disaster safety communication network is established by the telecommunication service providers and operated by the government agency, it is essential to conduct tests in accordance with the requirements from the initial construction stage. The performance test for core equipment of the disaster safety communication network has been required to define a new call model including the existing LTE service and MCPTT service. The MCPTT system capacity is 480,000 BHRA and 820,000 BHCA based on 400,000 subscribers, and the MME capacity is 3200 CPS and 44,300 TPS for 400,000 simultaneous subscribers. After testing the call model presented in Chapter 2 in a system with the same hardware/software version as the equipment built, it has been confirmed that the performance test was satisfactory. Therefore, the proposed call model can be used as a reference model for the performance tests of core equipment that must be considered when establishing a disaster safety communication network.
This paper proposed the call model and test methods in commercial PS-LTE, which have contributed to the world's fastest commercialization and stable operation of a disaster safety communication network able to be utilized.

Discussion
Since the disaster safety communication network is established by the telecommunication service providers and operated by the government agency, it is essential to conduct tests in accordance with the requirements from the initial construction stage. The performance test for core equipment of the disaster safety communication network has been required to define a new call model including the existing LTE service and MCPTT service. The MCPTT system capacity is 480,000 BHRA and 820,000 BHCA based on 400,000 subscribers, and the MME capacity is 3200 CPS and 44,300 TPS for 400,000 simultaneous subscribers. After testing the call model presented in Section 2 in a system with the same hardware/software version as the equipment built, it has been confirmed that the performance test was satisfactory. Therefore, the proposed call model can be used as a reference model for the performance tests of core equipment that must be considered when establishing a disaster safety communication network.
This paper proposed the call model and test methods in commercial PS-LTE, which have contributed to the world's fastest commercialization and stable operation of a disaster safety communication network able to be utilized.

Field Test Scenarios for MCPTT and eMBMS
The MCPTT function is one of the most important functions among DSCN services, and ensuring stability for this function is important because it is an essential service to the field in case of various disasters. eMBMS is a service that can expand call capacity by switching to eMBMS when hundreds of people gather within a cell coverage during a rally or disaster situations, and DSCNs and call congestions occur through unicast communication.

MCPTT Field-Test Scenarios
As for MCPTT test was performed such as group voice, group videos, all call paging, interception, emergency calls and multicall group reception functions. An MCPTT group call operates in a prearranged group call method in which command devices assign members to an affiliated group and call the group. Furthermore, as for floor control, if the push button is pressed, a request for permission to talk is sent to the floor control server, which sets in motion a mechanism for giving the right to transmit to the member with the highest priority among those who request permission to talk within a specific time [14,15]. According to the requirements for group calls, users select a call group by operating terminal buttons, switches, LCD panel touch, etc., and the terminals which are selected for a specific call group automatically receive all the calls of the corresponding call group. Only one user at a time is allowed to transmit to avoid collisions between transmission and reception during a group call. When a user is pressing the PTT button, it should be displayed as a separate pop-up to ensure that the user can selectively switch the reception mode without automatically disconnecting the group call [16,17]. Figure 5 shows an example concerning the creation of an MCPTT voice channel of the terminal for a group call, and it can be seen step-by-step as follows. • Step 1 (channel information inquiry): after an originating terminal performs the authentication procedure for a subscriber, it sends an inquiry to the CSC about the channel information under the subscriber's management.

•
Step 2 (voice channel creation request): a request is made for the creation of a channel by transmitting a SIP:INVITE message containing the channel ID to the CSCF.

•
Step 3 (voice channel creation request): the CSCF forwards the message to the PTT server.

•
Step 4 (channel member inquiry): the MCPTT server acquires the member list of the corresponding channel from the CSC server.

•
Step 5 (voice channel participation request): the MCPTT server requests channel participation to the channel members by transmitting a SIP:INVITE message.
CSCF requests the application of quality of service (QoS) for voice channel through a policy and charging rule function (PCRF), and EPC (SAE-GW) and eNB are established by applying MCPTT voice QoS to the bearer for the voice channel. If the button for a group call is pressed from terminals included in a specific group, it is verified whether the voice or video call is maintained without being interrupted for 30 s in all the reception terminals within the group. As for interception, a terminal with a higher priority should be able to terminate an emergency call from a terminal with a low priority and should be able to preempt the right for transmission according to the priority levels preallocated to the terminals, including command devices, within the same group. During emergency calls within the same group, if another user with the same priority requests an emergency call, they are not allowed to preempt the right of transmission of the existing emergency call user. The reception function for a multicall group refers to an ability for one terminal to receive two or more call groups. In this case, the main call group should be able to transmit and receive, whereas the remaining subcall group is configured to only receive. To conduct tests, the PTT keys from a given pair of terminals, A1 and A2, are pressed simultaneously to verify if the respective main channels can communicate. Subsequently, the subchannel should be changed into the main channel to verify whether the call can be made in the same manner as before. As for the criteria for call connection tests, it should be confirmed whether the delay budget for each key performance indicator (KPI) is satisfied in the setup process for an MCPTT call presented in 3GPP TS 22.179 Release 13 [18].
• KPI 1 restricts the required time to acquire permission to talk to within 300 ms. • KPI 2 restricts the end-to-end call connection time (invite to 200 OK) to within 1000 ms. • KPI 3 restricts the end-to-end media transmission delay time (mouth-to-ear latency) to within 300 ms when encryption is not applied.
mode without automatically disconnecting the group call [16,17]. Figu ample concerning the creation of an MCPTT voice channel of the termin and it can be seen step-by-step as follows. • Step 1 (channel information inquiry): after an originating termina thentication procedure for a subscriber, it sends an inquiry to the CS nel information under the subscriber's management.

•
Step 2 (voice channel creation request): a request is made for the cre by transmitting a SIP:INVITE message containing the channel ID t • Step 3 (voice channel creation request): the CSCF forwards the m server.

•
Step 4 (channel member inquiry): the MCPTT server acquires the m corresponding channel from the CSC server.

•
Step 5 (voice channel participation request): the MCPTT server req ticipation to the channel members by transmitting a SIP:INVITE m  The approximate delay of KPI 1 is calculated as approximately 200 ms using the network section from UE to peer entity through eNB, including (60 + 10 ms) × 2 (two-way delay) + UE L7 delay time (60 ms). The test is conducted at a fixed point of the wireless environment medium electric field, which does not include additional delays such as terminal mobility and RAN sharing. KPI 1, 2, and 3 should be measured for each terminal using a measuring instrument, and the average value is derived after repeating the process more than 50 times to ensure statistical significance of the result.
The test environment is shown in Figure 6, and groups are set for each manufacturer so that they are not affected by other groups during the test. Local selection calls that can call all the terminals within a specific area should not be operated during the test. The resulting measurements of delay budget in group voice calls are shown in Table 4, and it can be observed that the results of testing at a fixed point of reference at a signal received power (RSRP) of −63 dBm satisfies KPI 1/2/3.

eMBMS Field Test Scenarios
The core component of eMBMS is multicast-broadcast single frequen (MBSFN) transmission technology, which can simultaneously transmit the from different base stations. Multiple cells can be operated as one large cell us technology by simultaneously transmitting the same data across multiple cells each other. The set of cells that transmit the same data through the MBSFN tr technology is called an MBSFN area, and several MBSFN areas are gathered eMBMS service area. MBSFN areas can be configured up to 256, and each c component of up to 8 different MBSFN areas. Unicast-based MCPTT group c allocate as many bearer resources as the number of terminals, but eMBMS-ba group calls should allocate as many bearer resources as the number of groups resource allocation of base stations for eMBMS services can freely switch betw and multicast methods, and DSCNs in the format of frequency division du allocated up to 60% of the resources. For group video and voice services p mobile group identity (TMGI), 600 and 80 Kbps of resources are allocated, r and TMGI has unique values for each eMBMS bearer, i.e., by group. Tests are on a total of 50 terminals: 15 from manufacturer A, 15 from manufacturer B, 10 ufacturer C, and 10 from manufacturer D. For organizing MCPTT groups in

eMBMS Field Test Scenarios
The core component of eMBMS is multicast-broadcast single frequency network (MBSFN) transmission technology, which can simultaneously transmit the same data from different base stations. Multiple cells can be operated as one large cell using MBSFN technology by simultaneously transmitting the same data across multiple cells adjacent to each other. The set of cells that transmit the same data through the MBSFN transmission technology is called an MBSFN area, and several MBSFN areas are gathered to form an eMBMS service area. MBSFN areas can be configured up to 256, and each cell can be a component of up to 8 different MBSFN areas. Unicast-based MCPTT group calls should allocate as many bearer resources as the number of terminals, but eMBMS-based MCPTT group calls should allocate as many bearer resources as the number of groups [19,20]. The resource allocation of base stations for eMBMS services can freely switch between unicast and multicast methods, and DSCNs in the format of frequency division duplex can be allocated up to 60% of the resources. For group video and voice services per temporal mobile group identity (TMGI), 600 and 80 Kbps of resources are allocated, respectively, and TMGI has unique values for each eMBMS bearer, i.e., by group. Tests are performed on a total of 50 terminals: 15 from manufacturer A, 15 from manufacturer B, 10 from manufacturer C, and 10 from manufacturer D. For organizing MCPTT groups in command devices, unicast group (50 terminals) and multicast group (50 terminals) are organized, and the area code for MBMS is set. For setting the eMBMS service area, the MBSFN area (1 cell) is configured with a single unit of radio unit for the National Information Society Agency (NIA) testbed, and for the MBSFN area, it is configured with a separate MBMS service area and an MBSFN sync area. For the call group in which the terminals are organized, the service area is set up in the BM-SC, and the MBSFN and synchronization areas are set in the MCE and digital unit. The configuration diagram of the service networks for the eMBMS test is shown in Figure 7. As shown in Figure 8, when MBMS service packets are analyzed, the MCPTT server that receives the message of MBMS listening status report should check the message (1-4 in Figure 8), which instructs the MCPTT client to use MBMS for receiving downlink data from the corresponding MCPTT service. Checking whether normal traffic is transmitted to multicast groups rather than unicast IP addresses is essential through the packet analysis of MBMS. For the eMBMS, as the networks are established and the terminals are released without testing the service networks and terminal, user-based testing should be performed in an area receiving electric waves from DSCNs. Moreover, handover testing between the DSCNs and commercial networks should be performed to check whether unicast and multicast traffics are received normally. The multicast channel scheduling period of the terminals and service networks should also be checked in advance to prevent the situation of replacing hardware due to nonsupporting functions in the future.

Discussion
MCPTT and eMBMS functions are the core services of the disaster safety communication network. This paper describes various functions of the MCPTT and field test scenarios and results for KPI 1/2/3. In addition, the test configuration scenarios and detailed test results for eMBMS are explained with examples, and in the future, they can be used as reference data for testing in countries that have established disaster safety communication networks. As shown in Figure 8, when MBMS service packets are analyzed, the MCPTT server that receives the message of MBMS listening status report should check the message (1-4 in Figure 8), which instructs the MCPTT client to use MBMS for receiving downlink data from the corresponding MCPTT service. Checking whether normal traffic is transmitted to multicast groups rather than unicast IP addresses is essential through the packet analysis of MBMS. As shown in Figure 8, when MBMS service packets are analyzed, the MCPTT server that receives the message of MBMS listening status report should check the message (1-4 in Figure 8), which instructs the MCPTT client to use MBMS for receiving downlink data from the corresponding MCPTT service. Checking whether normal traffic is transmitted to multicast groups rather than unicast IP addresses is essential through the packet analysis of MBMS. For the eMBMS, as the networks are established and the terminals are released without testing the service networks and terminal, user-based testing should be performed in an area receiving electric waves from DSCNs. Moreover, handover testing between the DSCNs and commercial networks should be performed to check whether unicast and multicast traffics are received normally. The multicast channel scheduling period of the terminals and service networks should also be checked in advance to prevent the situation of replacing hardware due to nonsupporting functions in the future.

Discussion
MCPTT and eMBMS functions are the core services of the disaster safety communication network. This paper describes various functions of the MCPTT and field test scenarios and results for KPI 1/2/3. In addition, the test configuration scenarios and detailed test results for eMBMS are explained with examples, and in the future, they can be used as reference data for testing in countries that have established disaster safety communication networks. For the eMBMS, as the networks are established and the terminals are released without testing the service networks and terminal, user-based testing should be performed in an area receiving electric waves from DSCNs. Moreover, handover testing between the DSCNs and commercial networks should be performed to check whether unicast and multicast traffics are received normally. The multicast channel scheduling period of the terminals and service networks should also be checked in advance to prevent the situation of replacing hardware due to nonsupporting functions in the future.

Discussion
MCPTT and eMBMS functions are the core services of the disaster safety communication network. This paper describes various functions of the MCPTT and field test scenarios and results for KPI 1/2/3. In addition, the test configuration scenarios and detailed test results for eMBMS are explained with examples, and in the future, they can be used as reference data for testing in countries that have established disaster safety communication networks.

Measurement of Wireless Coverage and Quality
Korea's DSCN project has a strategy to secure the target coverage by (i) establishing fixed base stations for national infrastructure, roads, and densely-populated areas; (ii) utilizing commercial networks for agricultural, fishery, and indoor/underground areas; and (iii) utilizing mobile base stations for mountainous areas. The coverage criteria for national infrastructure and government and public offices are the ground sections of national infrastructure designated according to the Framework Act on the Management of Disaster and Safety. Roads should give priority to service continuity, and in densely populated areas, the coverage range of fixed base stations includes residential, commercial, and industrial areas among the use areas according to the National Land Planning and Utilization Act. A mandatory coverage measurement test should be performed in the field to determine whether a single communications service infrastructure is properly established using four solutions (fixed base stations, commercial networks, mobile base stations, and LTE-based railway wireless communication system (LTE-R)/LTE-based maritime wireless communication (LTE-M)) [21]. Moreover, a quality test for call services should be conducted simultaneously with the coverage measurement. Preliminary testing of call services is required as many risk factors exist in using the newly established DSCNs immediately in various disaster situations. Areas that fall short of the criteria after the test should be retested after the optimization work of the communications business carriers.

Measurement of Coverage and Call Quality
The purpose of coverage measurement is to provide stable services of DSCNs and to check the call rights necessary for immediate response in the event of a disaster. As for the coverage verification method, the RSRP value is measured at 75 × 75 grid units based on submunicipal administrative divisions, and RF scanners are used to verify whether an area is present whose RSRP drops below −117 dBm in the coverage edge. As for coverage measurements, RF scanners installed in vehicles measure RSRP values on roads that allow vehicle entry within the measurement area. The level of frequency for PS-LTE (700 MHz band) is measured in seconds using an RF scanner and stored together with GPS coordinates. To ensure a sufficient number of inspections (four or more) per grid, the inspection is conducted within 60 km/h, and if the average of RSRP is less than −117 dBm, it is judged as an area with insufficient coverage. The problem of the terminal itself should be minimized by selecting the manufacturer's terminal with the most stable operation in DSCNs.
The terminal setting for the measurement is configured by dividing the terminals into those set up to communicate only via the PS-LTE (Band 28) frequency and those capable of RAN sharing. The composition of the terminals and measuring instruments for the coverage and quality of data/call tests is shown in Figure 9. For the measurement of coverage, the test should be conducted after the consultation of coverage paths for the target area so that the terminal can check the latest version of firmware and measure the signal strength for the target coverage proposed by the business carrier that built it. Electronics 2021, 10, x FOR PEER REVIEW 15 of 22 Quality measurement is performed by composing a scenario in the order of individual call voice → individual call text → individual call video (set to record call video on the receiving terminal when measuring a video call) → voice group call → text group call → video group call. This scenario is repeated more than 20 times for measurement, and the composition of the measured call time for each call is shown in Table 5. Because 405 s are required for one measurement, approximately 135 min are required for 20 measurements [22,23].

Item Private Currency (s) Group Calls (s) Voice
Text Video  Voice  Text  Video  Idle time  10  5  10  10  10  10  Setup time  20  10  20  30  30  30  Traffic time  65  10  65  30  10  30  Total  95  25  95  70  50  70 Data are measured by repeatedly uploading and downloading files 30 times or more from the NIA diagnostic monitor app installed in the terminal. The time required for one measurement is 37 s, i.e., ping measurement (11 s), upload (13 s), and download (13 s). The measurement indicators based on the use stage of service for individual and group voice calls in standards such as ITU-T, ETSI, and 3GPP are shown in Table 6. [24,25].

Indicators
Description Connection success rate The rate at which the connection is completed within a certain base time among the calls of a connection attempt.

Dropped call rate
The rate at which the call is cut off within a certain call time among the calls of a successful connection.

Voice quality
Measurement on a five-point scale for quality deterioration between original and received voices.
(ITU-T P.862 PESQ or P.863 POLQA standard algorithm applied). Quality measurement is performed by composing a scenario in the order of individual call voice → individual call text → individual call video (set to record call video on the receiving terminal when measuring a video call) → voice group call → text group call → video group call. This scenario is repeated more than 20 times for measurement, and the composition of the measured call time for each call is shown in Table 5. Because 405 s are required for one measurement, approximately 135 min are required for 20 measurements [22,23].  Idle time  10  5  10  10  10  10  Setup time  20  10  20  30  30  30  Traffic time  65  10  65  30  10  30  Total  95  25  95  70  50  70 Data are measured by repeatedly uploading and downloading files 30 times or more from the NIA diagnostic monitor app installed in the terminal. The time required for one measurement is 37 s, i.e., ping measurement (11 s), upload (13 s), and download (13 s). The measurement indicators based on the use stage of service for individual and group voice calls in standards such as ITU-T, ETSI, and 3GPP are shown in Table 6. [24,25]. Table 6. Verification indicators for individual/group voice calls.

Connection success rate
The rate at which the connection is completed within a certain base time among the calls of a connection attempt.
Dropped call rate The rate at which the call is cut off within a certain call time among the calls of a successful connection.

Voice quality
Measurement on a five-point scale for quality deterioration between original and received voices.
Call success rate Call rate of successful calls above MOS threshold among call attempts.

Measurement of End-to-End Call Services
The measurement of end-to-end call services aims to confirm that smooth services are always provided for the areas where DSCNs have been established. For the end-to-end verification procedure, the measurement of voice, data, and video should be conducted for each service within a single base station, among multiple base stations, within heterogeneous network base stations, and between DSCN base stations and heterogeneous network base stations. For the measurement area, various types of verification areas and routes are selected for measuring end-to-end call services on the heterogeneous base stations (Samsung and Nokia) and heterogeneous networks (commercial, railway, and maritime networks). Moreover, the five major morphologies of national infrastructure, roads, densely populated areas, mountainous areas, and agricultural and fishery areas should be included by dividing downtown and suburban areas. For the measurement items, voice, data, and videos are measured for the following five types of call services according to call types.

•
For interception, only voice and data are measured, excluding voice transmission delay, voice quality, and data call connection measurement, while command devices and measuring instruments are not connected. The vehicle environment of the terminal and measuring instrument for measuring end-to-end call services are shown in Figure 10.

Call success rate
Call rate of successful calls above MOS threshold among call attempts.

Measurement of End-to-End Call Services
The measurement of end-to-end call services aims to confirm that smooth servic are always provided for the areas where DSCNs have been established. For the end-t end verification procedure, the measurement of voice, data, and video should be co ducted for each service within a single base station, among multiple base stations, with heterogeneous network base stations, and between DSCN base stations and heterogen ous network base stations. For the measurement area, various types of verification are and routes are selected for measuring end-to-end call services on the heterogeneous ba stations (Samsung and Nokia) and heterogeneous networks (commercial, railway, an maritime networks). Moreover, the five major morphologies of national infrastructur roads, densely populated areas, mountainous areas, and agricultural and fishery are should be included by dividing downtown and suburban areas. For the measureme items, voice, data, and videos are measured for the following five types of call servic according to call types.

•
For interception, only voice and data are measured, excluding voice transmission d lay, voice quality, and data call connection measurement, while command devic and measuring instruments are not connected. The vehicle environment of the terminal and measuring instrument for measurin end-to-end call services are shown in Figure 10. Analysis of the measurement results of five end-to-end call services is as follows. • The success rate of text transmission is measured at 100%, and also measured are th completion rate of individual calls (voice-97%; video-97%), and the completio rate of group calls (voice-96%; video-99%).

•
Because the communication environment between a tunnel section (commercial ne works) and a ground section (disaster networks) changes frequently, the policy RAN sharing with commercial networks should be clearly defined. Analysis of the measurement results of five end-to-end call services is as follows.
• The success rate of text transmission is measured at 100%, and also measured are the completion rate of individual calls (voice-97%; video-97%), and the completion rate of group calls (voice-96%; video-99%).

•
Because the communication environment between a tunnel section (commercial networks) and a ground section (disaster networks) changes frequently, the policy of RAN sharing with commercial networks should be clearly defined.

•
As the radio resource control (RRC) connection release time is different for each business area, consultation among business operators is required; if no traffic is present in area A for 10 s, it transitions to the RRC_IDLE state due to the occurrence of RRC connection release.

•
Strengthening the examination of areas where DSCN services are weak is necessary, including the boundary area between disaster and commercial networks in mountainous areas, the boundary area between business operators, and the boundary area between regions (numerous call drops and network connection failures occur due to RAN-sharing failures).

Testing Mobile Base Stations
Mobile base stations provide coverage info through the main communication method in sparsely populated areas, such as mountainous areas and the open sea. Mobile base stations have a vehicle or a backpack type, and coverage inspection should measure frequency intensity (RSRP) while moving based on the coverage boundary provided in the mobile station equipment specification [23,26]. The test area should be selected for mountainous areas, urban areas (big-, small-, and medium-sized cities), and empty sites considering the diffraction, refraction, reflection, etc., of electric waves. Testing for data communication, voice, and video PTT functions should be conducted in terms of interoperability, call quality, and coverage through an efficient operation of mobile base stations and independent network configuration. Therefore, detailed test criteria of the mobile base station are shown in Table 7. The test environment for mobile base stations (generic vehicle and generic portable types) is shown in Figure 11. The test environment for mobile base stations (generic vehicle and generic portable types) is shown in Figure 11.

Coverage Test Results of Vehicle-Type Mobile Base Stations
The vehicle-type mobile base station (PCI: 500, antenna height of approximately 3 m) installed in the parking lot of the KT branch in Gangneung-si, Gangwon-do, has a low antenna height, which causes propagation loss due to neighboring buildings, and which is suitable to be used for traffic dispersion rather than securing coverage in urban areas. The coverage of the vehicle-type mobile base station is valid for up to 245 m in a straightline distance. The portable mobile base station (PCI: 348, antenna height of approximately 1.5 m) also has a low antenna height and is blocked by neighboring buildings, and the output is also low; therefore, the portable mobile base station with a coverage of approximately 50 m is not suitable for use in urban areas.

Test Results of the Network Performance of Vehicle-Type Mobile Base Stations
The RAN-sharing test in the downtown area is conducted between fixed base stations ↔ mobile base stations, commercial network base stations ↔ mobile base stations, and maritime network base stations ↔ mobile base stations. Figure 12 shows that when the PS-LTE terminal is set to the RAN-sharing mode in the P2 area and moves among the other communication networks, the signal level of the serving cell decreases while the terminal is connected to the maritime networks (Band28, PCI 466); therefore, it is handed over to the commercial networks (Band3, PCI 469). Because handover occurs when the signal level of the serving cell is in the range of −117 dBm recommended by DSCNs, it satisfies the design value. The P1 section is a densely populated area with a large floating population due to Anmok Beach (Gangneung Cafe Street), which is a target coverage area for DSCNs. However, although it is located at a close distance of 1.7 km from the Songjeong-dong base station (PS-LTE) under the coverage shadow area of the DSCNs and call fails/drops occurs frequently, the signal level is very low (RSRP −100 to −120 dBm); thus, optimization is required to secure coverage.

Coverage Test Results of Vehicle-Type Mobile Base Stations
The vehicle-type mobile base station (PCI: 500, antenna height of approximately 3 m) installed in the parking lot of the KT branch in Gangneung-si, Gangwon-do, has a low antenna height, which causes propagation loss due to neighboring buildings, and which is suitable to be used for traffic dispersion rather than securing coverage in urban areas. The coverage of the vehicle-type mobile base station is valid for up to 245 m in a straight-line distance. The portable mobile base station (PCI: 348, antenna height of approximately 1.5 m) also has a low antenna height and is blocked by neighboring buildings, and the output is also low; therefore, the portable mobile base station with a coverage of approximately 50 m is not suitable for use in urban areas.

Test Results of the Network Performance of Vehicle-Type Mobile Base Stations
The RAN-sharing test in the downtown area is conducted between fixed base stations ↔ mobile base stations, commercial network base stations ↔ mobile base stations, and maritime network base stations ↔ mobile base stations. Figure 12 shows that when the PS-LTE terminal is set to the RAN-sharing mode in the P2 area and moves among the other communication networks, the signal level of the serving cell decreases while the terminal is connected to the maritime networks (Band28, PCI 466); therefore, it is handed over to the commercial networks (Band3, PCI 469). Because handover occurs when the signal level of the serving cell is in the range of −117 dBm recommended by DSCNs, it satisfies the design value. The P1 section is a densely populated area with a large floating population due to Anmok Beach (Gangneung Cafe Street), which is a target coverage area for DSCNs. However, although it is located at a close distance of 1.7 km from the Songjeong-dong base station (PS-LTE) under the coverage shadow area of the DSCNs and call fails/drops occurs frequently, the signal level is very low (RSRP −100 to −120 dBm); thus, optimization is required to secure coverage. Data transmission speed is measured in the downtown area where the PS-LTE mobile, fixed, marine network, and commercial network base stations are installed, centering on the KT Gangneung branch. For the download speed in Figure 13, although RSRP and signal-to-interference ratio (SINR) are good for the coverage section of the vehicle-type mobile base station (PCI 500) in the P1 section, the download speed is 0~1 Mbps, owing to the limit of the satellite backhaul bandwidth (2 Mbps). The download speed in the P2 section is 2~4 Mbps, owing to the SINR degradation caused by wave interference between PS-LTE and LTE-M. Although the P3 section is the coverage shadow section of PS-LTE and LTE-M (RSRP, −104 dBm and SINR, −1 dB), the download speed is measured at 6 Mbps, which is higher than in the area of the wave interference. Evidently, the resolution of the wave interference between PS-LTE and LTE-M should be considered a priority. Data transmission speed is measured in the downtown area where the PS-LTE mobile, fixed, marine network, and commercial network base stations are installed, centering on the KT Gangneung branch. For the download speed in Figure 13, although RSRP and signal-to-interference ratio (SINR) are good for the coverage section of the vehicle-type mobile base station (PCI 500) in the P1 section, the download speed is 0~1 Mbps, owing to the limit of the satellite backhaul bandwidth (2 Mbps). The download speed in the P2 section is 2~4 Mbps, owing to the SINR degradation caused by wave interference between PS-LTE and LTE-M. Although the P3 section is the coverage shadow section of PS-LTE and LTE-M (RSRP, −104 dBm and SINR, −1 dB), the download speed is measured at 6 Mbps, which is higher than in the area of the wave interference. Evidently, the resolution of the wave interference between PS-LTE and LTE-M should be considered a priority.
The summary and analysis of the test results of the coverage in disaster safety communication network and mobile base station are as follows:

•
The target of service area for disaster safety communication network is densely populated areas including national infrastructure and roads with two or more lanes. However, as a result of the coverage test, there are many areas share with commercial network due to the shaded sections; hence, it is necessary to improve and optimize the coverage.

•
As the area where the disaster safety communication network and LTE-R RAN Sharing are not applied is measured, the voice call quality of the disaster safety communication network service is very poor. Therefore, early interworking measures through consultation with involved organizations are required. • When measuring disaster safety communication network and LTE-M RAN-sharing, because there are many cases where the Physical Resource Block is not allocated and data cannot be transmitted at the transmission rate (DL, UL) 0 Mbps, it is necessary to check the RAN-sharing parameter and the system.

•
Due to the coverage shadow section between the disaster safety communication network and LTE-M and the instability of RAN sharing, poor voice call quality, such as call drops and call fails, occurs; hence, optimization work is needed to improve the coverage between public networks. • The overall voice call quality in the boundary area between different types is good. However, because of poor coverage in some areas and quality deterioration due to the base station, output abnormality occurs; hence, equipment inspection and coverage optimization are required.

Summary and Analysis of Test Results
The summary and analysis of the test results of the coverage in disaster safety communication network and mobile base station are as follows: • The target of service area for disaster safety communication network is densely populated areas including national infrastructure and roads with two or more lanes. However, as a result of the coverage test, there are many areas share with commercial network due to the shaded sections; hence, it is necessary to improve and optimize the coverage.
• As the area where the disaster safety communication network and LTE-R RAN Sharing are not applied is measured, the voice call quality of the disaster safety communication network service is very poor. Therefore, early interworking measures through consultation with involved organizations are required.

•
When measuring disaster safety communication network and LTE-M RAN-sharing, because there are many cases where the Physical Resource Block is not allocated and data cannot be transmitted at the transmission rate (DL, UL) 0 Mbps, it is necessary to check the RAN-sharing parameter and the system.

•
Due to the coverage shadow section between the disaster safety communication network and LTE-M and the instability of RAN sharing, poor voice call quality, such as call drops and call fails, occurs; hence, optimization work is needed to improve the coverage between public networks. • The overall voice call quality in the boundary area between different types is good. However, because of poor coverage in some areas and quality deterioration due to

Summary and Analysis of Test Results
As described above, many problems occur in interworking between public networks or heterogeneous networks in the initial stage of disaster safety communication network. Therefore, it is necessary to solve problems after performing that coverage of the entire area, end-to-end and mobile base stations test.

Discussion
This chapter presents detailed scenarios for voice call quality measurement for coverage and disaster safety communication network services and scenarios for end-to-end testing in the same environment as actual users. In addition, test items and results for mobile base stations used as main communication methods in mountains and maritime areas increase success rate for late-entry construction organizations.

Conclusions
This paper described the testing and verification methods that should accompany the establishment of DSCNs in Korea. For the performance test on the core equipment, a new call model and the field test method of MCPTT and eMBMS as the representative services of DSCNs were proposed. Our experience regarding unit equipment tests, interoperability tests, and coverage/quality measurements was discussed along with use cases to confirm whether the network was established to satisfy the requirements of the client organization. The call model required for testing PS-LTE and the testing methods for coverage/quality provided herein can help reduce trial and error for countries that establish DSCNs in the future. Therefore, we expect that the test methodology and know-how for wired and wireless networks in DSCNs, wireless coverage, and quality measurements will be utilized by countries establishing DSCNs in the future.
In addition, to consistently maintain and manage service quality in the process of operating the disaster safety communication network, research on methods to establish indicators for a Service Level Agreement and manage sensor traffics for Internet of Things services should be conducted in the future.