1. Introduction
Implementing and improving assessment mechanisms of the quality of service (QoS) and quality of network (QoN) in digital telecommunications have resulted in the dynamic development of cellular networks and a large diversity of services provided to subscribers and users. It is worth noting that QoS and QoN metrics are the basis for evaluating subsequent generations of mobile networks. For example, for the fifth-generation (5G) New Radio (NR), so-called International Mobile Telecommunications (IMT) for 2020 (IMT-2020), parameters such as user experienced or peak data rates, latency, area traffic capacity, connection density, and spectrum and network energy efficiency are used to show significant improvement in network performance and QoS compared to fourth-generation (4G), i.e., Long-Term Evolution (LTE) or IMT-Advanced, networks [
1]. This type of approach and these metrics are also used in political, social, and economic solutions, e.g., in determining the targets and objectives for the development of the digital environment in Europe for 2030, the so-called European Digital Decade [
2].
QoS assessment is carried out by mobile network operators (MNOs), providers of services and mobile applications, telecommunications market regulatory authorities [
3], and, increasingly often, users of networks and services. QoS measurement from the MNO or user sides can be performed at the network level. Generally, the first approach is only available to MNOs. It is used to optimize the allocation of radio resources to individual users/services [
4,
5]. However, the second approach can be used by both users and professional companies that provide QoS measurement and MNO evaluation services, e.g., on behalf of regulatory authorities or MNOs themselves [
6,
7]. Such measurements conducted by independent companies can give the MNOs objective information on QoS and QoN. From the viewpoint of the average network user, assessing QoS (e.g., specific values of throughput or latency) is usually too technical and incomprehensible. Therefore, the so-called quality of experience (QoE), related to the subjective perception of the service, is increasingly analyzed. It is crucial when assessing the smoothness and quality of video or voice transmissions, teleconferencing, online games, website and social media content’s downloading speed, or data transmission speed. The need for user involvement to evaluate QoE is a significant drawback of this approach. Hence, automatons (i.e., machines) or artificial intelligence algorithms are increasingly used for this purpose [
8,
9,
10,
11].
Implementing a specific service, e.g., data transmission or video streaming, is necessary to assess the QoS or QoE for such a service. From the viewpoint of professional companies providing services in the QoS/QoE/QoN evaluation, this forces the use of the so-called active measurements. For this purpose, user equipment (UE) operating in the network of a selected MNO is used to realize a specific service, which allows for the QoS/QoE assessment. This approach does not make it possible to assess QoN or MNO continuously and independently of the service provided. Hence, when performing active QoS/QoE measurements, the so-called passive measurements are carried out using scanners. Then, the power and quality of reference signals transmitted by the base stations of a selected MNO are measured, regardless of the provided connections or services [
7].
In modern communication networks, QoS has witnessed substantial advancements, with a focus on addressing the challenges posed by the deployment of 4G, 5G, and emerging technologies. One of the innovative directions of development is the integration of machine learning (ML) models to optimize network performance [
12,
13]. A. Al-Thaedan et al. propose an ML framework for predicting downlink (DL) throughput in 4G and 5G networks, showcasing the need for advanced design and optimization methodologies [
13]. Similarly, D. Minovski et al. adopt a comprehensive approach by utilizing ML models for throughput prediction in LTE and 5G networks, with a particular emphasis on real-life measurements and network slices’ performance benchmarking [
12].
Another prevalent area of investigation in the literature is the performance analysis of mobile broadband networks in diverse scenarios. A.A. El-Saleh et al. provide a detailed analysis of mobile broadband performance in urban settings [
14]. The authors of [
15,
16,
17] show results of drive test campaign for LTE and 5G networks in Malaysia. Throughput measurements performed in urban [
16] and rural [
15,
17] environments allowed us to assess the benefits and transition potential from 4G to 5G non-standalone (NSA) networks. Meanwhile, Ref. [
18] focuses on 5G performance measurements in the context of bus transportation systems, emphasizing the need for tools to identify and enhance services. S. Horsmanheimo et al. focus on studying 4G/5G mobile network technologies applied to smart city unmanned aerial vehicle (UAV) services [
19]. The results indicate that 5G is more sensitive to altitude changes than 4G, which should be considered in future mobile network planning. On the other hand, an analysis of specific services could be valuable from the end-user perspective. Therefore, some measurement campaigns concentrate on selected services, such as WhatsApp [
20], YouTube [
7], and web browsing [
21].
Aspects of QoE evaluation and measurement, in addition to QoS, are often considered in the literature, e.g., [
22,
23,
24]. A mapping model of 5G key performance indices (KPIs) to QoS and, finally, to QoE metrics is proposed in [
23]. Meanwhile, the authors of [
24] analyze the impact of software components and virtualization usage on 5G network performance. Furthermore, the authors of [
22] delves into the measurement processes of QoS and QoE in 5G technology, highlighting the diverse approaches and the absence of specific standards for 5G. The paper emphasizes the importance of tailoring measurement processes to the specific location and conditions of 5G implementation, offering valuable insights for effective QoS and QoE assessments in diverse scenarios.
KPI measurements in a 5G NSA network are considered in [
25]. The paper compares the results against actual hands-on 4G network measurements, confirming the expected superiority of 5G in basic KPIs, including up- and DL throughput, latency, and jitter, using different tools, e.g., iPerf, iPerf3 [
26], and Cisco TRex. Moreover, utilization of other KPI measurement tools to assess the throughput and latency for mobile phones is described in [
27].
Signal metrics are often the basis for estimating QoS metrics, especially those related to transmission rate or spectrum efficiency. It results from certain theoretical premises, i.e., the Shannon–Hartley theorem [
28]:
where
and
are the channel capacity and bandwidth, respectively; and
is the signal-to-noise ratio (SNR).
Capacity is the boundary value of the throughput that can be achieved for the given bandwidth and signal quality (i.e., SNR). On the other hand, the parameter
is referred to as spectrum efficiency. Equation (1) implies that throughput should be directly related to a signal quality measure. In the case of mobile networks, the primary metric determining the received signal quality is the signal-to-interference-plus-noise ratio (SINR). However, in practice, a parameter determining the received signal power, i.e., reference signal received power (RSRP), is more often used to estimate throughput. This approach is proposed by M. Lorenz [
29], an expert from Rohde & Schwarz (R&S) company. A similar methodology was also used to determine the capacity (i.e., boundary throughput) in LTE and 5G networks by the Polish regulatory authority (i.e., the Office of Electronic Communications) when developing requirements for MNOs related to a 5G auction for the C band [
30]. However, we propose a novel approach: using RSRP and SINR for throughput and jitter estimation in UL and DL, respectively.
The main aim of this paper is an analysis of relationships between radio reference signal parameters (such as RSRP, SINR, or Reference Signal Received Quality (RSRQ)) and QoS metrics (i.e., throughput and jitter) in the 5G network. We analyzed the scenario of throughput assessment by using the iPerf application [
26,
31,
32] and a measurement testbed based on R&S equipment. Measurements in Warsaw for four MNOs were performed by the Systemics-PAB company as part of the drive test campaign. Systemics-PAB provides QoS/QoE assessment services in mobile networks. In the analyzed scenario, throughput and lost datagrams rates are measured for Transmission Control Protocol (TCP) and User Datagram Protocol (UDP) in both uplink (UL) and DL. Additionally, jitter is measured for UDP. In the drive tests, the measurement of signal metrics, i.e., RSRP, SINR, and RSRQ, was performed. The measurements were carried out at the 2.1 and 2.6 GHz bands. In the mobile network, the coverage and capacity issues are connected with the QoS and radio signal parameters. However, based on the collected measurement data, we cannot unambiguously draw conclusions about coverage and capacity. From a propagation viewpoint, the range of base stations for the analyzed bands are similar. Therefore, we may expect the areal aspect of the coverage to be similar for these bands and all MNOs. We calculate correlation coefficients for various pairs of measured parameters, similarly to [
13]. The obtained results are the basis for selecting the best signal parameters for estimating throughput or other signal metrics. This approach can also be used in modeling some parameters based on others. Finally, the obtained results allow for the assessment of the MNOs regarding 5G network availability and provided QoS.
Our key contributions are as follows:
Selection of measurement data and correlation analysis of QoS metrics for iPerf scenario, four MNOs, TCP and UDP, and UL and DL based on drive-test measurement campaign in Warsaw;
Analysis of linear regression between QoS parameters, which can be used in their modeling or estimation;
Linking the obtained results with radio resources used by individual MNOs and their radio access network (RAN);
Discussion of the results and MNO comparison based on the assessment of the correlation level of QoS parameters;
Original approach to the classification/identification of MNOs in Poland based on the RSRQ-5G measurement.
The remainder of the paper is organized as follows. In
Section 2, we describe the measurement scenario, testbed, iPerf application, and the QoS and reference signal metrics measured in the drive test campaign. The correlation analysis and relationships between QoS parameters are presented in
Section 3 and
Section 4, respectively. A discussion of the results, including the comparison of MNOs, is contained in
Section 5.
Section 6 provides a summary.
2. Drive Test Campaign
2.1. Measurement Scenario
The QoS analysis was carried out based on a 33.5-h drive test campaign in an urban environment that was performed on 23–27 September of the 2021 year by the Systemics-PAB company, which provides a QoS/QoE assessment in mobile networks for MNOs, regulatory authorities, or other entities [
33]. The time scheme of the tests is presented in
Table 1.
The measurement campaign was carried out for four MNOs: Orange, T-Mobile, Play, and Plus, called Orange Polska, T-Mobile Polska, P4, and Polkomtel, respectively. They are global MNOs providing 4G/5G services in Poland. In the following, we use the shorter brand-name MNOs.
Nowadays, the MNOs use 5G NSA technology. Two methods of ensuring transmission for 5G services are implemented in their networks [
33]:
Dynamic sharing of 10 or 15 MHz channels within a frequency band at 2.1 GHz for 5G or LTE services, used by Orange, T-Mobile, and Play;
Providing a 40 MHz channel within a frequency band at 2.6 GHz used by Plus.
The detailed description of frequency allocation for each MNO is presented in
Appendix A.
The purpose of the drive tests was to collect statistically representative measurement data enabling the assessment of the network readiness of four Polish MNOs in Warsaw to support 5G services concerning the service scenarios defined in ‘Mobile and wireless communications Enablers for the Twenty-twenty Information Society-II’ (METIS II) project [
34]. It was possible by conducting a complete set of tests, such as interactive data transmission, iPerf-supported tests, sequential web browsing, and multimedia streaming transfers based on YouTube service.
Table 2 presents the number of measurements (so-called bins) for the tested services per each MNO. The total number of performed measurements (bins) was 46,608, of which 340 were incorrect (i.e., lack of a complete set of measured parameters).
Figure 1 presents the percentage of the analyzed service bins for each MNO and the percentage of MNO bins in the iPerf test. In this paper, we only analyzed the measurement results for the iPerf tests. We will continue our studies for other services shortly.
The measurement campaign was carried out in the Capital City of Warsaw.
Figure 2 illustrates the measurement vehicle route during the drive test campaign [
33]. The average speed of the measurement vehicle varied between 0 and 105 km/h.
2.2. Measurement Testbed
Drive tests were realized using the mobile testbed installed on special cars’ equipped with a set of professional measuring R&S equipment (see
Figure 3 [
35]), such as [
33,
35]:
R&S Smart Benchmarker Rel. v20.3.95;
Measuring terminals (UEs) based on Samsung Galaxy S21 + 5G (SM-G996BDS) (Suwon, Republic of Korea) working as active test probes;
R&S SwissQual QualiPoc software(Rel. 20.3.121);
Passive Radio Frequency (RF) scanner R&S TSME6 (Munich, Germany);
Global Positioning System (GPS) receiver with the GPS antenna.
The R&S Smart Benchmarker Rel. v20.3.95 worked as the main control, synchronization, and measurement unit that controlled and managed all devices of the test station within the test campaigns. The used measuring equipment was fully operational and performed reliable measurements, which were checked and verified during pretests. The measuring stations were operated by qualified, experienced personnel from the Systemics-PAB company. The measurement terminals were used to ensure a fair assessment of all MNOs, enabling the aggregation of all frequency bands used by the measured networks. The main aim of using the passive scanner was to measure the QoS metrics that represented the quality and power of radio reference signals, e.g., RSRP, SINR, or RSRQ. The scanner supports all frequency bands used in tested mobile networks.
The measurements, data processing, and data analysis were carried out under standards defined by ETSI [
36,
37,
38,
39,
40,
41,
42] and best industry practices described by R&S in [
43,
44].
During the tests, controlled measurement conditions were ensured:
The measuring terminals were in conditions with a uniform, controlled temperature;
The parameters of the measurement terminals affecting the measurement quality, such as processor load or processor/battery temperature, were continuously monitored;
The equipment was mounted in special housings in the roof box at a height of about 1.8 m.
2.3. iPerf Tool
The iPerf tool is used for network performance measurement, testing, and tuning. It is free (i.e., based on Berkeley Software Distribution (BSD) license) and available for various operating systems, including CentOS Linux, FreeBSD, macOS, OpenBSD, Android, other Linux distributions, and Windows. The iPerf is compatible with various protocols, i.e., TCP, UDP, and Stream Control Transmission Protocol (SCTP) with both Internet Protocol (IP) version 4 (IPv4) and 6 (IPv6), and offers a range of customizable parameters. This tool is a staple for network administrators and engineers seeking to troubleshoot network problems, enhance performance, and undertake network experiments. Furthermore, the iPerf plays a crucial role in the performance testing and benchmarking of various network devices. In December 2023, the iPerf3-16 was released [
26,
31,
32].
Tests using the iPerf application were aimed at examining the stability and performance of the radio transmission link. They were carried out for two types of ISO/OSI (International Organization for Standardization/Open Systems Interconnection reference model) transport layer protocols, i.e., TCP and UDP. The set data collected during tests were sequenced into UL and DL streams. This approach allows us to present the behavior of applications using TCP and UDP.
TCP provides connection transmission using acknowledgment mechanisms for uniquely numbered transferred packets called three-way handshakes. It means that the established connection is open for the duration of data transmission, which guarantees a relatively high degree of reliability of flow control connections and increases transmission security. It comes at the cost of information overhead on packet numbering and the need to check the correct transmission of each packet, which reduces the efficiency of data transfer. TCP supports the applications for which the certainty of data delivery is crucial, requiring transfer integrity control, but not time-critical, such as applications based on the following protocols, Hypertext Transfer Protocol (HTTP), File Transfer Protocol (FTP), Secure Shell (SSH), Simple Mail Transfer Protocol (SMTP), and Internet Message Access Protocol (IMAP) 4 [
45].
UDP is a connectionless protocol without mechanisms to control the correctness of data flow, which affects the transfer speed and efficiency. UDP implementation ensures faster data transfer, but there is no guarantee of data delivery. This protocol is used by real-time applications, mainly those that exchange large volumes of traffic and do not expect confirmations. It is used primarily by videoconferences, data streaming, and network gaming, as well as applications based on Voice over Internet Protocol (VoIP) or Domain Name System (DNS) [
46].
The parameter values adopted in the iPerf tests are contained in
Table 3. In TCP, the following test profile was used during iPerf testing: 3 streams of 40 Mbps (7 s transfer time) and 3 streams of 8 Mbps each (5 sec transfer time) for DL and UL, respectively.
UDP is mainly used for network stress testing because TCP automatically limits the throughput to adapt to the available bandwidth. To this aim, a bandwidth for UDP should be chosen far above what the connection can handle, e.g., if a user wants to stress a 10 Mbps connection, he/she should send about 100 Mbps of traffic [
31]. Hence, the Lost Datagrams Rate (LDR) for UDP obtained in the test is very high compared to TCP.
2.4. Measured QoS and Signal Metrics
In the analyzed scenario, we evaluated QoS and radio signal quality measures for UL and DL, using the iPerf application for two protocols, TCP and UDP. As basic QoS metrics, we use the following:
Additionally, using the R&S TSME6 scanner, the following radio signal quality parameters are measured:
RSRP (dBm)—reference signal received power (RSRP) is utilized mainly to rank among different candidate cells in accordance with their signal strength. Generally, the reference signals on the first antenna port are used to determine this parameter. However, the reference signals sent on the second port can also be used in addition to the radio stations on the first port if UE can detect that they are being transmitted [
47]. Basically, RSRP is defined as the linear average over the power contributions (in watts) of the resource elements that carry cell-specific reference signals within the considered measurement frequency bandwidth [
48,
49].
SINR (dB)—Signal-to-interference-plus-noise ratio (SINR) is defined as the linear average over the power contribution (in watts) of the resource elements carrying cell-specific reference signals divided by the linear average of the noise and interference power contribution (in watts) over the resource elements carrying cell-specific reference signals within the same frequency bandwidth [
47]. In fact, it is only measured for the reference signal. Therefore, it is called a Reference Signal-SINR (RS-SINR) [
48,
49].
RSRP-5G (dBm)—RSRP for 5G NR is widely named as a Second Synchronization-RSRP (SS-RSRP) [
50,
51]. It is defined as the linear average over the power contributions (in watts) of the resource elements that carry secondary synchronization signals. It represents the quantitative measure of SS-RSRP. The RSRP-5G range is from –156 to –31 dBm [
52]. This parameter gives possibilities to compare the strengths of signals from individual cells in 5G networks, which is key for cell selection or handover. SS-RSRP is the equivalent of RSRP used in LTE [
53].
RSRQ-5G (dB)—Reference Signal Received Quality (RSRQ) for 5G NR is named as a Second Synchronization-RSRQ (SS-RSRQ) [
50,
51]. It is defined as the ratio of N×SS-RSRP/NR carrier Received Signal Strength Indication (RSSI), where N is the number of resource blocks in the NR carrier RSSI measurement bandwidth. The measurements in the numerator and denominator shall be made over the same set of resource blocks. The SS-RSRQ measurements can be used for cell selection, cell reselection, and mobility procedures. The RSRQ-5G range is defined from –43 to 20 dB [
52]. SS-RSRQ is used in 5G NR networks to determine the quality of the radio channel. RSRQ, unlike RSRP, also includes interference level due to the inclusion of RSSI in the calculation. This parameter is also used for cell selection and handover if the RSRP is insufficient. It happens mainly in border parts of a cell. It is similar to RSRQ determined in LTE [
53].
SINR-5G (dB)—SINR for 5G is often called a Second Synchronization-SINR (SS-SINR) [
50,
51]. It is defined as the linear average over the power contribution (in watts) of the resource elements carrying secondary synchronization signals divided by the linear average of the noise and interference power contribution (in watts). In 5G networks, SINR-5G is reported as a coded value via measurement report to a base station. The reported range is from 0 to 127, total 128 values, where values 0 and 127 represent SINR < −23 dB and SINR > 40 dB, respectively [
52].
The method of determining the radio signal quality parameters in the 5G-NR system is defined by the 3GPP standard, particularly by the specification [
50,
51].
6. Summary
In this paper, we analyzed the QoS metrics and reference signal parameters used in LTE and 5G for the iPerf scenario, which were obtained in the drive test campaign conducted by Systemics-PAB in the urban area for four Polish MNOs. These measurement data were the basis for the correlation analysis performed for TCP and UDP, and UL and DL. Based on PCC, we selected the best signal parameters that can be used to estimate throughput for TCP/UDP, and jitter only for UDP. In this regard, we propose to use RSRP for UL and SINR for DL. In terms of assessing the relationship between radio signal metrics (i.e., RSRP and SINR), we find that these metrics are independent of the provided services and can generally be analyzed independently of the MNO. Only the Plus for the relationship between RSRP-5G and SINR-5G is characterized by a certain deviation.
On the other hand, analyzing the relationship between RSRQ-5G and SINR-5G makes it possible to identify MNOs based on averaged measurement data and reflects their KPIs in a broader scope. The analysis of the measurement results confirmed that the use of the same RAN contributed to a very similar QoS assessment for two MNOs, i.e., Orange and T-Mobile. On the other hand, the additional use of the 40 MHz band at the 2.6 GHz band by Plus for the needs of the emerging 5G NR network allowed for higher RSRQ-5G and significantly greater throughput for DL. The remaining MNOs do not have radio resources in this frequency band. Before the auction for the C band, these MNOs (i.e., Orange, Play, and T-Mobile) could only use bands typical for LTE or older generations (i.e., 900, 1800, 1900, or 2700 MHz) for the 5G NR network. This approach is possible thanks to the principle of technological neutrality.
In the near future, the authors plan to analyze other scenarios for QoS tests in Polish mobile networks.