Research on Performance Metrics and Environmental Conditions for 5G MIMO OTA

Abstract

For Multiple-input, multiple-output (MIMO) user equipment (UE), over-the-air (OTA) testing of radiated multi-antenna reception performance is crucial to guarantee actual network performance and is mandatory in North America, Europe and China. For 5G MIMO OTA, 3GPP has specified the clustered delay line (CDL) channel models as the reference channel models, which present higher order and directivity than those of 4G. Therefore, which kinds of performance metrics and environmental conditions are feasible and necessary to better characterize the MIMO performance should be studied. In this paper, MIMO OTA measurement results of different figures of merits (FoMs) under both UE noise-limited and interference-limited environmental conditions have been compared and discussed, respectively. In addition, the impact of different channel models on MIMO OTA throughput performance as well as the variance per azimuth are compared. Based on the analysis, Peak-Null can better reflect the difference between the 12 azimuth positions than Variance. Total Radiated Multi-antenna Sensitivity (TRMS) under both channel models presents nearly the same performance trends at different thresholds. On the contrary, MIMO Average Radiated SIR Sensitivity (MARSS) exhibits a high dependence on channel models. The MARSS under UMi channel model shows a much smaller performance variation between UEs than UMa channel model. TRMS under 3GPP UE noise-limited environmental condition exhibits a stronger ability to distinguish between good and bad performing devices than that of MARSS under CTIA interference-limited environmental condition. The discrepancy can reach 4 dB at most. This discrepancy does not come from different average manners between TRMS and MARSS, but originates from environmental condition itself. Therefore, it is proposed to adopt TRMS under UE noise-limited environmental condition as the unique or baseline test condition in 5G FR1 MIMO OTA. Peak-Null can be considered as a secondary FoM to characterize the variance per azimuth.

1. Introduction

A multiple-input, multiple-output (MIMO) system is adopted in 4G and 5G mobile communications to achieve higher channel capacity and data throughput [1,2,3,4,5]. The requirements for high rates of wireless communication networks grow exponentially with the applications of smart terminals and the development of MIMO and antenna arrays [6,7,8,9,10,11,12]. It is reported in [2] that with 8 MIMO antennas for 8 × 8 MIMO operation in one smartphone, the channel capacity can reach 37 bps/Hz at a 20 dB signal-to-noise ratio (SNR), which is much larger than that of a single antenna for single-input, single-output (SISO) operation. The data throughput can reach up to 7.4 Gbps with a bandwidth of 200 MHz centered at 3.5 GHz for Long Term Evolution (LTE) MIMO operation. Two LTE MIMO antennas are required in 4G user equipment (UEs), and more MIMO antennas will be installed in 5G smartphones and beyond [13,14,15,16,17,18].

For MIMO UE, OTA testing of radiated multi-antenna reception performance is crucial to guarantee actual network performance and has been mandated by the Cellular Telecommunication and Internet Association (CTIA) and the 3rd Generation Partnership Project (3GPP) as the standard method to evaluate the transceiver performance of MIMO wireless devices [19,20,21].

For LTE MIMO OTA, 3GPP and CTIA use different figures of merits (FoMs) and environmental conditions to evaluate MIMO performance of smartphones under Spatial Channel Model Extended (SCME) UMa and UMi channel models. In 3GPP, Total Radiated Multi-antenna Sensitivity (TRMS) under noise-limited environmental condition is selected as the FoM with both UMa and UMi channel models. In CTIA, MIMO Average Radiated SIR Sensitivity (MARSS) under interference-limited environmental condition is defined as the FoM with only UMi channel model. Under noise-limited environmental condition, UE MIMO throughput performance is evaluated in terms of the minimum downlink power at a low downlink signal level. Under interference-limited environmental condition, UE MIMO throughput performance is evaluated in terms of the minimum signal to interference ratio (SIR) at a relatively high downlink signal power from the emulator while the level of an additive white gaussian noise (AWGN) power is varied. A comparison of MIMO OTA methods in 3GPP and CTIA is listed in

Table 1. A comparison of MIMO OTA methods in 3GPP and CTIA.

Parameters	3GPP LTE	CTIA LTE
Test Methodology	MPAC (reference)	MPAC (reference)
MIMO	2 × 2	2 × 2
Environmental Condition	UE Noise-Limited	Interference-Limited
DUT (devices under test) Placement	sub 1 GHz: black box 1 GHz above: white box	sub 1 GHz: black box 1 GHz above: white box
FoM	TRMS	MARSS
Average Method	Inverse Linear Average	Linear Average
Outage Levels	95%/90%/70%	95%/90%/70%
Channel Models	SCME UMa &UMi	SCME UMa

. For 5G MIMO OTA, 3GPP has specified the clustered delay line (CDL) channel models as the reference channel models. Compared with LTE (2 × 2), higher order (4 × 4) is required in 5G channel models. In addition, the channel environment in 5G presents higher directivity due to higher operating frequency, which requires new FoMs characterizing variance per spatial orientation. So far, environmental conditions and FoMs for 5G MIMO OTA have not been determined yet. Therefore, analysis and performance comparison of environmental conditions and related FoMs aforementioned are of great importance to determine which ones should be adopted in 5G MIMO OTA.

The objective of this paper is to compare and define the performance metrics and environmental conditions for the verification of the radiated multi-antenna reception performance of 5G MIMO DUTs. In this paper, MIMO OTA measurement results of different FoMs under both UE noise-limited and interference-limited environmental conditions have been compared and discussed, respectively. At first, two new FoMs, i.e., Peak-Null and Variance, are proposed and compared to characterize the variance of performance per spatial orientation. Secondly, the impact of different channel models on UE MIMO OTA throughput performance is analyzed. Based on the analysis, it can be concluded that TRMS under UE noise-limited environmental condition has a stronger ability to distinguish between good and bad performing UE devices than that of MARSS under CTIA interference-limited environmental condition. At last, the mechanism of this phenomenon is explored based on an average manner analysis. This discrepancy in capability does not come from the average manner adopted by MARSS and TRMS, but originates from the environmental condition itself, i.e., the difference in signal level between UE noise-limited and interference-limited environmental conditions. In summary, it is proposed to adopt TRMS under UE noise-limited environmental condition as the unique or baseline test condition in 5G MIMO OTA. Peak-Null can be considered as a secondary FoM to characterize the variance per azimuth.

2. Experimental Specification

In this paper, 10 commercial smartphones available on the Chinese market have been randomly selected and a MIMO OTA measurement has been performed at the China Academy of Information and Communications Technology (CAICT) Multi-probe Anechoic Chamber (MPAC) using test methods adopted by the 3GPP LTE MIMO OTA specification TR37.977 and the CTIA MIMO OTA Test Plan [17]. The system configuration and measurement environment are illustrated in Figure 1, where 8 dual-polarized probes are uniformly distributed around the DUT as an azimuth ring with 16 channels of spatial channel emulation. All DUTs are measured with both UMi and UMa channel models under different environmental conditions. The “white box” test approach has been adopted for LTE MIMO OTA at frequency bands higher than 1 GHz in order to ensure consistent DUT placement, whereas a black box approach has been selected for 5G Band 5 (881.5 MHz) scenarios with larger than 30 cm test zones. All DUTs are fully contained within the test zone, the physical center of which is aligned with the center of the test zone. The experimental setup is summarized in

Table 2. A summarization of the MIMO OTA testing setup.

Parameters	Experimental Setup
DUTs	10 Smartphones
Test Methodology	MPAC (reference)
MIMO	2 × 2
Band	Band 5 (881.5 MHz)
Environmental Condition	UE Noise-Limited & Interference-Limited
DUT Placement	Black Box
FoMs	TRMS &MARSS
Outage Levels	95%/90%/70%
Channel Models	SCME UMa &UMi

. For a comprehensive analysis, the measured raw data are averaged in both TRMS and MARSS manners, respectively.

The average TRMS of free space data mode portrait (FS DMP), free space data mode landscape (FS DML) and free space data mode screen up (FS DMSU), is defined as the FoM of LTE MIMO OTA in 3GPP. The average shall be done in a linear scale for the TRMS results at these device positions, as shown in Equation (1):

$$\mathrm{T}\mathrm{R}\mathrm{M}{\mathrm{S}}_{\mathrm{a}\mathrm{v}\mathrm{e}\mathrm{r}\mathrm{a}\mathrm{g}\mathrm{e},\mathrm{x}}=10\log \left[3/\left(\frac{1}{{10}^{{\mathrm{S}}_{\mathrm{F}\mathrm{S} \mathrm{D}\mathrm{M}\mathrm{P},\mathrm{x}}/10}}+\frac{1}{{10}^{{\mathrm{S}}_{\mathrm{F}\mathrm{S} \mathrm{D}\mathrm{M}\mathrm{L},\mathrm{x}}/10}}+\frac{1}{{10}^{{\mathrm{S}}_{\mathrm{F}\mathrm{S} \mathrm{D}\mathrm{M}\mathrm{S}\mathrm{U},\mathrm{x}}/10}}\right)\right]$$

(1)

where:

$${\mathrm{S}}_{\mathrm{M}\mathrm{O}\mathrm{D}\mathrm{E},\mathrm{x}}=10\log \left[12/\left(\frac{1}{{10}^{{\mathrm{S}}_{\mathrm{M}\mathrm{O}\mathrm{D}\mathrm{E},\mathrm{x},0}/10}}+\frac{1}{{10}^{{\mathrm{S}}_{\mathrm{M}\mathrm{O}\mathrm{D}\mathrm{E},\mathrm{x},1}/10}}+\cdots +\frac{1}{{10}^{{\mathrm{S}}_{\mathrm{M}\mathrm{O}\mathrm{D}\mathrm{E},\mathrm{x},11}/10}}\right)\right]$$

(2)

Such that MODE is one of {FS DMP, FS DML, FS DMSU}, x is one of thresholds (for example {[70%, 90%, 95%]}), and {P_MODE,x,0, …, P_MODE,x,11} are the measured sensitivity values at each azimuth position.

The MARSS is defined as the FoM of LTE MIMO OTA in CTIA. The sensitivity point is defined as the SIR value (in dB) required to reach 70%, 90%, and 95% of the theoretical maximum throughput of the test case. The SIR value (in dB) associated with each outage point at the m_th EUT rotation is defined as the effective throughput SIR sensitivity (P_ETSS,70,m, P_ETSS,90,m, and P_ETSS,95,m).

The MARSS is computed as a linear average from the effective radiated sensitivity measurements across all EUT rotations and is given as follows:

$${\mathrm{P}}_{\mathrm{M}\mathrm{A}\mathrm{R}\mathrm{S}\mathrm{S},\mathrm{x}}=10{\log }_{10}\left(\frac{1}{\mathrm{M}}{\displaystyle \sum _{\mathrm{m}=1}^{\mathrm{M}}{10}^{{\mathrm{P}}_{\mathrm{E}\mathrm{T}\mathrm{S}\mathrm{S},\mathrm{x},\mathrm{m}}}}\right)$$

(3)

where M is the total number of azimuth positions, i.e., 12, x is one of thresholds (for example {[70%, 90%, 95%]}), and P_ETSS,x,m is the measured effective throughput SIR sensitivity at each azimuth position.

3. Experimental Results and Analysis

In our experiments, limited by the testing time, only 7 of 10 DUTs were tested with alternative channel models (i.e., Uma for TRMS under 3GPP UE noise-limited environmental condition and UMi for MARSS under CTIA interference-limited environmental condition).

3.1. Mode Analysis under 3GPP Environmental Condition

There are three modes in the 3GPP TRMS procedure under UE noise-limited environmental condition, namely, FS DMP, FS DML, and FS DMSU. On the contrary, there is only one mode in the CTIA MARSS procedure under interference-limited environmental condition, i.e., Portrait 45, the position of which is the same as FS DMP. In this section, a TRMS comparison between the three modes and the averaged TRMS at all thresholds is performed for comparison. Figure 2 and Figure 3 are for the UMa and UMi channel models, respectively.

It can be observed that for both channel models, the averaged TRMS is almost the same as TRMS in FS DML for all DUTs at three thresholds, with a maximum deviation of around 1.6 dB. We can conclude that TRMS performance is mainly determined by FS DML mode under UE noise-limited environmental condition, which can be seen as the dominant mode. In later analysis, TRMS in FS DML mode will be selected to be compared to MARSS for variation analysis per 12 azimuth positions.

3.2. Variation Analysis per 12 Azimuth Positions between TRMS in FS DML and MARSS

For smartphone applications, it is expected that the radiation performance variance across all angles of arrivals (AoAs) is close to zero. Traditionally, performance metrics, i.e., TRMS and MARSS, are the average values of 12 AoAs, which cannot reflect the variation between different AoAs. In this section, two new FoMs, i.e., Peak-Null and Variance, are introduced to analyze the variations in the 12 azimuth positions between TRMS in FS DML and MARSS. Peak-Null is determined by evaluating the total peak-to-peak variations between the 12 azimuth positions, which emphasizes the delta in the AoAs of signal. Variance stands for the variance of the measured values at each azimuth position.

It can be seen from Figure 4 and Figure 5 that the Peak-Null results and the Variance results show similar trends between different thresholds under each combination of channel models and environmental conditions. However, the Peak-Null results are at least 1.5 dB higher than the Variance results for the same DUT, which indicates that Peak-Null can better reflect the difference between AoAs. In addition, the maximum variations for both Peak-Null and Variance between DUTs are also analyzed. It can be seen from Figure 4a that the maximum variations are 2.6 dB and 1.6 dB between DUTs for Peak-Null and Variance respectively. A similar delta of around 1 dB can also be achieved in other plots, which indicates that Peak-Null can better reflect the difference between the 12 azimuth positions than Variance.

3.3. Comparison between TRMS and MARSS

TRMS and MARSS results under both UMi and UMa channel models are compared in Figure 6 and Figure 7.

Under 3GPP UE noise-limited condition, different channel models present nearly the same UE performance trends. TRMS-UMa (blue curves) in Figure 6 always presents worse MIMO OTA performance with a consistent performance degradation around 5 dB with respect to TRMS-UMi (blue curves) in Figure 7. In addition, both TRMS-UMa and TRMS-UMi results show a similar performance variation of around 5 dB across all DUTs at all three thresholds (i.e., TP@95%, TP@90%, and TP@70%). Therefore, if TRMS under 3GPP UE noise-limited condition is adopted for NR FR1 MIMO OTA, selecting only one threshold of TP@70% is sufficient to distinguish between UEs’ performance. TP@95% should also be considered as a criteria for checking whether the UE can pass 11 of total 12 positions.

Under CTIA interference-limited condition, MARSS results show nearly the same UE performance trends at all thresholds under the same channel model. However, different channel models present very different DUT performance trends. Meanwhile, the performance degradation for MARSS-UMa (orange curves) in Figure 6 with respect to MARSS-UMi (orange curves) in Figure 7 is not consistent. The offset is between 5 to 9 dB across DUTs. The maximum UE performance variation across DUTs is around 4.2 dB at the threshold of TP@95% with CTIA testing method (MARSS-UMa in Figure 6), which is similar to TRMS results at the threshold of TP@95% under 3GPP UE noise-limited condition. However, the maximum variation of MARSS-UMa (orange curves) in Figure 6 at the threshold of TP@70% is 2.6 dB, much smaller than the maximum offset at the threshold of TP@95%. We can conclude that, for MARSS-UMa under CTIA interference-limited condition, different thresholds present different performance variations among DUTs. Furthermore, for MARSS-UMi, the maximum performance variation is ~1.1 dB under all thresholds, which is much smaller than that of MARSS-UMa. Therefore, it can be concluded that MARSS performance is highly dependent on channel models.

In summary, TRMS under UE noise-limited condition has a stronger ability to distinguish between good and bad performing DUTs than MARSS under CTIA interference-limited condition. If MARSS under CTIA interference-limited condition is adopted for NR FR1 MIMO OTA, a proper channel model shall be carefully selected to differentiate UE performance. Meanwhile, both thresholds of TP@70% and TP@95% shall be defined.

3.4. Influence Analysis of Different Average Manners

For a comprehensive analysis, the measured raw data of both TRMS and MARSS are averaged pursuant to both TRMS and MARSS manners, i.e., the averaging shall be done as an inverse linear average and a linear average, respectively. As can be seen from Equations (1) and (3), the average methods of TRMS and MARSS favor toward the peak (like traditional TRP/TIS) and the null (emphasize the defect) of the 12 azimuth positions, respectively. Therefore, the influence from SIR substitution/penalty is less in TRMS due to favor in peak than in MARSS manner.

Figure 8 and Figure 9 illustrate the impact of different average methods to the same set of raw data. In Figure 8, different average methods are applied to TRMS raw data of UMi and UMa models under 3GPP UE noise-limited environmental condition, respectively. In Figure 9, similar processing is performed under CTIA interference-limited environmental condition. Under either combination of environmental condition and channel model, a maximum discrepancy of no more than 0.3 dB can be observed, which suggests that different average methods adopted in TRMS and MARSS, i.e., inverse linear average and linear average, have little influence on the discrepancy in ability to distinguish between well- and poorly performing DUTs.

In conclusion, TRMS under UE noise-limited environmental condition has a stronger ability to distinguish between well- and poorly performing DUTs than MARSS under CTIA interference-limited environmental condition. This discrepancy originates from the environmental condition itself. Under interference-limited environmental condition, UE MIMO OTA throughput performance is evaluated in terms of the minimum SIR at a relatively high downlink signal power from the emulator while the level of an AWGN signal (which is used to emulate co-channel interference in an actual network) is varied. The initial SIR is set to 30 dB. On the contrary, under noise-limited environmental condition, UE MIMO OTA throughput performance is evaluated in terms of the minimum downlink power at a low downlink signal level. In such an environmental condition, DUT’s low-signal performance is typically limited by its internal noise floor while antenna efficiency is the dominant role in TRMS. The measured SIRs under noise-limited environmental condition are around 20 dB level at all thresholds, which are 100 dB higher than the measured minimum downlink signal power in amplitude. Accordingly, the transformation of weak signals from a linear domain to a logarithmic domain will create much greater divergence across samples.

4. Conclusions

This paper analyzes the performance of TRMS and MARSS based on the MIMO OTA measurement among 10 commercial smartphones on the Chinese market. Two new FoMs, i.e., Peak-Null and Variance, have been compared to analyze the performance of 12 azimuth positions. The impact of different channel models on UE MIMO OTA throughput performance as well as the variance per azimuth positions are compared. Peak-Null can better reflect the difference between the 12 azimuth positions than Variance. TRMS under UE noise-limited environmental condition has a stronger ability to distinguish between good and bad performing DUTs than MARSS under CTIA interference-limited environmental condition. This discrepancy comes from environmental condition itself. Meanwhile, different channel models present nearly the same UE performance trends under 3GPP UE noise-limited condition. On the contrary, MARSS performance presents highly dependence on channel models. In conclusion, it is proposed to adopt TRMS under UE noise-limited environmental condition as the unique or baseline test condition in 5G FR1 MIMO OTA. Peak-Null can be considered as a secondary FoM to characterize the variance per azimuth.