Ray Tracing Simulators for 5G New Radio Systems: Comparative Analysis Through Urban Measurements at 27 GHz

Abstract

The use of millimeter-wave spectrum in fifth-generation (5G) systems is increasing the need for accurate prediction of received power and coverage in real deployment scenarios. In this context, ray tracing (RT) is a promising approach for site-specific analysis, although its reliability depends on how accurately different tools reproduce measurements in complex urban environments. This work presents a comparative assessment at 27 GHz of three RT tools: in-house Exact tool based on Vertical Plane Launching (VPL), Matlab 5G and open-source Sionna RT based on Shooting and Bouncing Rays (SBR). The comparison relies on a large outdoor walk-test campaign, including about 14,725 measurement points collected in a real urban area around a 27 GHz mMIMO base station, using real operator-provided antenna radiation patterns. Measured and simulated power levels are compared using statistical metrics, including Mean Absolute Error (MAE), Root Mean Square Error (RMSE), and a planning-oriented coverage-rate metric. The results show a reasonable agreement between simulations and measurements, with RMSE and MAE values around 10–12 dB, highlighting tool-specific behaviors related to boundary effects, interaction modeling, and high-power overestimation. This work confirms that RT is a flexible support for 5G preliminary network design, reducing the need for extensive drive tests.

1. Introduction

The 5G and beyond technologies are opening a new horizon for telecommunication, enabling the connection not just between people but also between things [1], with lower latencies than previous generation systems. To meet these new requirements and in response to the continuous growth in demand for higher data rates, 5G systems employ new approaches and technologies. In particular, the key novelties of 5G systems are the use of millimeter-wave (mm-Wave) spectrum, ultra-densification of cells and massiveMIMO (mMIMO) technology with beam steering and beamforming techniques [2]. The mm-Wave spectrum, consisting of frequencies from 30 GHz to 300 GHz, allows the use of a wider bandwidth, enabling the connection of many devices with a high data transmission rate. However, moving to higher frequencies increases signal propagation losses due to the interaction between mm-Wave and the surrounding environment (building, vegetation, and vehicles) and to atmospheric phenomena, such as rainfall, fog or clouds [3,4,5].

For this reason, to ensure adequate radio coverage, cell dimensions in 5G networks decrease from several kilometers in radius, as in previous generations’ systems, to about 100 m [2,6]. In addition, mMIMO techniques change base station characteristics, introducing many antennas that generate narrow and directional beams (beamforming technique). With the beam steering technique, it is also possible to sweep the beam direction and focus the transmitting power to a single connected user, guaranteeing a good connection to the base station. Several measurement campaigns have been carried out near 5G base stations [7,8,9], assessing power levels in indoor and outdoor scenarios. However, measuring 5G signal is challenging due to variations in signal strength over time and space, together with the need for very complex equipment, such as vector spectrum analyzers. Experimental investigation on high-frequency 5G radio coverage is also limited by the low availability of deployed antennas operating in the FR2 band (24.25 GHz–52.6 GHz). Theoretical and numerical studies about 5G networks have gained significant improvement thanks to the deterministic technique of RT [10,11]. Indeed, due to the complexity of 5G systems and the many features they introduce, developing a highly accurate predictive method for assessing power levels is extremely important. Indeed, as an example, narrow beams, beam steering and different downtilts and azimuth angles usually complicate 5G network modeling since they make small geometric errors costly.

The aim of this work is to assess the potential and limitations of RT techniques for 5G system deployment scenarios, providing some useful insight for the practical use of different available RT tools. For this purpose, three different tools (Exact, Matlab and Sionna), representative of academic, commercial, and open-source solutions based on distinct algorithms, have been considered, and computational analysis has been carried out by independent laboratories. Matlab 5G toolbox is a commercial suite tool that is currently widely used in the literature for both radio coverage and exposure assessment [12,13,14], while Exact is an academic tool developed by the University of Naples Federico II for research purposes [15,16]. Sionna, instead, is an open-source, recently introduced, GPU-accelerated toolbox that includes a ray tracing engine (Sionna RT) for site-specific propagation modeling in complex three-dimensional environments [17]. Moreover, an extensive measurement campaign, with more than about 15,000 measurement points, has been carried out in collaboration with a mobile operator near a proprietary 27 GHz mMIMO antenna, located in a real complex urban electromagnetic environment. The antenna frequency is located in the 3GPP n258 frequency band and corresponds to the real commercial European deployments [18].

Preliminary research on the currently published literature has highlighted that a comprehensive comparison of ray tracing tools in 5G FR2 band scenarios, including the comparison with measurements, has not been addressed. Moreover, most of the studies are focused on the FR1 band or at 28 GHz in controlled/indoor environments with only a few measured points. This study aims to address this gap with the following research questions: test the accuracy of ray-based tools, understand their practical usability for operators and define useful metrics for network design. Indeed, the extensive measurement campaign considered in this work represents added value to the analysis and comparison of the numerical simulation tools. An in-depth analysis of measured and simulated results is presented, considering different metrics to enhance the robustness of the comparison. Several challenges in the 5G network modeling are discussed, including boundary effects, accuracy in Line-of-Sight (LOS) and Non-Line-of-Sight (NLOS) regions, and coverage metrics. Outcomes showed a general good agreement between the predicted and measured values, despite the complexity of the selected scenario and the different features of the tools considered. This suggests that the RT technique, whatever algorithm is considered, represents valuable support for the actual deployment of 5G networks. Moreover, the coverage analysis introduced significant metrics that can be employed for practical network planning. Overall, the present work extends the current knowledge on radio propagation modeling at mm-Waves and supports the widespread deployment of 5G networks.

The paper is organized as follows: Section 2 reports a literature review on different studies focused on ray tracing modeling combined with measurement campaigns; in Section 3, the measurement campaign is presented, while Section 4 introduces the ray-based algorithms employed in this work; and in Section 5, the results are shown, while their discussion is contained in Section 6. Finally, Section 7 summarizes the main outcomes of this study and their impact.

2. Related Works

Several works on RT tools validation have been published, and most of them are based on the comparison between simulated results and measured values. Danping He et al. [10] introduced a tutorial on how to calibrate and design RT tools for 5G applications. They presented different commercial and academic software tools based on RT techniques, and from measurement campaigns, made in two different locations at 3.5 GHz and 28 GHz, they recalibrated and validated the open-source CloudRT tool. However, they did not perform an intercomparison between the software tools introduced, and the measurement campaign accounted for around 4080 measured points collected in an area of around 300 m from the BS site. Tiago Osório et al. [19] presented a comparison study on RT tools for the design of solar concentrating collectors. Seven tools were used, including one commercial, two open-source and four in-house software. From the comparison, it was possible to highlight the main capabilities and limitations of each software, understanding the best use case for each of them. Nevertheless, the comparison was not supported by any on-site measurement. Like in the previous case, S. Incerti et al. [20] performed an intercomparison between three different RT software (two open-source and one in-house) to validate the reliability of a new tool, without any comparison with real measurements. Additionally, E. Aksoy et al. [21] presented an intercomparison between two different RT tools (one commercial and one open-source) at THz frequencies, supported by indoor measurements too. However, channel (not field level) measurements were performed, and only 24 experimental points were considered. On the other hand, in [22], Zhu et al. propose a benchmarking framework to compare the computational efficiency of heterogeneous ray-based simulation methods for urban wireless propagation in electromagnetic digital-twin scenarios at 28 GHz. They employed two commercial solutions (Wireless InSite and Matlab5G Toolbox) and one open-source model (Sionna RT). More recently, measurement-supported studies based on open-source RT frameworks have started to appear in the literature. In [23], Li et al. investigated Sionna-based RF signal mapping in the CBRS band (3.55–3.7 GHz), validating the approach through an extensive real-world measurement campaign. In [24], Lee et al. analyzed UAV coverage prediction in a rural foliage-dominated scenario at 3.3–3.4 GHz, comparing RT results with measurements collected on the AERPAW platform. In [25], Manukyan et al. considered a dense urban scenario in Rome using real cellular measurements in low-to-mid bands, showing that the agreement with Sionna simulations strongly depends on antenna placement, orientation, and scene accuracy. These works confirm the growing interest in open-source RT tools validated against measurements. Additionally, in [26], the authors presented a ray tracing analysis supported by on-site measurements in an indoor environment at 3.7 GHz. Two different commercial tools were considered (Narda EFC-400 and Matlab 5G ToolBox), and a few measurement spots were analyzed. Similarly, in [27], one in-house RT software and one commercial software (Wireless InSite) are employed and compared with outdoor measurements at 3.5 GHz. Furthermore, in [14], two commercial RT solutions are compared (Matlab ToolBox and Wireless InSite) in an outdoor environment at 3.5 GHz, without performing a measurement campaign.

An overview of the presented study is reported in

Table 1. Literature overview of studies accounting for ray-based tools intercomparison.

Work	Frequency	Measurements (y/n)	Tools	Calibration (y/n)
[10]	3.5 GHz, 28 GHz	y (outdoor)	CloudRT	y
[14]	3.5 GHz	n	Matlab and Wireless InSite	n
[21]	300 GHz	y (indoor)	Wireless InSite and Sionna	n
[22]	28 GHz	n	Wireless InSite, Matlab and Sionna	n
[23]	3.66 GHz	y (outdoor)	Sionna	n
[24]	3.4 GHz	y (outdoor)	Sionna	y
[25]	3.6 GHz	y (outdoor)	Sionna	y
[26]	3.7 GHz	y (indoor)	Narda EFC-400 and Matlab	n
[27]	3.5 GHz	y (outdoor)	In house and Wireless InSite	n

, with the corresponding tools involved. Among them, only NARDA EFC-400 and Matlab do not specify the GPU as a requirement. As can be seen, most of the works that rely on measurement data are conducted in the FR1 band due to the easier and more widespread availability of operating FR1 band antennas. On the other hand, studies that focus on the FR2 band are usually performed at 28 GHz and account for a few experimental points. Different from their approach, our work relies on about 15,000 measurement-based data and includes real antenna parameters at 27 GHz, in which material and scattering behavior are slightly different, to enhance model accuracy. The measurement campaign is carried out in a large, outdoor complex environment that is still uncommon in the 5G literature. Moreover, published works in some cases also consider a calibration procedure or add empirical corrections, while this study uses default RT model settings.

3. Measurement Campaign

A measurement campaign was performed in Turin (Italy), on a city area served by one Base Station (BS) site with three 27 GHz mMIMO antennas, each covering a section of 120° around the site, identified as PCI4, PCI5 and PCI6, where PCI stands for Physical Cell Identity. The selected urban scenario with the BS position is reported in Figure 1. The scenario is mainly composed of buildings, making it suitable for the presented RT analysis. However, the selected scenario is complex from an electromagnetic point of view, due to the presence of vehicles, people, and many other unpredictable environmental factors that may occur during the measurement campaign. This aspect is crucial when mm-Waves are employed and makes it important to test ray-based tools in such critical scenarios. Measurements were performed in walk-testing mode during a sunny day to record the received power indicator. This approach allowed for the collection of detailed signal data while moving through the environment, providing a realistic insight into network coverage. Additional details about the measurement campaign are described in the following.

• Measurement Setup: The instrument setup is illustrated in Figure 2 and consists of a Rohde&Schwarz TSMA6B Mobile Network Scanner, a Rohde&Schwarz TSME30DC ultra compact down converter for the 24 GHz–30 GHz input band, a TSME-Z20 omni-directional antenna operating in the 26 GHz–40 GHz band with maximum external diameter of 46 mm, vertical polarization and gain from 2 to 4 dBi, a TSME-ZKC cable for up to 40 GHz to connect the omni-directional antenna to the down converter, and a laptop running Rohde&Schwarz Romes software. The scanner operates with an internal sampling rate of up to 122.88 Msps. This frequency is the standard for handling channels with a bandwidth of up to 100 MHz, typical of 5G configurations. For the walking test, the walking speed was around 3 km/h, the receiver height was around 1.7 m, and the instrument, equipped with a GPS antenna to record geographical information and a battery backup, was placed in a backpack, as shown in Figure 2.
• Measurement Procedure: The measurement process followed three main steps: an initial system setup, a continuous data acquisition during the walk test, and a final post-processing phase. During measurements, the receiving antenna was kept at a proper distance from the operator’s body to mitigate body-induced signal perturbation, minimize body shadowing, and reduce signal fluctuations caused by pedestrian-induced fading during the walk. Given the high scanner sampling rate relative to the walking speed, multiple measurements were available for each unique latitude–longitude position. Each segment was traversed only once, thereby avoiding repeated measurements of the same segment and ensuring consistent measurement conditions across the entire route. The measurement was conducted, monitoring the broadcast Synchronization Signal Block (SSB) signals using a scanner. By utilizing the scanner’s ability to simultaneously measure the SSB signals of different PCI within the limits of its sensitivity, the measurements were collected independently of the mobility rules configured in the network for handover management.
• Data Handling: As reported in [28,29], the measurement uncertainty level of the scanner is lower than 1.5 dB. In addition to this, some studies with a similar setup quantify the overall error source as 2.39 dB [30] and 3.70 dB [31]. GPS accuracy was taken into account by considering that the scanner sampling rate was sufficiently high relative to the walking speed. As a result, multiple measurements were collected for each unique latitude–longitude position. These samples were then averaged per position. Consequently, any errors due to occasional samples being associated with slightly different locations are statistically negligible and do not affect the average results.
• Measurement Indicators and Objectives: Overall, the measurement campaign provides the received power SSS-RSRP (Secondary Synchronization Signal-Reference Signal Received Power) indicator recorded during the walk test, while monitoring the broadcast SSB signals associated with the different PCI and recording the geographical coordinates of the measurement positions. This received power indicator provides experimental reference for comparison with simulations and network coverage assessment, while PCI and geographical coordinates enable the association of each sample with the corresponding sector and measurement position.

4. Ray Tracing Analysis

Three different tools based on the RT technique were considered in numerical analysis: Exact 1.1, Matlab 5G Toolbox r2024b and Sionna v0.19.2. The main tool characteristics are reported in

Table 2. Tool Model.

Tool	Type	RT Model
Exact	In-house	VPL
Matlab 5G Toolbox	Commercial	SBR
Sionna RT	Open Source	SBR

.

4.1. Description of the Tools

Exact is an in-house proprietary tool developed by the University of Naples Federico II [16] based on the Vertical Plane Launching (VPL) approach [32,33]. This approach makes the tool computationally efficient while maintaining a high accuracy in urban areas. Accordingly, it is suitable to compute the electromagnetic field level over a dense grid of points. The tool is written in IDL (Interactive Data Language), and it is able to simulate different kinds of scenarios. In principle, it can support any frequency in the range from a few hundred MHz to several tens of GHz, once a vector file describing the buildings, a raster file describing the terrain topography, and the antenna radiation diagram are available. One of the key characteristics of this tool is that simulations stop when the amplitude of each ray goes below a threshold that can be independently set according to the user’s needs. Such a criterion provides speed and accuracy performances adequate for 5G cell planning.

Matlab 5G Toolbox [34] supports RT for simulating signal propagation in 5G New Radio (NR) systems, compliant with 3GPP Release 15, 16, and 17 [35,36,37]. This feature models wave behavior in urban and indoor scenarios using geometric propagation models to compute multipath trajectories. It considers reflections, diffractions and scattering while integrating real-world maps via Site Viewer and OpenStreetMap [38]. Advanced MIMO and beamforming modeling enable massive-MIMO analysis in line with 3GPP specifications. The toolbox also supports mm-Wave communications in FR2 bands, incorporating high antenna directivity, atmospheric attenuation and path loss models from 3GPP TR 38.901 [39]. The RT functionality, implemented via the Shooting and Bouncing Ray (SBR) method, aids in analyzing coverage, signal strength and beamforming strategies. These capabilities support research and development in wireless communication, offering a detailed approach to studying interference, coverage and network performance in diverse deployment scenarios.

Sionna is an open-source library for link- and system-level simulation that includes a GPU-accelerated ray tracing engine (Sionna RT) [40]. The RT module is based on path tracing with multiple interactions between the rays and the surrounding 3D environment, allowing for multiple reflections and transmissions at dielectric interfaces. High-resolution three-dimensional scenes, built, for example, from OpenStreetMap data and modeled in Blender, can be imported and combined with realistic antenna radiation patterns and polarization. Sionna RT operates in the mm-Wave band and supports directional antenna arrays, enabling the analysis of site-specific propagation effects, coverage, and exposure in complex urban scenarios. Similar to Matlab, the Sionna RT functionality is implemented via the Shooting and Bouncing Ray (SBR) method, modeling each propagation path as a cascade of scattering processes occurring along the three-dimensional scene, including specular reflections, refractions, diffractions, and diffuse scattering.

4.2. Numerical Simulations Set-Up

The simulated scenario is reported in Figure 3, and it consists of about 230 buildings of different heights with a total map size of around 600 m × 650 m.

With regard to the input antenna radiation diagram, the radiation pattern envelope was used for each of the three mMIMO antennas of the BS. The antenna pattern has been imported into the tools in a .txt format file, with an angular resolution of 1° and by setting vertical polarization. Detailed antenna characteristics have been provided directly by the mobile operator (Vodafone) and reported in

Table 3. Antenna Parameters.

Parameter	Value
Frequency	27 GHz
Gain	26.98 dBi
Height	33.5 m
Position	395,517.19; 4,988,683.86
Downtilt (PCI4, PCI5, PCI6)	5°, 5°, 3°
Azimuth (PCI4, PCI5, PCI6)	85°, 150°, 310°
Input Power	3 dBm

. Simulations were performed considering also the diffraction phenomenon for each RT tool, modeled according to the Uniform Theory of Diffraction (UTD) [41]. Scattering and wall transmissions were not considered. Concerning the maximum number of interactions between a ray and a scene object, in

Table 4. Simulation Parameters.

Tool	N. of Iterations
Exact	No limits. Procedure stops when for each ray the field level goes below −135 dBm
Matlab 5G Toolbox	10 (Reflections) + 1 (Diffraction)
Sionna RT	10 (Reflections and Diffraction)

, the setup for each tool is shown. As can be seen, Exact is characterized by a threshold-based stopping criterion approach. In this scenario, the threshold was set to −135 dBm according to the sensitivity of the employed measurement equipment. It should be expected that this stopping criterion is comparable to the number of reflections set in the other tools, since it mainly affects rays that are reflected multiple times, whose contribution to the total simulated power becomes progressively smaller. In contrast, the main rays characterized by a higher power are accounted for in both cases.

Additionally, while in Matlab and Sionna, simulated power levels have been computed directly (and only) in the measurement points; in the Exact tool, they have been computed on a regular grid with a pixel spacing of 1 m × 1 m, providing a full electric field map in the selected scenario, and power levels in measurement coordinates have been extracted by a linear interpolation. In terms of ray generation, Matlab and Sionna are based on different parameterizations to achieve a uniformly distributed ray launching over the 3D sphere. In Matlab, the ray density was determined by setting the angular resolution parameter to 0.2695°. Otherwise, in Sionna, the initial density was governed by the total number of rays generated via a Fibonacci lattice, which was set to 150,000 rays. As regards building materials, they are simulated as concrete ($\epsilon =5.24-0.2i$). For all the tools, the terrain was modeled as flat. Simulations have been carried out by different independent laboratories. Due to the portability and computational efficiency of the tool, Exact simulations have been performed on a medium-level laptop (i7 processor with 8 GB RAM), with a computational time of the order of an hour. Otherwise, Matlab and Sionna simulations were performed on a server equipped with dual AMD EPYC 73F3 16-core processors, yielding a total of 32 physical cores, and 128 GB of DDR4 RAM operating at 3200 MT/s. Furthermore, Sionna’s computational workloads were accelerated using a dedicated NVIDIA L4 GPU with 24 GB of VRAM. With this hardware, computational times were in the order of several hours for Matlab and a few minutes for Sionna.

5. Results

In this section, experimental and numerical results are presented. A comparative analysis of RT tools’ performance is carried out by means of different statistical metrics: percentiles, Root Mean Square Error (RMSE), Mean Absolute Error (MAE), mean and standard deviation, similar to what has been proposed in our previous works [17,42]. Additionally, the tools’ efficiency in terms of radio coverage planning is considered.

5.1. Experimental Results

As discussed in Section 3, measurements have been performed in a walk-testing mode, following the route shown in Figure 4.

The selected scenario is served by one BS site covering a large area of around 1 km². The discrete measurement points along the route are denoted by blue-colored circles (see Figure 4). As can be observed, the measurement path is very extensive and covers several areas in obstructed line-of-sight (OLOS), but mostly in non-line-of-sight (NLOS) conditions with respect to the antenna, within a rather complex environment. Indeed, the points in OLOS are estimated to be approximately around 30%. The scenario is primarily characterized by buildings, but the intermittent presence of vehicular traffic (cars and trains) and the limited presence of vegetation introduce dynamic challenges for mm-Wave radio signal propagation. The total number of measurement points was 14,725. The mobile network scanner recorded the average power of the Resource Elements (REs) of the SSS within the SS/PBCH (Synchronization Signal Physical Broadcast CHannel) block (parameter SSS-RePower). Due to the lower walking speed compared to the sampling rate of the scanner, some consecutive samples shared identical geographical coordinates. To remove duplicates, as previously discussed, the SSS-RSRP values corresponding to the same location were averaged. Power levels in dBm recorded during the walk are shown in Figure 5 for the different cells served by the BS: PCI4, PCI5 and PCI6. For a better understanding of the results, a moving average (MA) filter with a window size of 20 samples has been applied to the raw data.

5.2. Comparative Analysis

5.2.1. Metrics Definition

To provide a clear analysis of the RT tools’ performance, several standard statistical metrics are considered to compare simulated received power $P_{sim}$ against measured received power $P_{meas}$ across measurement points. All metrics are computed considering the deviation $\Delta P$ for each measurement location i, defined as:

$$\Delta P_i=P_{sim}^i-P_{meas}^i \left[dB\right]$$

(1)

The percentile and their respective differences are calculated to assess the tools’ ability in reproducing the statistical tail of the coverage distribution. The mean (µ) and standard deviation (σ) characterize the first and second moments of the error distribution and quantify the average prediction offset and error dispersion, respectively. The RMSE and MAE provide complementary overall accuracy measures: RMSE penalizes large errors more heavily due to quadratic weighting, while MAE provides a linear error measure robust to outliers. Expressions are given below:

$$\mu =\overline{P}={\displaystyle \frac{1}{N}}{\displaystyle \sum }_{i=1}^N{P}_i \left[\mathrm{dB}\right]$$

(2)

$$\sigma =\sqrt{{\displaystyle \frac{1}{N}}{\displaystyle \sum }_{i=1}^N{\left(P_i-\overline{P}\right)}^2}\left[\mathrm{dB}\right]$$

(3)

$$MAE={\displaystyle \frac{1}{N}}{\displaystyle \sum }_{i=1}^N\left|\Delta P_i\right| \left[\mathrm{dB}\right]$$

(4)

$$RMSE=\sqrt{{\displaystyle \frac{1}{N}}{\displaystyle \sum }_{i=1}^N{\left(\Delta P_i\right)}^2} \left[\mathrm{dB}\right]$$

(5)

These metrics enable comprehensive and statistical evaluation of RT predictions for 5G network planning applications, from statistical distribution matching (percentiles) to point-wise error characterization (RMSE/MAE). Moreover, a visual comparison of received power is provided along the measurement route in the selected scenario, allowing for a better interpretation of results. The Pearson correlation coefficient (CORR) is also considered to quantify the linear agreement between simulated and measured received power values across the measurement points. This metric evaluates the capability of the RT tools to reproduce the spatial trend of power variations. It is defined as follows:

$$CORR={\displaystyle \frac{{{\displaystyle \sum }}_{i=1}^N\left(P_{sim}^i-{\overline{P}}_{sim}\right)\left(P_{meas}^i-{\overline{P}}_{meas}\right)}{\sqrt{{{\displaystyle \sum }}_{i=1}^N{\left(P_{sim}^i-{\overline{P}}_{sim}\right)}^2}\sqrt{{{\displaystyle \sum }}_{i=1}^N{\left(P_{meas}^i-{\overline{P}}_{meas}\right)}^2}}}$$

(6)

where ${\overline{P}}_{sim}$ and ${\overline{P}}_{meas}$ are the average simulated and measured power values, respectively.

Finally, an additional metric, defined as coverage rate (CR), is introduced to quantify the accuracy of RT tools’ results in the practical network design. Indeed, this last metric quantifies the ability of the different tools to reliably predict areas that respect specific requirements for the access to network services. In particular, this parameter is computed as follows:

$$CR={\displaystyle \frac{{{\displaystyle \sum }}_{i=1}^N{1}_{\{P\left(i\right)>threshold\}}}{N}} \times 100$$

(7)

where N is the number of points considered in the area, P is the power level measured or simulated, and threshold is the power level necessary for the access to different network services [43,44].

5.2.2. Assessment of RT Models

As previously described, the results are presented comparing received power values obtained with the three ray tracing tools (Exact, Matlab 5G Toolbox, and Sionna RT) against experimental measurements, collected across the entire BS coverage area, and applying the different metrics described. First of all, percentiles are computed from the 50th to the 95th, since these ranges are the most relevant for network design. Results are shown in Figure 6. As can be seen, an overall agreement is found between computational and experimental results. This outcome is confirmed by looking at

Table 5. Differences [dB] between simulated and measured percentiles.

Percentile	Exact	Matlab	Sionna
50p	3.0	1.6	3.1
60p	2.9	1.0	1.4
70p	1.3	−0.7	0
80p	−1.1	−3.3	−2.5
90p	−0.33	−4.0	8.0
95p	1.9	−4.4	8.4

, in which differences between percentiles are quantified, remaining mostly limited to a few dB, often comparable to the overall measurement and modeling uncertainties. For the sake of completeness, absolute error $\Delta P_i$ at 50p is considered, yielding values of 9.3 dB for Exact, 7.6 dB for Matlab, and 9.9 dB for Sionna.

Moreover, Figure 7, which reports data means and standard deviations, indicates that all tools exhibit limited mean and moderate standard deviation values, suggesting a relatively uniform error distribution without dominant outliers. Indeed, the experimental and simulated average power values are practically the same. Figure 8a,b report MAE and RMSE between simulated and experimental power levels of around 10–12 dB. Furthermore, the results highlight stable predictions since both RMSE and MAE are comparable among the different tools. The Pearson correlation coefficients, equal to 0.53, 0.46, and 0.50 for Exact, Matlab 5G Toolbox, and Sionna RT, respectively, indicate a moderate linear association between simulated and measured received power values. This suggests that all tools capture the general spatial trend of the received power along the measurement route, while still showing non-negligible local deviations. Exact provides the highest correlation, with Sionna yielding a very similar result, whereas Matlab shows a slightly lower coefficient. Overall, the Pearson metric confirms a comparable trend-reproduction capability among the considered RT tools. Power maps along the measurement routes in Figure 9 show good alignment between simulated and measured points across numerous path segments, particularly OLOS zones relative to the BS. Larger deviations appear in NLOS segments and near the simulated area boundaries.

Additionally, CR is estimated for experimental and numerical data and reported in

Table 6. CR for the access to network services.

Threshold	Measured	Exact	Matlab	Sionna
−115 dBm	79.9%	73.0%	81.8%	71.4%
−110 dBm	70.1%	66.7%	75.5%	65.3%
−105 dBm	58.4%	61.4%	65.6%	61.5%
−100 dBm	44.7%	53.5%	49.5%	54.2%
−95 dBm	31.3%	38.6%	29.9%	33.1%

. Threshold values are selected, as they enable various Modulation and Coding Scheme (MCS) potentially supporting different service levels in FR2 deployments, as evidenced in [45]. Specifically, −95 dBm is related to the MCS27-256QAM (Quadrature Amplitude Modulation), −100 dBm/−105 dBm to the MCS19-64QAM, −110 dBm to the MCS10-16QAM and −115 dBm to the MCS4-QPSK (Quadrature Phase Shift Keying). As can be seen, the values confirm an acceptable agreement between the results, and this aligns with 5G NR QoS (Quality of Service) requirements for service access. CR sensitivity to ±1 dB threshold changes was also analyzed, since this represents a relevant aspect for practical planning needs. The results (see

Table 7. CR sensitivity to ±1 dB threshold changes.

Measured	Exact	Matlab	Sionna
2.5%	2.0%	2.8%	2.3%

) indicate that the CR parameter is relatively stable, as evidenced by the narrow confidence intervals of the mean CR at each ±1 dB threshold.

Distance-binned statistics (RMSE and MAE) were first analyzed for points located at distances <100 m, 100–200 m, and >200 m from the antenna position, in order to assess whether the prediction error varies with distance. As reported in

Table 8. Distance-binned statistics.

Distance (d)	Simulator	MAE [dB]	RMSE [dB]
d < 100 m	Exact	13.97	15.86
	Matlab	14.34	16.01
	Sionna	13.08	18.54
100 m ≤ d ≤ 200 m	Exact	8.67	10.88
	Matlab	9.22	11.35
	Sionna	9.39	12.32
d > 200 m	Exact	11.08	13.78
	Matlab	8.34	10.83
	Sionna	10.86	13.30

, the agreement between simulations and measurements does not change monotonically with distance; rather, all the considered tools achieve their lowest MAE and RMSE values in the intermediate range between 100 m and 200 m, whereas larger discrepancies are observed for the closest points, and also for the farthest ones. This result suggests that distance alone is not sufficient to explain the prediction error, which is also influenced by local propagation conditions and by the specific characteristics of each simulator. For this reason, a separate analysis was carried out by distinguishing OLOS and NLOS points, as shown in Figure 10. In this case, both MAE and RMSE are significantly lower under OLOS conditions, with differences of about 4 dB compared with NLOS points. Consistently, the percentile differences reported in

Table 9. Differences [dB] between simulated and measured percentiles in OLOS region.

Percentile	Exact	Matlab	Sionna
50p	4.1	5.8	4.7
60p	3.7	5.3	5.0
70p	1.9	4.9	1.8
80p	0.8	4.7	7.1
90p	1.5	4.7	8.2
95p	2.3	4.8	9.2

and

Table 10. Differences [dB] between simulated and measured percentiles in NLOS region.

Percentile	Exact	Matlab	Sionna
50p	0.2	4.3	1.0
60p	6.8	4.3	4.7
70p	7.9	4.6	5.9
80p	8.2	5.3	4.9
90p	8.6	6.1	8.2
95p	9.7	6.9	14.4

confirm a better agreement in OLOS regions, while larger deviations are generally observed in NLOS, especially at higher percentiles.

To conclude, simulated and measured results are compared in the different PCIs, with results reported in Figure 11, in terms of percentiles, and in Figure 12, in terms of mean and standard deviation. As can be seen, the results in PCI6, which is the only one characterized by a tilt angle of 3°, exhibit larger deviations with respect to the measured value, especially for Sionna. This behavior suggests a higher sensitivity of the predictions to the specific propagation conditions of that sector.

6. Discussion

The results presented in Section 5 demonstrate reasonable agreement between numerical simulations obtained with the three different ray tracing tools and measurement data collected across the entire BS coverage area, despite the complexity of the considered urban scenario. Indeed, as previously mentioned, it should be considered that the selected scenario is very challenging for radio propagation assessment due to dimensions, topology and frequencies involved. Nevertheless, all metrics highlight that, although some discrepancies exist, RT tools represent a flexible and useful method for 5G network design. Indeed, percentile analysis in Figure 6 and Table 5 confirms that differences between simulated and measured values remain predominantly within a few dB up to the 95th percentile, compatible with the estimated combined setup measurement uncertainty (below 1.5–2.4 dB up to 3.7 dB). Additionally, maximum discrepancies between simulated and measured results of around 8 dB are reasonable, compared to measurement uncertainties quantified as 7–10 dB in NLOS for scenarios with the same level of complexity [45]. This indicates that all three tools adequately capture the received power statistical distribution, particularly in the mid-to-high power regions relevant for 5G FR2 coverage KPIs (Key Performance Indicators) and QoS. Aggregate error metrics (mean, standard deviation, RMSE, and MAE shown in respective figures) align with percentile outcomes: mean values and standard deviations are acceptable and comparable among the different tools. Indeed, RMSE values of around 12 dB are in line with results reported in the literature for simpler, more controllable indoor scenarios in the FR1 band, where signal propagation is less critical [43], and also for similar frequencies in the FR2 band in a walk-testing mode, specifically at 26 GHz, without calibration or empirical corrections [46]. In addition to this, the mean and standard deviation show symmetric error distributions with relatively contained tails, together with highly accurate predictions of average power levels. This metric also highlights that the three tools, compared to the measures, provide stable and consistent results, with similar deviations. Sionna exhibits a slight systematic tendency to overestimate power levels at higher percentiles compared to Exact and Matlab, visible both in the differences in Table 5 and in the coverage estimates in Table 6, where all tools slightly overestimate the serviceable area, but with deviations acceptable for practical planning. In addition to the aggregate metrics discussed above, the Pearson correlation coefficients indicate that all three tools capture the overall spatial trend of the received power with moderate and comparable agreement. The additional analyses reported in Table 8, Table 9 and Table 10 and Figure 10, Figure 11 and Figure 12 further show that the residual mismatch is strongly scenario-dependent, since it does not vary monotonically with distance, and is consistently lower in OLOS than in NLOS conditions. Therefore, these complementary metrics suggest that local propagation features and sector-specific characteristics affect the agreement more than distance alone.

Overall, it should be noted that the measurement campaign was carried out during a sunny day, minimizing the effect of rain and atmospheric attenuation that could be very impactful at 27 GHz. Indeed, such critical weather conditions may introduce significant additional losses and noticeably underestimate actual measurement results, an aspect that should be considered in ray tracing tools. Moreover, the presence of dense vegetation is not accounted for in this scenario, since it is mainly composed of buildings. However, in heavily vegetated areas, the 5G signal would be susceptible to foliage attenuation, thus requiring accurate modeling of scattering effects within ray tracing simulators. In addition to this, the presence of random vehicles and obstacles is not considered in numerical simulations, since it is very challenging to accurately forecast the exact scenario configuration at the time of measurement. This aspect is also difficult to reproduce in determinist models; thus, to closely match real-world scenarios, it is most appropriate and meaningful to neglect such unpredictable elements. Clearly, this can lead to small discrepancies in experimental outcomes; therefore, this point should be taken into account when performing comparisons, and it should be considered that, for network design, simulation results may slightly overestimate the actual power levels due to the lack of stochastic effects.

Spatial power maps along measurement routes reveal that RT simulations, as expected, faithfully reproduce local power variations under OLOS conditions, where direct path contributions dominate, while larger deviations emerge in NLOS regions and near simulated scenario boundaries. Most stable results are obtained with Matlab, which is less prone to underestimation of power levels. On the other hand, the main sources of discrepancies for the Exact tool are related to low power levels for measurements carried out near the boundaries of the simulated area. Indeed, as can be seen from Figure 9, the main areas of discrepancy occur at the scenario boundaries. These discrepancies are related to the loss of reflection and diffraction contributions from buildings that are not considered due to scene clipping. To confirm this, we present the impact of buildings at the edge, slightly extending the scenario considered by Exact: in Figure 13, a comparison between results in the extended and reduced scenarios is reported together with measurement results. It is evident that by increasing the number of buildings at the scene borders, the simulated power levels are much higher and closer to both measurements and predicted values by other tools. Indeed, the mean error between measured and simulated results decreases from 1.9 dB (in the reduced scenario) to 0.75 dB in the extended scenario.

Moreover, the comparison between measurements and Sionna simulations reveals a tendency of Sionna to overestimate received power levels, particularly evident in the percentile differences (Table 5), coverage estimates (Table 6) and power maps along measurement routes (Figure 9). While Exact and Matlab exhibit limited bias across the full dynamic range (as confirmed by mean and standard deviation analyses in respective figures), Sionna shows larger positive deviations at higher percentiles (e.g., above 90th percentile) and consistently higher predicted power levels in several route segments (e.g., city square with trees), with differences reaching several dB. These discrepancies are consistent with recent validation studies on Sionna-based ray tracing, which indicate that simulation-to-measurement mismatches are often driven more by antenna configuration, scene accuracy, foliage/blockage representation, and uncertain system parameters than by solver settings alone. In particular, Manukyan et al. [25] show that the agreement with measurements is weakly sensitive to several solver hyperparameters, while being much more affected by antenna placement and orientation. They also reuse a spatial subset of a public Rome low-to-mid-band dataset from a broader campaign, including outdoor walking measurements, whereas the present analysis compares Exact, Matlab 5G Toolbox, and Sionna RT under a common validation framework supported by urban walk-testing at 27 GHz in an FR2 scenario. Likewise, Lee et al. [24] report that Sionna can reproduce the overall behavior of measured coverage in a foliage-dense rural scenario, although local deviations remain due to blockage, antenna modeling, and the dynamic environment. A similar interpretation is also consistent with [23], where measurement-assisted calibration is introduced even for the Sionna-based baseline to compensate for uncertain system parameters, and where terrain, foliage, and human/vehicle blockage are explicitly recognized as relevant factors for path-gain prediction. Therefore, although our data do not allow us to attribute Sionna’s positive bias at high percentiles to a single mechanism, the observed mismatch is plausibly related to residual limitations in scenario representation and site-specific modeling rather than to one demonstrable solver-related factor.

Nevertheless, in our 27 GHz comparison results, the RMSE and MAE values for Sionna remain acceptable for practical planning but are consistently higher than those of Exact and Matlab, suggesting that site-specific calibration may be required for accurate mm-Wave network planning in mixed street-canyon/open-area urban deployments. Sionna’s contained systematic overestimation suggests the need for site-specific calibration when employing this GPU-accelerated platform in digital network planning or digital-twin workflows, particularly for sensitive coverage thresholds such as −95 dBm. From a practical usability perspective, Matlab 5G Toolbox provides a relatively direct workflow for engineering analyses, since ray tracing scenarios can be handled within the Matlab environment through Site Viewer and map-based urban scene support. By contrast, Sionna RT is more oriented toward flexible research workflows and typically requires a more articulated scene-preparation process, where the three-dimensional environment is generated or edited externally, for example, through OpenStreetMap-based data and Blender, before being used in the simulation pipeline. As a consequence, the learning curve of Matlab is generally lower for users already familiar with engineering simulation environments, whereas Sionna usually requires greater familiarity with Python-based workflows, 3D scene preparation, and GPU-oriented simulation settings. In this sense, Matlab appears more immediate for conventional engineering studies, while Sionna offers higher flexibility for customized research developments and advanced modeling extensions.

Overall, an important aspect in interpreting the observed mismatch is that the final error budget includes both measurement uncertainty and modeling uncertainty. In the present campaign, the scanner uncertainty is reported to be below 1.5 dB, while studies based on similar measurement setups indicate an overall experimental uncertainty of about 2.4–3.7 dB, so a representative value of approximately 3 dB can be reasonably assumed for the measurement contribution. Since the overall RMSE observed in this work is on the order of 10–12 dB, the residual discrepancy is mainly attributable to modeling uncertainty, including scene simplifications, imperfect material and antenna representation, and the static treatment of a dynamically varying urban environment. More in detail, the error related to the scenario modeling can be considered by looking at the analysis carried out in Exact, which showed an improvement of 1.2 dB by extending the number of buildings. Additionally, preliminary analysis in simplified scenarios with comparable frequencies showed that by varying the real part of permittivity from 2 to 7, discrepancies of around 2–3 dB can be found in simulated power levels.

Finally, the consistency of coverage rates among measurements and the three simulation tools represents a key factor for reliable network planning, since strong deviations in this parameter may lead to inaccurate predictions of service availability in 5G systems. This is particularly crucial at mm-Waves, where the signal attenuation is very high and may negatively impact the QoS. Overall, results confirm the practical utility of RT techniques for FR2 coverage prediction, despite complex scenario challenges such as obstacles and NLOS regions.

7. Conclusions

This work presented a comprehensive evaluation of RT tools in 5G FR2 scenarios, based on an extensive measurement campaign at 27 GHz performed in a complex urban area. Measurements were performed in terms of received power levels, recording the SSS-RSRP indicator emitted by the mMIMO BS. The performance of different RT-based tools was assessed by comparing simulation results from Sionna RT (open-source), Matlab 5G Toolbox (commercial), and Exact (in-house proprietary tool). All the analyzed tools achieve an overall good agreement between measured and simulated data. Indeed, our findings showed that simulation tools provide reliable and robust predictions, which are sufficiently accurate for most practical applications, such as coverage prediction and preliminary network design. The discrepancies are acceptable for 5G network design, considering the critical environment under investigation, particularly at mm-Waves. Furthermore, the analysis also revealed that precise characterization of the scenario, especially for buildings and obstacles at the simulation boundary, is relevant for achieving accurate RT-based power level results. In summary, our work confirms that RT can be a powerful and versatile method for evaluating 5G network performance, regardless of the specific algorithm employed. This means that these tools can effectively support 5G deployment strategies, with results not affected by the uncertainties that may arise from measurement campaigns, due to the variability of radio environments, especially at mm-Waves. The observed agreement between simulated and measured data can also help to reduce the number of measures necessary to ensure good coverage of the area and thereby minimize the associated operational costs and time requirements. Additionally, according to our findings, we can conclude that:

• The Exact tool is generally more suitable when EMF levels must be calculated across a large number of discrete points (e.g., on a regular grid of points covering the whole scene);
• The Matlab tool is suggested for a low number of evaluation points, to obtain more accurate punctual results;
• Sionna is particularly attractive when an open-source, GPU-accelerated RT engine is desired, for example, to integrate site-specific propagation modeling with data-driven methods and digital-twin workflows.

These findings support the integration of RT tools into 5G network design and their deployment in complex urban environments, while also suggesting their potential use in 6G-oriented digital-twin workflows for site-specific network planning and performance evaluation. Future work may explore AI-assisted calibration and tuning of RT model parameters to further improve the agreement between simulated and measured power levels.