Development and Application of a Vertical-Agnostic Methodological Assessment Framework for Evaluation of 5G-Based Use Cases

Abstract

This paper addresses the industrial adoption gap of 5G/6G by presenting a novel, vertical-agnostic Methodological Assessment Framework (MAF). The MAF bridges the Network Key Performance Indicators (KPI) of 5G networks with user-centric User-KPIs and User-KVIs (Key Value Indicators) to quantify the techno-economic and societal value propositions of industrial 5G use cases from an end-user perspective. First, a detailed description of the MAF and its underlying principles is given, explaining how a use case’s value proposition can be captured. Second, the MAF is applied to three different industrial use cases from the verticals manufacturing, construction, and automotive utilizing the individual User-KPI and User-KVI for the quantification of the individual value propositions. The results show that the use of 5G can lead to enhanced process capability and reproducibility as well as increased insights into different processes. In addition, it is shown that the MAF objectively quantifies user value across diverse verticals and is able to strengthen interdisciplinary alignment over different verticals.

1. Introduction and Motivation

The fifth generation of mobile communication (5G) as well as its upcoming technological successor (6G) both offer tremendous innovation potential in a variety of application scenarios and settings [1,2,3]. For instance, Mertes et al. achieved a reduction in the down times of machine tools and an overall increase in operational safety through the integration of 5G-enabled sensors into machining processes [4]. Lundgren et al. came to the conclusion that the use of 5G technology in manufacturing can positively influence key characteristics like productivity, maintenance performance, and flexibility, as well as quality and aspects related to ecological sustainability [5]. However, clarity on the achievable benefits of the industrial application of the technology is still missing. Potential end users of the technology in different industries state that they lack transparency about what specifically can be gained in their respective domains by adopting the technology so that they can assess whether the necessary investments are worthwhile [6,7]. Therefore, it can be stated that, even though companies are generally interested in the adoption of 5G/6G, one of the main barriers for technology adoption is still the lack of clarity regarding the potential value proposition that can be realized with the technology to justify the required investments for the end user.

Another aspect that must be considered is that the industrial use of 5G requires experts from different technical domains to collaborate to achieve a shared goal. If, for instance, 5G is supposed to be deployed in a production line, network experts with domain knowledge in networking and communications must collaborate with manufacturing experts that usually do not have detailed knowledge of networking and communication. One central challenge in interdisciplinary technology development and application is the lack of common understanding among experts from different domains, which can make communication difficult [8]. A striking example of this circumstance is the term “KPI (Key Performance Indicator).” The term is used in almost every professional environment to measure the performance of, for example, processes, procedures, or products and services [9]. However, experts from the network and communications industry and experts from the manufacturing industry consider completely different aspects when defining and using the term KPI in their respective domains. To collaborate, they must establish a common ground of understanding and discussion to be able to speak the same technical language.

To address the described aspects, a Methodological Assessment Framework (MAF) was developed by the authors of this paper with the aim to provide a domain/vertical-agnostic approach that captures the value proposition of industrial 5G use cases from different domains/verticals. The approach aims at building the bridge between the characteristics of a 5G network, the use case utilizing this network, and the value proposition that can be achieved for the end user of the considered use case. Therefore, the working hypothesis of this paper is that an end-user-focused MAF can be developed to enable the quantification of a variety of benefits of 5G-based use cases. This working hypothesis shall be verified through the exemplary application of the MAF to three use cases from different verticals. The contributions of the MAF can be summarized as follows: the MAF focuses on user benefits and presents a stepwise approach to quantify the value propositions of 5G-based use cases from a techno-economic as well as a societal perspective, utilizing User-KPIs (User Key Performance Indicators) and User-KVIs (User Key Value Indicators). The MAF helps to identify how the use of 5G can bring real added value, for example, by making processes more transparent or efficient through the increased availability of real-time data. The MAF also bridges the way from Network KPIs (e.g., end-to-end latency) to user benefits like reduced cycle times of manufacturing processes. Furthermore, the MAF is designed in a vertical-agnostic way and can be applied in a variety of different application scenarios and settings.

The paper at hand is structured as follows. Section 2 describes the MAF and its underlying principles in detail while also presenting the defined User-KPI and User-KVI that enable the capturing of a use case’s value proposition. Section 3 illustrates the capturing of use cases’ value propositions. For this purpose, the MAF is applied to three different use cases originating from three different verticals that employ 5G. The first use case “Inline Quality Assurance for Machining” originates from the manufacturing vertical and utilizes 5G to establish real-time monitoring of critical quality parameters in machining processes. The second use case is titled “5G for Energy Analytics” and was developed in the construction vertical. This use case aims at the data-based quantification of a construction operation’s environmental footprint and uses 5G to wirelessly transmit metering data from a construction machine to an edge cloud for analysis and monitoring purposes. The third use case is called “Predictive Quality of Service for Tele-operated Vehicles”. This use case aims at showing how bidirectional interaction between automotive applications and 5G networks can enhance the efficiency of tele-operated driving. It employs concepts of 5G functions, such as the Network Exposure Function (NEF) and Network Data Analytics Function (NWDAF), to enable such interaction. As the focus of the use cases is on the evaluation, no in-depth description of use case parameterization is provided in the paper at hand. These descriptions can be found in [10,11,12]. Evaluation results based on the MAF are presented for each use case. In Section 4, a summary of the use case evaluations and an outlook regarding future research needs are given.

2. Methodological Assessment Framework

One of the biggest barriers to the widespread use of 5G in industrial application scenarios is the clear quantification with user-centric metrics of the added value (the value proposition) for users employing 5G. Although a variety of approaches have already addressed this issue, the lack of clarity regarding the benefits remains, which is also indicated by the low adoption rate of 5G technology in the industry [13,14]. Previous approaches to quantifying the added value of industrial use cases have often focused on the deployment, operating costs or technical aspects of 5G or 6G networks [15,16,17], or have examined very specific use cases in detail [18,19]. While such studies also provide valuable insights for the adoption of the technology, they do not focus on the value proposition that results for the user of a use case. Furthermore, although a detailed analysis of a specific use case provides insights into that particular use case, it does not provide insights into other use cases. As a result, the transferability (across different use cases as well as across different verticals) of the evaluation results is often limited, which means that other use cases cannot be examined with minimal effort across different verticals. Therefore, a novel approach was developed containing a stepwise methodology to capture the value proposition of a variety of different 5G-based industrial use cases in a vertical-agnostic way aiming at building a bridge between the network domain (5G/6G) and the use domain. The MAF has been developed in the EU-funded research project TARGET-X by the authors of the paper at hand [20] and it was also validated in the 5G Alliance for Connected Industries and Automation (5G-ACIA) to incorporate the feedback from potential industrial users, the results of which have been published in a Whitepaper [21]. The development process of the MAF was accompanied by the technical development of the use cases to be evaluated with it, while also ensuring that MAF could be made so universally applicable that it could be easily applied to other use cases. The value proposition provided by the successful application of an industrial 5G use case to the user is at the center of the MAF. For this reason, the perspective of the potential user of a use case is the main focus of an evaluation conducted by employing the MAF. For a manufacturing use case, this perspective is as follows: if a production manager is contemplating the integration of 5G into their operations, the production manager’s focus is very likely on the impact the use of 5G might have on key metrics that describe the performance of the production system. While the production manager should naturally be an expert in their domain, it is unlikely that they will also have expert knowledge in the area of communication networks, so they will most likely not be able to discuss network-specific metrics like latency and data rates with network experts. This circumstance can complicate communication, coordination, and alignment between network experts and domain experts (such as production) and contribute to further delays in adopting the technology. As a result, the following principle was defined to guide the development of the MAF: “Take the perspective of the end user of the use case and describe what advantages/benefits the implementation and execution of the use case promises compared to the current state of the art”. Based on this principle, the MAF was developed. Figure 1 illustrates the MAF in detail.

The use case which is to be evaluated is located in the center of the MAF. If successfully implemented and applied, a product is created as the outcome of the application of the use case. This product does not necessarily have to be a physical object; it can also be the successful transportation of a passenger with a tele-operated vehicle with the tele-operation being enabled by 5G communication. On the left side of the MAF, the use case requirements and evaluation scenario are located. The use case requirements define the technical requirements that must be fulfilled by the network so that the use case can be applied successfully. The way in which the use case is executed is defined by the evaluation scenario, which describes the sequential execution of the steps required to ensure that the result of the use case is the desired product. This product is then evaluated using pre-defined User-KPIs and User-KVIs. User-KPIs are designed to determine to which degree techno-economic goals, like an increase in the throughput of a production system, are achieved, while User-KVIs are designed to determine to which degree societal goals, like a decrease in a use case’s environmental footprint, are achieved [22,23]. Both the User-KPIs and the User-KVIs have been developed together with experts in industrial 5G use cases to capture the value proposition of a use case from the user’s perspective. For each User-KPI and User-KVI, an equation has been defined to enable data-driven quantification of the degree to which techno-economic and societal goals have been achieved. If read from the left-hand side to the right-hand one, it becomes obvious how the MAF can build the bridge between Network KPIs and User-KPIs/-KVIs. Network KPIs describe the capabilities of the employed 5G network, thereby enabling the crucial assessment of whether the network can fulfill the defined use case requirements. Following the evaluation scenario, the use case is executed, producing the desired product which is then evaluated using the User-KPI and User-KVI to determine whether added value for the end user (the value proposition) is achieved. In this way, a clear and precisely defined connection between network parameters, like latency, uplink, etc., and a value proposition for the end user of the use case, like the increased capability of their processes, is created. The creation of this connection is one of the main contributions of the MAF to the objective of supporting the broad rollout of 5G technology in industrial settings. To enable the broad applicability of the MAF, the following algorithmic description of the procedure to apply the MAF is given:

(1)

Start with the value proposition from the user’s perspective: determine the value proposition that is to be achieved through use case implementation (e.g., lower cycle time of a manufacturing process) as well as the product as the outcome of the successful use case application.

(2)

Match the desired value proposition that is to be achieved with the User-KPI and/or User-KVI from

Table 1. Selection of User-KPIs for industrial 5G use cases.

Techno-Economic Goals	User-KPI
Expanding Process Insights	Accuracy of Process and Product Data
	Completeness of Process and Product Data
	Consistency of Process and Product Data
	Reliability of Process and Product Data
	Timeliness of Process and Product Data
	Uniqueness of Process and Product Data
	Validity of Process and Product Data
Increasing Operational Capability	Process Performance Index (Ppk)
	Process Capability (cp & cpk)
	Process Variability
Increasing Process Efficiency	Cycle Time
	Throughput
	First-pass Yield
	Overall Equipment Efficiency (OEE)
	Error Rate
	Quality Rate
	Worker Efficiency
Increased Profitability	Net Present Value (NPV)
	Return on Investment (RoI)

and

Table 2. Selection of User-KVIs for industrial 5G use cases.

Societal Goals	User-KVI
Improvement of Safety-Related Aspects	Work Accident Rate Manufacturing
	Work Accident Rate Construction
	Absolute Number of Prevented Traffic Accidents
Transparency About Ecological Impacts	Global Warming Potential
	Water Consumption
	Ozone Depletion
	Photochemical Ozone Formation
	Depletion of Abiotic Resources (Minerals and Metals)
	Depletion of Abiotic Resources (Fossil Fuels)
Digital Inclusion	Digital Literacy

by answering the question of how the value proposition can be quantified with the User-KPI and User-KVI.

(3)

Determine how the selected User-KPI or User-KVI can be calculated (e.g., through customized equations based on process data) or estimated based on expert estimates.

(4)

Define the use case requirements that need to be fulfilled by the employed 5G network so that the use case can be carried out successfully and the User-KPI and User-KVI, which have been selected in Step 3, can be calculated.

(5)

Define the evaluation scenario as the step-by-step instructions for the execution of the use case under consideration, including information on what data is to be acquired through measurements.

(6)

Evaluate the use case by calculating/estimating User-KPI sand/or User-KVIs and quantify its value proposition.

The procedure makes a differentiation between the calculation and estimation of the User-KPI and User-KVI. While, in many cases, it is possible to measure specific parameters that can be used to calculate the KPI, this is not the case for some applications. In this case, targeted questions regarding the expected impact of faster and more reliable communication enabled by 5G can be asked, for example, in the form of expert interviews to make estimates. This procedure to utilize expert estimates is described in more detail in [21]. For the MAF to be applied, expert knowledge on the domain in which the use case that is to be assessed must be combined with expert knowledge on 5G networks. The domain expert (e.g., a production manager) must be able to define which objectives are to be achieved through the implementation of the use case, while the network expert must ensure that the 5G network allows for the technical implementation of the use case. The description of the MAF and how it is applied indicate that its novelty mostly lies in the development of a structured procedure integrating existing and reliable evaluation metrics and bringing experts from the application domain and the network domain together, while no radically new evaluation method was developed.

In order to enable the domain/vertical-agnostic evaluation of industrial 5G use cases, lists of User-KPIs and User-KVIs have been developed. Table 1 and Table 2 provide an overview of both the defined User-KPI and User-KVI of TARGET-X.

The User-KPIs have been defined to determine the degree to which four overarching techno-economic goals are achieved by a use case. The first of these goals, expanding process insights, was defined based on the assumption that highly performant and reliable wireless networking with 5G will significantly increase the data-driven insight in an industrial setting. Since 5G enables the fast transmission of valuable process and product data with a high quality and quantity, which can be captured directly in a running industrial process, data-driven insights can be gained to learn more about ongoing processes and their performance. The second goal, increasing operational capability, addresses the possibility that the use of 5G will lead to more stable processes that are highly likely to produce the same output parameters with the same input parameters. This fact is addressed by the term process capability, which characterizes a process in terms of its ability to meet (customer) specification limits so that the process outcome remains within those limits [24]. In the context of use case evaluation, this can, for instance, be the dimensions of a manufactured component or the travel time for a route covered by a tele-operated vehicle. In this context, the two capability indices c_p and c_pk are of high importance to calculate the capability of a considered process [24,25]. The index c_p describes the spread or distribution of the process outcome in relation to the tolerance range. The index c_pk describes the centering of the mean of the measured values. The process for which c_p and c_pk are calculated is generally considered capable if both indices are greater than 1; in industrial manufacturing, for instance, the required value for both indices is usually greater than 1.33 [25,26]. The analysis of process capability stems from manufacturing and is an essential part of quality management. The principle can also be applied outside of the manufacturing context and transferred to other verticals, since the main idea to achieve reproducible process outcomes can be applied universally [27,28]. Therefore, process capability is also applied to different verticals here. Increasing process efficiency, which was defined as the third goal, is closely associated with the goal of increasing the operational capability as well as the goal of expanding process insights. Based on improved insights and increased process capability, overall improvements in the performance of the considered systems are expected, as the systems and their underlying relations can be understood in more detail based on the increased amounts of available process data, while the controllability of the processes is increased as well. Following the hypothesis that the improvements enabled by the use of 5G lead to increased efficiencies and less waste in the use cases, improvements in profitability are to be expected, since the desired results (e.g., to produce a certain product or execute a certain process) can be achieved with less effort and resources in a leaner way. This fact is expressed by the fourth and final goal, increased profitability.

The User-KVIs from Table 2 are designed to address certain elements of the 17 Sustainable Development Goals (SDGs) defined by the United Nations [29]. The first societal goal, improvement of safety-related aspects, was defined based on the hypothesis that fast and reliable 5G communication enables the real-time supervision of safety-critical aspects, like the work and safety zones of robots that operate on construction sites while human workers are also present. The second societal goal, transparency about ecological impacts, was defined because 5G data transmission offers the potential to equip processes (e.g., manufacturing or construction processes) with additional measurement devices to measure the consumption of electrical energy. In this way, the ecological footprint of operations can be calculated, allowing points of reference for optimizing this footprint to be determined at the same time. The third and final societal goal, digital inclusion, was defined due the fact that 5G enables the use of innovative digital technologies in application areas that have so far been excluded in certain application scenarios. For instance, the use of extended reality (XR) applications on construction sites has not been possible in many cases because wired communication was not feasible on dynamically changing construction sites, and the lack of wireless communication options meant that data could not be exchanged over medium and long distances. 5G addresses this issue by enaling the digital inclusion of workers that have so far been left out.

The listed User-KPIs and User-KVIs illustrate how the principle of focusing on the value proposition from a user’s perspective was prioritized in the design of the MAF. The focus on the individual value proposition and the broad applicability of the User-KPI and User-KVI of a use case ensure that the described approach is vertical-agnostic. It should be noted that, in order to apply the MAF to a use case, process experts must precisely determine the individual value proposition for this individual use case. This follows the premise that a use case to be implemented must have a value proposition for the user that must be quantifiable. This is to ensure that potential users of the technology understand the technical potential they can realize by adopting it. In this way, the developed framework enables a multi-perspective evaluation approach of industrial 5G-based use cases and thus contributes to the goal of increasing the industrial use of 5G. In the following section, the MAF is exemplarily applied to three different 5G-based industrial use cases from different verticals to demonstrate its functionality in different domains. For this purpose, a selection of the User-KPIs and User-KVIs from Table 1 and Table 2 is employed.

3. Application of the Methodological Assessment Framework

3.1. Application of the MAF to a Manufacturing Use Case

3.1.1. Description of the Use Case

One central benefit of using 5G communication in manufacturing is the ability to seamlessly integrate quality control and assurance functionalities into production processes [30,31]. The use case considered in this section was developed to utilize 5G-TSN (Time-Sensitive Networking) in order to enable an inline quality assurance system for machining based on deterministic real-time communication. The use case was developed in the research project TARGET-X and is described in more technical detail in [20,32]. Test results of the use case have already been published in [10]. Using Frame Replication and Elimination for Reliability (FRER) [33] within the communication pipeline, bounded latency and jitter of the communication with a guaranteed latency of 10 ms for 99.99% of transmitted packets was achieved. The system architecture diagram of the use case is shown in Figure 2.

For the implementation of the use case, a 5G-enabled smart sensor is integrated into a milling machine measuring acoustic emission signals during machining. The measured data is wirelessly transmitted to an edge cloud system, where it is analyzed and used to implement an application control logic system to guarantee the desired quality of the machined product. The feedback from this system is then fed back into the machine control, creating a seamlessly integrated quality control. For this process control to function, strict requirements regarding latency and reliability must be met. For the use case under consideration, it was determined that 99.99% of all transmitted data packets must be transmitted within 10 ms. The (simplified) FRER setup employed to guarantee this requirement is pictured in Figure 3. It uses a redundant communication setup in which the exact same packet is sent twice via two streams to guarantee that at least one packet arrives on time, which, in this case, is below 10 ms.

3.1.2. Description of the Selected User-KPI and Evaluation of the Use Case

Given the core requirement that the use case must deliver the required data packets via 5G in a timely manner for subsequent latency-critical applications, such as inline process control, the User-KPI timeliness and completeness as well as process capability have been selected from Table 1 to evaluate the use case. Process capability (c_p and c_pk) is calculated according to the following equations for the process capability indices [24]:

$${\mathrm{c}}_{\mathrm{p}}={\displaystyle \frac{\mathrm{U}\mathrm{T}+\mathrm{L}\mathrm{T}}{6\mathsf{\sigma }}}$$

(1)

$${\mathrm{c}}_{\mathrm{p}\mathrm{k}}={\displaystyle \frac{\min (\mathrm{U}\mathrm{T}-\overline{\mathrm{x}};\overline{\mathrm{x}}-\mathrm{L}\mathrm{T})}{3\mathsf{\sigma }}}$$

(2)

In both equations, UT and LT describe the upper and lower tolerance limit for the latency, while $\overline{\mathrm{x}}$ describes the mean and σ the standard deviation of the measured latency. To fulfill the set requirements of the use case, the latency must be within the tolerance specification. Since latency cannot be too low for this use case, only the upper tolerance limit is of relevance here so that a calculation of c_pk is sufficient and c_p does not have to calculated to prove capability [24]. Figure 4 shows the measured latency per packet for the considered use case. For the measurement, data packets are transmitted from the milling machine to the edge cloud in a redundant way, according to Figure 3, while the timestamps of each individual packet were logged. Based on these logs, latency calculations were conducted for each packet. As indicated by the line visualizing the mean, the average latency value was measured to be 6.512 ms, while the 99.99th percentile was transmitted with a latency of 8.778 ms. Therefore, the technical setup is able to fulfill the latency requirement. This is also proven by the calculation of c_pk. With an upper tolerance limit of 10 ms, c_pk was calculated to be 1.34, fulfilling the requirement of being greater than 1.33 (see Section 2). Thus, it could be shown that the FRER setup enables stable and controllable data transmission, fulfilling the communication requirements of the use case. Since 99.99% of data have been transmitted within 8.778 ms or less, the requirements regarding the timeliness and completeness of data have also been satisfied, as the required data is transmitted in a sufficient amount, respecting the latency limits.

The determination of the User-KPI completeness and timeliness as well as the proof of process capability shows that the technical goals to expand process insights (i.e., learn more about the internal relationships within the use case) and to increase operational capability can be achieved for this use case based on 5G wireless communication, as more valuable data can be acquired on time. FRER is utilized to fulfill latency requirements, which are justified by the aspect of controllability in the event of quality problems. Following the MAF depicted in Figure 1, the product of successful use case execution is the real-time transmission of the required data packets to enable the inline quality assurance of the machining process. Based on the calculated value for c_pk = 1.34, it can be concluded that the use case is able to provide a concrete value proposition to a user, as the critical variable of the use case (latency) is under control.

3.2. Application of the MAF to a Construction Use Case

3.2.1. Description of the Use Case

Another core benefit of 5G technology is the potential to establish wireless connectivity in areas that can otherwise not be connected, for instance, expansive and dynamic environments such as construction sites [34]. In these settings, 5G acts as an enabler for the utilization of digital technologies that can improve existing processes [35,36]. To demonstrate the potential of the MAF, an energy-related use case from a construction site is described in this section. The use case is titled “5G for Energy Analytics” and has been described in [20,37]. The use case aims to create energy awareness through the integration of a 5G-enabled metering device (Meter-X) for measuring the power consumption of electrically powered construction machinery like lifts, cranes, or deconstruction robots. The development and application of Meter-X is described in detail in [11]. Usually, the electrical consumption of construction machinery cannot be monitored centrally, as most of the machinery does not offer the functionality to extract and aggregate the data from the machine control. As a result, the environmental footprint of construction sites cannot be calculated precisely, which also contributes to the fact that approaches for reducing the environmental footprint have not yet been broadly implemented in the construction vertical. The 5G for Energy Analytics use case addresses this issue. For its implementation, the metering device is connected to the electricity consumer (material lift) so that the power consumption (voltage and current) can be measured. The metering device is connected to a server through 5G so that the acquired metering data can be logged and analyzed in real time. One key advantage of the Meter-X is the fact that any construction worker can connect it to the construction machinery whose energy consumption is to be measured. No certified electrician is required due to the fact that Meter-X does not require interference with the machine’s internal circuit, as it is a plug-and-play solution that can be connected to any electrical consumer. Based on the measurement of power consumption, the environmental footprint of the use case (User-KVI addressing the societal goal of creating transparency around ecological impacts in Table 2) can be calculated. Following the approach of the MAF, the product of the successful application of the use case is an energy profile for the characterization of energy consumption, which can also be employed to derive optimization approaches in subsequent steps. The system architecture diagram is shown in Figure 5. The measurement values of the voltage and current are sent to the edge cloud from the metering device, which is attached to the material lift. On the edge cloud, the received data is analyzed and visualized so that an energy profile and the calculation of the selected User-KVI are created.

3.2.2. Description of the Selected User-KVI and Evaluation of the Use Case

For this use case, the User-KVI addressing the societal goal of creating transparency around ecological impacts from Table 2 is selected. This was done due to the fact that there are currently no established solutions that enable the environmental footprint of construction operations to be calculated [11]. Therefore, the use of 5G in this setting offers a significant improvement exploiting the technology’s key characteristics to reliably transfer data wirelessly in dynamic environments. The calculated values of the User-KVI for a lifting operation carrying 12 Ytong blocks (20 kg each) from the ground to a height of approximately 4 m and back to the ground are listed in

Table 3. Calculated User-KVIs for the 5G for Energy Analytics use case.

User-KVI	Calculated Values	Unit
Global Warming Potential	7.22 × 10−3	kg CO2-eq.
Water Consumption	2.89 × 10−3	m3
Ozone Depletion	1.43 × 10−10	kg CFC 11-eq.
Photochemical Ozone Formation	2.20 × 10−5	kg NMVOC-eq.
Consumed Electricity Per Kilogram of Transported Mass	1.19 × 10−4	kWh/kg

. For the entire operation, a total power consumption of 2.86 × 10⁻² kWh was measured. The User-KVIs have been calculated based on this consumption value in combination with the emission factors for electrical energy produced in Germany, based on the Life Cycle Assessment (LCA) methodology according to ISO 14040/44 [38,39].

The energy profile of the material lifting operation is pictured in Figure 6. The x-axis shows the time, and the y-axis shows the measured values for active power in kW, with the plot showing different segments of peaks. One peak indicates the active power consumption for the upward movement of the material lift from approx. 15:44:30 to approx. 15:44:45, and the other shows the downward movement of the material lift from approx. 15:45:00 to approx. 15:45:15. As indicated in Table 3, the upward and downward movement of the lift caused a total of 7.22 × 10⁻³ kg CO₂-eq., which represents the carbon footprint of this operation over the total execution time of approximately 30 s. Thus, the 5G-enabled metering devices, which can be directly connected to a variety of power consumers, enable the operators of construction sites to calculate the environmental footprint of their operations, creating energy transparency and awareness based on distributed metering systems and centralized energy analysis and monitoring. In this way, it can be shown that the described use case is able to contribute to the overall goal of creating transparency around ecological impacts on construction sites.

3.3. Application of the MAF to an Automotive Use Case

3.3.1. Description of the Use Case

The third use case that the MAF is applied to is titled “Predictive Quality of Service for Tele-operated Vehicles”, in which 5G is used to realize the tele-operation of two tele-operated vehicles (ToV). The operation of ToVs enabled by 5G is considered to be a major benefit of the technology, offering solution approaches to currently existing challenges [40,41]. For the considered use case, both tele-operated vehicles (ToV A and ToV B) travel on the same route. The route is covered by a 5G network, which enables remote vehicle operation. During the journey, a degradation of the Quality of Service (QoS) occurs at some point, potentially interrupting the tele-operation due to reduced connectivity quality. To prevent the ToV from stopping on the road, predictive QoS is used by ToV B, allowing it to proactively reroute and avoid areas with low network performance. The predictive QoS concept relies on information provided by the network through network exposure APIs. To quantify the advantage of the predictive QoS, ToV A is not equipped with this functionality and therefore remains susceptible to connectivity degradation. The successful execution of the use case demonstrates ToV B’s capability to anticipate and avoid low-coverage areas, maintaining operational continuity and on-time arrival at the planned destination. The use case is thoroughly described in [12,20]. Test results of the use case have already been published in [42]. The system architecture for the realization of the use case is pictured in Figure 7.

3.3.2. Description of the Selected User-KPI and Evaluation of the Use Case

One of the main challenges in tele-operated driving is maintaining reliable network coverage and avoiding potential interruptions caused by QoS degradation, which can result in ToVs becoming immobilized in the middle of the road. The presented use case addresses this issue by establishing a technical solution that enables a capable and stable process that consistently produces the same output (timely arrival of the vehicle) under identical input conditions (fixed route with estimation for the required travel time). For this reason, the User-KPI “Process capability (c_p & c_pk)” was selected from Table 1. The calculation of c_p and c_pk allows for the evaluation of whether a stable and capable process can be achieved through the integration of 5G connectivity/features and QoS prediction. To evaluate this aspect, ToV A and ToV B traveled along various routes. The experiment was repeated 100 times with different network conditions, and for each iteration, the travel times of ToV A (without predictive QoS) and ToV B were measured. The results for two different routes (route one and route two) are pictured in Figure 8 and Figure 9 for both vehicles.

The calculated values for cp and cpk are presented in

Table 4. Calculated values for c_p and c_pk for routes one and two for ToV A and B.

Calculated cp and cpk Values	Route One	Route Two
ToV A, cp	0.228	0.153
ToV A, cpk	0.228	0.153
ToV B, cp	1.082	2.838
ToV B, cpk	1.082	2.838

. For both routes, ToV B, which is supported by the predictive QoS functionality, achieved lower and more consistent travel times, as the majority of the achieved travel times are oriented around the mean values (30.931 min for route one and 27.458 min for route two). Table 4 summarizes the calculated c_p and c_pk for ToV B for both routes, derived using Equations (1) and (2). For both routes, the upper and lower tolerance limits were defined by adding or subtracting 5 min to the mean value. This was achieved based on the assumption of the domain experts (technical developers of the use case) that a deviation of up to ± 5 min was estimated to be acceptable for passengers when traveling in a ToV with the mean total travel time being approximately 30 min.

As indicated by the values for c_p and c_pk, the proposed solution utilizing predictive QoS enables reliable and stable use case outcomes for ToV B. For manufacturing processes, c_p and c_pk values greater than or equal to 1.33 are usually required to guarantee low scrap rates and high yields [24]. For route two, this target range is achieved and even exceeded with the calculated values of 2.838. Although the target range (designed initially for manufacturing processes) for route one is not achieved, it can be concluded that the use case is able to mostly function within its specification limit of ± 5 min of travel time. The value of 1.082 for c_p and c_pk is equivalent to 0.118% of cases in which the travel time specification limits are exceeded by more than 5 min, which is significantly low for such a use case. When compared to the values of c_p and c_pk for ToV A, the significant improvement achieved by predictive QoS becomes obvious. The comparison of ToV A and ToV B shows that ToV A, in contrast to ToV B, is not able to achieve any capability to produce reliable outcomes within the tolerance specifications caused by a lack of early detection of network degradation. The evaluation of the use case shows the potential of 5G to positively affect the outcome of use cases when empowered with innovative solutions such as predictive QoS based on network exposure APIs.

4. Summary and Outlook

The paper at hand presents a Methodological Assessment Framework that was developed to capture the value proposition attainable through the implementation of industrial 5G use cases. The use cases have been developed and deployed in an interdisciplinary manner, as they originate from different verticals. For the quantification of the value propositions, a variety of User-KPIs and User-KVIs, which focus on the added value for the end user of the use case, have been derived. These User-KPIs and User-KVIs enable a holistic and cross-vertical evaluation of use cases that, since they originate from different verticals and application domains, have very different technical backgrounds. Therefore, the main contribution of the developed MAF is the potential to conduct a vertical-agnostic analysis of use cases to check in which way the considered use case addresses key technical, economic, or societal goals. In this way, the MAF aims at contributing to an accelerated adoption of 5G technology in the industry. By quantifying potential value propositions, the MAF can be utilized to illustrate the potential advantages that the use of 5G has to offer. Furthermore, the MAF can also make important contributions to an improved understanding between network experts and domain experts, for instance, production or construction managers. By building the bridge between Network KPIs, like latency, uplink, downlink, etc., and the value proposition for the end user of the use case through User-KPIs and User-KVIs, the MAF presents a method that can support the exchange between experts from different disciplines and domains by creating a common basis for understanding and discussion.

To demonstrate the applicability of the MAF, this paper describes and analyzes three industrial use cases. These use cases, collaboratively developed and deployed within an interdisciplinary research project, represent the manufacturing, construction, and automotive sectors. The analysis results are expressed through the presented User-KPIs or User-KVIs. For all three use cases, it was shown that the use of industrial 5G provides quantifiable benefits. For the manufacturing and the automotive use case, the calculation of the process capability shows that 5G can improve the capability of specific applications, so that the overall operational capability of, e.g., an organization, can be positively influenced by the adoption of the technology. For the construction use case, it was shown that 5G also enables the enhancement of data-driven insights into different processes. This is illustrated by the Energy Analytics use case, which utilizes a plug-and-play solution to acquire energy consumption data that can then be utilized to calculate the environmental footprint of a construction operation. The working hypotheses that an end-user-focused Methodological Assessment Framework can enable the quantification of a variety of benefits of 5G-based use cases can therefore be confirmed. In addition, the application to different verticals also shows that the developed MAF is agnostic to the vertical to which it is applied, underlying its broad applicability in different industrial scenarios.

So far, the MAF has been applied to 12 prototypical use cases within the research project TARGET-X and to 3 use cases from the 5G-ACIA consortium. Thus, it has been applied to the verticals of manufacturing, robotics, energy, construction, and automotive. Therefore, it can be said that the approach is vertical-agnostic in industrial settings and can be applied to a broad variety of different industrial scenarios. To validate the MAF further and enable it to make greater contributions to the broad rollout of 5G technology in the industry and, therefore, the digitalization of the European economy, the methodology should be applied to further use cases which are used by industrial end users. Further research activities should also investigate whether and how the simultaneous implementation of several parallel use cases (e.g., multiple use cases within one production facility) can be evaluated with the MAF, or whether, for example, the User-KPIs or User-KVIs need to be adjusted in order to also make economies of scale measurable. As the assessment with the MAF heavily relies on expert knowledge and expert assumptions (e.g., regarding which User-KPIs and User-KVIs are selected), future research activities should also investigate the extent to which subjectivity of the involved experts influences the analysis results. This can, for instance, be done by having different expert teams conduct the same analysis simultaneously and then comparing the calculated results with each other. Another subsequent analysis should examine how variances in network performance (e.g., changes in throughput or latency) influence the User-KPIs and User-KVIs, so that network dynamics are also taken into consideration. In this way, more insights can be generated into how different network parameters and differences in network performance influence the value proposition of an industrial use case for the end user. By carrying out the described future research activities, it can be checked whether the calculation of User-KPIs and User-KVIs can really contribute to the broader adoption of 5G technology in the industry. As User-KPIs and User-KVIs can also be applied to new generations of wireless communication technologies like 6G, the presented MAF should also be applicable in the future. However, further thorough testing of the applicability must first verify this hypothesis.