Adaptive Microservice Architecture and Service Orchestration Considering Resource Balance to Support Multi-User Cloud VR

Abstract

Recently, in the field of Virtual Reality (VR), cloud VR has been proposed as a method to address issues related to the performance and portability of Head-Mounted Displays (HMD). Cloud VR offers advantages such as lightweight HMD, telepresence, and mobility. However, issues such as Motion-To-Photon (MTP) latency and the handling of large-scale traffic due to continuous video streaming persist. Utilizing edge computing is considered a potential solution for some of these issues. Nevertheless, providing this in a cloud–edge continuum environment for simultaneous users presents additional issues, such as server rendering load and multi-user MTP latency threshold. This study proposes an adaptive MicroServices Architecture (MSA) and a service orchestration based on it to effectively provide multi-user cloud VR in a cloud–edge continuum environment. The proposed method aims to ensure the MTP latency threshold for each user while addressing network congestion, even when the application is provided to multiple users simultaneously in a resource-constrained edge network environment. Furthermore, it aims to maintain high edge applicability for microservices through efficient edge resource management. Simulation results confirm that the proposed method demonstrates better performance in terms of networking and MTP latency compared to other edge resource-management methods.

1. Introduction

Virtual Reality (VR) is a technology that provides immersive virtual environments, primarily used in education and training simulations, remote medical services, and psychological therapy [1,2,3]. However, Head-Mounted Displays (HMD), essential for VR, face significant constraints related to high specification requirements, battery life, portability, and mobility. To overcome these constraints, cloud VR has been proposed [4,5,6,7]. Cloud VR operates by transmitting motion data, such as head rotation and movement detected by the HMD, to a cloud server. The server then performs screen rendering based on the motion data and streams the result back to the HMD for playback. This method allows users to experience high-quality content on low-spec HMD by offloading the rendering tasks to remote servers with higher hardware specifications.

In cloud VR operations, the time from when the user motion input is received to when the rendered video based on this input is sent back to the HMD for playback is referred to as Motion-to-Photon (MTP) latency. The most common and critical issue with cloud VR is that if this MTP latency exceeds 20ms, it can cause VR sickness, severely degrading the user experience to an unusable level [8,9]. The largest portion of MTP latency is attributed to the network latency between the HMD and the remote rendering server. Reducing this latency is a key challenge for the feasibility of cloud VR. Additionally, the high network bandwidth requirement for continuous video streaming between the local HMD and the remote cloud server also presents an issue [10].

A practical approach to addressing these issues is utilizing edge computing by deploying rendering servers within the edge network where the user is connected [11,12,13]. However, utilizing the edge network infrastructure involves issues such as the location of edge devices that depend on network topology, their relatively lower hardware specifications, and resource capacity compared to cloud servers, and the heterogeneity of edge devices [14,15,16]. Moreover, when considering multi-user cloud VR (MCVR) applications, such as metaverse or remote conferencing, meeting the MTP latency threshold for all users distributed across single or multiple networks becomes a significantly challenging issue [17,18,19,20,21,22]. Considering these requirements, the monolithic service model in the MCVR environment has limitations in simultaneously meeting the computing resource constraints of the edge environment and the MTP latency threshold.

1.1. Motivations

The conceptual diagram of MCVR operation in a scenario with three users is depicted in Figure 1. Each user HMD samples the user motion input converts it into a data stream, and transmits it to a remote server for logic processing. Upon receiving the Field of View (FoV) stream, which is the screen rendering output corresponding to the direction in which the user HMD is facing, it plays the stream on the internal display. Tasks with high computational loads, such as logic processing and FoV rendering, which were traditionally performed on a local HMD or a tethered PC, are now performed on a remote server. The transmission of motion data to the remote server and the reception of the FoV rendering result stream are conducted over wired or wireless communication networks [23]. To support MCVR on the edge network, the following issues should be considered.

First, the role of the remote server in MCVR can be divided into logic processing and rendering. For rendering tasks, even with identical content, each user has a different current location and viewing direction, which necessitates a user-specific FoV rendering. Thus, the role of the remote server includes logic processing and user-specific rendering, which means the computing load on the server increases linearly with the number of simultaneous users. Providing MCVR based on a single server becomes a challenge due to the high computing resource requirements, which makes finding suitable edge devices to execute it difficult.

Second, MTP latency should be considered to be a user-specific threshold rather than an average value, and its satisfaction depends on the network delay between each user and the rendering server. Reducing the average network distance between users and the rendering server is beneficial, but the computing power and available resources of edge networks are not high. If computing resources within the network become insufficient or the number of users increases, which results in higher overall resource demands, rendering servers for some users might not enter the edge network. This means those users will be unable to use the application.

Third, although MTP latency is influenced by network distance, it is negligible if the motion-based rendering server is within the edge network. Instead, the path between users and servers will consistently carry high volumes of streaming traffic, which increases linearly with the number of users in MCVR, potentially causing congestion in the edge network if not controlled. Consequently, queuing delays are likely to occur due to high volume streaming traffic depending on the server’s location. In addition, edge devices exhibit heterogeneous characteristics, which cause frequent mismatches between computing resource availability and resource demands. This mismatch results in computing resource wastage within the network and adversely affects the efficiency of rendering server deployment. Therefore, for efficient traffic management and resource allocation of edge resources, it is essential to consider the resource balance between rendering servers and edge devices.

1.2. Contributions

When providing MCVR in a cloud–edge continuum and addressing the aforementioned challenging issues, the most appropriate approach considered in this study is implementing the application following a MicroServices Architecture (MSA). This study proposes a method for adaptively configuring microservices based on situational needs and service orchestration to minimize the negative effects on the edge network while maximizing the deployment of microservices within it. The contributions of this approach are as follows.

• To address scalability issues when providing MCVR through a single server, a design based on multiple microservices operating as individual units can be considered. The logic processing in MCVR includes handling core elements such as content logic, physics engine, and AI computations. This requires data collection and synchronization for all user inputs at a single point. The proposed method configures this as a single Logic service per application, primarily operating in the cloud. In addition, the function of rendering the Field of View (FoV) screen by processing user-specific motion data and streaming it is configured as a user-specific Render service operating in the edge network.
• Render services generate high volumes of traffic for users and have significant computing requirements. If such render services cannot be deployed within the edge network where the user is connected due to the resource constraints of edge devices, it will be impossible to meet the MTP latency threshold. In such situations, it is necessary to split the render service into two smaller services, Motion and Augment services. Motion services handle tasks such as video cropping or applying motion parallax based on depth information, which involves relatively low computational loads and can be placed closer to the user. Augment services transmit data to the motion service, which includes render data for a wide FoV screen or depth information for individual objects. This allows the motion service to operate independently.
• The MSA configuration of MCVR considered in this study consists of a single logic service and user-specific render or augment and motion services. Render services or augment and motion services can be deployed anywhere within the edge network as long as they meet the MTP latency threshold for the corresponding user without having a direct effect. We focus on the generated traffic between users and these services, and we perform service orchestration to minimize this traffic. Simultaneously, to maximize the number of services deployed within the edge network, resource balance is considered proportionally according to the situation.

2. Related Works

Our previous study [24] proposed a resource provisioning method that comprehensively considers content types, client usage patterns, and network metrics to meet user experience requirements for cloud VR applications. However, to provide cloud VR within the edge for all users while satisfying MTP latency, adopting an MSA configuration is inevitable. The high computing resource requirements of applications make it impossible to always meet these requirements for all users by simply applying the single server-based resource provisioning. As in previous studies, if the rendering function is provided as a divided service into augment and motion services, it may lead to computational and network resource wastage compared to providing it as an integrated service. Additionally, deploying services sequentially based on priority cannot consider multiple combinations, which may reduce resource utilization efficiency. To address this, this study proposes an adaptive microservice configuration method for the rendering function, which requires significant computing and networking resources. Furthermore, a user-specific service deployment strategy is established, followed by a heuristic selection method for combining these strategies. Through this approach, resource efficiency has been improved compared to previous studies, thereby ensuring user-specific MTP latency constraints more effectively.

Du et al. [25] proposed collaborative virtual reality, a cloud-based multi-user VR headset system that enables real-time communication between users in an interactive VR environment. Hou et al. [26] aimed to overcome the limitations of HMD through an edge and cloud computing-based wireless VR system and proposed suitable responses to bitrate and latency requirements. Zhao et al. [5] constructed a VR testbed using cloud servers, commercial VR headsets, and standard Wi-Fi Access Points (AP), analyzing cloud VR application traffic patterns and Quality of Service (QoS) characteristics to propose ways to enhance content performance and user experience. These studies assume that cloud VR applications are provided based on a single server for a single user or a small number of users, which does not properly address the computing resource requirements of the service.

Alencar et al. [27] divided VR into modules with user information (i.e., device, buffer, playback quality) and application modules to handle VR content, which created microservices. Their deployment considered latency to satisfy the quality of experience (QoE) of VR streaming but did not consider the computational load of each module. While the study split applications into microservices and deployed them at the edge, it lacked sufficient consideration of computing resource requirements, which may fail to meet latency requirements in multi-user environments or when computing resource demands are high.

Wang et al. [28] and Li et al. [29] conducted studies to reduce overall application latency. They proposed utilizing sufficient available computing resources at edge nodes to reduce computing latency when it is determined that specific tasks cannot be processed within their deadlines by the local system, even if it means sacrificing transmission delay. This approach is effective for reducing total latency through local offloading but has limitations when applied to multi-user cloud VR applications that necessarily require a server.

Kumar et al. [30] proposed a method to maximize resource utilization while minimizing service maintenance costs using game theory. This approach effectively addresses the Edge Resource Allocation (ERA) problem by appropriately pricing multi-tenant edge servers. Yan et al. [31] proposed a dynamic mode selection that uses evolutionary game theory to formalize the competition among potential user space groups as a dynamic evolutionary game and resolves the game through evolutionary equilibrium. The approach considered node location, cache size, and delay costs to derive reward formulas for both fog access points and devices. Qiang et al. [32] and Phu et al. [33] proposed game theory-based algorithms and heuristic approaches for performing distributed processing at the edge. These methods improve scalability while providing individual benefits for each application user and provider. These studies focused on the efficiency of managing computing and networking resources within edge networks but did not consider constraints for multi-user or time-sensitive services.

Velasquez et al. [34] proposed a service placement architecture for IoT environments, with the service orchestrator as the main module. The architecture includes models and implementation details for service placement tasks, deploying services to minimize hop count while considering network resources. However, this method did not consider computing resources and may fail to meet network requirements for all users due to insufficient computing resources of nodes in a multi-user environment. Taneja et al. [35] sorted edge nodes and servers to be deployed based on remaining computing resources and computing resource demands, respectively. They planned deployment by checking the applicability of nodes in the sorted order. This allows quick and simple deployment planning but can pose problems when deploying latency-constrained services such as cloud VR, as it does not consider latency during planning.

3. Adaptive MSA Configuration and Service Orchestration for Cloud–Edge Continuum-Based MCVR Offerings

This study proposes an adaptive MSA configuration and service orchestration for providing MCVR in a cloud–edge continuum. The goal is to meet each user’s MTP latency threshold, reduce congestion in the edge network, and improve the edge deployment applicability of microservices.

3.1. Adaptive Configuration of Microservices for MCVR

The MSA configuration proposed for the MCVR application in this study involves configuring one logic service per application and configuring one render service or augment and motion services for each user. Figure 2 illustrates an example of the microservices configuration with three simultaneous users. It shows the operation of each service and the information exchanged between them.

The logic service handles the core logic processing of the application, such as content logic, physics engine, and AI computations. The logic service processes inputs from multiple users in the order they arrive to compute the results displayed in the next frame on each user screen. It transmits these results to the corresponding render or augment service for each user. To synchronize all user inputs, this service is configured as a single service, which also handles input validation and data storage.

The render service renders the HMD screen for the corresponding user and streams it to the user. Specifically, the render service renders the screen to be displayed in the current FoV based on the motion input calculated from the logic processing results received from the logic service. To meet the MTP latency threshold, this service should be deployed close to the user within the edge network. However, if the computing resource requirements for this service increase or if available resources within the edge become insufficient, additional measures are needed.

In special situations where the render service cannot operate within the edge, it can be split into augment and motion services. The motion service performs low-load rendering based on the user motion input and the augmented FoV stream near the user. Since it has lower resource requirements than the render service, it can be easily deployed close to the user. The augment service streams augmented FoV rendering results to the motion service to enable low-load rendering by the motion service. This service can be configured to handle rendering pipelines more efficiently than the render service, thus reducing resource requirements. However, the augmented FoV stream generated between the augmented service and the motion service may contain more information than the render service output stream, potentially increasing traffic load.

The implementation of augment and motion services can vary depending on the type of application. For example, in a panoramic video-based application, the augment service renders a full panoramic view screen centered around the user or a wide FoV screen that includes a broader area than the user FoV and transmits it to the motion service. The motion service crops the FoV based on the current motion input and sends it to the user. In another example, for stereoscopic-based applications, the augment service performs binocular rendering for a wide FoV and integrates depth information for key objects before transmitting it to the motion service. The motion service adjusts and interpolates the positions of the key objects based on the depth information to create motion parallax according to the motion input and streams this to the user. In both cases, the output from the motion service is a temporary screen corresponding to the current motion input, with the normal FoV screen being displayed after a few frames.

In certain situations, the render service can be divided into augment and motion services to meet the MTP latency threshold. However, the combined computing resources required by the augment and motion services may exceed those required by the render service, which can potentially lead to resource wastage in the edge network. Additionally, the traffic load of the augmented FoV stream from the augment service to the motion service exceeds the traffic load requirements of the render service. Due to this additional overhead, it is ideal to adopt a strategy where the render service is only divided into augment and motion services when it cannot be executed within the edge network.

Figure 3 illustrates the process of handling a user motion input in the proposed MSA-based MCVR. For users served by augment and motion services, the user motion input is first sent to the motion service, which generates a motion-processed temporal FoV stream. This output temporarily processes the current motion input based on the rendering results from a few frames earlier, provided by the augment service. The same motion input is then forwarded from the motion service to the augment service, which updates and returns the augmented FoV stream to the motion service. Depending on the service environment and conditions, this interaction may introduce a delay of a few frames. Consequently, the user sees the accurate frame after a few frames, but does not notice the brief delay because a temporarily processed frame has already been displayed on the HMD in the meantime. The same motion input is also sent to the logic service, which computes synced data processed by the logic for that user and other users. These data are then sent to the corresponding augment or render service for each user.

Figure 4 illustrates an example of configuring microservices in MCVR from a cloud–edge continuum infrastructure perspective. The application operator configures the application through a cloud interface. Users of the application are geographically distributed across three edge networks, which necessitates the operation of the logic service within the cloud infrastructure. In the left edge network, the application is provided through a render service. This scenario represents an ideal situation where the edge network has sufficient computing and networking resources. In the middle edge network, users are served by augment and motion services. When edge network resources are insufficient, the augment service may run in the cloud, but the motion service remains in the edge network, which meets the MTP latency threshold. However, high volumes of traffic occur between the cloud and the edge network, and users might sometimes perceive a temporary FoV stream, which can result in quality of experience degradation. The right edge network shows a situation where resources are severely lacking, and the render service runs in the cloud. In this case, the MTP latency cannot be satisfied, which makes it impossible for users to use the application normally. To avoid scenarios like the one shown in the right edge network, where the application becomes unusable, we propose a service orchestration for MSA-based MCVR.

3.2. Service Orchestration for MSA-Based MCVR to Satisfy MTP Latency Thresholds and Reduce Network Congestion

The goal of the proposed algorithm is to meet the MTP latency threshold for each MCVR user while reducing congestion in the edge network. Although application users may exist across multiple networks, this subsection assumes a single edge network. This assumption is reasonable in the absence of user mobility between networks or federated resource configurations with adjacent networks. Furthermore, it is assumed that all edge nodes possess general-purpose computing resources and container runtime environments, which makes them usable as edge devices.

As mentioned, the algorithm can be divided into two main goals. The first consideration, which significantly affects MTP latency, is the deployment location of the render or motion service. Specifically, the network distance between the corresponding user and the service is crucial. This study assumes that the MTP latency threshold can be met if the render or motion service is within the same network as the corresponding user. Therefore, from an MTP latency perspective, there is no need to overextend operations to deploy these services as close to the user as possible.

The next consideration is network congestion, which is also affected by the deployment location of microservices. The path between the render service and the user, as well as the path from augment to motion service, generates high volumes of traffic. To reduce congestion, it is essential to minimize the length of these paths.

Referring back to the first goal, to meet the MTP latency threshold, the render or motion service must be deployed within the edge network where the user is connected. To achieve this, the available resources of all devices within the network must be able to accommodate the computing resource requirements of all services. Considering the heterogeneity of edge devices and the diversity of applications, the usage of available resources and the resource demands of services may not align by resource type. For example, if memory usage is at 100% while CPU and GPU usage are each at 10%, it would be challenging to deploy additional services on that device, which results in underutilized CPU and GPU resources. This resource wastage can decrease the service deployment feasibility of the edge network. In this study, we consider this by adding a separate factor called resource balance.

While the problem might seem like a bin-packing problem where services are assigned to devices, specific types of resources cannot be shared by multiple services simultaneously. Hence, we aim to solve the problem using the Vector Bin-Packing (VBP) algorithm [36]. In VBP, the vectorized bin represents the available resources of an edge device, and the vectorized item represents the resource requirements of a microservice. The smaller the angle between the two vectors, the more balanced the distribution of resource availability and requirements. Therefore, deploying services on the device with the smallest angle minimizes computing resource wastage from a network perspective. If edge devices and microservices need to consider new types of resources, this can be easily addressed by simply adding dimensions to the vectors.

Algorithm 1 shows the planning method for deploying MCVR application microservices within an edge network. In summary, it first generates deployment strategies for each user-specific service within the edge network. For example, after creating a strategy to deploy a render service for a specific user to a particular node, it derives metrics for the expected traffic and resource balance of the deployment node. The created strategy prioritizes all strategies based on these metrics and examines the applicability of applying each strategy in order of priority, ultimately designating applicable strategies as the planned deployment strategies. The algorithm concludes successfully if it successfully plans the service deployment strategy for all users.

Algorithm 1 PlanDeploymentStrategies

1: $w_{rb}\leftarrow$ the weight factor of the resource balance for calculating strategy selection metrics, initialized as 0  2: $q_{\sigma }\leftarrow$ the priority queue for strategies  3: $P\leftarrow$ the list of planned strategies, which contains as many strategies as users  4: $\mathrm{\Sigma }\leftarrow$ CreateStrategies()  5:   6: while $w_{rb}\le 1$ do  7:     clear the collections P and $q_{\sigma }$  8:     for all $\sigma \in \mathrm{\Sigma }$ do  9:     ${\sigma }_m\leftarrow w_{rb}·{\displaystyle \frac{{\sigma }_{\theta }-\min \left(\theta \right)}{\max \left(\theta \right)-\min \left(\theta \right)}}+(1-w_{rb})·{\displaystyle \frac{{\sigma }_t-\min \left(t\right)}{\max \left(t\right)-\min \left(t\right)}}$ 10:     $q_{\sigma }$.enqueue($\sigma$) 11:    end for 12:    while $!q_{\sigma }.\mathrm{empty}\left(\right)$ do 13:     $\sigma \leftarrow q_{\sigma }.\mathrm{dequeue}\left(\right)$ 14:     if ${\sigma }_{user}$ has planned strategy then continue 15:     $\rho \leftarrow {\sigma }_{pairs}$ 16:     ${\mathbf{r}}_{n_0}\leftarrow {[{\rho }_0.n^{cpu},{\rho }_0.n^{gpu},{\rho }_0.n^{mem},...]}^{current}$ 17:     ${\mathbf{r}}_{s_0}\leftarrow [{\rho }_0.s^{cpu},{\rho }_0.s^{gpu},{\rho }_0.s^{mem},...]$ 18:     if $\left|\rho \right|=2$ then 19:      ${\mathbf{r}}_{n_1}\leftarrow {[{\rho }_1.n^{cpu},{\rho }_1.n^{gpu},{\rho }_1.n^{mem},...]}^{current}$ 20:      ${\mathbf{r}}_{s_1}\leftarrow [{\rho }_1.s^{cpu},{\rho }_1.s^{gpu},{\rho }_1.s^{mem},...]$ 21:     end if 22:     if ${\mathbf{r}}_{s_0}\le {\mathbf{r}}_{n_0}\wedge \left(\right|\rho |=1\vee \left(\right|\rho |=2\wedge {\mathbf{r}}_{s_1}\le {\mathbf{r}}_{n_1}\left)\right)$ then 23:      P.add($\sigma$) 24:      ${\rho }_0.n^{cpu,gpu,mem,...}-={\mathbf{r}}_{s_0}$ 25:      if $\left|\rho \right|=2$then${\rho }_1.n^{cpu,gpu,mem,...}-={\mathbf{r}}_{s_1}$ 26:      update $\forall {\sigma }_{\theta ,m}\in \mathrm{\Sigma }$ associated with ${\rho }_0.n$ or ${\rho }_1.n$ 27:     end if 28:    end while 29:    if all users have planned strategies then break 30:    else $w_{rb}\leftarrow w_{rb}+0.1$ 31: end while 32:  33: ProcessUsersWithoutStrategy() 34: DeployLogicServices() 35:  36: Output: 37: return P

The proposed method can handle dynamic user join and leave patterns as follows. When a user leaves, no additional action is taken if there are no unallocated services within the edge. If there are unallocated services or a new user joins, the user services are deployed using only the remaining resources from the existing plan. If this additional process fails, the entire algorithm is restarted.

Algorithm 2 shows the procedure for creating strategies for each user $u\in U$. It first assumes that the user is served by a render service. In Line 5, all nodes in the network are considered candidates for deploying the render service, and in Line 6, a new strategy $\sigma$ is formulated. In lines 7–10, the resource balance is considered. The available resource size of the candidate node $n_r$ and the resource requirements of the render service $s_r$ are vectorized. If a node cannot accommodate the service, it is ignored. Otherwise, the algorithm calculates the angle between the vectors and stores it in the strategy. In Line 11, the traffic from the candidate node $n_r$ to the user node $n_u$ is added and stored in the strategy. This process creates the render service deployment strategy for all users. In Line 14, the deployment strategy for augment and motion services is established using the node $n_u$ directly connected to user u and the node $n_r$, where the render service $s_r$ is currently being considered for deployment.

Algorithm 2 CreateStrategies

1: $\mathrm{\Sigma }\leftarrow$ the list of strategies  2: for all $u\in U$ do  3:   $n_u\leftarrow$ the edge node to which the user u is directly connected, $n_u\in N$  4:   $s_r\leftarrow$ the render service corresponding to the user u  5:  for all  $n_r\in N$ do  6:    $\sigma \leftarrow$ the deployment strategy for the render service $s_r$ for the user u  7:    ${\mathbf{r}}_n\leftarrow [n_r^{cpu},n_r^{gpu},n_r^{mem},...]$  8:    ${\mathbf{r}}_s\leftarrow [s_r^{cpu},s_r^{gpu},s_r^{mem},...]$  9:    if ${\mathbf{r}}_n<{\mathbf{r}}_s$then continue 10:    ${\sigma }_{\theta }\leftarrow arccos(({\mathbf{r}}_n·{\mathbf{r}}_s)/\left(\right|{\mathbf{r}}_n\left|\right|{\mathbf{r}}_s\left|\right))$ 11:    ${\sigma }_t\leftarrow$ the expected traffic from $n_r$ to $n_u$ by $s_r$ 12:    ${\sigma }_{pairs}\leftarrow \left[(s_r,n_r)\right]$ 13:    $\mathrm{\Sigma }$.add($\sigma$) 14:    $\mathrm{\Sigma }$.add(CreateAltenativeStrategies($n_u,n_r$)) 15:  end for 16: end for 17:  18:  Output: 19: return $\mathrm{\Sigma }$

Algorithm 3 illustrates the method for creating an alternative deployment strategy for augment and motion services for a specific user. This algorithm operates alongside the creation of a render service deployment strategy for a specific user. It assumes that the augment service is deployed to the same node $n_a$ as the render service node $n_r$. Then, all other nodes are considered to be candidate nodes $n_m$ for deploying the motion service, and a strategy is created accordingly. As indicated in lines 14–16, $n_m$ only considers the nodes between $n_r$ and $n_u$. In lines 19–23, the resource balance factor for the strategy is calculated by taking the larger of the angles between the vectors of $s_a$ and $n_a$, and $s_m$ and $n_m$. This approach better reflects the effect of a specific strategy on the network. Lines 24–26 calculate the expected traffic for the strategy by adding the traffic generated by $s_a$ from $n_a$ to $n_m$ and the traffic generated by $s_m$ from $n_m$ to $n_u$. Line 27 creates and stores two (service, node) pairs for the strategy.

Algorithm 3 CreateAlternativeStrategies

1: Input:  2: $n_u\leftarrow$ the node directly connected to user u  3: $n_r\leftarrow$ the node that the render service $s_r$ is currently being considered for deployment  4:  5: ${\mathrm{\Sigma }}_{alt}\leftarrow$ the list of alternative strategies  6: $n_a\leftarrow n_r$  7: $s_a\leftarrow$ the augment service for the user u, paired with $s_m$  8: $s_m\leftarrow$ the motion service for the user u, paired with $s_a$  9: ${\mathbf{r}}_n^a\leftarrow [n_a^{cpu},n_a^{gpu},n_a^{mem},...]$ 10: ${\mathbf{r}}_s^a\leftarrow [s_a^{cpu},s_a^{gpu},s_a^{mem},...]$ 11: ${\theta }_a\leftarrow arccos(({\mathbf{r}}_n^a·{\mathbf{r}}_s^a)/\left(\right|{\mathbf{r}}_n^a\left|\right|{\mathbf{r}}_s^a\left|\right))$ 12:  13: for all $n_m\in N$ do 14:   $d_{r-u}\leftarrow$ the network distance between $n_r$ and $n_u$ 15:   $d_{m-u}\leftarrow$ the network distance between $n_m$ and $n_u$ 16:   if $d_{r-u}<d_{m-u}$ then continue 17: 18:   $\sigma \leftarrow$ the deployment strategy for the augment service $s_a$ and the motion service $s_m$ for the user u 19:   ${\mathbf{r}}_n^m\leftarrow [n_m^{cpu},n_m^{gpu},n_m^{mem},...]$ 20:   ${\mathbf{r}}_s^m\leftarrow [s_m^{cpu},s_m^{gpu},s_m^{mem},...]$ 21:   if ${\mathbf{r}}_n^m<{\mathbf{r}}_s^m$then continue 22:   ${\theta }_m\leftarrow arccos(({\mathbf{r}}_n^m·{\mathbf{r}}_s^m)/\left(\right|{\mathbf{r}}_n^m\left|\right|{\mathbf{r}}_s^m\left|\right))$ 23:   ${\sigma }_{\theta }\leftarrow \max ({\theta }_a,{\theta }_m)$ 24:   $t_{a-m}\leftarrow$ the expected traffic from $n_a$ to $n_m$ by $s_a$ 25:   $t_{m-u}\leftarrow$ the expected traffic from $n_m$ to $n_u$ by $s_m$ 26:   ${\sigma }_t\leftarrow t_{a-m}+t_{m-u}$ 27:   ${\sigma }_{pairs}\leftarrow [(s_a,n_a),(s_m,n_m)]$ 28:   ${\mathrm{\Sigma }}_{alt}$.add($\sigma$) 29: end for 30: 31: Output: 30: return ${\mathrm{\Sigma }}_{alt}$

Returning to Algorithm 1, we use the created strategies to create the deployment plan. Each created strategy includes the user, resource balance, expected traffic, and service–node pairs. To set the priority for each strategy, the weight factor $w_{rb}$ of the resource balance is used in Line 9 to derive a metric by adding the normalized resource balance and the expected traffic. The initial value of $w_{rb}$ is 0, but it gradually increases under certain conditions, which means the focus is initially on reducing network traffic. If the MTP latency cannot be satisfied, the resource balance is gradually given more consideration. A smaller vector angle suggests more balanced resource usage, and the expected traffic should be minimized. Hence, strategies with lower metrics are given higher priority. Initially, only MTP latency is considered, but when services cannot be deployed, $w_{rb}$ is gradually considered to balance resource utilization and increase the number of users supported at the edge. All strategies with derived metrics are managed in a priority queue $q_{\sigma }$, examined in order of increasing metric values.

In Line 14, if the user of the strategy being examined has already been processed, the strategy is ignored. In lines 15–22, the applicability of the current strategy is examined. If the service requirements exceed the currently available resources of the paired node for any of the pairs in the strategy, the strategy is considered inapplicable. The currently available resources of a node refer to the total available resources of the node, subtracting the requirements of services that other strategies plan to deploy on that node. If the strategy is applicable, the strategy is added to the final plan P, and the requirements of the services within the strategy are subtracted from the current resources of the nodes within the strategy. Then, the vector angles and metrics of the remaining strategies within $q_{\sigma }$ associated with those nodes are recalculated, and the queue is updated accordingly.

After examining all strategies, if a deployment strategy exists for every user, then it means that all services of the MCVR applications can be successfully deployed within the edge network. If there are users for whom services could not be deployed, it is determined that resource usage was unbalanced, so the weight factor $w_{rb}$ of the resource balance is increased, and the process restarts from the metric derivation step. If services cannot be deployed for all users, even when the weight factor is set to 100%, it indicates that the cloud must be utilized. For users without a planned strategy, a strategy is planned to deploy only the motion service at the edge while deploying the augment service in the cloud. If deploying the motion service at the edge is also not possible, the motion service will be deployed in the cloud. Lastly, in Line 34, application-specific logic services are deployed at the edge if resources permit; otherwise, they will be operated in the cloud.

Figure 5 illustrates the operation of the algorithm using a simple case scenario. When there are multiple users in the network, the first step is to create strategies for each user. Here, both strategies that involve only the render service and those that include augment and motion services are presented, with the expected traffic and resource balance as associated factors for each strategy. In the second step, the priority metrics for each strategy are derived, and the strategies are sequentially examined according to their priority to select the most appropriate strategy for each user in the final plan. The right side of the figure shows the results of applying the final planned strategies to the network.

4. Simulation Results

4.1. Environmental Setup

The proposed algorithm was evaluated through simulations against various existing methods from multiple perspectives. The simulation environment was configured to reflect various scenarios, and the key metrics for comparative evaluation are as follows.

• Traffic Load per User: The average network traffic generated during service provision for each user is calculated as the arithmetic mean of the sum of traffic on the paths between microservices and between each microservice and its corresponding user.
• MTP Latency Threshold Satisfaction Rate: The rate of users satisfying the MTP latency threshold is represented by the percentage of users for whom the render or motion service is deployed within the edge network.
• Network Distance per User: This metric represents the path length between the node where the render or motion service operates for the user and the user node, which indicates how close each management method can deploy services to the user. It indirectly reflects the satisfaction rate of the MTP latency threshold and the degree of network congestion management.
• Edge Deployment Feasibility Index: This metric represents the rate of microservices that can be deployed within a constrained edge network, which indicates the percentage of planned strategy services that are deployed within the edge network.

The simulation assumes a cloud–edge network with a single cloud server and 25 edge nodes. All nodes within the edge network have general-purpose computing resources capable of running containerized microservices. Since the edge network has a flat structure, MCVR users can connect to any edge node. The cloud server is assumed to have unlimited computing resources, but the network distance between the cloud and the edge gateway is set to 10.

Simulation parameters are defined as follows. The computing resources and traffic requirements for each microservice of the MCVR application are uniformly distributed within the ranges specified in

Table 1. Simulation parameters for each microservices of MCVR application.

Type	CPU	Memory	GPU	Traffic
Logic	[20, 30]	[20, 30]	[20, 30]	[0.1, 1]
Render	[40, 60]	[40, 60]	[80, 100]	[8, 10]
Augment	[36, 54]	[36, 54]	[72, 90]	[10, 12]
Motion	[16, 24]	[16, 24]	[32, 40]	[8, 10]

. Computing resources are abstract values represented as workloads without specific units [24]. The function of the render service cannot be completely partitioned, and each container requires a minimum amount of base resources to operate. Therefore, it is assumed that the computing resource requirements for augment and motion services are, on average, approximately 0.9 and 0.4 times that of the render service, respectively. From a traffic perspective, the result of the render and motion services is the FoV screen stream, while the augment service outputs an augmented FoV stream, which is assumed to generate about 1.2 times more traffic.

To reflect the heterogeneity of edge devices, the total computing resources of the devices are assumed to follow a normal distribution with a mean of 700 and a standard deviation of 100. Individual resources follow a normal distribution with a mean of 150 and a standard deviation of 50, and the total of each resource matches the total computing resources determined by the normal distribution. To ensure each device can perform minimal functions, the minimum value for individual resources is set to 100. The default number of users in the network is set to 60 for simulations related to locality and 40 for others.

The simulation parameters were set with a high degree of randomness for variables, except for those directly involved in the simulation, to reflect various scenarios. In each run, the computing resource values of edge nodes are set differently and uniformly applied to all nodes. Each simulation result is averaged over 100,000 runs to mitigate the effects of randomness.

The proposed method is compared not only to our previously proposed method but also to other management methods [34,35]. Our previously proposed method is denoted as PVBP and considers both traffic and computing resource balance to deploy edge servers on nodes close to the user. The method from [34] deploys edge servers on nodes with the minimum hop count from the client and is denoted as IC (Information Collection). The method from [35] deploys edge servers on nodes with the most remaining resources, starting with those with the lowest computing resource requirements, and is denoted as MM (Module Mapping). The BF algorithm, a traditional Best-Fit method, deploys edge servers on nodes with the most matching available resources, starting with those with the lowest resource requirements.

Although all comparative methods were originally proposed without MSA, they have been redesigned to adhere to MSA for a fair comparison with the proposed method. Methods labeled with dash-2 assume the application is provided with a render service for each user, while methods labeled with dash-3 assume the application is provided through augment and motion services for each user. All methods are assumed to consider the cloud only if a suitable device cannot be found within the edge network.

4.2. Simulation Results and Discussion

Figure 6 illustrates the key network indicator variations with user count changes. Figure 6a shows the change in traffic per user as the number of users increases. All methods tend for user traffic to increase steeply after a certain point. This is because an increase in the number of users leads to an increase in the number of generated services, which causes a steep rise in traffic once edge resources are exhausted and some services start operating from the cloud. For methods of type dash-2, which provide applications to individual users using only render services, there is a relatively lower traffic load because only the FoV screen stream exists between the render and user nodes. On the other hand, methods of type dash-3, which employ augment and motion services, generate higher traffic loads due to the augment FoV stream between augment and motion services. The proposed method adaptively selects deployment strategies based on the state of edge resources, initially maintaining low traffic levels but showing moderate load during the phase when traffic starts to increase steeply.

Figure 6b shows the rate of users satisfying the MTP latency threshold. All comparison methods fail to sufficiently consider resource balance during the service deployment process, which results in a significant amount of wasted computing resources within the edge. In contrast, the proposed method carefully considers resource balance even in situations where resources are scarce, which enables a higher number of render and motion services to operate within the edge network.

Figure 6c illustrates the changes in network distance between users and their respective render or motion services as the number of users increases. This indirectly shows the MTP latency satisfaction rate and the performance of network congestion control. For dash-2 methods, edge devices capable of handling the relatively large resource blocks required by render services are quickly exhausted, leading to operations shifting to the cloud and resulting in higher network distances. In the case of the dash-3 methods, although the resource requirements for motion services are lower, allowing better utilization of the edge network, the combined deployment of motion and augment services quickly depletes edge resources. The proposed method initially focuses only on traffic but increasingly considers resource balance as resources become scarce. Consequently, to maintain low traffic levels, it reduces network distances and minimizes resource wastage, significantly delaying the point at which cloud resources need to be utilized compared to other methods.

Figure 6d shows the variation in the edge deployment feasibility index of all microservices as the number of users increases. While the previous metrics show the quality of service provided to users, this metric indicates how efficiently edge network resources are managed. Although all methods show a decrease in edge acceptance rates with increasing user numbers, the proposed method, through effective consideration of resource balance, maintains the highest edge deployment feasibility index in most scenarios.

The proposed method successfully controls network congestion and maintains a high MTP latency threshold satisfaction rate, a crucial metric for MCVR, even as the number of users and managed microservices increases. Additionally, through efficient utilization of edge resources, the method keeps the network distance between users and motion-related services to a minimum, which ensures that the largest number of microservices operate at the edge in most scenarios.

Figure 7 evaluates the ability to handle various types of MCVR content with different resource requirements. The types considered include normal quality content (N; Normal), high-resolution and high-frame-rate content generating high volumes of traffic (HT; High Traffic load), high-level 3D content with significant rendering load (HC; High Computing load), and content with high demands on both network and computing resources (HTC). The HT type is assumed to require twice the expected traffic of the N-type, while the HC type is assumed to require twice the computing load of the N-type. This simulation setup reflects that network and computing resource requirements vary significantly depending on the type of MCVR content. High-resolution and high-frame-rate content leads to substantial network traffic, whereas complex 3D content results in heavy rendering loads. Consequently, resource management and service deployment strategies within the edge network become important.

Figure 7a shows that the proposed method exhibits a traffic load that is located between the levels of dash-2 and dash-3 methods in most scenarios. This indicates that the adaptive service configuration of the proposed method works effectively across various content types. Figure 7b demonstrates that the proposed method significantly outperforms others in terms of satisfying the MTP latency threshold across all content types. In particular, even in situations where individual services have high resource requirements, such as with HC and HTC, the proposed method effectively considers resource balance, which allows the deployment of more motion-related services at the edge.

Figure 7c shows that the proposed method maintains a low network distance in all scenarios. The network distance for HC and HTC is overwhelmingly high for most methods due to the large number of services operating in the cloud. Figure 7d demonstrates that the proposed method effectively utilizes edge resources by considering adaptive service configuration and resource balance metrics.

Figure 8 illustrates the key network indicator variations with changes in the computing resource capacity of the infrastructure. The computing resources of the nodes are adjusted using a resource scaling factor, which is applied to the maximum boundaries of each resource as shown in Table 1. Figure 8a shows the average traffic load per user as the factor changes from 2 to 0.5. As available resources decrease, the proportion of services operating in the cloud increases, leading to a rise in the average traffic load per user. For BF and MM methods, despite having sufficient resources, the failure to consider both traffic and resource balance results in high traffic loads. In contrast, PVBP and IC methods maintain lower traffic loads by considering traffic during service deployment.

Initially, the proposed method exhibits a traffic load similar to PVBP and IC methods. However, as computing resource capacity decreases, the traffic load is located between the BF and MM methods and the PVBP and IC methods. Additionally, as shown in Figure 8b, the proposed method maintains a higher MTP latency threshold satisfaction rate compared to other methods, even when absolute available resources are limited. This indicates that although the rate of motion services operating at the edge and augment services operating in the cloud increases as node computing resources decrease, the proposed method still deploys services to ensure that MTP latency is maximally maintained. Consequently, while the average traffic load per user may increase slightly due to the traffic loads between services, MTP latency remains as well maintained as possible.

Figure 8c,d show that even in resource-constrained situations, the proposed method can orchestrate services to operate at the edge as much as possible in most situations compared to other methods. The simulations confirm that as the computing resource capacity of the edge network decreases, the appropriate MSA configuration and the proposed orchestration method significantly enhance the MCVR user experience and the efficient operation and management of edge network resources.

In some cases, users within the network may concentrate on specific nodes, which creates hotspots. In this simulation, the Zipf distribution was utilized to effectively reflect the hotspot scenario, where requests are concentrated on specific nodes within the network [37]. Figure 9 presents the simulation results for this scenario, with user distribution following a Zipf distribution. The probability of a user connecting to the k-th node is given by $P\left(k\right)=(1/k^s)/({\mathrm{\Sigma }}_{n=1}^{N}1/n^s)$, where N is the number of nodes, k is the k-th node, and s is the exponent characterizing the distribution. The ranking of nodes is randomized in each simulation.

Figure 9a shows the key network indicator variations with the user locality changes. With the proposed method, as locality increases, the traffic load per user also increases, but it effectively controls traffic low by maintaining as many services as possible within the edge network and considering service characteristics during deployment. Figure 9b illustrates the deployment ratio of motion-related services within the edge network according to user locality. The BF and MM methods consider only the remaining resources of edge devices, which leads to a similar level of performance regardless of user locality. In the case of PVBP and IC methods, although they try to deploy services near users by considering traffic, edge devices near hotspots cannot handle the resource demand, which results in services being deployed further away and disrupting resource balance. This reduces the deployment ratio of motion-related services within the edge. The proposed method ensures that motion-related services operate within the edge, even as user locality increases.

Figure 9c shows the average network distance according to user locality. Most methods increase the network distance when hotspots occur, as they deploy services farther from users. The MM method deploys services on nodes with large available resources regardless of user location. Uniquely, as user locality increases, users cluster in one area, which may reduce the distance between connected nodes and service nodes by chance. The proposed method is more likely to deploy motion-related services near users even when hotspots occur, maintaining a lower network distance than other methods, even in extreme situations. Figure 9d illustrates the changes in edge service acceptance rates according to user locality. The proposed method consistently operates edge resources more efficiently than other methods, regardless of user locality. In summary, while the efficiency of the proposed method may slightly decrease as user locality increases, it still performs better than other methods in most situations.

Figure 10 presents the simulation results assuming that nodes have computing resources with locality, which means certain nodes have significantly more computing resources than others. In this scenario, the total amount of resources within the edge remains the same, but resource-concentrated nodes are chosen according to the Zipf distribution. Each time a node is selected as a resource-concentrated node, it receives an additional 5% of the total available resources in the network for each resource type, while the resources of other nodes are uniformly reduced by the same amount.

Figure 10a shows the changes in traffic load with respect to resource locality. When computing resources within the edge are concentrated on a single node rather than being distributed across multiple nodes, the possibility of resource wastage slightly decreases. As a result, an increase in locality leads to a slight reduction in traffic per user. However, the dash-3 methods tend to have higher traffic loads compared to the dash-2 methods because the total resource requirements for the deployed services are higher, which leads to more deployments to the cloud. According to Figure 10b, even when resource demands are high and locality increases, the proposed method maintains a high MTP latency threshold satisfaction rate by effectively considering resource balance. Figure 10c,d also demonstrate that the proposed method consistently performs well and is not significantly affected by resource locality.

5. Conclusions and Future Work

Supporting MCVR involves addressing issues with MTP latency thresholds caused by network distances between multiple users and remote servers, and network congestion resulting from continuous video streaming. Even with edge computing, using a single server to handle multiple users is challenging in terms of meeting resource requirements and user-specific MTP latency thresholds. Therefore, this study proposes an adaptive MSA approach for MCVR applications and a user-specific service deployment strategy to address these issues. The proposed service-orchestration method ensures that MCVR users meet their MTP latency thresholds, reduces congestion within the edge network, and balances edge resource usage to increase service capacity. This study focused on the feasibility of MSA-based MCVR applications, particularly on the edge network capacity to deploy services, which has the most impact on MTP latency.

The proposed method was compared and evaluated from various perspectives against other recent studies through simulations. In providing MCVR through an MSA approach, the key metrics from a network perspective included user traffic load per user, MTP latency threshold satisfaction rate, network distance between users and motion services, and the edge deployment feasibility index of services. By applying the proposed MSA configuration and service-orchestration method, MCVR users can have their MTP latency requirements met effectively even with limited edge resources while maintaining a low level of network congestion within the infrastructure.

In the future, we plan to build an MSA-based MCVR testbed in a cloud–edge continuum environment, considering various additional factors, such as throttling due to hardware conditions, wireless network interference, and signal attenuation, to conduct empirical research. Specifically, we aim to evaluate the feasibility of the MSA configuration through various user experiments. Additionally, we will address the limitations of the current study, such as the static aspect of service resource utilization and research methods to reduce the overhead of online orchestration. Furthermore, considering the passthrough view-based outdoor MCVR operation environment, we will address issues related to microservice migration and the utilization of federated edge resources.